A probabilistic framework for automatic term recognition

Article type: Research Article

Authors: Wong, Wilson | Liu, Wei | Bennamoun, Mohammed

Affiliations: School of Computer Science and Software Engineering, University of Western Australia, Crawley, WA, Australia. E-mail: wilson@csse.uwa.edu.au, wei@csse.uwa.edu.au, bennamou@csse.uwa.edu.au

Abstract: Term recognition identifies domain-relevant terms which are essential for discovering domain concepts and for the construction of terminologies required by a wide range of natural language applications. Many techniques have been developed in an attempt to numerically determine or quantify termhood based on term characteristics. Some of the apparent shortcomings of existing techniques are the ad-hoc combination of termhood evidence, mathematically-unfounded derivation of scores and implicit assumptions concerning term characteristics. We propose a probabilistic framework for formalising and combining qualitative evidence based on explicitly defined term characteristics to produce a new termhood measure. Our qualitative and quantitative evaluations demonstrate consistently better precision, recall and accuracy compared to three other existing ad-hoc measures.

Keywords: Term recognition, termhood, term characteristic, Bayes' theorem, word distribution models, term extraction

DOI: 10.3233/IDA-2009-0379

Journal: Intelligent Data Analysis, vol. 13, no. 4, pp. 499-539, 2009