A variant of n-gram based language-independent text categorization

Article type: Research Article

Authors: Graovac, Jelena

Affiliations: Faculty of Mathematics, University of Belgrade, Studentski Trg 16, 11000 Belgrade, Serbia. Tel.: +38 164 865 0063; E-mail: jgraovac@matf.bg.ac.rs

Abstract: A technique for automated categorization of text documents, based on byte-level n-gram profiles and a new dissimilarity measure between profiles is presented. K nearest neighbors classifier is used. The technique is language independent. It has been applied to four document collections in English, Chinese and Serbian: Reuters-21578 newswire articles, 20-Newsgroups, Tancorp and Ebart. The evaluation was done by using the micro- and macro-averaged F1 function. The results obtained confirm that the presented technique, although very simple, in the case of Tancorp and 20-Newsgroups corpora achieves better results than other n-gram based techniques. As compared to other state-of-the-art methods, it performs better than “bag-of-words” K nearest neighbors classifier and in the case of 20-Newsgroups corpus it works even better than “bag-of-words” Support vector machines classifier. It can be successfully used in a variety of related problems.

Keywords: Byte-level n-gram, text categorization, kNN classifier

DOI: 10.3233/IDA-140663

Journal: Intelligent Data Analysis, vol. 18, no. 4, pp. 677-695, 2014