Scalable document classification

Article type: Research Article

Authors: Lee, Jae-Moon^{a; b} | Calvo, Rafael A.^b

Affiliations: [a] School of Information and Computer Engineering, Hansung University, Korea | [b] Web Engineering Group, School of Electrical and Information Engineering, University of Sydney, Australia. E-mail: rafa@ee.usyd.edu.au

Abstract: This paper describes the design and implementation of new naive Bayes and k-Nearest Neighbour methods that are highly scalable and efficient for document classification. Three methods for improving scalability are analysed: a change in the data representation and therefore in the algorithms' implementation, a partitioning mechanism that breaks down the problem into smaller parts, and a buffering mechanism to improve memory efficiency for large datasets. The classifiers were tested over two Reuters datasets: ModApte a popular but small benchmark, and RCV1 a new large collection of news stories, and compared to more standard implementations of these methods, both experimentally and analitically.

Keywords: language technologies, news stories, document classification, naive Bayes, k-Nearest Neighbour

DOI: 10.3233/IDA-2005-9404

Journal: Intelligent Data Analysis, vol. 9, no. 4, pp. 365-380, 2005