An iterative refinement approach for data cleaning¹

Article type: Research Article

Authors: Karmaker, Amitava | Kwek, Stephen

Affiliations: Department of Computer Science, University of Texas at San Antonio, TX 78249, USA. E-mail: akarmake@cs.utsa.edu, kwek@cs.utsa.edu

Note: [1] This research is supported by NSF grant CCR-0208935.

Abstract: Data cleaning is an important step in the data mining process. Successful data mining applications require good quality data. In this paper, we propose a data cleaning technique that smoothes out a substantial amount of attribute noise and handles missing attribute values as well. Our approach is inspired by the Expectation-Maximization (EM) algorithm. It iteratively refines each attribute-value using a predictor constructed from the previously refined values (known values in the first iteration). We demonstrate the effectiveness of our technique in smoothing out attribute noise and corroborate the efficacy of our technique by showing improved classification accuracy on a number of real world data sets from UCI repository [2]. Moreover, we show that our technique can easily be adapted to fill up missing attribute-values in classification problems more effectively than other standard approaches.

Keywords: Missing attribute values, noise smoothing, classification problems, EM algorithm

DOI: 10.3233/IDA-2007-11507

Journal: Intelligent Data Analysis, vol. 11, no. 5, pp. 547-560, 2007