Determining noisy instances relative to attributes of interest

Article type: Research Article

Authors: Khoshgoftaar, Taghi M.^{a; *} | Van Hulse, Jason^b

Affiliations: [a] Taghi M. Khoshgoftaar, Empirical Software Engineering Laboratory, Department of Computer Science and Engineering, Florida Atlantic University, Boca Raton, FL 33431, USA | [b] Florida Atlantic University, Boca Raton, FL, USA

Correspondence: [*] Corresponding author. Tel.: +1 561 297 3994; Fax: +1 561 297 2800; E-mail: taghi@cse.fau.edu.

Abstract: An important issue in the analysis of data is that of data quality. Algorithms for the detection of instances with class noise and, to a lesser degree, attribute noise have been presented in the literature. We propose a novel technique to detect noisy instances relative to an Attribute of Interest. The attribute of interest can be any feature from the dataset as defined by the domain-specific practitioner. The proposed technique determines those instances that contain noise relative to the chosen attribute of interest. This approach can be iterated for any number of user-specified attributes. Our methodology is demonstrated with empirical case studies using real-world datasets and is verified by an expert-based validation of the results. Additional studies show the effectiveness of our technique in detecting noise injected into instances. For detecting noise relative to the class or dependent variable, our technique is compared to the well-known classification and ensemble filters and outperforms both techniques on a real-world dataset with known class noise. Based on the results of a wide variety of case studies presented in this work, we conclude that our methodology for ranking noisy instances relative to an attribute of interest is an effective and useful noise handling procedure.

Keywords: Noise detection, data cleaning, attribute of interest

DOI: 10.3233/IDA-2006-10305

Journal: Intelligent Data Analysis, vol. 10, no. 3, pp. 251-268, 2006