Affiliations: [a] Istituto TeCIP, Scuola Superiore Sant'Anna, Ghezzano (PI), Italy | [b] Politecnico di Torino, Corso Duca degli Abruzzi, Torino, Italy
Correspondence:
[*]
Corresponding author: Nicola Matarese, PERCRO, Istituto TeCIP – Scuola Superiore Sant'Anna – Via L. Alamanni 13D, 56010 Ghezzano (PI), Italy. Tel.: +39 050882511; E-mail: n.matarese@sssup.it.
Abstract: Data preprocessing is a main step in data mining because real data can be corrupted for different causes and high performance data mining systems require high quality data. When a database is used for training a neural network, a fuzzy system or a neuro-fuzzy system, a suitable data selection and pre-processing stage can be very useful in order to obtain a reliable result. For instance, when the final aim of a system trained through a supervised learning procedure is to approximate an existing functional relationship between input and output variables, the database that is exploited in the system training phase should not contain input-output patterns for which the same input or similar input sets are associated to very different values of the output variable. In this paper a procedure is proposed for detecting non-coherent associations between input and output patterns: by comparing two distance matrices associated to the input and output patterns, the elements of the available dataset, where similar values of input variables are associated to quite different output values can be pointed out. The efficiency of the proposed algorithm when pre-processing data coming from an industrial database is presented and discussed together with a statistical assessment of the obtained results.
Keywords: Data preprocessing, filtering technique, distance evaluation
