Detecting concept drift: An information entropy based method using an adaptive sliding window

Article type: Research Article

Authors: Du, Lei^{; *} | Song, Qinbao | Jia, Xiaolin

Affiliations: Department of Computer Science and Technology, Xi'an Jiaotong University, Shaanxi, China

Correspondence: [*] Corresponding author: Lei Du, Department of Computer Science and Technology, Xi'an Jiaotong University, Shaanxi, China. E-mail: dulei.323@stu.xjtu.edu.cn.

Abstract: Concept drift in data stream poses many challenges and difficulties in mining this tradition-distinct database. In this paper, we focus on detecting concept drift in evolving data stream. We propose a novel method to detect concept drift using entropy over an adaptive sliding window. In the method, the sliding window is not fixed but dynamically determined. Another distinct property is that the method integrates an algorithm to find the exact timestamp for retraining the classifier whenever a concept drift is detected. In the experiments, we evaluate our method on publicly available data streams containing various types of concept drifts, and then compare it with four well known concept drift detection methods. The experimental results show that compared with four benchmarks, the proposed method is better than or comparable with other methods for most cases.

Keywords: Data stream, concept drift detection, sliding window, Hoeffding bound

DOI: 10.3233/IDA-140645

Journal: Intelligent Data Analysis, vol. 18, no. 3, pp. 337-364, 2014