Voting experts: An unsupervised algorithm for segmenting sequences

Article type: Research Article

Authors: Cohen, Paul^a | Adams, Niall^b | Heeringa, Brent^c

Affiliations: [a] Information Sciences Institute, University of Southern California, CA, USA | [b] Department of Mathematics, Imperial College London, UK | [c] Department of Computer Science, Williams College, MA, USA

Abstract: We describe a statistical signature of chunks and an algorithm for finding chunks. While there is no formal definition of chunks, they may be reliably identified as configurations with low internal entropy or unpredictability and high entropy at their boundaries. We show that the log frequency of a chunk is a measure of its internal entropy. The Voting-Experts exploits the signature of chunks to find word boundaries in text from four languages and episode boundaries in the activities of a mobile robot.

DOI: 10.3233/IDA-2007-11603

Journal: Intelligent Data Analysis, vol. 11, no. 6, pp. 607-625, 2007