A metric for unsupervised metalearning

Article type: Research Article

Authors: Lee, Jun Won^{; *} | Giraud-Carrier, Christophe

Affiliations: Department of Computer Science, Brigham Young University, Provo, UT, USA

Correspondence: [*] Corresponding author. Tel.: +1 801 422 3027; Fax: +1 801 422 0169; E-mail: p3man4@gmail.com.

Abstract: We argue the value of unsupervised metalearning and discuss the attendant necessity of suitable similarity, or distance, functions. We leverage the notion of diversity among learners used in ensemble learning to design a distance function for the clustering of learning algorithms. We revisit the most popular measures of diversity and show that only one of them, Classifier Output Difference (COD) is a metric. We then use COD to produce a clustering of 21 learning algorithms, and show how this clustering differs from a clustering based on accuracy, and how it can be used to highlight interesting, sometimes unexpected, similarities among learning algorithms.

Keywords: Unsupervised metalearning, behavior distance, clustering

DOI: 10.3233/IDA-2011-0498

Journal: Intelligent Data Analysis, vol. 15, no. 6, pp. 827-841, 2011