Pruning extensions to stacking

Article type: Research Article

Authors: Rooney, Niall | Patterson, David | Nugent, Chris

Affiliations: Faculty of Engineering, University of Ulster at Jordanstown, Newtownabbey, BT37 OQB, UK. E-mail: nf.rooney@ulster.ac.uk, wd.patterson@ulster.ac.uk, cd.nugent@ulster.ac.uk

Abstract: In this paper we investigate an algorithmic extension to the technique of Stacking for regression that prunes the ensemble set before application based on a consideration of the training accuracy and diversity of the ensemble members. We evaluate two variants of this approach in comparison to the standard Stacking algorithm, one of which is a static approach that prunes back the ensemble to the same constant size; the other of which is a variable approach prunes the ensemble to an appropriate level based on measures of accuracy and diversity of the ensemble members. We show that on average both techniques are robust in performance to their non-pruned counterpart, while having the advantage of producing smaller and less complex ensembles. In the latter respect, the static approach proved more effective, but we show that the variable approach lends itself better for further optimization.

Keywords: Machine learning, ensemble learning, stacking, pruning

DOI: 10.3233/IDA-2006-10104

Journal: Intelligent Data Analysis, vol. 10, no. 1, pp. 47-66, 2006