Affiliations: [a] CSL@BS Research Group, University of Borås, Borås, Sweden | [b] Department of Computer and Systems Sciences, Stockholm University, Stockholm, Sweden
Correspondence:
[*]
Corresponding author: Tuve Löfström, CSL@BS Research Group, University of Borås, Borås, Sweden. Tel.: +86 46 033 435 4236; E-mail:tuve.lofstrom@hb.se
Abstract: Conformal prediction (CP) is a relatively new framework in which predictive models output sets of predictions with a bound on the error rate, i.e., the probability of making an erroneous prediction is guaranteed to be equal to or less than a predefined significance level. Label-conditional conformal prediction (LCCP) is a specialization of the framework which gives a bound on the error rate for each individual class. For datasets with class imbalance, many learning algorithms have a tendency to predict the majority class more often than the expected relative frequency, i.e., they are biased in favor of the majority class. In this study, the class bias of standard and label-conditional conformal predictors is investigated. An empirical investigation on 32 publicly available datasets with varying degrees of class imbalance is presented. The experimental results show that CP is highly biased towards the majority class on imbalanced datasets, i.e., it can be expected to make a majority of its errors on the minority class. LCCP, on the other hand, is not biased towards the majority class. Instead, the errors are distributed between the classes almost in accordance with the prior class distribution.
Keywords: Conformal prediction, imbalanced learning, class bias
