Robust weighted general performance score for various classification scenarios

Article type: Research Article

Authors: Pandey, Gaurav^{a; b} | Bagri, Rashika^c | Gupta, Rajan^{a; d} | Rajpal, Ankit^{c; *} | Agarwal, Manoj^e | Kumar, Naveen^{c; *}

Affiliations: [a] Research and Analytics Division, Analyttica Datalab, Whitefield, Bangalore, Karnataka, India | [b] School of Artificial Intelligence and Data Science, IIT Jodhpur, Jodhpur, Rajasthan, India | [c] Department of Computer Science, University of Delhi, Delhi, India | [d] Artificial Intelligence and Innovation Lab, Universidad Autonoma de Tamaulipas, Mexico | [e] Department of Computer Science, Hans Raj College, University of Delhi, Delhi, India

Correspondence: [*] Corresponding authors: Ankit Rajpal, Department of Computer Science, University of Delhi, Delhi, India. E-mail: arajpal@ cs.du.ac.in. Naveen Kumar, Department of Computer Science, University of Delhi, Delhi, India. E-mail: nkumar@cs.du.ac.in.

Abstract: Traditionally, performance measures such as accuracy, recall, precision, specificity, and negative predicted value (NPV) have been used to evaluate a classification model’s performance. However, these measures often fall short of capturing different classification scenarios, such as binary or multi-class, balanced or imbalanced, and noisy or noiseless data. Therefore, there is a need for a robust evaluation metric that can assist business decision-makers in selecting the most suitable model for a given scenario. Recently, a general performance score (GPS) comprising different combinations of traditional performance measures (TPMs) was proposed. However, it indiscriminately assigns equal importance to each measure, often leading to inconsistencies. To overcome the shortcomings of GPS, we introduce an enhanced metric called the Weighted General Performance Score (W-GPS) that considers each measure’s coefficient of variation (CV) and subsequently assigns weights to that measure based on its CV value. Considering consistency as a criterion, we found that W-GPS outperformed GPS in the above-mentioned classification scenarios. Further, considering W-GPS with different weighted combinations of TPMs, it was observed that no demarcation of these combinations that work best in a given scenario exists. Thus, W-GPS offers flexibility to the user to choose the most suitable combination for a given scenario.

Keywords: Performance measures, classification, imbalanced data modelling, noisy data modelling, coefficient of variation

DOI: 10.3233/IDT-240465

Journal: Intelligent Decision Technologies, vol. 18, no. 3, pp. 2033-2054, 2024