A confidence paradigm for classification systems with out-of-library considerations

Article type: Research Article

Authors: Leap, Nathan J. | Bauer Jr., Kenneth W.^{; *}

Affiliations: Air Force Institute of Technology, AFIT/ENS, Wright-Patterson AFB, OH, USA

Correspondence: [*] Corresponding author. E-mail: kenneth.bauer@afit.edu

Abstract: There is no universally accepted methodology to determine how much confidence one should place in the output of a classification system. Leap and Bauer [15] present a confidence paradigm based on the assumptions that system confidence acts like, or can be modeled as value and that indication confidence can be modeled as a function of the posterior probability estimates. This paper extends the paradigm to include out-of-library considerations. In addition, a novel out-of-library detector is presented. Developing the out-of-library detector involves bounding and discretizing the feature space and assigning each discrete point to either in-library or out-of-library classes based upon Mahalanobis distance from the in-library target classes. Application of the confidence paradigm to the out-of-library detector leads us to the demonstration of a new concept called out-of-library non-declarations. The extended paradigm is applied to a synthetic data set as well as an automatic target recognition data set. In all cases, the results show performance that tracks well with previous studies found in the literature and demonstrate positive steps toward fuller development of a theoretical framework that unites the viewpoints of the classification system developer and its user.

Keywords: Confidence, classification, pattern recognition, automatic target recognition, out-of-library

DOI: 10.3233/IDT-2012-0119

Journal: Intelligent Decision Technologies, vol. 6, no. 1, pp. 1-25, 2012