Affiliations: University of California, San Diego, CA, USA | Lawrence Livermore National Laboratories, Livermore, CA, USA
Note: [] Corresponding author: Seemeen Karimi, University of California,
San Diego, California, CA, USA. Tel.: +1 617 953 1662; E-mail:
seemeen.karimi@gmail.com
Abstract: BACKGROUND: Imaging systems used in aviation security include
segmentation algorithms in an automatic threat recognition pipeline. The
segmentation algorithms evolve in response to emerging threats and changing
performance requirements. Analysis of segmentation algorithms' behavior,
including the nature of errors and feature recovery, facilitates their
development. However, evaluation methods from the literature provide limited
characterization of the segmentation algorithms. OBJECTIVE: To develop segmentation evaluation methods that measure systematic errors such as
oversegmentation and undersegmentation, outliers, and overall errors. The
methods must measure feature recovery and allow us to prioritize segments. METHODS: We developed two complementary evaluation methods using
statistical techniques and information theory. We also created a semi-automatic
method to define ground truth from 3D images. We applied our methods to
evaluate five segmentation algorithms developed for CT luggage screening. We
validated our methods with synthetic problems and an observer evaluation. RESULTS: Both methods selected the same best segmentation algorithm.
Human evaluation confirmed the findings. The measurement of systematic errors
and prioritization helped in understanding the behavior of each segmentation algorithm. CONCLUSIONS: Our evaluation methods allow us to measure
and explain the accuracy of segmentation algorithms.
