Blinded Clinical Evaluation for Dementia of Alzheimer’s Type Classification Using FDG-PET: A Comparison Between Feature-Engineered and Non-Feature-Engineered Machine Learning Methods
Affiliations: [a] School of Engineering Science, Simon Fraser University, Burnaby, Canada
| [b] Department of Psychiatry and Behavioral Sciences, Feinberg School of Medicine, Northwestern University, Chicago, IL, USA
| [c]
Jewish General Hospital, Montreal, Canada
Correspondence:
[*]
Correspondence to: Mirza Faisal Beg, ASB 8857, 8888 University Drive, Simon Fraser University, Burnaby, BC, V5A1S6, Canada. Tel.: +1 778 782 5696; E-mail: faisal-lab@sfu.ca.
Note: [1] These authors contributed equally to this work.
Abstract: Background:Advanced machine learning methods can aid in the identification of dementia risk using neuroimaging-derived features including FDG-PET. However, to enable the translation of these methods and test their usefulness in clinical practice, it is crucial to conduct independent validation on real clinical samples, which has yet to be properly delineated in the current literature. Objective:In this paper, we present our efforts to enable such clinical translational through the evaluation and comparison of two machine-learning methods for discrimination between dementia of Alzheimer’s type (DAT) and Non-DAT controls. Methods:FDG-PET-based dementia scores were generated on an independent clinical sample whose clinical diagnosis was blinded to the algorithm designers. A feature-engineered approach (multi-kernel probability classifier) and a non-feature-engineered approach (3D convolutional neural network) were analyzed. Both classifiers were pre-trained on cognitively normal subjects as well as subjects with DAT. These two methods provided a probabilistic dementia score for this previously unseen clinical data. Performance of the algorithms were compared against ground-truth dementia rating assessed by experienced nuclear physicians. Results:Blinded clinical evaluation on both classifiers showed good separation between the cognitively normal subjects and the patients diagnosed with DAT. The non-feature-engineered dementia score showed higher sensitivity among subjects whose diagnosis was in agreement between the machine-learning models, while the feature-engineered approach showed higher specificity in non-consensus cases. Conclusion:In this study, we demonstrated blinded evaluation using data from an independent clinical sample for assessing the performance in DAT classification models in a clinical setting. Our results showed good generalizability for two machine-learning approaches, marking an important step for the translation of pre-trained machine-learning models into clinical practice.
