Utility of Machine Learning Approach with Neuropsychological Tests in Predicting Functional Impairment of Alzheimer’s Disease

INTRODUCTION

Neuropsychological tests provide useful information in assessing current and future functional abilities in dementia [1, 2]. Accurate assessment of daily functional abilities is crucial in depicting the clinical significance of dementia and distinguishing Alzheimer’s disease (AD) from mild cognitive impairment (MCI) [3]. As biomarkers of dementia became widely available, the ecological validity and real-world utility are becoming more highlighted in the practice of neuropsychological assessment [4, 5]. Although functional disturbances or a loss of independence are defined as a critical boundary in distinguishing the significance of cognitive impairment, a formal model in how multiple neurocognitive functions buttress normal daily functioning has not been clear [6–8]. Due to a weak foundation of formal tools for assessing functional ability, clinical characterization of AD requires further improvement [9].

Levels of functional impairment in dementia generally correspond with overall cognitive test performances. The neuropsychological battery consists of multiple scores of subtests and the scores. The battery typically provides a summary index by summing the individual scores into a composite of the total score. This approach has been widely used in representing one’s global cognitive function, specifically in the neuropsychological battery for dementia assessment [10, 11]. The total scores as a global cognitive function have shown utility in assessing multiple clinical signatures of dementia, including early detection, prognostic trajectory, daily functional disturbances, differential diagnosis, and neurodegeneration [12–16]. Though the diagnosis of dementia can be made based on a deficit in a single cognitive domain, a combination of multiple test scores may provide a more reliable estimate of a cognitive construct than any single measure in characterizing cognitive impairment [17–19].

Despite the utility of the total scores of neuropsychological batteries, a single summary may overly simplify complex features of the test elements. The summarized score typically sums the “no-hold” tests sensitive to the brain pathological changes. However, the construction of a summary score is mostly confirmed based on theoretical analysis rather than an empirical comparison of the multiple possible summary approaches. Thus, each subtest may differ in its metric properties, leading to suboptimal weighting to the total score, and summing the raw scores may fail to reflect the differential importance of each subtest score. Previous studies have shown that a set of subtests can be selectively augmented based on specific clinical purposes [20].

Another issue in the total score method is the use of linear summation. This method summarizes multiple tests by adding each score with predefined weights without additive effects (e.g., interaction effect, higher-order effect). Based on this linear modeling approach, the decrease in a specific subtest will lead to a corresponding change in the total score as a function of the given weight. However, in the levels of clinician’s intuition, the neuropsychological test performances may not always be linearly informative in explaining the real-world impairment [21]. The complex nature of cognitive impairment implies that the nonlinear combination of multiple cognitive tests characterizes a meaningful clinical state. While some tests are interpreted as clinically significant with the presence of a complete task failure, other tests indicate clinical risk with subtle variabilities within the normal range [22]. In some cases, subtests signal cognitive impairment of dementia only when combined with a deficit in other domains [23, 24]. These qualitative patterns that clinicians often read from the neuropsychological test profile may not be fully summarized with the current method of total scoring, and a more fine-tuned summary with profound complexity needs to be developed.

With the goal of personalized medicine, the machine learning approach has prompted the development of clinical and behavioral models with higher complexity [25, 26]. One of the fundamental concepts in machine learning is finding an optimal balance between flexibility and robustness in predicting the target of interest. Machine learning algorithms effectively stretch a model into higher complexity and generate various models that explain a given dataset. In addition, the cross-validating procedure effectively rules out the overfitted models by highlighting the predictability in a novel dataset.

Especially the machine learning approach can explore how complex a model should be in predicting a meaningful dementia outcome and the prerequisites for achieving such predictive performance. If the predictability of the nonlinear model outperforms standard linear summation models, for example, there may be additional higher-order nonlinearity and interaction effects that a high dimensional model can capture but a standard linear model does not [27]. Machine learning, however, may not always provide enhanced utility in the prediction of dementia and there are only few studies that explicitly compared the effect of model complexity [28]. Complex models based on machine learning performed better than classical linear models in some clinical cases [29, 30], whereas several studies showed that using models with higher complexity and nonlinearity did not improve the predictive accuracy in classifying cognitive impairment of dementia [31, 32]. Moreover, most previous studies have focused on whether or not the machine learning algorithms successfully predict the target outcome, rather than evaluating the extent to which high dimensional models show a better predictive performance than classical linear models [33]. Few studies have directly compared how specific algorithms performed better and what it implies with the predictive attributes of behavioral instruments. The lack of rationale for the choice of algorithms may hinder the meaningful discussions of using the machine learning approach in the practice of AD assessment.

Another constraint in utilizing the machine learning approach is the necessity of the large-scale dataset. The advantage of machine learning algorithms generally depends on the sample size. The datasets used for predictive modeling should be sufficiently large for an algorithm to build a model that exploits complex characteristics of clinical outcomes [34, 35]. Thus, machine learning approaches should suggest a rationale for sample size needed in exceeding the performance of the classical statistical model, rather than merely identifying the presence of predictability. This issue has been thoroughly dealt with by the predictive performance across varying sample sizes when constructing complex machine learning models. The predictive power tends to show improvement as the sample size increases, and endpoint saturation of this maximum accuracy reflects the potential utility in using nonlinear models of machine learning. This procedure explores how many data points should be acquired in constructing the most predictive model. In the examples of image classification, using nonlinear models (i.e., kernel support vector machine or neural network) more benefited from the large-scale dataset than linear models (i.e., linear support vector machine, logistic regression) [34]. Machine learning in the clinical assessment of dementia can also benefit from large-scale data [36], but the empirical evidence of the extent to which the larger sample size benefits in utilizing the neuropsychological test remains unknown.

This study will examine how the multiple scores of the neuropsychological battery can be enhanced in predicting the clinical outcome of dementia based on the nonlinear modeling of the large-scale dataset. Clinical rating of functional impairment and survey score of daily activities were used as a clinical outcome of interest that cognitive test attempts to predict. The single total score and multiple subtest scores of the test battery were used as the main predictors. We utilized a support vector machine, a classic algorithm widely used to predict neurological and psychiatric disorders [37, 38]. The predictive utility of the linear and nonlinear model was compared across different sample sizes.

METHODS

RESULTS

The stages of clinical impairment rating (CDR-SOB ranges) corresponded with the score of IADL (Table 3). The total score of CERAD-K showed the highest rank-order correlation with the CDR-SOB stages. The subtests representing the cognitive domain of episodic memory and language showed moderately high association with the CDR-SOB.

After predictive models are fitted within the training dataset (2/3 of the dataset), the predictive weights were applied in the testing dataset (1/3 of the dataset). In the testing dataset, the accuracy metrics (correlation and MAE) between the predictive score based on neuropsychological performance (NP-predicted score) and the actual measure of the functional outcome (CDR-SOB and IADL) were calculated (Fig. 1). Also, the 10-times iteration of the prediction showed stable variability (Fig. 2). In linear models (Model 2), using the multiple subtest scores added a minimal increase in the predictive correlation compared to the single total score (Model 1). On the contrary, the predictive model using nonlinear models (Model 3, 4) showed higher accuracy than the linear models.

Fig. 1

Prediction scatter plots in the testing dataset (1/3 of the total sample, n = 881). X-axis: Actual score of CDR-SOB and IADL. Y-axis: Predicted scores based on neuropsychological performance (NP-predicted) in the training dataset (2/3 of the total sample). Averages of 10-iterated predictions (correlation coefficient r) are also noted.

Fig. 2

Iterated prediction accuracy of the neuropsychological test in predicting CDR-SOB and IADL scores (correlation r between predicted score and actual score) (Test set size = 881, 1/3 of the total sample). Each dot indicates iterated prediction (10 times). Model 1: Simple linear regression with the single total score. Model 2: Multiple linear regression with subtests. Model 3: nonlinear support vector regression of low complexity with subtests. Model 4: nonlinear support vector regression of high complexity with subtests.

We also examined specific sample size conditions under which machine learning algorithms provide benefits in utilizing neuropsychological tests. The result showed that linear regression (Model 1, 2) showed lower predictive errors when the sample size was relatively small (n < 200). However, the linear models showed a minimal benefit of adding the sample size, and the accuracy reached the asymptotic maximum at a specific dataset size (n = 500).

Fig. 3

Sample size effects on the predictive accuracy of Clinical severity (CDR-SOB) and Instrumental daily functioning (IADL). Sample size increased from 100 to 2,600 by 100 units.

On the other hand, the nonlinear kernel SVR (Model 3, 4) gradually showed enhanced predictive accuracy as the sample size increases. Even until the largest sample size (n = 2,600), the gradual improvement in the predictive accuracy did not reach the maximum level. Moreover, the SVR model with higher complexity (Model 4) showed a steeper improvement, leading to the highest predictive performance with a more extensive sample size (n > 1500). The number of averaged supporting vectors representing the trained data points in determining the regression tube was greater in SVR with higher complexity (Model 4, CDR-SOB: 1,713, IADL: 1,706) than SVR with lower complexity (Model 3, CDR-SOB: 1,447, IADL: 1,357).

The amount of predictive contribution was assessed with regression coefficients and feature importance of each subtest (Fig. 4). The result showed that the feature importance of Model 3 generally coincided with linear coefficients of Model 2. How-ever, the Word List Recognition and Word List–delayed tests showed an attenuated contribution in Model 3. Moreover, the feature importance in Model 4 was highly variable across iterations, and the salience of the specific feature importance (e.g., Fluency first session, Word List Recognition, Word List Recall delayed) became unclear. On the contrary, the tests with initially weak univariate association with functional level (e.g., Constructional Praxis–copy) showed a considerable improvement in the strength of feature importance.

Fig. 4

Predictive weights of CERAD-K subtests. Each dot represents 10-iterated predictions. Upper: CDR-SOB prediction. Lower: IADL prediction. Left: Multiple regression coefficients (Model 2). Middle: Feature importance of support vector regression with lower complexity (Model 3). Right: Feature importance of support vector regression with higher complexity (Model 4). Flu 1∼4: Animal fluency (4 sections), boston: Boston Naming Test, wr1/2: Word List Recall immediate/delayed, wrecog: Word List Recognition, cons1: Constructional Praxis Copy.

We also evaluated the extent to which other neuropsychological tests (TMT-A/B, Constructional Recall) and demographic information (age, education) add accuracy in predicting functional level. The result showed that adding five of the predictors more improved the predictive accuracy in both linear regression (Model 2 approach) and nonlinear SVR (Model 4 approach) (Fig. 5). Moreover, the improvement of prediction accuracy was greater in the nonlinear model than merely adding the predictors in the linear model, indicating the multiplicative properties in the predictive benefit. TMT-A showed a high contribution in the predictive weights, and the feature importance of the age effect became stronger in the nonlinear model (Supplementary Figure 1).

Fig. 5

Enhanced predictive accuracy when predictors of test scores and demographics are added. The scatter plots represent a single iterated prediction result in Model 4 (SVR-high) with 14 predictors included. NB: Nine subtests of the neuropsychological battery used for CERAD-K total score summation, TMT: Trail Making Test A and B, Cons2: Construction Recall, Demo: Demographic information (age, education, sex).

We also confirmed that the predictive benefit requires subtest combination rather than sole algorithmic complexity. When the CERAD-K total score was modeled with a nonlinear SVR-high (Model 4) algorithm, the predictive accuracy showed only slight improvement (MAE in CDR-SOB = 0.478, IADL = 0.471) compared to the simple linear regression method (MAE in CDR-SOB = 0.535, IADL =0.543). However, the accuracy was still worse than the model with the full nine subtests (Model 4 SVR-high MAE in CDR-SOB = 0.347, IADL = 0.356).

The finding was consistently maintained when the same analysis was conducted within the dementia diagnosis subgroups (MCI or AD group) (Table 4). Likewise, the predictive accuracy minimally differed between gender groups of males (n = 958; MAE in CDR-SOB = 0.29, IADL = 0.30) or females (n = 958, subsampled; MAE in CDR-SOB = 0.31, IADL = 0.32).

DISCUSSION

The neuropsychological battery provides essential information in assessing the levels of functional impairment of dementia. However, unfavorable sources of information and the idiographic context of lifestyle often hinder valid assessment of one’s real-world functioning. In the current study, we aimed to find translational methods that can aid in interpreting the complex nature of the functional status based on neuropsychological test performance. We examined how the classical summary of neuropsychological battery, namely the total score method, can be improved with the machine learning approach. We first showed that the linear regression models that use a single total score or multiple subtests performed moderately well in predicting functional variabilities of dementia. However, the performance showed a profound improvement when using a nonlinear algorithm that exploits the data pattern with higher complexity. This enhancement was explicitly observed when the dataset was sufficiently large. As the sample size used for training the model increases, the linear models were rapidly saturated with minimal additive benefit. On the contrary, nonlinear models showed a gradual increase in the predictive accuracy until the largest dataset available, especially when using the nonlinear model with higher complexity.

We first confirmed that the total sum score of the neuropsychological battery moderately predicts functional impairment associated with AD dementia. When comparing the two linear models, the combination of multiple subtests of the neuropsychological battery in a linear fashion showed a marginal increase in predictive accuracy, indicating that the current method in summing the subtests as the total score is generally optimal in describing functional outcomes of dementia. However, some scaling biases and prediction errors seem to require more precisions. Each subtest differs in its metric properties, including skewness and floor/ceiling effect across the levels of functioning. Also, the predictive importance may not be uniform across the range of functional abilities. These differences indicate that the linear modeling may not be optimal in finding the predictive test elements in every functional range of dementia, requiring a more flexible model in prediction.

One of the key findings of this study was identifying specific conditions under which machine learning algorithms provide benefits in utilizing neuropsychological tests for functional assessment in dementia. Using the radial basis kernel function of the support vector machine, additive terms with higher-degree or interaction effects may have effectively generated and flexibly fit into the given dataset as a high-dimensional hyperplane. The numerosity of supporting vectors in the fitted SVR model indicates the complex data pattern that the models captured. In contrast to the linear models, applying a nonlinear algorithm (SVR) led to a gradual increase of predictive accuracy as a function of sample size. Moreover, the learning curve did not saturate to a maximum accuracy even with the largest sample size (n = 2,600), suggesting that adding more datasets will further improve the predictability. While some previous studies showed discouraging results in the nonlinear machine learning approach [31, 32], such results may have been due to the lack of sample size. Our findings suggest that a relatively small dataset (n < 500) may be insufficient in exploiting the advantage of complex nonlinear algorithms.

In using the neuropsychological tests to predict the functional outcome of dementia, multiple reasons may explain the benefit of the nonlinear models. The stages of clinical impairment can be reified with the concepts of discrete and ordinal categories (i.e., early or late MCI, preclinical AD), rather than a fine-grained measure of a real entity of human attribute [59, 60]. Thus, if the significance of the cognitive deficit differs across the stages of clinical ordinality, predictive application of linear model may partly fail since they assume uniform predictability of test elements across a long range of clinical severity. Flexible generation of a nonlinear model can be effective in such cases.

One of the practical difficulties in capturing the nonlinear pattern is that the predictive benefit starts to emerge at a relatively larger dataset. Our findings noted that utilizing higher-order nonlinear algorithms requires a great amount of data points, and this difficulty may be related to the signal-to-noise ratio of the predictors. In the previous study, the slowly saturating pattern of the asymptotic learning curve was also observed when predicting other physical properties or discrete image categories [34]. However, the required sample size in the study grew larger in identifying the benefits of nonlinear models, especially as the amount of random noise increases. The noise in the trained dataset minimizes the benefit of nonlinear models, and the linear models may instead perform better when a considerable noise is mixed in the model training. Similar to this previous study, the classical summation of the total score in the current study provided a more reliable and accurate prediction on the functional outcome when the predictive summary score was generated with a relatively smaller sample size (n < 500). On the contrary, the worse performance of nonlinear models with small sample sizes indicates that the model easily overfits uninformative noise. These results suggest that the prerequisite of the benefit is a large-scale dataset that overcomes the confounding noise effect. Although the source of the noise in the neuropsychological test is not identified in the current study, sound data quality based on standardized administration and scoring procedures will be critical in improving the clinical prediction of dementia.

The use of predictive modeling implies further practical applications in the field of clinical assessment [25]. For example, recent attempts showed a promising advancement of using predictive pattern scores of dementia in the multidimensional dataset, such as genetic polymorphisms or brain structural imaging [61, 62]. Likewise, translation from cognitive task performance to real-world functioning requires complex translation. Despite the criticisms on using summary scores and their theoretical basis in neuropsychological assessment [63, 64], the current study showed that multiple cognitive tests could be effectively transformed into a single summary score. Furthermore, the summary approach can be used to serve other purposes of clinical assessment (e.g., differential diagnosis, specific domain of functional outcome) [65]. Varying degrees of one’s functional ability can result from multiple nonlinear combinations of cognitive processes, and clinically meaningful features may not be captured with human-level theories. Based on an optimal combination of the clinical information, some evidence suggests that the machine learning model can outperform the clinician’s predictive expertise in a specific domain of functional outcome [66]. The current study implies that score patterns can be validly augmented with specific purposes and may aid the clinicians’ implicit intuition.

Our study also made use of a machine learning advantage in translating group-level datasets toward personalized indices. In evaluating the utility of neuropsychological test instruments, machine learning provides precision metrics at an individual level, rather than merely rejecting null-hypothesis testing as a group-level difference [25, 38]. Based on the accuracy metrics (i.e., mean absolute error), clinicians can interpret the current study results in how precise or imprecise the summary score will be when assessing a single individual. Our study was able to evaluate the extent to which the classical summary scores and machine learning prediction scores track functioning levels at an individual level.

Consistent with the recent attempts of developing a performance-based assessment of instrumental activities, validating an objective instrument of real-world functioning will be an essential task in translating the use of neuropsychological battery [67]. A valid assessment of the presence and severity of functional ability is crucial in older adults with neuropsychiatric conditions. However, a functional assessment may face practical difficulties when information is gathered under unfavorable informants or patients with low daily motivation. The clinicians should often discern actual ability (i.e., maximum potential ability) from the motivational propensities in daily activities. Also, the significance of the functional difficulties differs across specific informants and may be prone to subjective criteria of the symptom severity. Moreover, the contextual meaning of one’s ‘impairment’ can profoundly vary depending on the social demands (e.g., rural or urban). The current study suggests that the predictive summary score of CDR-SOB and IADL based on neuropsychological test performance would be less affected by confounding factors and may provide ‘expected’ levels of real-world functioning.

Our findings support the utility of using multiple tests in the neuropsychological assessment of dementia. According to the actuarial neuropsychological approaches in MCI diagnosis, applying a multivariate definition of the cognitive impairment led to a more reliable and valid prediction of dementia prognosis and biomarkers [17, 18, 68]. Interestingly, the current study showed that using a single score is insufficient in generating an optimal predictive model. Moreover, the benefit of combining multiple subtests grew larger when exploiting nonlinear patterns of predictors rather than linearly adding the predictors. These findings further support using multiple cognitive tests rather than a single test score as a clinical cutpoint. The multiplicative combination seems to be a critical factor in narrowing the remaining errors that linear methods cannot explain.

Another issue in using neuropsychological scores in dementia assessment is the norm-based decisions. Identifying the deviant deficit based on the expected performance of age and education can sensitively detect subtle inflection in the early stages of cognitive disorders. Accordingly, we observed large variabilities of the total scores, especially within the range of subtle impairment. These participants consisted of highly heterogeneous educational attainment. It is possible that many of the older adults with low education may have maintained the preexisting function while showing significantly low cognitive performances [69, 70]. The lower test specificity in low-educated elderly may have degraded the prediction accuracy, especially in the linear model method [71]. This study found that the additive effect of demographic information was more significant only in the SVR-high complexity model, indicating the predictive role of demographics in an interactive and conditional way. Though the benefits of using norm-adjustment in predicting clinical impairment have been unclear [72–74], utilizing test interpretation that reflects a premorbid functional ability will be crucial in making a theoretically valid distinction between preexisting functional abilities and progressive deterioration [6].

In the subgroup analyses of dementia diagnosis (MCI and AD), the main findings remained consistent. Since the functional assessment measures are strongly associated with the diagnostic criterion of dementia, the predictability could have only reflected diagnostic variation. However, this additional analysis confirmed that the benefits of SVR also held for the variabilities within the same clinical diagnosis of MCI or AD. MCI group showed lower prediction accuracy, possibly due to lower variability of IADL and CDR-SOB, but a meaningful improvement of SVR models was consistently observed. Likewise, the predictive benefit held for both gender groups. Notably, the predictability showed a minimal decrease even with a significant loss sample size of the training dataset in the subgroup analysis. The overall difference in the prediction accuracy was observed, especially in the metric of correlation coefficient (r), while absolute deviance of the prediction (MAE) showed a minimal difference. This difference may be due to the scaling effect between mild levels of impairment (MCI group, low variance, low predictive correlation) and severe range of impairment (high variance, AD group, higher predictive correlation). The correlation metric in predictive modeling has been criticized for being sensitive to the scaling of the data, and a high correlation can exist even when predicted values are discrepant from actual values [26]. The predictability may systematically differ across other clinical stages (e.g., MCI converter versus non-converter, controls versus AD, controls versus MCI) [28] and future study can specify whether the development of predictive models requires a homogeneous training dataset.

When examining the regression coefficients and feature importance, we generally confirmed that subtests of verbal episodic memory, processing speed, category fluency contribute to the prediction of functional impairment [28, 75]. However, it is notable that the variability of feature importance grows flat, and subtests with initially weak weights came to have similarly high contributions. For example, constructional praxis showed minimal predictive contribution in the linear model (Model 2), but SVR identifies the test as a similarly significant predictor (Model 3 and 4). In other words, even though some test remains the most robust predictors (auditory verbal learning test), low-ranking predictors (e.g., visuospatial ability, demographic information) play a role in the higher-order models. Such multiplicative effect implies that the clinical prediction of AD requires combining multiple cognitive domains [17–19], rather than nominating cognitive sign of a single impairment. Moreover, our result warrants caution that a selective set of cognitive domains can only reveal a fraction of the functional outcomes of AD.

Several limitations in the study should be noted. First, the outcome measure used for the prediction target is not the ultimate gold standard in evaluating the predictability of models. The two outcome measures used to train the models can also be influenced by multiple factors of physical illness and socioeconomic constraints that typically concur in older adults. While the administration guideline of CDR explicitly requires not to consider functional disturbances unrelated to cognitive disorders, the confluent nature of one’s real-world functioning may obscure the clear-cut ratings. Moreover, the lifelong division of gender roles in housework blur the boundary between the propensity and ability of daily activities [49]. Neuropsychological test scores are insufficient in accounting for such idiographic social context. The given dataset in the current study is not free from the inherent bias of the prediction target. The results require cautious interpretation in whether the predicted score purely captured the concept of interest. Secondly, we note that the current dataset only confined a specific type of dementia, namely AD. Due to the symptom specificity of AD, the difficulties and functional impairment can differ with other types of neurodegenerative disease or neuropathological conditions. Thus, the specific cognitive domain such as long-term memory function may have been accentuated in the predictive weight. In the future study, modeling various neurocognitive disorders will identify a more generalized theory that explains the association between cognitive test performance and real-world functioning. Lastly, a nonlinear model with high dimensional features is difficult to translate into human-understandable theories. The classic theoretical models typically highlight which of the specific subtest was most sensitive and specific in discriminating a particular clinical condition, but the current study suggests that the true theoretical model of real-world functioning may be much more complex than designed with human inference [76]. Flexible generation of a highly complex model accurately predicted human behavior, but the exact explanatory mechanisms remain in the ‘black box’ models. Further study is needed in decoding how the machine learning model improves predictive accuracy and what specific latent patterns are implied.