Validation of Random Forest Machine Learning Models to Predict Dementia-Related Neuropsychiatric Symptoms in Real-World Data

1INTRODUCTION

Neuropsychiatric symptoms (NPS) are the leading cause of the social burden of dementia as they constitute the key factor in both loss of quality of life in patients and caregivers and families’ giving up on keeping patients at home [1, 2]. It could be thought that they are well acknowledged as 8% of new drugs currently under evaluation for Alzheimer’s disease are intended to treat NPS [3]. On the other hand, their population impact is underestimated as they are not well coded in health records [4– 6], implying inadequate clinical management [7]. In order to break the vicious circle established by the underestimation of the impact and the lack of proper recording of these symptoms, there is a need for tools providing information to monitor intervention plans from a population perspective [1, 8, 9].

The prevalence of NPS has been measured in clinical samples using questionnaires such as the Neuropsychiatric Inventory (NPI) that are rarely applied in clinical practice [4, 10], and the prevalence found varies between 56% and 98% [1, 11]. The problem with these figures is that they cannot be extrapolated to populations as a whole due to their heterogeneous distribution of dementia stages [9]. This distribution could be measured by analyzing a random sample of the general population using a door-to-door survey [4]. Notably, however, another study design is now feasible based on anonymized databases built from electronic health records (EHRs). This approach based on real-world data (RWD) has already been used to validate the diagnosis of dementia [12, 13] and the presence of agitation [6] but not to explore rates of NPS. On the other hand, validation studies are required in order to systematically use RWD as a source of epidemiological information [7, 14]. RWD have been described in an Organisation for Economic Co-operation and Development report as “broad data” because they cover large populations but include limited amounts of outcome and exposure data [15]. In line with this, machine learning (ML) tools have been postulated as having more capacity to predict complex clinical conditions like NPS [15] and being able to convert RWD into “smart data” [15, 16]. An example of this would be the calculation of the prevalence of NPS in population samples. While this design has been previously applied in cardiovascular research [17] and Alzheimer’s disease neuroimaging [18, 19], no examples have been reported of its use to measure features of dementia-related NPS at population level [20]. Therefore, the objective of this study was to construct and validate predictive models based on ML tools to identify the presence of psychotic and/or depressive symptoms in dementia-diagnosed patients from administrative and clinical databases that cover entire populations to inform social care and healthcare provision for NPS.

2MATERIALS AND METHODS

3RESULTS

The resulting dataset contained 62 variables and 4,003 cases, the main features of which are described in Table 1. Psychotic symptoms were documented in the EHR of 58% of the population and depressive symptoms in 59%. Psychotic symptoms were more common in men than in women. The pattern with age was different: the group with psychotic disorders being older in age and those with depressive disorder younger. Living in a nursing home was strongly associated with both types of symptoms. The dataset was randomly divided into a training set (N1 = 3003) and a test set (N2 = 1000). Supplementary Table 6 summarizes the pharmacological treatments given based on data from the EHR review by type of symptom, while Supplementary Table 7 compares patient characteristics in training and test sets.

Fig.1

Importance of variables in selected models predicting psychotic and depressive symptoms.

Table 2 shows the performance of the models tested for psychotic and depressive symptoms in both training and test sets. The iterations and modelling variables tested in each model are also summarized in this table. The analysis of the raw data without other variables indicates that antipsychotic prescribing is more specific and antidepressant prescribing more sensitive for identifying NPS. Notably, the models seeking to predict psychotic symptoms perform better, reaching an AUC of 0.80, than the depressive symptom models (maximum AUC of 0.74). Other statistics, namely, the kappa index and accuracy, also evidenced that the psychotic cluster models had better discriminatory power.

Figure 1 and Supplementary Table 8 indicate the importance of variables in the selected models. The most important ones in the psychotic symptom model are use of risperidone and level of sedation achieved by all the drugs prescribed, followed by use of quetiapine and haloperidol and the number of antipsychotics prescribed. Other important variables were the prescribing of trazodone, and sulpiride and number of changes from antidepressants to antipsychotics. In the depressive symptom model, the most important variable is number of antidepressants prescribed followed by use of escitalopram, level of sedation, and age. In this model, other important variables were the prescribing of Sertraline, duloxetine, mirtazapine, and citalopram, and the number of changes from antipsychotics to antidepressants.

Fig.2

Calibration plot for selected models predicting psychotic and depressive symptoms in the test set.

Figure 2 shows the calibration plot for the test dataset. If the model calibration was to be perfect, the points should lie on or around the 45° line of the plot. In our case, the models had poorer predictive power when the probability of NPS was lower than 0.25. Finally, Fig. 3 included in the Supplementary Material shows the ROC curves obtained in the test dataset for each of the models. As measured with the AUC (Table 2 and Fig. 3), the curves demonstrate graphically the better predictive ability of the psychotic symptom model.

Fig.3

Area under the curve (AUC) for selected models predicting psychotic and depressive symptoms in the test set.

4DISCUSSION

To our knowledge, this is the first time that ML techniques have been applied to build and validate models based on real-world population data to estimate the prevalence of NPS in dementia. This study shows that ML-based models are good at predicting dementia-related NPS and opens the prospect of applying these techniques in population databases. The raw results showed that more than half of the sample had NPS as identified by the presence of key terms in their EHR. This fits well with findings in the population-based literature. Halpern et al. found evidence of agitation in 44.6% of all patients when analyzing the EHR notes of dementia patients [6]. Similarly, a systematic review of 12 studies reporting NPS based on NPI score estimated rates of 40 to 50% for symptoms that could be included in the depressive and psychotic clusters [27], while in the Cardiovascular Health Study, 62% of dementia patients exhibited clinically significant NPS [28].

Considering AUCs of ≥0.80 to indicate excellent and 0.70– 0.79 acceptable discrimination, the psychotic symptom model can be classified as excellent and the depressive symptom model as acceptable [29]. Given these classifications, it would be valid to apply the results to the whole population database. They are consistent with the expected higher specificity of models for psychosis. When data collected from EHRs are applied to research, avoiding false-positive diagnoses may be more important than avoiding false negatives. In longitudinal studies, for example, false positives can dilute observed effects and reduce statistical power [30]. Therefore, our approach is valid for epidemiological research on dementia-related NPS. Consistent with the AUC values, other statistics used, namely, accuracy and the kappa index, also indicated that the model predicting psychotic symptoms performed better.

As previously mentioned, however, the calibration was poor when the probability of NPS estimated by the model was low. This implies that both models systematically underestimated symptom rates observed in EHRs. The explanation for this may be that when the symptoms are of recent onset the recording of symptoms in the EHR is not yet accompanied by pharmacological treatment. Nonetheless, as the disease moves forward, clinicians treat such symptoms, and hence, the calibration line overlaps the 45° line of the plot meaning that observed and predicted cases fit well.

Our framework for analyzing the NPS was based on distinguishing between two broad categories or constructs in individual clinical symptoms, because analyzing individual symptoms would lead to overly complex analysis. In 2015, the International Psychogeriatric Association developed a provisional consensus definition of agitation [31]. Currently, this association is updating the criteria for psychosis in major and mild neurocognitive disorder and has described agitation and depression as specific subtypes. The different level of importance of the variables in the two models corroborates the usefulness of the two-category classification. Nonetheless, there is agreement between the models on the importance of changes in medication and their sedative properties. As in the Chinese metaphor about cats, perhaps it does not matter whether a drug is an antidepressant or an antipsychotic so long as it produces sedation. Instead of being avoided as an adverse effect of treatment [32], it seems that sedation is the mechanism sought to reduce agitation. The other key variable is patients changing from one medication to another (antidepressant to antipsychotic or vice versa). Day-to-day fluctuations in clinical symptoms are interpreted as treatment failure, which boosts the recording of symptoms in the EHR and leads to the prescribing of new drugs.

The plot showing the importance of variables for psychotic symptoms highlights the role of risperidone and quetiapine in their pharmacological treatment. It was to be expected that risperidone would be the drug that contributed the most predictive value to the psychotic symptom model because it is the only one that has official approval from the European Drug Agency. The great gap between recommendations and reality is evidenced by the role of quetiapine, which is the second most important medication even though its indication for dementia is off label. On the other hand, the level of sedation achieved by the medications used also had a high explanatory capacity, and medication changes contributed considerably to the predictive value of the psychotic symptom model. NPS affect almost all individuals with dementia (97%) over the course of the disease [3], and although they fluctuate [4], they rarely disappear, and this makes level of sedation particularly important. Families seldom have access to specialized clinics to deal with such fluctuations and family caregivers often manage them by adjusting doses by trial and error. If that does not work, families turn to the most accessible doctor to change the drug prescribed because they cannot cope with the stress. This explains the great importance in the model of the variables indicating the number of antipsychotic treatments and changes from antidepressant to antipsychotic drugs. As described in the literature, antidepressant drugs also play a relevant role managing agitation [33]. Not only trazodone but also sertraline, paroxetine, mirtazapine, and escitalopram showed importance in the model. The most important antidepressants in the psychotic cluster model were mirtazapine and trazodone, which are most likely used for their sedative effect. In contrast, the prescribing of sertraline and paroxetine could have more to do with their antidepressant effects and ease of use due to fewer side effects. We believe that their role reflects a history of both psychotic and depressive symptoms in some cases. A few non-pharmacological variables showed importance, namely, comorbidities (Parkinson’s disease and stroke) and living in a nursing home. Nonetheless, their explanatory power was very weak compared with that of pharmacological variables.

The depressive cluster model was based on different variables, the most important one being the number of antidepressants prescribed. Escitalopram is the first individual drug in the ranking, and is followed by level of sedation and age. This model seemed to be more homogeneous as it relied much less on antipsychotic drugs and comorbidities. Moreover, living in a nursing home played a trivial role. Despite this homogeneity, its predictive power was much lower, especially because of the lower specificity. The model identified positive cases well, but failed to rule out negative ones.

Various authors have underlined the need to integrate evidence from heterogeneous sources including clinical trials, cohort data, and RWD to evaluate disease progression and build health economic models for dementia treatment [15, 34, 35]. Nonetheless, RWD lack consistency in the collection of outcomes and monitoring of disease severity. We recognize the limitations of our study associated with failure to include other comorbidities and our dataset lacking measures of cognitive function and/or autonomy, such as walking speed and performance of activities of daily living, which are not currently recorded in a structured format in EHRs. It is likely that inclusion of the such variables would have improved the predictive power of the models. Additionally, it is widely known that different feature selection techniques to compute variable importance can lead to different results [36]. In this sense, random forest-based technique has been deemed of giving biased results when using high-dimensional datasets with possible multicollinearity [36]. Considering the nature of our data, we believe that it does not significantly affect the presented results. Nonetheless, the comparison with other variable selection techniques in future works would be worth to consider. In this scenario, validation of variables available from EHRs seems to be a key first step towards an AD/dementia integrated curated data environment fed from multiple sources [15]. Brayne et al. pointed out the crucial importance of approaches to dementia research being anchored in the true population as selective participation in observational studies may systematically bias findings [37].

In addition to the aforementioned limitations of this study (related to variables not included in the models), a limitation is that no validated scale such as the NPI was used to identify the presence of NPS in patients with dementia in the whole sample [10]. Finding specific terms in an EHR review only reveals that a physician recorded symptoms linked to the presence of behavior and depressive disorders. The identification of dementia cases could also be deemed problematic, as some authors have questioned the use of Medicare claims to identify dementia [38]. On the other hand, the aim of such claims is to allow physicians to be paid. In contrast, our database is obtained directly from a unified EHR used by all healthcare professionals (physicians and nurses) to document all patients’ contacts with the health service in all care settings (primary, emergency, inpatient, home and outpatient care). The system includes an automatic coding system (ICD-10) managed by physicians when they provide care to patients. Specialists and primary care doctors are not able to move forward in the EHR if an episode is not assigned a diagnosis that is automatically coded. Canadian researchers have applied a similar approach for identifying Parkinson’s disease and dementia with good results [13, 39].

We have applied a binary classification that simplifies the heterogeneous way in which doctors describe NPS in the EHR. Moreover, we have consciously avoided including sleep disturbances within the scope of the research due to the bidirectional relationship between sleep disturbances and dementia [5]. Given that it is unclear whether dementia is a cause or consequence of sleep disturbance, we believe that the interpretation of predictive models based on the use of hypnotics would be very difficult, and hence, for the time being, have focused on psychotic and depressive clusters.