Comparing the Electronic and Standard Versions of the Montreal Cognitive Assessment in an Outpatient Memory Disorders Clinic: A Validation Study
Abstract
The Montreal Cognitive Assessment (MoCA) has become widely used as a brief test of cognitive function in patients with neurological disease. More convenient application of the MoCA might increase its use and enhance its utility. An electronic version of the MoCA has recently been developed. To establish validity of the electronic version (eMoCA), discrepancy scores, concordance correlation coefficients (CCC), and root mean squared differences (RMSD) were calculated between each administration method in a sample of 43 new adult patients presenting with primary memory complaints. The CCC was 0.84 and the RMSD was 2.27, with 76% of the sample having a difference score within 2 points. Overall, this study establishes adequate convergent validity between the MoCA and eMoCA among an adult population presenting with memory concerns.
INTRODUCTION
Cognitive assessment is a critical component of clinical care for individuals with neurological disease. The Montreal Cognitive Assessment (MoCA) [1] is a validated brief paper-and-pencil cognitive screening measure that is commonly employed in clinical practice and research settings. The MoCA takes approximately 10 minutes to administer and evaluates several cognitive domains including visuospatial ability, executive functioning, language, memory, attention, and orientation. The MoCA has been demonstrated to have adequate construct validity as compared to standard neuropsychological tests measuring similar cognitive constructs [2, 3] and to be sensitive in identifying cognitive impairment in a number of clinical populations including mild cognitive impairment [1, 4–6], Alzheimer’s disease [1, 4], and Parkinson’s disease [7, 8].
An electronic version (eMoCA) of the measure, administered on a tablet computer, has recently been developed by the authors of the MoCA. With increasing advances in technology, integration of electronic assessments into clinical practice has the potential to improve the reliability and efficiency of cognitive testing as well as assisting integrating scores into the electronic health record. Although the measure is purported to be the same as the current pencil-and-paper standard, the method of administration may influence test performance and the resulting scores, particularly given that the assessment is typically administered in an older adult population who may have less familiarity with utilizing a stylus pen and tablet computer.
If research can demonstrate that the electronic version of the MoCA performs similarly to the pencil-and-paper version, the ability to utilize a tablet-based assessment would facilitate electronic data capture, and increase both reliability and clinical efficiency by automating scoring procedures.
Prior to clinical implementation of any electronic measure, however, the validity of any new assessment method should be established. The aim of the present study is to determine the validity of the eMoCA by comparing it to the standard paper-and-pencil MoCA in an outpatient memory disorders clinic. Given that content is identical and only the method of administration differs, it was hypothesized that there would be no significant differences between the total score or individual domain scores between the analog version and the electronic version and a high rate of agreement between tasks, thereby providing evidence of convergent validity.
MATERIALS AND METHODS
Participants
This study was conducted in accordance with guidelines on human subjects research and approved by the Cleveland Clinic Institutional Review Board. Participants included 43 new adult patients (Age: M = 71.6, SD = 8.8; range = 47–89) referred for neuropsychological testing after presenting to an outpatient neurology specialty clinic with primary memory complaints. As these patients were new referrals, diagnostic information was not available; differential diagnoses included probable Alzheimer’s disease, frontotemporal dementias, mild cognitive impairment, vascular dementia, and subjective cognitive impairment. Participants had a mean of 14.8 years of education (SD = 2.8; range = 10–20) with a mean estimated premorbid intelligence of 105.7 (SD = 12.8) based on a single-word reading test. The sample was 60.5% female and predominantly Caucasian (86%). Participants who agreed to participate were consented and randomly assigned to receive either the MoCA or eMoCA during their clinical appointment with neuropsychology. Two weeks later, participants retuned to clinic for a research appointment and received the alternate version. All study tests were administered by trained psychometricians. Participants were tested on average 14.1 (SD = 1.8; range = 9–18) days apart. Exclusion criteria included: Non-English speaking, severe mental illness, substance abuse, history of traumatic brain injury, recently prescribed memory loss medication and severe cognitive impairment that would preclude their ability to provide informed consent. In addition, patients with possible dementia with Lewy bodies as determined at their initial neurology visit were excluded due to possible fluctuations in cognition. Post hoc record review showed that one participant had sustained head trauma and thus was excluded from subsequent analysis, resulting in a final analyzed sample of 42 cases. All neuropsychological assessments were completed between August 2015 and May 2017.
Measures
Montreal Cognitive Assessment (MoCA)
The MoCA should be administered and interpreted by healthcare providers with appropriate training. The measure consists of 12 individual tasks, grouped into seven cognitive domains: 1) visuospatial/executive; 2) naming; 3) attention; 4) language; 5) abstraction; 6) memory; and 7) orientation. Scores are summed generating totals for the individual domains and an overall total score out of a possible maximum score of 30. There is also an educational correction (i.e., one point added for individuals with 12 years of education or less). A cutoff score has been established with total scores below 26 indicating cognitive impairment [1]. There are three versions of the paper and pencil MoCA. The original version (7.1) was administered to all participants in this study.
Electronic version of the Montreal Cognitive Assessment (eMoCA)
The electronic version of the MoCA is downloaded as an app and administered by a trained technician on a tablet computer. For the purposes of the present study, a beta version of the app was used, though item content and instructions were identical to the production version. Administration and instructions are similar to version 7.1 of the standard paper-and-pencil MoCA. The test instructions are listed directly on the tablet. A stylus pen is used for responding directly on the tablet by the examiner. The only tasks that require that the patient utilizes the stylus on the tablet are the trail making test, cube copy, and clock drawing. Scores for the individual cognitive domains and total score are automatically tabulated and a pdf copy of the test formatted in the same manner as the paper version is produced. Currently, there is one version of the eMoCA app available.
Statistical analysis
Analyses were conducted using IBM Statistics for Windows (Version 23). Discrepancy scores were calculated by subtracting the total score of the eMoCA from the MoCA for each participant. Concordance correlation coefficients (CCC) and root mean squared differences (RMSD) were calculated between total scores based on an absolute definition of agreement (i.e., X = Y) [9, 10]. The largest possible value of the CCC is 1. The closer to 1, the better the agreement (i.e., CCC = 1 when the scores for the two measurements are identical). The CCC is a preferred method for establishing the rate of agreement between continuous measures, compared to the Pearson correlation coefficients and interclass correlations, because the model takes into account differences in means and variances for the two measures [9, 11, 12]. The RMSD, which is the square root of the average squared difference score, indicates the average amount by which scores tend to differ [9]. The lowest possible RMSD value is 0, and the maximum possible value is the difference between the highest and lowest possible scores on the measures [9]. In order to evaluate for systematic bias, discrepancy scores were used to generate a Bland Altman Plot and to test the difference from zero (i.e., perfect agreement) in a one-sample t-test, and a linear regression was conducted to test for proportional bias. Pearson Correlations were conducted between individual domain scores and total scores on the MoCA and eMoCA, and paired sample t-tests were conducted to compare mean differences between scores.
RESULTS
The average total score on the MoCA was 23.6 (SD = 4.2; range 11–30) and the average total score on the eMoCA was also 23.6 (SD = 4.0; range 16–30); see Table 1. Using conventional standards, performance on both methods ranged from impaired to average, with the majority of participants performing below the 26-point cutoff. Discrepancy scores ranged from –5.0 to 5.0 with a mean difference of 0.07 (SD = 2.3). As shown in Fig. 1, 54% of the sample have a difference score of 1 point or less and 76% of the sample have a difference score within 2 points. The CCC comparing total scores between the MoCA and eMoCA was 0.84, which reflects a high degree of absolute agreement. The RMSD was 2.27 indicating that the average range of differences between total scores on the two measures was just over 2 points. The mean discrepancy between total scores on the two measures was not significantly different from zero (t (41) = 0.20, p = 0.84), indicating that there is no fixed bias, and the Bland Altman Plot (not shown) does not indicate the presence of a proportional bias (R2 = 0.006, F (1, 40) = 0.24, p = 0.63; mean β= 0.08, p = 0.63). Pearson correlation coefficients were all significant and there were no significant mean differences between any of the domain scores or the total score (see Table 1).
Table 1
Means, standard deviations, and correlations between individual domain scores and total scores on the standard and electronic versions of the MoCA
| Domain | MoCA | eMoCA | r | ||
| Mean | SD | Mean | SD | ||
| Visuospatial/Executive | 3.64 | 1.19 | 3.38 | 1.29 | 0.68*** |
| Naming | 2.81 | 0.55 | 2.81 | 0.46 | 0.73*** |
| Attention | 5.36 | 0.88 | 5.36 | 0.79 | 0.41** |
| Language | 2.10 | 0.91 | 2.19 | 1.11 | 0.66*** |
| Abstraction | 1.36 | 0.73 | 1.33 | 0.75 | 0.45** |
| Delayed Recall | 2.69 | 1.87 | 2.71 | 1.85 | 0.84*** |
| Orientation | 5.45 | 0.89 | 5.55 | 0.80 | 0.36* |
| Total Score | 23.64 | 4.19 | 23.57 | 4.02 | 0.84*** |
n = 42. eMoCA, electronic version of the MoCA. *p < 0.05; **p < 0.01; ***p < 0.001.
Fig.1
Frequency distribution of discrepancy scores between total scores on the standard and electronic versions of the MoCA. n = 42. eMoCA, electronic version of the MoCA.
Ten participants had a difference score of 3 points or greater between the MoCA and eMoCA. Secondary analyses compared this subset of participants (high discrepancy group) with those whose difference score was within 0–2 points (n = 32; low discrepancy group). Analysis of variance revealed significant group differences in education (F (1, 40) = 5.35, p = 0.03), such that the high discrepancy group had higher levels of education (M = 16.6, SD = 1.7) compared to the low discrepancy group (M = 14.4, SD = 2.9), and in MoCA total score (F (1, 40) = 5.84, p = 0.02), with the high discrepancy group having lower total scores (M = 21.0, SD = 4.3) than the low discrepancy group (M = 24.5, SD = 3.9). There were no significant group differences in age (F (1, 40) = 0.0, p = 0.98) or eMoCA total score (F (1, 40) = 2.65, p = 0.11).
DISCUSSION
In comparing the electronic version of the MoCA to the established paper-and-pencil version in a clinical patient population presenting with memory concerns, the majority of individuals perform comparably. The CCC was 0.84 out of a possible score of 1 (which would be obtained if every individual had the same total score on both measures), and all scores were highly correlated. In determining the strength of agreement for CCC, it has been proposed that scores should be above 0.90 to indicate adequate strength [13]. However, this interpretive guideline was based on comparing biochemical concentration levels, whereas one may expect greater variability than when comparing human performance on psychometric assessments on different days. In the current sample, the substantial majority of test takers obtained a total score difference of 2 points or less between the two measures. This is consistent with previously reported test-retest reliability of the paper version of the MoCA [1], suggesting that administering the test on an electronic tablet does not significantly alter the reliability of the measure. Importantly, the mean score on the visuospatial/executive domain (trail making, cube copy, clock drawing), the only tasks that required the participant to respond directly on the tablet, did not differ between the MoCA and eMoCA suggesting that the use of a tablet had negligible effects on performance.
While the majority of participants showed minimal difference in performance between the MoCA and eMoCA, there were 10 individuals whose total scores differed by 3 to 5 points, which likely impacted the overall CCC score. Individuals who had greater variability in performance between the MoCA and eMoCA had lower total scores on the MoCA and higher levels of education. Given that individuals with higher education typically perform better on the MoCA [1] (thus the addition of the education correction for those with 12 years of education or less), it may be the case that a greater burden of cognitive impairment may lead to greater variability in test performance. Alternatively, individuals with higher education or worse performance at baseline may have demonstrated more robust practice effects across testing sessions [14, 15]; however, direct measures of practice were not included as part of the study design so this remains a point for further inquiry. Additional post hoc review of these cases revealed that in most instances single points were lost over multiple domains rather than a several points difference in a single domain, which may be accounted for by typical test-retest variability as opposed to a function of the administration method.
This is the first study to compare the novel electronic version of the MoCA to the standard paper-and-pencil version, and there are limitations. While every participant was instructed on how to use the stylus and tablet, specific information on prior tablet experience was not collected. Future research comparing differences in the MoCA and eMoCA as they relate to prior tablet experience will be useful. Furthermore, the current sample is relatively small and is comprised mostly of Caucasian adults with higher levels of education; it is thus unclear to what extent these findings would generalize to less educated and or more demographically diverse populations. In addition, while it is a majority older adult sample, the age range varied broadly. However, this sample does represent a typical referral stream that is likely to be encountered in a general memory disorders clinic, which enhances the generalizability of these findings to similar clinical services. While all participants in the current sample reported primary memory complaints, specific diagnostic information was not available. Exploring differences between specific diagnostic groups or stage of the patients’ neurocognitive disorder will also be important.
Overall, this study demonstrated adequate convergent validity between the well-established paper-and-pencil version of the MoCA and the new electronic version (eMoCA) among an adult population presenting with memory concerns. Utilizing the electronic version of the MoCA in clinical practice and research settings has the potential to improve both the reliability and efficiency of this well-established cognitive screening tool.
ACKNOWLEDGMENTS
Research reported in this publication was partially supported by Keep Memory Alive and an Institutional Development Award (IDeA) from the National Institute of General Medical Sciences of the National Institutes of Health under grant number 5P20GM109025.
Authors’ disclosure available online (https://www.j-alz.com/manuscript-disclosures/17-0896r1).
REFERENCES
[1] | Nasreddine ZS , Phillips NA , Bédirian V , Charbonneau S , Whitehead V , Collin I , Cummings JL , Chertkow H ((2005) ) The Montreal Cognitive Assessment, MoCA: A brief screening tool for mild cognitive impairment. J Am Geriatr Soc 53: , 695–699. |
[2] | Lam B , Middleton LE , Masellis M , Stuss DT , Harry RD , Kiss A , Black SE ((2013) ) Criterion and convergent validity of the Montreal cognitive assessment with screening and standardized neuropsychological testing. J Am Geriatr Soc 61: , 2181–2185. |
[3] | Vogel SJ , Banks SJ , Cummings JL , Miller JB ((2015) ) Concordance of the Montreal cognitive assessment with standard neuropsychological measures. Alzheimers Dement (Amst) 1: , 289–294. |
[4] | Freitas S , Simões MR , Alves L , Santana I ((2013) ) Montreal cognitive assessment: Validation study for mild cognitive impairment and Alzheimer disease. Alzheimer Dis Assoc Disord 27: , 37–43. |
[5] | Markwick A , Zamboni G , de Jager CA ((2012) ) Profiles of cognitive subtest impairment in the Montreal Cognitive Assessment (MoCA) in a research cohort with normal Mini-Mental State Examination (MMSE) scores. J Clin Exp Neuropsychol 34: , 750–757. |
[6] | Smith T , Gildeh N , Holmes C ((2007) ) The Montreal Cognitive Assessment: Validity and utility in a memory clinic setting. Canad J Psychiatry 52: , 329–332. |
[7] | Hoops S , Nazem S , Siderowf A , Duda J , Xie S , Stern M , Weintraub D ((2009) ) Validity of the MoCA and MMSE in the detection of MCI and dementia in Parkinson disease. Neurology 73: , 1738–1745. |
[8] | Gill DJ , Freshman A , Blender JA , Ravina B ((2008) ) The Montreal cognitive assessment as a screening tool for cognitive impairment in Parkinson’s disease. Mov Disord 23: , 1043–1046. |
[9] | Barchard KA ((2012) ) Examining the reliability of interval level data using root mean square differences and concordance correlation coefficients. Psychol Methods 17: , 294. |
[10] | Schuck P ((2004) ) Assessing reproducibility for interval data in health-related quality of life questionnaires: Which coefficient should be used? Qual Life Res 13: , 571–585. |
[11] | King TS , Chinchilli VM ((2001) ) A generalized concordance correlation coefficient for continuous and categorical data. Stat Med 20: , 2131–2147. |
[12] | Lin LI ((1989) ) A concordance correlation coefficient to evaluate reproducibility. Biometrics 45: , 255–268. |
[13] | McBride G ((2005) ) A proposal for strength-of-agreement criteria for Lin’s concordance correlation coefficient, Hamilton, New Zealand. |
[14] | Rapport LJ , Brines DB , Theisen ME , Axelrod BN ((1997) ) Full scale IQ as mediator of practice effects: The rich get richer. Clin Neuropsychol 11: , 375–380. |
[15] | Cooley SA , Heaps JM , Bolzenius JD , Salminen LE , Baker LM , Scott SE , Paul RH ((2015) ) Longitudinal change in performance on the Montreal Cognitive Assessment in older adults. The Clin Neuropsychol 29: , 824–835. |
