Equivalence, modality use and patient satisfaction of telepractice administration of the scenario test in persons with primary progressive aphasia

2.1Aim of the study / Research questions

This study aims to examine telediagnostics in PPA regarding the criteria of equivalence of total scores, the use of communicative modalities, and participant satisfaction for the Scenario Test. The following research questions are sought to be answered:

The hypotheses (H1-H3) of the present paper are:

2.2Study design

The present study was a feasibility study since no study results were available on using the Scenario Test in PPA or on telediagnostics using the Scenario Test in people with aphasia at the time of study initiation. The evaluation was based on the scheme according to Dekhtyar et al. (2020), with participants undergoing face-to-face and telediagnostics in a Cross-Over-Design (see Fig. 1). At the beginning, participants were randomized into two groups. Participants of group 1 started with the telediagnostic assessment; participants of group 2 were tested face-to-face first. Participants took part in the second assessment after 7-14 days. This was considered enough time for a washout period. This procedure was chosen to balance the learning effect of the assessment. As this was a feasibility study, the sample size was calculated to be 20 participants.

Fig. 1

Study programme. First row: Group 1 (D1-11), Second row: Group 2 (A1-10).

2.3Inclusion / Exclusion criteria

To be included in the study, participants had to have already undergone speech-systematic testing in our Memory Clinic at the time of study entry and fulfil the diagnostic criteria of PPA, according to Gorno-Tempini et al. (2011). All PPA variants were included to reflect the heterogeneity of the disorder in the study sample. All levels of PPA severity were included because the Scenario Test is suitable for mild to severe aphasia. Since people with advanced PPA are underrepresented in studies, there is a particular need to include this group in new studies (Rao et al., 2022). Participants were required to have access to a computer and a webcam and to have support of a relative or caretaker during the teleassessment. In case potential participants did not have access to the required equipment, relatives or their outpatient speech and language therapist (SLT), who are regularly involved in the participants clinical care, were contacted and asked to provide the equipment and support for the telediagnostics for the according participant. In case this was not feasible either, participants were asked to take part in a room in the Memory Clinic where the required equipment was available while the examiner was in another room. Further inclusion criteria were an intact visual acuity (if necessary, with visual aid) and an intact hearing function (if necessary, with a hearing aid). Exclusion criteria were other neurological diseases affecting communication ability, such as stroke, Parkinson’s disease or multiple sclerosis. Participants who did not communicate verbally and/or were fully dependent on the assistance of their conversational partner were excluded for ethical reasons as the test situation can be very stressful and pose a high burden to the person (Rogers & Alcaron, 1998). Information from relatives and treating outpatient SLTs served as an indicator for this classification. Participants were not required to have previous experience with the use of an online video platform (Rao et al., 2022).

2.4Ethical aspects

The study was conducted in the Memory Clinic of the Department of Psychiatry and Psychotherapy (University Medical Center of the Johannes Gutenberg University Mainz) from January to March 2022. Participation in the study was for all participants and did not influence further diagnosis or therapy. There was no conflict of interest for the examiner of the study. The study was conducted in accordance with the Declaration of Helsinki and approved by the Ethics Committee of the Federal Medical Association of Rhineland-Palatinate (15960 and 24.11.2021).

Persons with dementia, such as PPA, may not be able to give informed consent. Therefore, the elements of informed consent according to Beauchamp and Childress (2013) were considered. If the SLT and examiner or the doctors involved had doubts about the participant’s ability to give consent, relatives and outpatient SLTs were consulted. During the test sessions, the examiner constantly paid attention to signs of excessive demands (e.g., meta-linguistic comments, changes in facial expressions/gestures, no response). In such cases, the examiner explained again that the participant could withdraw from the study at any point and that this would not have any negative consequences for the participants.

Following the completion of both assessments, a debrief appointment could be scheduled at the request of the participant and/or their relative. The purpose of this appointment was to provide comprehensive information about the diagnostic results and to answer open questions. Further, outpatient SLTs were contacted to inform them about the diagnostic findings and to fulfil the duty of care.

2.5Procedure

Between December 2021 and January 2022, 37 of the Memory Clinic’s patients with PPA were contacted via telephone. Following the telephone information session, written information was sent to interested patients by mail. 21 gave verbal consent to participate and were randomly assigned to two groups. Group 1 was to be initially examined in the telediagnostic setting. The participants were therefore named D1-D11. Group 2 was to receive the diagnosis in the face-to-face setting first, and the participants were consequently designated A1-A10. After randomization, the written consent form was sent in, and a physician explained the aims and benefits of the study as well as the effort associated with the study. After the medical education, the first testing took place. According to the manual of the Scenario Test, 15-45 minutes were estimated for the examination (Nobis-Bosch et al., 2020). Seven to 14 days later, the second testing was conducted.

The same unblinded SLT, referred to as “examiner”, performed the test administration and evaluation. For pragmatic reasons and because validation studies of the Scenario Test show a high correlation between different assessors, this methodological weakness was accepted (Nobis-Bosch et al., 2020).

2.6Setting

Telediagnostics took place via the video conference software Big Blue Button (BBB; invokable GmbH, 2022), which guarantees users 100% data protection and compliance with the Basic Data Protection Regulation. In preparation for the telediagnostics, a link to the BBB study room was sent to the participants via e-mail. Video conferencing via BBB allowed sharing of uploaded slides while communication partners could still see each other in the video. Participants were asked to provide a self-assessment of the statements in a short self designed questionnaire. The answeres were captured through the examiner. A PowerPoint presentation was used during the videoconference. This included the Scenario Test’s items with a blank slide between each item to avoid participants pointing at the images, according to the Scenario Test’s hand instructions. Participants and their supporting relative or SLT were asked to have a paper and a pen ready before the test to enable multimodal communication during the assessment telediagnostics. If the participant had an Augmentative and Alternative Communication (AAC) book for expressive communication, this was also provided. During the assessment, relatives were asked to always stay in the background but within reach to assist in case of technical difficulties. Following the instructions of the Scenario Test, the examiner shared the questionnaire by using the “share screen” function. Participants were then asked to answer each question.

The face-to-face testing took place in an examination room of the Memory Clinic. The room was furnished with as little stimuli as possible and was in a quiet environment so that the participants’ attention was not disturbed. Participants were seated at the table opposite of the examiner and were provided with a paper and a pen. Furthermore, the test documents with the items of the Scenario Test were placed on the table.

The video camera, which documented the face-to-face and telediagnostics, was placed in a way to capture both the examiner’s and participant’s facial expressions and gestures during the diagnostic process.

2.7Outcome measures

The Scenario Test, which measures communicative-pragmatic skills using six everyday situations represented by line drawings, was used as the primary measurement. The instructions and test administration followed the German manual instructions (Nobis-Bosch et al., 2020). There are three items per situation leading to a maximum total score of 54 points. The total score reflects how much help was needed by the participant to communicate. At the proposition level, verbal and nonverbal responses were coded. Further, the responses were coded in terms of their modality. The test was stopped if the person with PPA did not respond to the practice items and the first scenario (Nobis-Bosch et al., 2020). Video recordings of the test situation enabled a subsequent evaluation with a simultaneous collection of quantitative and qualitative results.

As a secondary measurement tool, a short questionnaire was used to evaluate participant satisfaction with the assessment. The questionnaire comprised eight items and was self-designed based on a participant satisfaction questionnaire by Hill et al. (2009). The same questionnaire was used at both assessment conditions. If necessary, the examiner assisted in answering the questions by reading the question aloud and clarifying the principle of scale assessment in simple terms. The first item of the short questionnaire measured the affinity for technology. The following seven items measured satisfaction with the previously conducted examination. Satisfaction with the acoustic and visual presentation was evaluated. Furthermore, well-being, subjective satisfaction, willingness for further participation, comfort in the respective diagnostic setting, and satisfaction with the own performance were queried. The questionnaire used a five-point Likert scale ranging from “do not agree at all” to “fully agree”. For better readibililty, each questionnaire item was printed on one page in a large font and landscape format. If a participant answered “do not agree at all” to continue participation, the assessment was discontinued. In the event of negative satisfaction scores after the first assessment, participants were also informed again that participation was completely voluntary and that they could withdraw from the study at any time.

2.8Data analysis

To evaluate the results of the Scenario Test and short questionnaire, the data were imported into SPSS (version 27, IBM^®). Both, descriptive and inferential statistics were conducted. Various forms of presentation (boxplots, pie charts, G-G charts) were created. Further, a Wilcoxon test, equivalence test, McNemar test and Kendall-Tau-b were performed. Omni Calculator (Szczepanek, 2022) was used to perform the McNemar test. In the following, the quantitative analysis procedure regarding the equivalence of the total scores, the use of modalities and participant satisfaction will be described.

For the quantitative analysis, either an intention-to-treat (evaluation in the group after randomization) or a per-protocol analysis (evaluation when the study protocol was followed) was performed. The aim was to obtain as high external and internal validity as possible while counteracting statistical bias. Dropouts were not generally excluded from the analysis to reflect the heterogeneity of PPA in terms of severity. Instead, missing data were considered a noteworthy variable to capture the feasibility of telediagnostics in different settings and with different participants (Tabachnick & Fidell, 2013). Missing values were addressed depending on the research question and the failure process. Listwise case exclusion, pairwise case exclusion, and various imputation procedures were considered (Enders, 2010).

The primary target variables for the evaluation were the total scores in the Scenario Test and the modalities used according to the coding already described for face-to-face and telediagnostics, which were collected using the Scenario Test. The boxplot was used for the presentation. Further, various measures of location and dispersion were used for descriptive data analysis. A Q-Q plot was used to check whether the distribution of total scores conformed to a normal distribution (Field, 2018). Due to the specifics of the cross-over design, a preliminary test should be performed before inferential statistical analysis to verify that the washout period was sufficiently long and that no learning effects occurred between time point one and time point two. For this purpose, the sums of the measured values should be considered, and a test for independent samples should be used (Wellek & Blettner, 2012). In the case of a normal distribution, a t-test should be used at this point, whereas in non-parametric data, the Wilcoxon test would be indicated (Field, 2018). The test statistic used was p < 0.05.

Furthermore, an equivalence test was performed to check whether the difference between the two diagnostic settings was clinically relevant or whether the examination was equivalent in both diagnostic settings. For this purpose, the equivalence range was first defined, i.e., the range in which differences can be tolerated as clinically irrelevant (Wellek, 2002). According to the Scenario Test’s manual, ≥7 performance points in the total score was designated as clinically relevant (Nobis-Bosch et al., 2020). For this reason, the equivalence range (also MCID: minimal clinical important difference) was set at [-7; 7]. Subsequently, the confidence interval was calculated to check whether it lies within the equivalence range.

The use of the different modalities was evaluated descriptively and with the help of a frequency table. A circle diagram was used to visualise the results. The changes regarding the modalities used were additionally tested inferentially. Since the modalities used were related dichotomous variables at two test time points, the McNemar test was selected (Field, 2018). The alpha value was 0.05.

The secondary target variable was the degree of participant satisfaction, which was recorded with the aid of the short questionnaire. A bar chart was drawn to visualize the degree of agreement at item level. An equivalence test was used. A percent clinical agreement of > 80% has been described as a high level of agreement by Hill et al. (2009). At the same time, the same authors have defined the critical difference of one scale point as a relevant difference for five-point Likert scales. For this reason, the MCID in this study is set at [-1; 1].

3.1Description of the sample

After completion of recruitment and randomisation, six participants dropped out and five participants were excluded. Dropouts occurred due to personal reasons like feeling of excessive demand, marked progression of symptoms, or an upcoming relocation. Two participants (D3, D5) had severe language impairments, so that the Scenario Test was discontinued according to the criteria described in the manual and two participants (D4, A5) withdrew from the study. One participant (D8) experienced an increase of psychiatric symptoms between the two assessment timepoints, so that he/she had to be admitted to an inpatient facility and study participation was no longer feasible. Four participants from the total sample (D3, D4, D5, A5), took part in both test sessions but showed difficulties completing the short questionnaires, resulting in missing measurements. At the end of the study (03/2022), ten complete data sets and five data sets with partially missing information were available (see Fig. 2). In the following, those participants are described who were able to take part in at least one testing and whose data were included in the statistical analysis (N = 15). The mean age of participants was 69.27 (SD: 6.31) with a mean age of 65.27 at symptom onset (SD: 6.24). The sample consisted of 60% men and 40% women. The years of education was on average 13.67 (SD: 2.54). Of 15 participants, 33.33% had a lvPPA, 13.33% had a nfvPPA and 53.33% had a svPPA. Seven out of eight participants with svPPA had causal Frontotemporal Lobar Degeneration, and four out of five participants with lvPPA had underlying Alzheimer’s pathology. The only participant with nfvPPA in the sample showed AD as the aetiology of PPA. In 53.3% of the sample, depression was also present. 80% of the participants were undergoing speech therapy at the start of the study. 93.3% of the participants were monolingual German speakers, one participant was an English native speaker. An overview of the demographic data of the sample is given (Table 1).

Fig. 2

Flow chart for the recruitment of the participants.

3.2Evaluation of equivalence, modality use and participant satisfaction

The primary outcomes of the study participants (N = 15) are presented below (Table 2). Analysis of the primary outcome measure showed intra-individual clinically relevant differences in total scores of Scenario Test (MCID≥7 performance points) between the two test time points of six participants. The other nine did not show any clinically relevant differences. While two participants in group 1 showed ceiling effects (D6, D7), no one in group 2 reached the maximum score. Two participants in group 1 met the dropout criteria and thus achieved 0 total points. These factors led to a wide dispersion in group 1 (see Fig. 3). The Wilcoxon test reveals no significant relationship between the order of the examinations (face-to-face at time point one compared to time point 2) and the total score (p = 0.407). The confidence interval of the median calculated by bootstrapping is [-4.5; 0] and thus lies within the defined equivalence range of [-7; 7].

Fig. 3

Boxplot for group comparison in a telediagnostic (red) and face-to-face (orange) setting of total scores after intention-to-treat analysis (N = 15). A6 seen as outlier (∘) with significant lower scores in both diagnostic settings (telediagnostic: 2; face-to-face: 6).

A more considerable variation in the use of different modalities was seen in the face-to-face, compared to telediagnostics (see Fig. 4). The preferred mode of communication in both diagnostic settings was speaking. The results of the McNemar test showed no significant differences between the use of modalities in face-to-face and telediagnostics (p = 0.09 - 0.85).

Fig. 4

Modality use in a telediagnostic (left) and face-to-face setting (right).

For the evaluation of secondary outcome measures, the statement “do not agree at all” was assigned the value -2, “rather disagree” the value -1, “partly agree” the value 0, “rather agree” the value 1 and “fully agree” the value 2. Overall, the analysis showed positive satisfaction ratings for both settings, but higher satisfaction scores for the face-to-face setting (see Fig. 5). Group 2 reported higher overall satisfaction scores and showed less dispersion (face-to-face: M = 10; SD = 2; telediagnostic: M = 8.5; SD = 2.74) compared to group 1 (telediagnostic: M = 7; SD = 10.74; face-to-face: M = 7.25; SD = 9). The results of the bootstrapping show that in the present sample, no equivalence can be assumed concerning participant satisfaction in face-to-face and telediagnostics, as the confidence interval [-2;0] exceeds the equivalence range [-1;1].

Fig. 5

Participant satisfaction with telediagnostic (red) and face-to-face (orange) setting.

4.1Limitations

The test administration and evaluation by the same, unblinded person reduces the objectivity of the present work. In future studies, however, the evaluation should be carried out by two independent examiners. Further, the small sample of 15 respondents limits the significance of the statistical analysis. The implementation of larger equivalence studies is indicated to achieve higher external validity. The group of subjects was heterogeneous overall. Although the numerous drop-outs reduced the significance of the statistical results, new insights into the feasibility of telediagnostics for patients with PPA can be made. The influence of psychiatric symptoms on the feasibility of telediagnostics offers grounds for further research.

4.2Conclusion

The present study points out the opportunities and limitations of telediagnostics for people with PPA. Evidence was found that the Scenario Test in telediagnostics is feasible with PPA patients and that the results are comparable to face-to-face testing. This opens perspectives for communicative-pragmatic assessment of PPA in Dutch-, English-, and German-speaking countries. Although differences were non-significant, more communicative modalities were used in face-to-face compared to telediagnostics. This aspect should be investigated in future studies. Regarding participant satisfaction, a basically positive attitude was recorded for both settings, which is also in line with existing evidence (Barton, Morris, Rothlind & Yaffe, 2011). The trend towards a preference for the face-to-face setting in this sample is noteworthy and should be investigated further. Nevertheless, there is a need to investigate all research questions examined in the study in a larger, representative sample of patients with PPA.

Feedback from participants indicates that people with PPA positively receive telediagnostic solutions, as shown in other studies earlier. The dropouts contributed to a reduced sample. Nevertheless, the high drop-out rate in a neurodegnerative disease is not atypical and should be seen in connection with concomitant neuropsychiatric disorders. Seeing the high potential of telediagnostics to improve care for people living with PPA, there should be more research on developing telediagnostic solutions in speech therapy. To ensure the quality of telediagnostics in PPA, further training should in future also address special features of the digital setting and the use of the procedure in people with dementia-related speech disorders.