How should a virtual agent present psychoeducation? Influence of verbal and textual presentation on adherence
Abstract
BACKGROUND AND OBJECTIVE:
With the rise of autonomous e-mental health applications, virtual agents can play a major role in improving trustworthiness, therapy outcome and adherence. In these applications, it is important that patients adhere in the sense that they perform the tasks, but also that they adhere to the specific recommendations on how to do them well. One important construct in improving adherence is psychoeducation, information on the why and how of therapeutic interventions. In an e-mental health context, this can be delivered in two different ways: verbally by a (virtual) embodied conversational agent or just via text on the screen. The aim of this research is to study which presentation mode is preferable for improving adherence.
METHODS
: This study takes the approach of evaluating a specific part of a therapy, namely psychoeducation. This was done in a non-clinical sample, to first test the general constructs of the human-computer interaction. We performed an experimental study on the effect of presentation mode of psychoeducation on adherence. In this study, we took into account the moderating effects of attitude towards the virtual agent and recollection of the information. Within the paradigm of expressive writing, we asked participants (
RESULTS AND CONCLUSION:
We found that both the attitude towards the virtual agent and how well the psychoeducation was recollected were positively related to adherence in the form of task execution. Moreover, after controlling for the attitude to the agent and recollection, presentation of psychoeducation via text resulted in higher adherence than verbal presentation by the virtual agent did.
1.Introduction
Imagine a virtual embodied conversational agent asking you to confront your worst memories. How would the agent convince you to do this? This is the question we wish to address in this paper. In our research we studied a talking virtual embodied agent, a human-like graphical interface, controlled by a computer, that can help people with recollecting their worst memories. This virtual agent was a part of an autonomous e-mental-health (AEMH) application, which can be used to give expressive writing or Post-Traumatic Stress Disorder (PTSD) therapy. AEMH applications are designed to monitor, assist and treat mental health problems without any direct human involvement. With rising costs in health-care [1], long travel distances to clinics and access to care in rural areas [2], and stigma on seeking help from a therapist [3] all preventing people from seeking care, these applications can fill an important gap. Aside from being available remotely and being privacy sensitive [4], they are also cost-effective in use [5], and efforts are also being made to reduce cost in the development of these interventions [6]. Virtual agents can play a central role in AEMH applications, contributing to the effectiveness and acceptability of the system and intervention [7, 8, 9, 10]. One of the primary functions of our virtual agent is to maximise adherence, i.e. ensure that people recollect their most negative memories in as much detail as possible.
Adherence is a major factor in health care. The World Health Organization identifies it as a primary determinant of the effectiveness of treatment and defines it as “the extent to which a person’s behaviour – taking medication, following a diet, and/or executing lifestyle changes, corresponds with agreed recommendations from a health care provider” [11]. This importance of adherence is widely accepted. However, measuring to which extent behaviour follows recommendations is challenging and the exact way to do so it differs greatly. For this reason, many regular mental health interventions only consider whether a session was done at all, or rely heavily on self-report measures [12]. Applied to e-health, adherence refers to the extent to which the system’s recommendations are followed, because a health-care provider is not directly involved. This can be performing a session, showing a certain behaviour or even taking medication if the system advises to do so. In a review of the impact of adherence on e-therapies, Donkin et al. found many different measures of adherence, such as completed modules, the number of visits to a page, the number of log-ins and self-report measures [13]. The question arises though, if any of these measures fully describe adherence, as there is a difference in passively viewing information and actively applying its contents [14]. For this reason, we adopt a two-fold definition for adherence to a virtual agent. The first aspect of adherence is if recommendations of the agent are followed, i.e., whether tasks are executed (this could also mean the suppression of something). The second aspect is how well these tasks are performed, i.e., if they are not glossed over, and details are left out.
Psychoeducation is an important method to increase adherence in therapy for mental disorders. Psychoeducation can be an intervention of its own, informing patients on their disorder and how to deal with it [15, 16, 17]. It is, however, also used as part of more comprehensive interventions. In this case, it often entails information on how the therapy works and why patients will have to perform certain tasks [18, 19]. In the remainder of the article, we will use the term psychoeducation to refer to this latter form of information included in an intervention. Psychoeducation is especially relevant for therapies where patients need to perform difficult tasks such as changing habits or facing fears, as is the case in expressive writing or PTSD treatment. It aims to increase adherence by firstly explaining how to perform the tasks correctly. Additionally, it explains why performing these tasks will be beneficial [20]. Virtual agents can present psychoeducation in different ways. One possibility is that the virtual agent asks patients to read the psychoeducation text on the screen. Another option would be for the virtual agent to give the information verbally. Of course, human therapists generally present the information verbally, accompanied by a written brochure. Likewise, it is possible to include the option to read back the information on the screen in an AEMH application as well. Still, the choice for the initial presentation mode is an important one. Both options might have their advantages on maximising adherence to the virtual agent, which leads to our first hypothesis: presentation mode of psychoeducation influences adherence.
Several factors exist through which the mode of presentation might affect adherence. Firstly, we predict that textual presentation of psychoeducation will maximise recollection of the information. By better remembering the psychoeducation, specifically how and why actions need to be taken, adherence improves. Secondly, we predict that verbal presentation of psychoeducation will maximise a positive attitude towards the virtual agent, leading to improved adherence. These two factors can be considered mediators of the relationship between the presentation mode and adherence. Because attitude to the agent is predicted to be higher for verbal psychoeducation and recollection for textual psychoeducation, their combined result on adherence is unclear. This leads to our second hypothesis: the effect of presentation mode of psychoeducation on adherence is mediated by how well the information is recollected and the attitude towards the virtual agent. Together with hypothesis 1, we can visualise the two hypotheses as shown in Fig. 1.
Figure 1.
Hypotheses on how mode of presentation will influence adherence via the mediators of attitude towards the virtual agent and recollection of the information.
The first mediating factor predicted by our hypothesis is recollection, how well the given information is remembered. Webb and Wallon work already showed a difference between listening and reading, indicating that text which is read is remembered better [21]. Later work by Rubin et al. showed similar results, also measuring mental effort, which was higher for reading [22]. In the domain of human-computer interaction, verbal-visual learning aids were found to be more beneficial for retention than verbal aids [23]. However, work with virtual agents shows that people often prefer verbal delivery above text [24, 25]. Important to note though is that most work with virtual agents has focused on short assignments and messages while psychoeducation usually consists of much longer texts. It is also important to consider that the speech of a virtual agent might not be as easy to understand as that of a human because many agents use a text-to-speech system [26]. This study aims to provide insight into the effect of a text-to-speech voice on the recollection of longer texts.
The second mediating factor we predict is the attitude towards the virtual agent. Presenting psychoeducation verbally can increase credibility and trust in a virtual agent, as it presents itself as an expert [27]. Trust in a virtual agent is an important aspect of patient-provider health communications and patient adherence [28, 29, 30]. Moreover, both trust and perceived expertise are important aspects of therapeutic alliance [31]. This refers to the bond between a therapist and patient, which is important for both therapy outcome [31] and adherence [32, 33]. Therapeutic alliance has been shown to be relevant for virtual agents [34, 35] as well as human therapists. However, it is not exactly clear what effect mode of presentation of psychoeducation will have on trust and alliance, because of the many aspects involved. Studies show that factors such as the type of language used [36], interpersonal encounter skills [37], and introduction style [38] all influence alliance. However, these studies have been done with human therapists and might not translate to a virtual agent. Agent competence has been shown to affect trust in the agent, but this effect needs to grow over time [39]. Another factor is that virtual agents often use digital voices and do not necessarily bring the same social presence as a human would. Although some indications have been found that even the addition of a text-to-speech voice can increase trust, this effect has been found in the context of consumer trust, not health-care [40].
In this study, we aim to investigate how presentation mode of psychoeducation affects adherence to a virtual agent. Specifically, we compare verbal presentation via a digital voice spoken by a virtual embodied agent with presentation through text. We also consider the effect of the mode of presentation on how well the information is recollected, and the attitude towards the virtual agent. In the remainder of this paper, we present the methods of the experiment we conducted and conclude with the results of our experiment and discussion.
2.Method
The gold standard in evaluating e-health interventions is a coherent set of dedicated evaluations in the development stage to test program components, followed by a Randomized Controlled Trial (RCT) when the intervention is stable [41, 42, 43, 44]. One recommendation for these evaluations in the development stage is the use of proximal outcome measures such as self-efficacy and adherence [45]. These measures serve as a quicker and efficient way to test if system components have the proposed effect. These studies are usually performed with a non-clinical sample because clinical samples cannot be presented with only a small part of an intervention for ethical reasons. The validity of this choice is supported by the continuum hypothesis [46], which defines mental states on a continuous scale and has been supported for disorders such as psychosis [47], depression [48] and PTSD [49]. This means that many symptoms of clinical populations may also be present in the healthy population, be it to a lesser degree. In this paper, we wish to evaluate how a virtual agent can best present psychoeducation to maximise adherence. Doing this, we took the approach of evaluating a small part of a therapy (psychoeducation), using a proximal outcome measure (adherence) in a non-clinical sample. The psychoeducation was part of a larger intervention, included in the beginning and meant to increase adherence in the latter part.
A between-subjects experiment with two conditions was conducted. In one condition the psychoeducation was delivered verbally by the virtual agent via text-to-speech. In the other, the psychoeducation was presented as text on the screen. The virtual agent was present for the rest of the experiment in both conditions, and was the one prompting the participant to pick and describe a stressful memory. The design of this experiment was approved by the human research ethics committee of Delft University of Technology.
2.1Participants
Forty-six participants (38 males, age M 22.9, SD 3.8) were recruited from the university staff and student population and completed the experiment. All participants had Dutch as their first language. All participants were given experiment information and consent form prior to the experiment. One participant in the text condition voluntarily dropped out due to personal reasons related to the experimental task (not included in the 46).
2.2Therapeutic task
The subject of psychoeducation differs per therapy and application, as the goal is to inform patients on what that specific therapy is about. To test the effect of presentation mode on adherence, a therapeutic task was chosen which could be performed by a non-clinical person for whom it was yet unpleasant to perform so that a ceiling effect in adherence was avoided. This task was expressive writing. Expressive writing is a therapeutic tool [50] where healthy participants have to write about the most negative events in their life, which evoked the most stress. Expressive writing can have several beneficial effects such as reduced stress, a decrease in healthcare use, and an increase in study performances [51, 52, 53, 54]. In the traditional format, individuals write about their memories for 15 minutes on three consecutive days, although studies have also shown it to work for shorter time spans and with less time between sessions [55, 56].
2.3Therapy system, virtual agent and psychoeducation
The therapy application participants used was an adapted version of the Multi-Modal Memory Restructuring (3MR) system [57]. This application was originally developed for PTSD therapy, where the therapy is mainly focused on the recall and description of traumatic memories. The 3MR application is therefore very suitable for an expressive writing task, as it also revolves around describing negative memories.
The 3MR system has a diary function where people can describe their memories. At the top of the screen, a timeline of the user’s life is displayed. Memories can be added to this timeline and described in a digital diary. The user of the diary can, besides written text, also include several types of media such as pictures, music or videos. The images, video and music can be added from the computer or via internet sites such as Google image search and YouTube. Locations can be further described using Google maps and emotions can be added in the form of an emotion word. All media are added to the diary in the shape of a small thumbnail, which can be moved around on the screen.
Figure 2.
The digital diary filled in with an example memory. The items show text, maps, emotion words and images. The bar at the top shows the memory on a timeline. The virtual agent is shown on the right. This screenshot is translated into English; all participants worked with the original Dutch version.
An important aspect of the 3MR system is a virtual agent. Two versions of the virtual agent were used, one female and one male. Gender was matched with the participant to avoid possible effects of interacting with someone with a mismatched gender [58, 59, 60, 61]. The agents were rendered by Unity and displayed general idle behaviour and mouth movement when speaking. The voices of the virtual agent were generated with the text-to-speech engine Fluency, which specialises in Dutch voices.11 Figure 2 shows a screenshot of a diary with the timeline at the top and the virtual agent on the right.
The psychoeducation included in this experiment was based on existing research on expressive writing which can be found in [62]. The psychoeducation started with a small introduction, followed by possible benefits, how to write to attain the best effect and finished with a short conclusion. The brief introduction stated that extensive research was done on expressive writing and its possible benefits. After this, these benefits were listed: e.g., personal growth, less anxiety, a decrease in health-care utilisation and improved academic performances. After this, instructions were provided on how to maximise the benefits of expressive writing. For example, it was mentioned that it only works for negative memories which evoke stress, works best when writing about your deepest emotions and that the person had to link the memory to current and past events. All examples for both positive effects and instructions were chosen after reviewing previous studies on expressive writing [51, 55, 52, 53, 63]. The psychoeducation concluded with linking expressive writing to the digital diary, and people were asked to keep the information in mind during the remainder of the experiment.
2.4Procedure
At the start of the experiment, all participants received information on the privacy of their data. This included the information that their memory descriptions would be stored to gather statistics such as word count, but that nothing would be read by other people. This was done because we did not want participants to be influenced by privacy concerns during the task. Participants were also informed that they would be able to see exactly what statistics would be stored at the end of the experiment and that they could still withdraw consent at that time. After the information procedure, there was a 3.5-minute video on how the 3MR system worked. In this video, the 2 functions of the diary were explained. Afterwards, there was a short practice session with the diary led by the experimenter. The virtual agent was not yet present during this exercise.
After the practice session, the experimenter left the room, and the virtual agent took over. The virtual agent stayed present during the rest of the experiment, guiding the participant through the assignments with the system that formed the intervention.
The first assignment for the participants was to list their five most negative memories and add a one-sentence description. They were asked to number the memories from most to least stressful, and additionally, assign a number between 0 and 100 to every memory to indicate how stressful they were. In the remainder of the experiment, participants chose one of these five memories to describe in detail. This first task was inserted to get a measure of how stressful the eventually selected memory was. After five memories were listed, the participants received the psychoeducation. In one condition the virtual agent presented the information verbally via text-to-speech, in the other condition the information was present in the form of text on the screen and the virtual agent simply asked participants to read the text.
Next, participants had to pick one of their five memories to write about in detail. The virtual agent specifically asked the participants to pick one memory, and advised them to pick the most negative. It reminded them that expressive writing works best for the most negative memories, those that evoke stress. This is the recommendation for which we wish to maximize adherence. As a final step, the participants created a diary page for the chosen memory and were asked by the virtual agent to write about the memory in detail in the diary. This was the actual expressive writing task, slightly adapted from the original [50] as our participants only followed one session of writing for 15 minutes instead of three.
At the end of the 15 minutes, the virtual agent asked the participants to fill out several questionnaires on paper and afterwards call in the experimenter. The experimenter closed the experiment by allowing the participants to see which data would be stored, namely the diary descriptions and several statistics such as the word count and the number of emotion items. The data from the diary was shown to the participants in such a way that the experimenter could not see the content. No participants objected to their data being stored for analysis at this point.
2.5Measures
Several measures were collected using both behavioural data and questionnaires. A translation of all questionnaires can be found in [64]. All original questions were in Dutch.
2.5.1Primary measures
Suggestion adherence was the measure of how stressful the memory chosen for the detailed description was. This measure describes how well participants complied with the virtual agent’s suggestion to describe a very negative memory that evoked stress. The initial assignment asked to list the five most stressful memories experienced, of which only one was described in detail in the diary. To judge how negative the chosen memory was, the participants were asked to order the memories from least stressful to most stressful. This gave a ranking score to each memory. Additionally, each memory was rated on a scale from 0 to 100 on stressfulness to get a stressfulness score. This was added to control for interpersonal differences in the stressfulness of past experiences.
Task adherence refers to how well participants performed the task they were given by the virtual agent, namely describing their memory in detail. All participants spent 15 minutes describing their chosen memory in detail in the virtual diary. From this description, the total number of words typed, which emotion items were added to the diary and the total number of items added were recorded. Taken together these measures formed an index for task adherence.
Subjective adherence was measured with a questionnaire asking for the participants’ experiences in recollecting the memory. The questionnaire asked if people had truly described their deepest feelings, how confronting the memory was, how much they linked it to other memories and if it affected how they saw the memory now. These were all recommendations given in the psychoeducation, which the virtual agent asked the participant to follow. All four items were posed in the form of statements such as ‘I truly described my deepest emotions and feelings’. All questions were answered on a continuous scale from don’t agree to fully agree in the form of a cross mark on a line. The line had a distinct middle point and incremental marks.
2.5.2Mediating measures
Recollection of the psychoeducation was measured with three open questions. The first question asked for the term used for writing about negative experiences, the second in which areas it could have positive effects and the third what was important during the writing. All of these asked for information given as psychoeducation at the beginning of the experiment. For the first question, a correct answer yielded 2 points, a wrong answer 0. For the second and third question, all correct answers yielded 1 point, and for all concepts written down which were not correct, 1 point was deducted. This resulted in a score between 0 and 5 for questions two and three. The sum of these scores was used as a measure for how well the information was recollected.
Agent attitude was measured with a questionnaire asking for the participant’s attitude towards the virtual agent. The questions were on the topics of trust, expertise, realism, perceived amiableness and the adherence of the participant to the agent. All 22 questions were answered on a 7-point Likert scale.
2.5.3Exploratory measures
Psychoeducation attitude was measured to check for influences of how the information was received. A questionnaire was posed with the topics of trust in the information, how interesting it was, how it was presented and the degree of clarity. All 17 questions were answered on a 7-point Likert scale.
Function usefulness measures were taken for the psychoeducation, the virtual agent, the diary functions and the agent speaking. These four questions asked if these functions added something to the task or not. All questions were posed in the form of statements such as ‘The presence of the virtual agent added something to this exercise’. All questions were answered on a continuous scale from don’t agree to fully agree in the form of a cross mark on a line. The line had a distinct middle point and incremental marks. All lines were measured to determine where the cross stood and transformed to a value between
2.6Data preparation and analysis
Suggestion adherence was calculated from both the stressfulness and ranking score of the chosen memory. These two scores were found to be positively correlated
To create a single task adherence index, z scores were calculated for the number of diary items added, the number of specific emotion-word items and the total number of words written in the diary over all participants. These three scores were averaged to form a single index score.
Subjective adherence was measured on a continuous scale with four questions. These four questions measured the different aspects of adherence to the virtual agent. All lines were measured to determine where the cross stood and transformed to a value between
For the questionnaire on attitude towards the virtual agent and attitude towards the psychoeducation, the scores for the negatively phrased questions were inverted. Both questionnaires were tested with Cronbach
SPSS version 22 was used for data management and analysis. A Kolmogorov-Smirnov test for normality showed that suggestion adherence and the function usefulness measures deviated significantly (
Table 1
Descriptive statistics and one-sample t-test with bootstrapping results
| One-sample t-test | |||||||
|---|---|---|---|---|---|---|---|
| 95% CI | |||||||
| Measure | Type | Mean | SD | Neutral point | Sig | Lower | Upper |
| Agent attitude | 7 pt. Likert | 4.38 | 0.75 | 4 | 0.002 | 0.17 | 0.61 |
| Psychoeducation attitude | 7 pt. Likert | 4.96 | 0.63 | 4 | 0.001 | 0.79 | 1.14 |
| Function use diary | Analog | 0.42 | 0.37 | 0 | 0.32 | 0.53 | |
| Function use psychoeducation | Analog | 0.33 | 0.47 | 0 | 0.20 | 0.45 | |
| Function use agent | Analog | 0.05 | 0.60 | 0 | 0.586 | 0.21 | |
| Function use speak | Analog | 0.06 | 0.62 | 0 | 0.513 | 0.22 | |
| Subjective adherence | Analog | 0.02 | 0.36 | 0 | 0.756 | 0.12 | |
| Suggestion adherence | Behavior | 0.50 | 0.74 | ||||
| Chosen score | Behavior | 81.17 | 16.60 | ||||
| Chosen rank | Behavior | 4.54 | 0.81 | ||||
| Task adherence | Behavior | 0.00 | 0.65 | ||||
| Nr. items made | Behavior | 13.91 | 5.50 | ||||
| Nr. emo-items made | Behavior | 4.61 | 3.24 | ||||
| Wordcount | Behavior | 419.74 | 340.03 | ||||
| Recollection | Behavior | 3.65 | 2.14 | ||||
Figure 3.
Frequency Histograms of the perceived usefulness of the virtual agent and the fact that it spoke, as quizzed with the statement ‘The virtual agent is a useful addition to this exercise’ and ‘The fact that the virtual agent spoke was a useful addition to this exercise’.
3.Results
Table 1 shows the descriptive statistics for all measures collected. One-sample t-tests indicate that agent attitude, psychoeducation attitude, usefulness of the diary, and usefulness of the psychoeducation were all significantly higher than the neutral point on the test-scale. For subjective adherence, usefulness of the virtual agent, and usefulness of the virtual agent speaking, no significant difference from the neutral position was found. The values of usefulness of the agent and usefulness of the agent speaking were found to be highly correlated
Independent t-tests were done to check for differences between conditions for the primary, secondary and exploratory measures. The full results are shown in Table 2. A trend was found for agent attitude, which was more positive in the verbal condition. A trend was also found for perceived use of the diary, which was more positive in the text condition. However, no significant differences were found. This means that no effect of mode of presentation was found on agent attitude, recollection or adherence. As these are all premises for hypothesis 2, a mediation analysis was unnecessary to conclude that our data did not support hypothesis 2.
Table 2
Descriptive statistics and independent-sample t-test with bootstrapping checking for differences between conditions
| Mean | 95% CI | ||||
|---|---|---|---|---|---|
| Measure | Verbal | Textual | Sig | Lower | Upper |
| Subjective adherence | 0.01 | 0.03 | 0.86 | 0.20 | |
| Suggestion adherence | 0.48 | 0.51 | 0.90 | 040 | |
| Task adherence | 0.11 | 0.24 | 0.13 | ||
| Agent attitude | 4.58 | 4.18 | 0.08 | 0.81 | |
| Recollection | 3.6 | 3.70 | 0.66 | 1.11 | |
| Psychoed. attitude | 4.98 | 4.93 | 0.19 | 0.39 | |
| Function use diary | 0.52 | 0.32 | 0.06 | | 0.39 |
| Function use psychoeducation | 0.34 | 0.32 | 0.90 | 0.29 | |
| Function use agent | 0.06 | 0.04 | 0.90 | 0.39 | |
| Function use speaking agent | 0.15 | 0.36 | 0.52 | ||
Table 3
Correlations between measures
| Measures | Subjective | Suggestion | Task | Agent | Recollection | Psychoeducation |
|---|---|---|---|---|---|---|
| adherence | adherence | adherence | attitude | attitude | ||
| Subjective adherence | 1 | 0.46** | 0.26 | 0.11 | 0.05 | 0.25 |
| Suggestion adherence | 0.46** | 1 | 0.18 | 0.17 | 0.07 | 0.14 |
| Task adherence | 0.26 | 0.18 | 1 | 0.31* | 0.39** | 0.22 |
| Agent attitude | 0.11 | 0.17 | 0.31* | 1 | 0.64** | |
| Recollection | 0.05 | 0.07 | 0.39** | 1 | 0.13 | |
| Psychoed. attitude | 0.25 | 0.14 | 0.22 | 0.64** | 0.13 | 1 |
*The 95% Confidence Interval does not include 0; **The 99% Confidence Interval does not include 0.
Although no effect of mode of presentation on adherence was found, other aspects might still have an effect. To explore the relationships between the different measures, a correlation was performed on all primary and mediating measures, as well as on the attitude towards the psychoeducation. Results show that the suggestion adherence was significantly positively correlated with both suggestion and task adherence. Subjective and task adherence were, however, not correlated. This means that the negativity of the chosen memory was not related to how well that chosen memory was described. Agent attitude and psychoeducation attitude were also positively correlated. A correlation between recollection and adherence was found for task adherence. Similarly, agent attitude was significantly correlated with task adherence. Full statistics for the correlation analysis can be found in Table 3.
Given the correlations between recollection and task adherence, and between agent attitude and task adherence, these were considered extraneous variables affecting task adherence. Because presentation mode did not influence either recollection nor agent attitude, an ANCOVA with covariates recollection and attitude to the agent was performed on task adherence. The analysis found a significant effect for presentation mode on task adherence (
Figure 4.
Model showing the result that presentation mode effects adherence to the agent, and attitude to the agent and recollection are extraneous variables.
4.Discussion and conclusion
Overall participants held a positive attitude towards the psychoeducation and the virtual agent, as shown by the questions about their usefulness. However, room for improvement exists for both aspects, with average scores being closer to the neutral point than to the maximum. Another finding was the subjective usefulness of several functions during the given task. Firstly, both the digital diary and psychoeducation were considered useful additions to the task. More interestingly, there was no significant consensus on whether the virtual agent and the fact that it spoke were useful. The results indicated a divide between people finding the agent either useful or not useful, with few people in the middle. This indicates that a virtual agent might not be a suitable addition for everybody. Therefore inter-personal differences might have to be considered in the decision to add a virtual agent or not. However, as preference did not correlate with any of the adherence measures or recollection of the psychoeducation, the behavioural consequences of a preference for the agent are less clear. One option would be to let people themselves decide if they wish to have a virtual agent or plain text when working with an autonomous e-mental health application.
Considering the two hypotheses formulated, a relationship was found between agent attitude and task adherence, as well as between recollection and task adherence. After controlling for these variables, presentation mode had an effect on task adherence. Textual presentation resulted in better adherence than verbal presentation, confirming the first hypothesis. However, no such effect was found for the other two outcome measures: suggestion adherence and subjective adherence. This suggests that the condition had no effect on how difficult the chosen task was, but only on how well the eventual task was executed. More work is necessary to understand the relationship between different types of adherence fully. No correlation was found between suggestion adherence and task adherence. This indicates that these might have to be viewed as formative indicators for general adherence instead of reflexive indicators, e.g. that they are factors which together explain general adherence, even though they are not correlated [66]. The relationship between different types of adherence and treatment outcome also needs to be studied in more detail to know exactly what type of adherence has what effect.
No effect was found for presentation mode on either agent attitude or recollection. In other words, the difference found for presentation mode in adherence to the agent could not be explained by changes the mode had caused in people attitude towards the agent or in their ability better recollect the psychoeducation. This means that hypothesis 2 cannot be confirmed. Given the results, we can modify our initial model as shown in Fig. 4. Both attitudes towards the virtual agent and recollection of the psychoeducation effect task adherence and are extraneous variables and no longer included as explaining or mediating factors for presentation mode effect on people’s adherence to the task set by the agent.
The relationship between agent attitude and adherence corresponds with the prevailing notion that therapist alliance is related to adherence [32, 33]. However, despite the relationship between displaying expertise and being credible [27, 39], this study found no support for the theory that agent attitude can be influenced by presentation mode of psychoeducation. One reason for this is that expertise needs to be displayed continuously over time to have an effect on trust. Further work is necessary to study exactly how agent attitude could be maximised. First steps in this process might be to study which aspects of therapeutic alliance in human therapists could be translated to a virtual agent.
Our findings also show that recollection of the psychoeducation is positively related to task adherence. This matches with the main goal of psychoeducation, namely increasing adherence through explanation and information [20]. No relationship was found between presentation mode of psychoeducation and recollection. This might be explained by different processes being in place. While traditionally, text is remembered better than audio [21], for agents a preference of voice over text has been reported [25]. More research is necessary to understand the relationship between presentation mode of information by a virtual agent and recollection. The presentation of longer texts by a virtual agent, in particular, has not received much attention.
The finding that textual presentation of psychoeducation is preferable above verbal presentation cannot be explained by any of the measures taken in this experiment. None of the measures collected correlated with both presentation mode and task adherence. This means that the mechanism that explains why presentation mode changes people adherence to agents should be sought outside the factors studied. One possible explanation could be that written psychoeducation is more persuasive in getting people to execute the task thoroughly. Although many factors play a role, the written modality has been found to have a greater persuasive effect than audio or video for complicated persuasive messages [67]. This would also fit with the finding that presentation mode only had an effect after controlling for attitude towards the agent, which has been documented to influence the persuasiveness of messages [68]. Further research is necessary to test this hypothesis and to study the impact of other factors.
Although virtual agents in AEMH systems provide an opportunity to deliver care to patients which are not always reached by regular care, they also come with their own risks and responsibilities. Firstly, an AEMH system should be able to identify situations in which a patient is at risk, and ensure patient safety. If it is not possible to put this in place fully, patients should be screened by human caregivers before use with an AEMH, or monitored during use. Another concern unique to virtual agents is that people might develop an emotional bond with the agent and develop certain expectations. It is always important to be very clear about the abilities and limitations of a virtual agent and to manage the expectations of users. Finally, an AEMH system should be conscious of patient privacy, especially in systems collecting sensitive information [69].
Aside from these general concerns, we can identify some limitations to this particular study design which are necessary to appreciate the findings. Firstly, it only considers a computerised voice for the virtual agent. Another option is a recorded human voice, which could have a different effect. Moreover, as text-to-speech systems improve, results might change. The exact effect of a more realistic voice is yet unknown. Similarly, we did not compare the virtual agent to a video of a human presenting the psychoeducation. Research has shown these mediums to be different [70], but the effect of this difference on adherence to psychoeducation is still unknown to the best of our knowledge. Another limitation is the short duration of this experiment. Although it gives a good idea of a short exercise, many e-mental health applications are for more long-term use. This gives a different dynamic in the attitude towards the virtual agent and recollection of information given in the beginning [71]. This study was also not conducted in a double-blinded setting, although the experimenter was only present shortly, she was aware of the experimental condition during the initial program instructions. Finally, we found that the virtual agent was not necessarily considered a useful addition to the exercise, despite the general positive agent attitude. For this finding, it should be noted that the virtual agent had a very basic role in this experiment. It guided the participants through the experiment by telling them where to click and what to do, and in half of the cases told the psychoeducation. It had no interactive or personalising features such as in Gilani et al. [72], and did not assist in the actual recollection of the memory such as in Tielman et al. [73]. A more interactive and personalised virtual agent might also be perceived as more useful.
Psychoeducation is a tool used in a wide range of interventions for mental health problems to increase efficacy. Given the societal burden of mental health problems that lead to both direct and indirect cost [74], as well as loss of quality of life [75], good and effective mental health applications are very valuable. As more virtual agents are deployed in the field of mental-health, the need for good psychoeducation procedures therefore grows. This is relevant particularly for those virtual agents that have an active coaching, guiding or conversational role, taking on some of the tasks of a regular therapist, such as psychoeducation. Interventions with such virtual agents have been developed for a range of mental-health problems [76] such as insomnia [77], depression [78], social phobia [79] and PTSD [80]. For all of these, psychoeducation that maximises adherence is an interesting component. However, the question of how to present information is relevant even for applications outside of the scope of mental health. General health-care information has similarities to psychoeducation. Patients often need to know what medication to take, what exercises to do, or even need to change attitudes. Virtual agents are being developed for these uses [81, 82] and as with mental health care adherence to the agent is an important aspect [34]. Moreover, the results of this study can be relevant even for fields outside of health-care such as education or marketing. With agents teaching, explaining or convincing users, adherence to information is essential.
The main contribution of this study lies in identifying several aspects of psychoeducation that influence adherence. Firstly, how well the information is recollected has an effect. Secondly, the attitude towards the virtual agent, which asks people to perform the task, affects adherence. Thirdly, if the psychoeducation is presented in text it results in a better adherence than if a virtual agent, with a text-to-speech voice, offers it verbally.
Additionally, from this study, we can infer that the perceived usefulness of a virtual agent differs considerably between people dividing them into a pro and a con camp, but no indications of any effect of preference on adherence is found. These findings are a first step in determining how exactly psychoeducation in mental health applications with virtual agents influences adherence.
Notes
Acknowledgments
This work is part of the programme Virtual E-Coaching and Storytelling Technology for Post-Traumatic Stress Disorder, which is financed by the Netherlands Organization for Scientific Research (pr. nr. 314-99-104).
Conflict of interest
None to report.
References
[1] | Olfson M, Mojtabai R, Sampson NA, Hwang I, Druss B, Wang PS, et al. Dropout from outpatient mental health care in the United States. Psychiatric Services. (2009) ; 60: : 898-907. |
[2] | Wang PS, Lane M, Olfson M, Pincus HA, Wells KB, Kessler RC. Twelve-Month Use of Mental Health Services in the United States. Twelve-Month Use of Mental Health Services in the United States. (2005) ; 62: : 629-640. |
[3] | Lyons C, Hopley P, Horrocks J. A decade of stigma and discrimination in mental health: Plus ça change, plus c’est la même chose (the morethings change, the more they stay the same). Journal of Psychiatric and Mental Health Nursing. (2009) ; 16: : 501-507. |
[4] | Cartreine JA, Ahern DK, Locke SE. A roadmap to computer-based psychotherapy in the United States. Harv Rev Psychiatry. (2011) ; 18: (2): 80-95. |
[5] | Donker T, Blankers M, Hedman E, Ljotsson B, Petrie K, Christensen H. Economic evaluations of Internet interventions for mental health: A systematic review. Psychological Medicine. (2015) ; 45: (16): 3357-3376. |
[6] | Zhang M, Cheow E, Ho CS, Ng BY, Ho R, Cheok CCS. Application of low-cost methodologies for mobile phone app development. JMIR MHealth and UHealth. (2014) ; 2: : e55. |
[7] | Yee N, Bailenson JN, Rickertsen K. A meta-analysis of the impact of the inclusion and realism of human-like faces on user experiences in interfaces. In CHI. (2007) ; 2007. |
[8] | Blanson-Henkemans OA, van der Mast CAPG, van der Boog PJM, Neerincx MA, Lindenberg J, Zwetsloot-Schonk BJHM. An online lifestyle diary with a persuasive computer assistant providing feedback on self-management. Technology and Health Care. (2009) ; 17: : 253-267. |
[9] | Looije R, Neerincx MA, Cnossen F. Persuasive robotic assistant for health self-management of older adults: Design and evaluation of social behaviors. International Journal of Human-Computer Studies. (2010) ; 68: (6): 386-397. |
[10] | Andrade AD, Anam R, Karanam C, Downey P, Ruiz JG. An overactive bladder online self-management program with embedded avatars: A randomized controlled trial of efficacy. Urology. (2015) ; 85: (3). |
[11] | Organization WH. Adherence to long-term therapies: Evidence for action. World Health Organization; (2003) . |
[12] | Shumaker SA, Ockene JK, Riekert KA, editors. The handbook of Health Behavior Change: Springer; (2009) . |
[13] | Donkin L, Christensen H, Naismith SL, Neal B, Hickie IB, Glozier N. A systematic review of the impact of adherence on the effectiveness of e-therapies. Journal of Medical Internet Research. (2011) ; 13: (2). |
[14] | Gould RA, Clum GA. A meta-analysis of self-help treatment approaches. Clinical Psychology Review. (1993) ; 13: (2): 169-186. |
[15] | Coulthard K, Patel D, Brizzolara C, Morriss R, Watson S. A feasibility study of expert patient and community mental health team led bipolar psychoeducation groups: Implementing an evidence based practice. BMC Psychiatry. (2013) ; 13. |
[16] | Zhang MWB, Ho RCM. Tapping onto the potential of smartphone applications for psycho-education and early intervention in addictions. Frontiers in Psychiatry. (2016) ; 7. |
[17] | Taylor-Rodgers E, Batterham PJ. Evaluation of an online psychoeducation intervention to promote mental health help seeking attitudes and intentions among young adults: Randomised controlled trial. Journal of Affective Disorders. (2014) ; 168: : 65-71. |
[18] | Farooq S, Naeem F. Tackling nonadherence in psychiatric disorders: Current opinion. Neuropsychiatric Disease and Treatment. (2014) ; 10: : 1069-1077. |
[19] | Eker F, Harkin S. Effectiveness of six-week psychoeducation program on adherence of patients with bipolar affective disorder. Journal of Affective Disorders. (2012) ; 138: : 409-416. |
[20] | Rimer BK, Glanz K. Theory at a Glance – A Guide For Health Promotion Practice. (2005) . |
[21] | Webb WB, Wallon EJ. Comprehension by reading versus hearing. The Journal of Applied Psychology. (1956) ; 40: : 237-240. |
[22] | Rubin DL, Hafer T, Arata K. Reading and listening to oral-based versus literate-based discourse. Communication Education. (2000) ; 49: (2): 121-133. |
[23] | Sanchez E, Garcia-Rodicio H. The use of modality in the design of verbal aids in computer-based learning environments. Interact. (2008) ; 20: (6): 545-561. |
[24] | Fox J, Ahn SJ. Recommendations for designing maximally effective and persuasive health agents. In Int Conf Intelligent Virtual Agents; (2014) . |
[25] | Walter N, Ortbach K, Niehaves B. Designing electronic feedback – Analyzing the effects of social presence on perceived feedback usefulness. (2015) . |
[26] | Qiu L, Benbasat I. Evaluating Anthropomorphic Product Recommendation Agents: A Social Relationship Perspective to Designing Information Systems. (2009) . |
[27] | Fogg B, Marshall J, Laraki O, Osipovich A, Varma C, Fang N, et al. What makes web sites credible? A report on a large quantitative study. In CHI (2001) ; 2001. |
[28] | Fiscella K, Meldrum S, Franks P, Shields C, Duberstein P, McDaniel S, et al. Patient trust: Is it related to patient-centered behavior of primary care physicians? Medical Care. (2004) ; 42: (11): 1049-55. |
[29] | Bohnert ASB, Zivin K, Welsh DE, Kilbourne AM. Ratings of patient – provider communication among veterans: Serious mental illnesses, substance use disorders, and the moderating role of trust. Health Communication. (2011) ; 26: : 267-274. |
[30] | Safran DG, Taira DA, Rogers W, Kosinski M, Ware J, Tarlov A. Linking primary care performance to outcomes of care. Journal of Family Practice. (1998) ; 47: (3): 213-220. |
[31] | Muran JC, Barber JP, editors. The therapeutic alliance: An evidence-based guide to practice: Guilford. (2010) . |
[32] | Czobor P, Dorn RAV, Citrome L, Kahn RS, Fleischhacker WW, Volavka J. Treatment adherence in schizophrenia: A patient-level meta-analysis of combined CATIE and EUFEST studies. European Neuropsychopharmacology. (2015) ; 25: : 1158-1166. |
[33] | Leclerc E, Noto C, Bressan RA, Brietzke E. Determinants of adherence to treatment in first-episode psychosis: A comprehensive review. Revista Brasileira de Psiquiatria. (2015) ; 37: : 168-176. |
[34] | Baker S, Richards D, Caldwell P. Relational agents to promote ehealth advice adherence. In PRICAI; (2014) . |
[35] | Bickmore T, Gruber A. Relational agents in clinical psychiatry. Harv Rev Psychiatry. (2010) ; 18: (2): 119-130. |
[36] | Laska KM, Smith TL, Wislocki AP, Minami T, Wampold BE. Uniformity of evidence-based treatments in practice? Therapist effects in the delivery of cognitive processing therapy for PTSD. Journal of Counseling Psychology. (2013) ; 60: (1): 31-41. |
[37] | Anderson T, Ogles BM, Patterson CL, Lambert MJ, Vermeersch DA. Therapist effects: Facilitative interpersonal skills as a predictor of therapist success. Journal of Clinical Psychology. (2009) ; 65: (7): 755-768. |
[38] | Priebe S, Palumbo C, Ahmed S, Strappelli N, Gavrilovic JJ, Brenner S. How psychiatrists should introduce themselves in the first consultation: An experimental study. The British Journal of Psychiatry. (2013) ; 202: : 459-462. |
[39] | Kulms P, Kopp S. The effect of embodiment and competence on trust and cooperation in human-agent interaction. In Int Conf Intelligent Virtual Agents; (2016) . |
[40] | Qiu L, Benbasat I. Online consumer trust and live help interfaces: The effects of text-to-speech voice and threedimensional avatars. International Journal of Human-Computer Interaction. (2010) ; 19: (1): 75-94. |
[41] | Catwell L, Sheikh A. Evaluating eHealth intervention: The need for continuous system evaluation. Plos Medicine. (2009) . |
[42] | Jacobs MA, Graham AL. Iterative development and evaluation methods of mHealth behavior change interventions. Current Opinion in Psychology. (2016) ; 9: : 33-37. |
[43] | Brinkman WP. Current and future research directions in mental health computing. In Int Conf on Computation and Communication Advancement. (2013) . |
[44] | Looije R, Neerincx MA, Hindriks KV. Specifying and testing the design rationale of social robots for behavior change in children. Cognitive Systems Research. (2016) . |
[45] | Baker TM, Gustafson DH, Shah D. How can research keep up with eHealth? Ten strategies for increasing the timeliness and usefulness of eHealth research. Journal of Medical Internet Research. (2014) ; 16: (2). |
[46] | Lee HJ, Lee SH, Kim HS, Kwon SM, Telch MJ. A comparison of autogenous/reactive obsessions and worry in a nonclinical population: A test of the continuum hypothesis. Behaviour Research and Therapy. (2005) ; 43: (8): 999-1010. |
[47] | Murphy J, Shevlin M, Houston J, Adamson G. A population based analysis of subclinical psychosis and help-seeking behavior. Schizophrenia Bulletin. (2012) ; 38: (2): 360-367. |
[48] | Watson PJ, Sawrie SM, Greene RL, Arredondo R. Narcissism and depression: MMPI-2 evidence for the continuum hypothesis in clinical samples. Journal of Personality Assessment. (2002) ; 79: (1): 85-109. |
[49] | McDermut JF, Haaga DAF, Kirk L. An evaluation of stress symptoms associated with academic sexual harassment. Journal of Traumatic Stress. (2000) ; 13: (3): 397-411. |
[50] | Pennebaker JW, Beall SK. Confronting a traumatic event: Toward an understanding of inhibition and disease. Journal of Abnormal Psychology. (1986) ; 95: (3): 274-81. |
[51] | Baikie KA, Wilhelm K. Emotional and physical health benefits of expressive writing. Advances in Psychiatric Treatment. (2005) ; 11: : 338-346. |
[52] | Frattaroli J, Thomas M, Lyubomirsky S. Opening up in the classroom: Effects of expressive writing on graduate school entrance exam performance. Emotion. (2011) ; 11: (3): 691-696. |
[53] | Harris AHS. Does expressive writing reduce health care utilization? A meta-analysis of randomized trials. Journal of Consulting and Cli. (2006) ; 74: (2): 243-252. |
[54] | Morisano D, Hirsch JB, Pihl JBPRO, Shore BM. Setting, Elaborating and reflecting on personal goals improves academic performance. Journal of Applied Psy. (2010) ; 95: (2): 255-264. |
[55] | Burton CM, King LA. Effect of (very) brief writing on health: The two-minute miracle. British Journal of Health Psychology. (2007) ; 00: : 1-7. |
[56] | Chung CK, Pennebaker JW. Variations in the spacing of expressive writing. British Journal of Health Psychology. (2008) ; 13: : 15-21. |
[57] | van den Steen M, Brinkman WP, Vermetten E, Neerincx M. Design and usability evaluation of a multi-modal memory restructuring system for the treatment of combat-related PTSD. In ECCE 2010 Workshop on Cognitive Engineering for Technology in Mental Health Care and Rehabilitation; (2010) . |
[58] | Lee EJ, Nass C, Brave S. Can computer-generated speech have gender? An experimental test of gender stereotype. In CHI; (2000) . |
[59] | Baylor AL, Plant EA. Pedagogical agents as social models for engineering: The influence of agent appearance on female choice. In Conference on Artificial Intelligence in Education; (2005) . |
[60] | Baylor AL. Promoting motivation with virtual agents and avatars: role of visual presence and appearance. Philosophical Transactions of the Royal Society B. (2009) ; 364: : 3559-3565. |
[61] | Guadagno RE, Blascovich J, Bailenson JN, McCall C. Virtual humans and persuasion: The effects of agency and behavioral realism. Media Psychology. (2007) ; 10: : 1-22. |
[62] | Tielman ML, Neerincx MA, van Meggelen M, Franken I, Brinkman WP. How should a virtual agent present psychoeducation? Appendix part I - Psychoeducation. (2017) . |
[63] | Hemenover SH. The good, the bad, and the healthy: Impacts of emotional disclosure of trauma on resilient self-concept and psychological distress. Personality and Social Psychology Bulletin. (2003) ; 2003: (10): 1236-1244. |
[64] | Tielman ML, Neerincx MA, van Meggelen M, Franken I, Brinkman WP. How should a virtual agent present psychoeducation? Appendix part II - Questionnaires. (2017) . |
[65] | Tielman ML, Neerincx MA, van Meggelen M, Franken I, Brinkman WP. How should a virtual agent present psychoeducation? Appendix part III - Dataset. (2017) . |
[66] | Blunch NJ. Introduction to Structural Equation Modeling: Sage. (2013) . |
[67] | Chaiken S, Eagly AH. Communication modality as a determinant of message persuasiveness and message comprehensibility. Journal of Personality and Social Psychology. (1976) ; 34: : 605-14. |
[68] | Chaiken S. The heuristic model of persuasion. Social Influence: The Ontario Symposium. (1987) ; 5: : 3-39. |
[69] | Luxton DD. Recommendations for the ethical use and design of artificial intelligent care providers. Artificial Intelligence in Medicine. (2014) ; 62: (1): 1-10. |
[70] | Kang SH, Feng AW, Seymour M, Shapiro A. Study comparing video-based characters and 3D-based characters on mobile devices for chat. In MIG ’16 Proceedings of the 9th International Conference on Motion in Games; (2016) . |
[71] | Bickmore T, Picard R. Establishing and maintaining long-term humancomputer relationships. In ACM Transactions on Computer Human Interaction (ToCHI); (2005) . |
[72] | Gilani S, Sheetz K, Lucas GM, Traum D. What kind of stories should a virtual human swap? In Int Conf Intelligent Virtual Agents; (2016) . |
[73] | Tielman ML, van Meggelen M, Neerincx MA, Brinkman WP. An Ontology-based question system for a virtual coach assisting in trauma recollection. In Int Conf on Intelligent Virtual Agents; (2015) . |
[74] | Ho RC, Mak KK, Chua AN, Ho CS, Mak A. The effect of severity of depressive disorder on economic burden in a university hospital in Singapore. Expert Review of Pharmacoeconomics & Outcomes Research. (2013) ; 13: : 549-59. |
[75] | WHO. Global burden of mental disorders and the need for a comprehensive, coordinated response from health and social sectors at the country level. (2012) . |
[76] | Provoost S, Lau HM, Ruwaard J, Riper H. Embodied conversational agents in clinical psychology: A scoping review. Journal of Medical Internet Research. (2017) ; 19: : e151. |
[77] | Beun RJ, Brinkman WP, Fitrianie S, Griffioen-Both F, Horsch C, Lancee J, et al. Improving adherence in automated e-coaching. Persuasive Technology. (2016) ; 9638: : 276-287. |
[78] | Bresó A, Martínez-Miranda J, García-Gómez JM. Leveraging adaptive sessions based on therapeutic empathy through a virtual agent. In ICAART - Doctoral Consortium; (2014) . |
[79] | Hartanto D, Brinkman WP, Kampmann IL, Morina N, Emmelkamp PG, Neerincx MA. Home-based virtual reality exposure therapy with virtual health agent support. Pervasive Computing Paradigms for Mental Health. (2015) ; 204: : 85-98. |
[80] | Rizzo AA, Lange B, Buckwalter JG, Forbell E, Kim J, Sagae K, et al. An intelligent virtual human system for providing healthcare information and support. Medicine meets Virtual Reality. (2011) ; 163: : 503-509. |
[81] | Edwards RA, Bickmore T, Jenkins L, Foley M, Manjourides J. Use of an interactive computer agent to support breastfeeding. Maternal and Child Health Journal. (2013) ; 17: (10): 1961-1968. |
[82] | Ren J, Schulman D, Brian J, Bickmore TW. Supporting longitudinal change in many health behaviors. In CHI; (2014) . |
