Usability, Appeal, and Relevance of Music and Wellbeing Training for Carers of People with Dementia: A Think-Aloud Study

Participants

Purposive sampling was used to recruit 13 participants (9 female; 4 male) including one person living with dementia, 5 family carers, and 7 participants working in residential aged care (Table 1). Participants were recruited by emails to stakeholder mailing lists and consumer advisory groups. Care staff needed to have 2 or more years’ experience in aged care. All participants needed a sufficient level of English to be able to participate in discussions and access to a device with a reliable internet connection. No participants who expressed an interest were ineligible and there were no withdrawals from the study. Participants were offered a $30AUD voucher for participation.

2.2The training

Music playlists for people living with dementia consists of a number of different modules that are available depending on the role of the person undertaking the training, including care staff and lifestyle workers, family members or friends, executive staff and managers, and allied health professionals. Each module covers the 5 steps, with the Carer & Lifestyle module (designed for care staff and staff employed to coordinate leisure activities in aged care) being the most detailed, since these are likely to be the staff most involved in implementing music programs. The course in its entirety takes about 20-40 min to complete. While some small amounts of reading are involved, the training is primarily presented using video content and scenario-based learning to help users envisage real-life applications. Quizzes and interactive activities, such as assessing the likely effect of particular pieces of music on mood, consolidate learning and help users develop hands-on experience in selecting music for various care scenarios.

Procedures

After obtaining ethics approval from the Human Research Ethics Committee of Western Sydney University (Approval number: H14462), emails were sent to potential participants. When interest was expressed, eligibility was determined via email correspondence, and eligible participants provided with an information sheet before signing a consent form.

Sessions of approximately one hour were held individually over Zoom with participants joining the session from their home or work in an isolated room. Participants were provided with a link to the online training. The session facilitator explained the think-aloud procedure, requesting that participants share their screen for the video recording. Participants were informed that all forms of feedback, both negative and positive, would be helpful to the research team in improving the training in the future. Participants explored whichever modules of the course they were interested in (Table 1), independently navigating through the materials whilst articulating their thoughts out loud [15]. Participants were prompted to continue speaking if they were silent for 20 consecutive seconds. Other than that, the facilitator remained silent and off-camera in order to avoid disrupting the train of thought or influencing responses [16]. The facilitator also took notes throughout the session.

Once participants had explored as many modules of the training as they wished, a follow up discussion was conducted in which the session facilitator asked questions about events or responses that arose during the think-aloud protocol. Finally, participants completed a short online survey.

Materials

Most participants viewed the training course on a computer, although one viewed it on a smartphone and another on an iPad. Zoom record function captured the shared screen to allow researchers to see which part of the course the participant was viewing as they spoke.

A broad interview guide covered 6 topics including: 1) general impressions, 2) relative appeal of features, 3) needed changes or improvements, 4) relevance, clarity and comprehensiveness of content, 5) ease of use, and 6) visual appeal. The Qualtrics survey completed by participants consisted of 29 items rated on a scale of 1 (Highly disagree) to 5 (Highly agree). Questions were mostly drawn from the Questionnaire to Evaluate Online Training in the Workplace (CEFOAL) [17, 18] and the Mobile App Rating Scale (MARS) [19]. The CEFOAL was designed to assess online training across a range of workplace contexts on five dimensions: pedagogical design, tutor performance, virtual environment design, timing, and transfer of learning. Since items related to tutor performance were not relevant, 14 items relating to the other four dimensions were retained. Although the MARS was developed for evaluating smartphone applications, 10 items relating to visual design and course content which were not covered by the CEFOAL were used with minor adaptations (such as changing “app” to “course”). An additional 5 items were included to consider potential impact of the course on approaches to care such as increasing “awareness of how to use music”, and “motivation of people caring for someone with dementia to use music as part of the care routine”.

Data analysis

Quantitative data analysis. Due to the small sample size and relative homogeneity of responses, factor analysis could not be performed on the survey items. However, data were logically grouped into 3 primary dimensions of 9 items corresponding to dimensions of the CEFOAL: 1) pedagogical design, 2) virtual environmental design, and 3) transfer and impact. Reliability analyses returned Cronbach’s alphas of 0.63, 0.85, and 0.85 for these dimensions respectively. One question each addressed timing/duration (Item 7), and overall quality (Item 24).

Qualitative data analysis. Analysis followed principles of Framework Analysis [20], which is commonly used in health research and allows data to be analyzed both deductively with reference to an overall framework and inductively to derive novel themes. Think-aloud sessions and interviews were transcribed verbatim and screen shots of shared screens added into the transcriptions for reference. Frequency counts of usability issues were tallied to identify problematic sections by an independent researcher (non-author). Overall, responses to 139 screens in the training were analyzed. In a second wave of analysis, coding and thematic analysis was conducted by the first author and independently by a non-author as a check to reduce potential bias using the table function in Microsoft Word. Data were categorized according to three dimensions: 1) virtual environmental design, 2) pedagogical design, 3) transfer, impact and barriers to uptake. In a final stage of analysis, qualitative and quantitative results were considered together to derive an overall picture of participant response and identify key areas for improvement.

Reflexivity

Sessions were facilitated by a Research Assistant (non-author, female), since both authors were directly involved in developing the training, a fact which may have been known to participants. The use of a think-aloud method was utilized as a means of reducing bias as well, since participants have the opportunity to provide an initial response to the training without prompting from the session facilitator. None of the research participants were previously known to the session facilitator. No information about the session facilitator other than job title was provided to participants.

Both the first and second authors have previously conducted research about personalized music programs for people with dementia and were involved in developing the training, creating some risk of bias in data analysis. The use of a non-author to conduct the think-aloud sessions and as an independent coder was used in an attempt to reduce this risk.

Engagement, usability, and appeal

Quality of the training and duration each received an average rating of 4.92 (SD = 0.28, possible scores 1–5). With possible scores ranging from 9 to 45, the Virtual Environment Design dimension returned a mean of 42.23 (SD = 3.17), indicating a high level of satisfaction with the design.

Overall, 28 usability issues were identified in the think-aloud study, falling into two main categories: readability and navigation issues. Qualitative analysis of the data about the virtual environmental design identified 2 main themes and sub-themes including 1) visual elements, and 2) technical difficulties (Table 2).

Visual elements. In general, participants reported appreciating the simplicity of the overall visual design, which was “nicely laid out” (P2), “visually interesting” (P6), and “appealing” (P5). However, on some screens people reported that color choices or insufficient contrast made it difficult to read. Participants made suggestions about text or navigation keys that could be visually bolder so that they stand out, “I’d really like my arrows to be a bit bolder to tell me where I have to go” (P13).

Participants appreciated the personal touch that images of people provided but reported that they would have preferred animations rather than “static” images (P5), or would have preferred closer face shots to make the person more central. Others noted that the exclusive use of images of older people could make carers of people with younger onset dementia feel uncomfortable, “You’re defining dementia as an old person’s disease. My husband was 58 when he started on his dementia journey, so looking at old lady Shirley probably would be a turn-off” (P10). Others appreciated the use of graphs and other visualizations that communicated concepts without dense text.

Technical difficulties. Most participants said that the course had been easy to navigate, “My knowledge of anything technical is very limited, so I appreciated the fact that it was easy to use” (P7). However, others found it non-intuitive to navigate in places. One participant stated, “There were a couple of points where it wasn’t obvious which button I should press next” (P2). Participants noted that consistency was important in this regard, with one participant observing that it was confusing when buttons changed format:

Ease of navigation was considered to be particularly important given the diversity of technical know-how among potential users, with many being “people who are not tech savvy” (P13). Facilitator notes also noted a need for clarity about how to engage with the interactive elements of the course, recording several instances of participant confusion.

Pedagogical design and content

With possible scores ranging from 9 to 45, ratings of the Pedagogical Design returned a mean of 42.54 (SD = 2.37), indicating a high level of satisfaction. Participants commented that the messages were easy to understand and clearly communicated. One participant said, “I think it’s really insightful. I think it gives a lot of good information” (P5). Another stated:

Language. While some people thought the language used was clear, home-based carers tended to find the language too formal or academic especially when viewing the modules designed for aged care staff. Two participants in fact noted that they had not understood that the “Carer and Lifestyle” module was designed for aged care staff (P10, P12) and that this labelling itself was somewhat confusing. There were also some particular terms that users were not familiar with, for example expressions such as “partnering in care” (P8). However, some terminology was noted as potentially confusing for aged care staff as well, with one participant pointing out that, “There’s quite a lot of staff we have here that would not have a clue what you’re talking about when you say ‘playlists’. They still live in the 1990s” (P3).

Pedagogical techniques. Participants appreciated a number of techniques used in the training to communicate information, such as the use of scenario-based learning to help users see real-life applications of concepts. Interactive activities such as song selection, and quizzes to reinforce learning were also appreciated.

Content. The elements of content most frequently commented upon by participants were the focus on person-centered care and the need for personalization. One participant said, “I really like the focus on individual preferences and asking the person or those who know them really well” (P9).

A participant with dementia stated:

Others agreed that giving information about equipment and technology was useful and appreciated the inclusion of information about the potential for negative responses. One participant noted that it could be quite discouraging to carers if there was no acknowledgement of times that music may not work. She said:

Other content that was noted as useful included the importance of considering carer wellbeing, the need to observe responses rather than to put music on and then leave, suggestions for how family members can communicate with staff, and clear explanation of musical concepts for people without music training.

Transfer, impact, and barriers to uptake

The Transfer & Impact dimension returned a mean score of 39.70 (SD = 4.64, possible range 9 to 45), suggesting a high level of agreement that the training would be useful and relevant to participant roles as carers, and likely to achieve the intended purpose. Qualitative data in this data category revealed four broad themes: 1) incorporation in care plans, 2) changes to practice encouraged, 3) increased awareness, and 4) barriers to implementation (Table 4).

Incorporation in care plans. A number of participants liked the fact that the training talked about incorporating music use into the overall care plans of the individual, “It’s great that you mention the care plan, because it can be such a good tool” (P9). Others liked the fact that the training linked to national government standards in Australia and a recently introduced requirement that care plans outline non-pharmacological strategies to supporting people experiencing changes in behavior.

Changes to practice encouraged. Participants spoke about aspects of the training that they would incorporate into their own approach to care or that would be useful for encouraging change in other staff. One key area of change highlighted in the training was the need for personalization of a music program, rather than using generic forms of music in communal living areas. One participant described ideas that the training had sparked for her:

Another described her increased awareness about personalization saying:

Increased awareness. Participants also believed the training would support increased awareness of a number of important issues including consent, how the needs of people with dementia change over time, technology choices, and risks. In relation to consent it was noted that the training drew attention to the fact that it is important to make sure the person living with dementia consents to have headphones put on them. Several other participants noted that the training would help staff to be more aware that music programs need to be revised as needs change over time, “You need to change the playlists as the person goes on with their dementia progression. It may change the whole thing, the whole game. They may not like it anymore.” (P1)

Another issue that the training raised awareness of is the need for carers to consider other contextual factors before using music to address behaviors or moods. As one participant said, “You can’t just shove on a song and think that’s going to fix everything - there’s obviously other things that we’ve got to keep in mind to help them with. Like pain, depression, that sort of thing.” (P7)

Barriers to implementation. Two important barriers to implementation of the training were identified by participants: 1) an ingrained culture of focusing on physical needs, and 2) time-poor staff. Some staff argued that limited budgets were responsible for a focus on caring for physical needs, while others suggested this was a mindset that could slowly be shifted. Lack of time was also suggested as a contributing factor to the focus on necessities of care, with one staff member commenting: