Assessing Implementation Fidelity of a School-Based Crisis Prevention Program with an Ex-Post-Facto Design: The NETWASS FOI Assessment System

Materials Used to Conceptualize Core Components

We included all available sources of information to obtain a multi-method data pool and a broad perspective on program implementation:

Information from the manual. As a starting point, we examined the structured program manual for schools. The manual specified all program goals, activities and the sequence in which they were to be delivered. Furthermore, we obtained information from the structured trainer manual which provided additional information about training methodology, a detailed time-table, and learning goals.

Feedback from the trainers. To obtain a meta-perspective of the facilitating trainers, which were either members of our research team or multipliers with a background in school psychology or legal crime prevention, we engaged in regular exchange sessions by telephone to learn about their experiences with the training manual.

Discussions in our research team and conceptual papers. We conducted a series of documented discussions within our research team about program goals, assumed mechanisms, and the rationale behind the inclusion of specific activities and practices. We also reviewed a paper written by the program developers when applying for program funding. Following this, we presented and discussed our hypothesized core components with colleagues from Freie Universität Berlin who are engaged in other prevention projects, but were not involved in the NETWASS study.

Evaluation results and findings from follow-up interviews with schools. We received input from three schools that were part of a pilot trial and obtained qualitative data on implementation success and challenges from telephone-supported process monitoring from all schools. Additionally, we conducted guideline-based follow-up interviews in which the crisis prevention teams reflected on their experiences made with our case management procedure. Finally, we drew on quantitative results from the effectiveness study to contextualize and interpret the findings from our FOI assessment system.

Stage-Specific Conceptualization of Core Components and Program Theory

Since there is a potential for non-adherence to the program manual at any given stage of implementation, we conceptualized NETWASS core components for each of the three stages. Depending on the degree of standardization, a failure to implement the core processes or activities relevant for a specific stage, or altering the dosage or quality of the initial training sessions, could reduce program effectiveness. This also implies that FOI should be measured separately for each stage. Figure 1 illustrates stages of implementation and FOI dimensions.

Figure 1.

Evaluation design with stages of implementation and FOI dimensions.

After an in-depth review of the NETWASS materials, and a series of discussions within our research team on the underlying theory of change, we defined our core components more precisely and developed a more refined logic model that decomposed the program into its hypothesized active ingredients, as presented in Figure 2. According to Gould et al. (2014), we classified program core components into two types: Program core processes and program core activities. Program core processes refer to the manner in which the implementers engage with the teachers and the way in which the program material is taught to school personnel (e.g., training style, interaction behavior). While processes may be difficult to measure, they are critical to the program’s success because they influence the acceptance and overall compliance with the intervention.

Figure 2.

NETWASS program theory and core components.

Core activities are the practices as recommended in the curriculum. They are implemented and measured at all three stages of implementation: Trainings, implementation of structures and case identification, and case management. Since all core activities that should be implemented during the trainings are standardized by the material, we conceptualized only one core component for the first stage. The core activities for the stage of structural implementation and case identification were intended to cover all preconditions that are necessary for the school to recognize and exchange information when a student is experiencing a crisis. Core activities implemented at the third stage ensure that crisis prevention teams recognize early crisis indicators, conduct a structured case management that includes an evidence-based threat assessment, and develop student support measures based on a student’s specific risk and protection factors.

Measures to Assess FOI of Trainings

Our FOI assessment system, including a list of all data sources, is outlined in Table 2. The fifth dimension “differentiation” is incomplete, because it requires empirical evidence of the relations between specific components and program outcomes. Dosage of the initial trainings was determined by the training schedule and no additional measure was needed. Training participation was estimated by the pre-training questionnaire response rate. In order to protect participant confidentiality, there were no attendance lists, so we calculated a training participation quotient by dividing the number of pre-training questionnaires by the total number of teachers per school. In all cases, a pre-training questionnaire was followed by training participation. However, in some cases, a teacher did not participate in the study yet attended the training afterwards. Consequently, this assessment can underestimate the number of participants.

Trainer adherence scale. To obtain data on trainer adherence to core activities specified in the training manual, we had to rely on retrospective self-report from the trainers. Every trainer completed a self-report questionnaire at the end of the implementation period (seven-month follow-up) and assessed his or her adherence across all training sessions (ranging from 5–10). Instructors rated the extent to which they stuck to the training schedule or made adaptations. Five items of the Trainer Adherence Scale (Cronbach’s alpha = .93) covered adherence to the time-table, content, conducting case study exercises, and preparing an implementation plan with the training groups, e.g., “When delivering the training sessions, I usually discussed the topics as scheduled in the manual.”

Trainer adaptations scale. Additionally, four more items were rated on the Trainer Adaptations Scale (Cronbach’s alpha = .70), indicating adaptations made by the trainer, e.g., “When delivering the training sessions, I occasionally provided additional content or discussed issues that were not included in the manual.”

Training delivery scale. Adherence to core processes during the trainings was assessed by participants (i.e. all school staff) at the end of the training sessions. The Training Delivery Scale (Cronbach’s alpha = .90) included five items on all three core processes, e.g., “The trainer communicated the course content clearly and effectively to the participants.”

Training evaluation form. Participants (all school staff) filled out a four-item Training Evaluation Form (Cronbach’s alpha = .95) to assess the quality of trainings. Items covered overall training quality, benefit, participant interest level, and usefulness of the trainings, e.g., “Overall, how would you rate the training class in terms of usefulness?”

Participant engagement scale. To evaluate responsiveness to the trainings, all school staff answered the Participant Engagement Scale (5 items, Cronbach’s alpha = .92) at the end of the trainings, and marked their interest level in the topic, as promoted by the trainings, their motivation to use the new knowledge in day-to-day work, and the extent to which trainers were able to communicate relevance of the topic for an individual teacher’s work, e.g., “I feel encouraged to incorporate the newly learned skills into my working routine.”

Intervention compliance scale. In addition, a five-item Intervention Compliance Scale (Cronbach’s alpha = .75) was rated by all school staff members two weeks after the trainings. This scale had items concerning their attitudes towards the program and readiness to collaborate with colleagues, as well as concerns about increased work load, e.g., “I am willing to take on additional tasks and extra work to implement the program at our school.” We also asked crisis prevention teams to report their attitudes towards the program, and perceptions about its usefulness in daily work at the seven-month follow-up.

Measures to Assess Implementation of Structures and Case Identification

A set of core activities was supposed to be implemented by school staff on a school-wide basis in stage two (following the trainings). To assess schools’ adherence to manual-based recommendations on structural changes, research staff completed an observational rating. Based on process monitoring (mainly by telephone interviews), all research staff reported on each school’s implementation success, specifically whether a school had established a crisis prevention team and nominated a crisis prevention appointee.

Three specific instruments were created to evaluate the remaining core activities for stage two. Operationalization of these core activities was most difficult because they were not concrete structural changes or well-defined behaviors by a specific person. Additionally, the context and time-span of these core activities was less well-defined, since stage two started directly after training and lasted until the final follow-up survey. We also aimed to differentiate between a core activity as defined by the program manual (e.g., apply decision criteria for reporting cases) and an outcome (e.g., number of cases reported), with the latter being an effect of a properly implemented core activity. However, we had to put a major focus on the stage two core activities because they were an immediate consequence of training core components and outcomes. What is more, their full implementation was essential for implementing stage three core activities in case management. To take a step forward, we outlined the specific skills and cognitions of trained school staff that would reflect most accurately whether a core activity had been implemented properly.

Knowledge test. To assess the implementation of the core activity concerned with cognitive expertise, we evaluated school staff’s knowledge on the topic two weeks after training. Every teacher completed an 11-item knowledge test on the topic of severe targeted school violence, e.g., school staff should disagree with the following statement: “Checklists are the best way to identify potential perpetrators.” A total score was calculated from the number of correct answers. To determine whether a school staff member had made an attitude shift towards a more general preventive perspective, we administered two scales:

Attitudes towards prevention scale. School staff first indicated to what extent they believed certain school safety measures were effective and helpful. Three items (Cronbach’s alpha = .64) covered approaches specific to the NETWASS prevention model, such as information on risk factors and warning behavior, involving school psychologists or school social workers to ensure school safety, and providing individual student support and case monitoring.

Response to student problems scale. Teachers rated their general tendency to respond to student problems on a seven-item scale (Cronbach’s alpha = .77). Items included behavioral reactions that either increased information exchange in the school (e.g., sharing concerns with a colleague, informing the homeroom teacher or school principal), or offered direct student support (e.g., talking to the student or consulting a school psychologist). According to the NETWASS crisis prevention model, these teacher behaviors are assumed to be helpful to preventing an individual crisis, and promoting an open and trustful school climate.

Evaluation skills scale. Because direct monitoring of how a teacher applied the NETWASS criteria for case identification and reporting was not feasible, we obtained data about their decision-making with case vignettes. All school staff evaluated four case examples two weeks after the training. These case scenarios represented student risk factor patterns and warning behaviors. Four items per case covered the NETWASS decision criteria for reporting a case (one item per criterion). An evaluation skills score was computed by averaging the 16 items (McDonald’s ωt = 0.83). The ability of a single teacher to evaluate a case is an essential core activity at stage two of program implementation because it provides the necessary condition for triggering advanced case management.

Measures to Assess FOI of Case Management

A major challenge was to assess how the crisis prevention teams implemented core activities for stage three. It was not feasible to have observers attend or record case management sessions because cases arise sporadically and require immediate action. Furthermore, school teams did not want researchers to observe confidential discussions about a student’s family background, medical issues, or other private concerns. Therefore, we relied on written documents from these sessions.

Adherence to protocol scale. Additionally, we created a six-item self-report scale that covered the main standards that should be followed when discussing student cases. All crisis prevention team members answered the adherence to protocol scale (Cronbach’s alpha = .87) at the seven-month follow-up. Items included ratings on the extent to which information about a student was exchanged and documented in a structured procedure (“Our crisis prevention team adhered to the case management protocol when collecting and evaluating information about a student in trouble.”), whether the team followed guidelines on how to provide adequate support for a student, and whether the school’s professional network was involved in case management.

Program acceptance scale. To operationalize school-wide responsiveness to the intervention, we created a 9-item Program Acceptance Scale (Cronbach’s alpha = .84) from items administered to all school staff at the 7-month follow-up. All school staff rated their level of satisfaction with the management of cases conducted by the school’s crisis prevention team, perceived accessibility and reliability of the crisis prevention appointee, the effect of the implemented structures on institutional information exchange, as well as overall program acceptance.

Program usefulness scale. The 8-item Program usefulness scale (Cronbach’s alpha = .81) provided an additional long-term-perspective on structural implementation and responsiveness, and reflected the experiences of crisis prevention team members with structured case management according to the NETWASS principles, e.g., “The crisis prevention team is helpful in managing student cases and violence-related problem behavior.”

Table 2 gives an overview of the used instruments and measures.

Establishing FOI Benchmarks and Descriptive Analysis

We defined cut-offs for all instruments that represented the minimum standards we considered necessary for program effectiveness. Specifically, we defined numerical thresholds for high FOI, based on either an empirical cutoff (mostly the mean value), or based on theoretical standards as discussed by the team, e.g., a minimum percentage of school staff that should have participated in the NETWASS trainings to achieve school-wide sensitization for the topic, or a minimum degree of responsiveness and compliance we considered to be a necessary condition for sustainable program implementation. In seven cases the theoretically defined threshold was the same as the calculated statistical mean value of the respective aggregated scale score. To assess the degree of FOI within each school, we first calculated scale scores for individual teachers, and then aggregated individual scores to the school level.

Results on the Degree of FOI at all Stages of Implementation

We collected FOI data from 15 measures consisting of 91 items. Our measurement system included an observational rating, a calculated training-participation quotient, a knowledge test, one rating scale measuring evaluation skills based on case vignettes, and eleven rating scales reporting general attitudes, as well as attitudes towards the program, and self-reported past behavior. Table 3 displays the results of FOI across the schools, as well as the cut-off values for high FOI.

In 73 schools at least 60% of all school staff participated in the trainings. Sixty-three schools met the criterion of 75% staff participation. There were 14 schools which showed a participation rate below 10%. School staff’s mean rating for training quality was 5.32. Self-reported participant engagement during the trainings was rated 3.72, and intervention compliance after the trainings was rated 3.19 by school staff. Mean ratings for program usefulness and program acceptance were 3.11, and 3.34, respectively. The trainers’ self-reported mean ratings for adherence to core processes were 4.02, and 3.11 for adherence to core activities. An above-average knowledge level (mean > .65) was found for 86 schools, and 88 schools implemented the recommended structures with FOI (i.e. established a crisis prevention team and nominated a crisis prevention appointee).

School staff’s mean ratings for the effectiveness of a preventive perspective targeting student support were 3.42, and 2.84 for the general tendency to provide student support and promote a trustful school climate. School staff’s mean adherence to the recommended decision criteria for case identification and reporting was 3.58. The crisis prevention team members’ self-reported mean rating for adherence to the case management protocol was 2.91.