Accuracy of the Örebro Musculoskeletal Pain Questionnaire and Work Assessment Triage Tool for selecting interventions in workers with spinal conditions

2.1Study design

We conducted a historical comparative cohort study on patients with spinal conditions using data from the administrative and clinical databases of the Workers’ Compensation Board – Alberta (WCB-Alberta). This is a secondary analysis of a previously collected dataset used for evaluating the WATT among workers with a variety of musculoskeletal conditions. Details of the data collection procedures and variables in the dataset are available elsewhere [15]. In brief, all clinical measures in the dataset, including the ÖMPQ and WATT, were collected during routine clinical care as part of a return-to-work assessment conducted to determine patients’ ability to return to work or need for further treatment. We followed the Standards for QUality Improvement Reporting Excellence (SQUIRE 2.0) guidelines to report our results [17].

2.2Population

Data were extracted from the WCB-Alberta database on patients considered for rehabilitation between January 2013 and December 2016. We used ICD9 diagnostic codes to identify patients with specific or non-specific spinal conditions. Relevant diagnostic codes that were considered as spinal conditions and the actual number of cases in our study are provided in Supplementary Table 1. We excluded patients who did not have a spinal condition or who did not have a complete ÖMPQ score.

2.3Procedures

This study was approved by the University of Alberta Health Research Ethics Board. No patients were recruited/consented since this study relied on archived program evaluation data from the WCB-Alberta database. More details on the data extraction process are provided elsewhere [15]. From the existing data, we created the intermediate variables needed to perform the analysis.

2.4Measures

2.5Outcome

Our reference criterion was based on whether the intervention actually undertaken led to a successful complete or partial return to work. Return to work was measured using a surrogate indicator, reception of wage replacement benefits, and was available on 100% of our sample. Workers who were not receiving a full day of wage replacement benefits 30 days following assessment were considered to have returned to work. This was our main indicator of treatment success since wage replacement benefits are a commonly used measure within workers’ compensation jurisdictions and was used to evaluate the WATT tool previously [14, 21]. We also considered other potential wage replacement outcomes including whether the workers received wage replacement benefits (full or partial benefits) for up to 180 days after assessment in the follow-up year.

2.6Analysis

The sample was described using appropriate descriptive statistics for nominal, ordinal, and continuous variables. We used available demographic data to describe the patients with spinal conditions and a complete ÖMPQ score.

To assess the matching of risk-based recommended treatments to those actually undertaken versus whether that treatment was successful, we used the contingency table shown in Table 2. Columns reported whether the patients had successfully returned to work or not within 30 days and the rows reported the frequencies of whether the actual treatment received matched the recommended treatment or not. Sensitivity was calculated as the number of cases where the recommendation and the actual intervention matched and led to a successful return to work divided by the total number of cases with successful return to work. Specificity was calculated as the number of cases where the recommendation and the intervention mismatched and did not lead to a successful return to work divided by the total number of cases without successful return to work. Sensitivity and specificity were compared between decision tools and clinician recommendations using 95% confidence intervals (95%CI). The overall accuracy for each recommendation was calculated as the number of patients correctly classified divided by the total number of patients. Receiver Operating Characteristic (ROC) values were also calculated. These parameters were calculated overall for the ÖMPQ, WATT, and clinician recommendations, as well as separately within each risk level.

Our main comparison between recommendation sources was based on the positive likelihood ratio (LR+). LR+ was considered useful for identifying how the decision-making tools improved the initial probability of return to work (i.e., prevalence). It is interpreted as how importantly does following a recommendation change the initial probability (prevalence) of return to work. The size of the LR+ is interpreted as follows: 1 no difference, 2 small, 5 moderate, 10 large [22]. In addition, this parameter takes into account both sensitivity and specificity. It is calculated by the formula LR+= Sensitivity/(1-Specificity). We are interested in the LR+ because we want to know how much we increase the probability of return to work by using the appropriate intervention (i.e., by following a tool’s recommendation). The LR- is a secondary outcome because we are less interested in knowing the probability of return to work when we don’t follow a tool’s recommendation. The 95% CI was used to determine if the different tools had significantly different results (no superposition/overlapping).

In addition to completing analyses on the entire sample, we conducted sub-group analyses within relevant occupational and diagnostic groups. We examined results within the occupational group of ‘Trades, Transport and Equipment Operators and Related Occupations’ since this group had adequate sample size and has traditionally been at high risk of experiencing spinal conditions. We also examined results separately for workers with neck, back, non-specific, and specific diagnoses (see Supplementary Table 1 for relevant ICD9 codes representing these groups). Analyses were performed using IBM© SPSS© Statistics version 26 (Armonk, NY, USA).

3.1Patient characteristics

In the full database, 8,747 injured workers had a spinal condition. All of these had data on the clinician and WATT recommendations. However, only 2,872 (33%) had complete ÖMPQ data. In 32.8% of the cases, patients had low back pain (i.e. ICD9 diagnosis of lumbago, backache, or lumbar sprain). In 82.7% of cases, patients had a non-specific diagnosis. For those with spinal conditions and complete ÖMPQ data, descriptive characteristics are presented in Table 3.

3.2Treatment recommendations

The tools’ and clinicians’ recommendations cross-tabulated against the level of risk assigned to the interventions actually undertaken by each worker are shown in Table 4. Clinicians rarely recommended interventions considered suitable for ‘high risk’ workers (1.4% of recommendations made), whereas the WATT rarely recommended interventions considered suitable for ‘low risk’ workers (0.2% of recommendations made). The ÖMPQ was more balanced across the three risk levels, with some differences seen between the original and WCB-Alberta classifications.

3.3Tools’ accuracy

At 30 days after the assessment, all sensitivities and specificities were modest for ability to recommend interventions that lead to a successful outcome (see Table 5). The clinicians’ recommendations were the most sensitive (0.75, 95% CI 0.73–0.76) while the ÖMPQ with original cut-offs was the most specific (0.78, 95% CI 0.75–0.82). The LR+ (primary outcome) were small for each tool, but the LR+ of the ÖMPQ with the original cut-offs (1.51, 95% CI 1.26–1.82) was significantly higher than the WATT (0.93, 95% CI 0.83–1.05), the recommendations of clinicians (1.01, 95% CI 0.95–1.07) or the ÖMPQ with the alternate WCB-Alberta cut-offs (1.13, 95% CI 1.01–1.27) (Table 5). Matching interventions with low risk categories recommendations were much more accurate in predicting successful outcomes than the matching in the moderate and highrisk categories. For example, for the ÖMPQ with the original cut-offs the LR+ were 3.80 (95% CI 2.53–5.7), 0.96 (95% CI 0.75–1.24), 0.16 (95% CI 0.08–0.35), respectively, for matching interventions to the low, moderate and high risk recommendation categories.

Results did not change substantially on any of the subgroup analyses with the exception of slightly improved sensitivity of clinician recommendations within the back pain group (sensitivity increased from 0.75 to 0.83). However, all Likelihood Ratios as well as the ÖMPQ and WATT results did not change meaningfully (less than 10% change).

When we defined return to work success as receiving less than 180 days of compensation in the follow up year, results were not substantially different (results not shown) however there were no longer statistically significant differences between the tools. Overall, in most of the analyses the clinicians were the most sensitive (when return to work is a success clinical recommendations often matched actual treatments undertaken) and the ÖMPQ with the original cut-offs was the most specific (when recommended treatments don’t match those undertaken, the ÖMPQ had the highest number of failures).

4.1Main findings

Previous research indicates that screening questionnaires such as the ÖMPQ may help for predicting delayed return to work [9, 12, 23]. In this paper, we evaluated clinical decisions made based on the ÖMPQ risk recommendations against a computer-based decision support tool (WATT) and clinician treatment recommendations made following a thorough assessment. Matching risk recommendations to the actual treatments undertaken resulted in overall poor accuracy, however the ÖMPQ had significantly better results than the WATT and clinician recommendations. Therefore, ÖMPQ may also be helpful for selecting appropriate interventions. In fact, based on our observed likelihood ratios, following the ÖMPQ’s recommendations could increase the return to work probability and not following it could decrease the return to work probability. The original cut-off showed the best accuracy.

To our knowledge this is one of the first times that the ÖMPQ has been tested as a clinical decision support tool. Clinicians knew the ÖMPQ results and may have used it in their decisions, however the ÖMPQ alone still resulted in higher accuracy than clinician recommendations. Further research is needed to explore optimal cut-offs and what additional information or measures could improve the accuracy of treatment recommendations.

Overall, the difference between the different recommendation sources is small and the LR+ were often low. However matching treatments provided by the ÖMPQ recommendations with the original cutoffs offered significantly better prediction of treatment success than the clinician recommendations alone (current standard practice). Even if the differences between tools are low, the consequences of using the tool are important. For example, if we consider a return to work prevalence of 68.2% [24] at 30 days, following the clinicians’ recommendation would increase the rate of return to work by only 1% (95% CI = 0.5–1.8%), whereas following the ÖMPQ with the original cutoffs would improve the likelihood of return to work by 8% (95% CI =4.7–11.4%). When applied to an entire population of workers, this may result in sizeable cost savings through earlier return to work.

4.2Strengths and limitations

Relying on historical archived data from WCB-Alberta allowed us to achieve a large sample size, however it also had some limitations and resulted in a large amount of missing data. We had no control over what variables could be measured and were limited to the variables in the existing dataset. In addition, the WATT and the clinicians normally have 6 treatment categories but in order to make it comparable with the ÖMPQ risk levels, we merged treatment options into only 3 categories. This increases the accuracy parameters of matching the recommendations but decreases precision (it recommends a group of interventions, not a single one). Additionally, the ÖMPQ treatment classifications used have not been previously studied and different classification strategies may perform better. The generalization of results is limited due to the restricted location of the population. The analysis has been restricted only to the 33% of the patients with spinal pain who had a complete ÖMPQ, which could lead to a selection bias. On the other hand, the study has a relatively high number of participants that could improve external validity of results. Another limitation is the use of wage replacement benefits as our main outcome measure. Reception of benefits is a surrogate indicator of actual return to work and is influenced by several individual and contextual factors. However, this outcome is commonly used in studies of injured workers and is an important outcome within workers’ compensation jurisdictions. Instead of wage replacement outcomes, future prospective studies in this area could include patient-reported outcome measures that were not available in our dataset.