Validity, inter-rater reliability, and feasibility of the Chelsea Physical Assessment Tool for assessing physical function in post-acute COVID-19 patients: A cross-sectional study

1.Introduction

Patients with coronavirus 2019 (COVID-19) who survive after intensive care unit (ICU) support are at risk for several physical, mental, and psychosocial problems such as ICU acquired weakness, dysphagia, chronic inflammation, hypovitaminosis D, neuromyopathy, delirium, anxiety, and functional limitations, known as post intensive care syndrome [1, 2, 3, 4]. Post-acute COVID-19 patients are often transferred from the ICU to post-acute specialized COVID-19 rehabilitation units or other hospital wards. In this context, an adequate transfer of information regarding patients’ condition and functional status at the time of ICU discharge during handover is mandatory for both ICU and rehabilitation physicians [5]. After an increasing number of patients who display functional impairments after COVID-19 discharged from the ICU, there has been a surge in the literature regarding the evaluation of the functional status of COVID-19 patients, as well as rehabilitation interventions that can improve the functional status of COVID-19 patients after discharge from the ICU [6, 7]. As a result, practical, feasible, valid, and reliable tools for assessing the functional status of COVID-19 patients discharged from the hospital should be used to make a definite conclusion about the efficacy of rehabilitation interventions.

The present assessment tools to assess the functional status of patients following ICU discharge could be categorized as strength test, walk tests, functional tests, and health related quality of life (HRQOL) assessment [8]. Among manual muscle strength tests, Medical Research Council (MRC) sum score is the suggested tool to screen ICU acquired weakness. Walk tests such as six-minute walk test or timed up and go test could be difficult to carry out as the patients could not even sit due to severe impairments during the early period [8, 9]. Additionally, these tests require space to perform and may require the management of several drips, drains, and oxygen delivery systems, which complicates test execution while the patient is walking and turning. Because of the increase in patient admissions due to the unexpected pandemic, hospital rooms have become crowded. This made it difficult to perform walk tests like these due to limited space in patient rooms.

Physical functional assessments include evaluation of muscle strength, mobility, physical and functional activity [8]. Among functional tests, the Physical Function in ICU Test (PFIT), Functional Status Score for the ICU and the Chelsea Critical Care Physical Assessment Tool (CPAx) are specifically developed to assess function after ICU stay. Barthel and Katz indexes are general tools to assess functional daily activities in various diseases [8]. According to a systematic review of 26 different outcome measures, the CPAx and the PFIT demonstrated the strongest clinimetric properties with established reliability, validity, and responsiveness; however, the PFIT has a greater floor effect than CPAx [10, 11]. The PFIT and CPAx may be more suitable for the assessment of patients who may never reach the ability to perform submaximal exercise tests [12].

Among these functional assessment tools, CPAx is easy to use in the clinical setting which covers respiratory function, coughing capacity, moving within the bed, sitting, and standing balance, sit to stand, transferring from bed to chair, stepping, and grip strength. Duration of assessment time is short and relatively few equipment (hand dynamometer for grip strength) is required. Many numbers of patients who were delivering non-invasive ventilation (e.g. continuous positive airway pressure) had to be discharged from the ICU to other outpatient wards because ICU beds had to be reserved for COVID-19 patients requiring invasive ventilation [13]. Although there are several tools that were constructed to measure physical function during ICU stay and following ICU discharge [14, 15, 16], they had not been validated in patients with COVID-19. To define rehabilitation needs of post-ICU COVID-19 patients who had been stepping down to acute wards and onwards into rehabilitation there was a need for a reliable, valid and feasible tool. We aimed to investigate the validity and the inter-rater reliability of CPAx in the assessment of the functional status of COVID-19 patients discharged from the ICU. Our secondary aim was to investigate feasibility of commonly used assessment tools for assessing physical function after ICU in COVID patients discharged from the ICU.

2.Materials and methods

3.Results

Demographic, clinical characteristics, and baseline data of 16 patients with COVID-19 transferred from the ICU to the ward are presented in Table 1. The flowchart of the study participants is depicted in Fig. 1.

Figure 1.

Flowchart of the study participants.

CPAx total score was strongly correlated with MRC sum score, Barthel and Katz indexes (Table 2).

Area under the ROC curve demonstrated that cut off score for CPAx was ⩽ 12 to discriminate patients with MRC sum score < 48 scores, with a sensitivity and a specificity of 100% and 63%, respectively (AUC = 0.859, p< 0.001) (Fig. 2).

Interclass correlation coefficients were high for CPAx, MRC sum score, Barthel and Katz indexes, Glasgow Coma Scale, and hand grip strength measurement, with the highest value observed for CPAx (Table 3). ICC of CPAx was calculated as 0.96 (95% CI, 0.71–0.98) indicating excellent inter-rater reliability. Each item of the CPAx demonstrated good to excellent inter-rater reliability (Table 4). κw values for “respiratory” and “moving within the bed” were calculated as highest among the all items (κw= 1, p= 0.0001; κw= 0.85, p= 0.0001, consecutively) indicating excellent agreement. “cough” and “supine to sitting on the edge of the bed” items were the two items with utmost lowest κw value among all items (κw= 0.64, p= 0.004; κw= 0.62, p= 0.004, consecutively) demonstrating good agreement. Floor effect was observed at Barthel Index scores (25%) and Katz score (37.5%) while no floor or ceiling effect was observed at CPAx.

Among the 16 patients, only 4 (25%) of the patients were physically able to complete the 5 time sit-to-stand, 30 seconds sit-to-stand tests. 2 (12.5%) of the patients were unable to complete the MRC sum score. All patients could perform grip strength test and CPAx.

4.Discussion

This cross-sectional study demonstrated that CPAx is a valid, reliable, and feasible tool with a high inter-rater reliability that can be used in the functional assessment of COVID-19 patients during the 48 hours following discharge from the ICU. CPAx showed a strong correlation with MRC sum score, which is an important screening tool for ICU-acquired weakness. The validity of CPAx as a functional assessment tool after ICU is further confirmed by its strong correlation with Barthel and Katz indexes which were previously found suitable for this population [35]. In addition, contrary to what is seen in the Barthel and Katz Indices, no ceiling and floor effects were observed in CPAx scores.

Figure 2.

Receiver operating characteristics (ROC) curve for Chelsea Critical Care Physical Assessment Tool to discriminate post-ICU COVID-19 patients with MRC sum score < 48.

Validity is the degree that a measured result reflects the measured content. The more consistent the measured result is with the measured content, the higher is the validity [36]. The construct validity results including the correlation coefficient between the CPAx score and MRC-score in the present study were considered good (n= 0.832) and correspond to earlier studies by Zhang et al. (r= 0.60) [36], Eggmann et al. (r= 0.809) [19] and Corner et al. (r= 0.65) [37]. Functional Independence Measure (FIM), Barthel and Katz indexes assess the ability to perform ADL [35]. They are frequently used for clinical decision-making for rehabilitation plans. Also, unlike previous validation studies of CPAx, in the current study, its validity has been demonstrated by also measuring the correlation coefficient between CPAx, Barthel, and Katz indexes which are commonly used tools in rehabilitation settings for measuring functional independence. Similarly, Eggmann et al. has shown high correlation coefficient between the CPAx and FIM indicating good construct validity [19]. The clinical implication of these results is that the CPAx seems to be a valid tool in assessing the functional status of COVID-19 patients after discharge from the ICU. Reliability results for CPAx were detected as excellent in this study and were consistent with previous studies. [11, 19] Different from previous studies, comparative reliability of the existing tools in assessing function after ICU including MRC sum score, Barthel and Katz indexes, Glasgow Coma Scale, and hand grip strength measurement were calculated. CPAx showed the highest reliability result. Among items of CPAx, inter-rater reliability assessed by Kappa value was highest for the respiratory item. “Cough” and “supine to sitting on the edge of the bed” items showed fair to good interrater reliability. Inter-rater reliability of the respiratory item was also found to be one of the items with the highest reliability in previous studies [11, 19, 38]. Similar to the results of the present study, Milton et al. detected moderate agreement for cough and supine to sitting items and almost perfect agreement for respiratory function [38].

Contrary to Barthel and Katz indexes, items for respiratory function and cough in the CPAx provides important information regarding the respiratory status of the critically ill patients who are recently discharged from the ICU. In case of acute respiratory distress patients such as COVID-19, majority of the patients continue to suffer from respiratory symptoms, as 75% of our study population had MRC dyspnea 3 or greater. These items displayed good-excellent reliability between two raters. Similar to the results of the present study, inter-rater reliability of individual CPAx items was detected high in critically ill patients and in patients who discharged from the ICU [11, 19].

Another advantage of CPAx is the inclusion of hand grip strength, which is an important indicator for general health, sarcopenia, and mortality [39]. This item might increase the value of CPAx as a prognostic indicator. It might have a role in screening the patients at risk for ICU acquired weakness. In our study population, a cut off value of 12 points in CPAx was found 100% sensitive and 63% specific to identify patients with MRC sum scores lower than 48. Zhang et al. have found that the cut-off value for diagnosis of ICU acquired weakness for CPAx is 31.5 with a sensitivity of 87% and specificity of 77% [31]. In the present study cut off value of CPAx for ICU acquired muscle weakness were detected lower than the study by Zhang et al. [31] but sensitivity was 100%. Sensitivity refers to a test’s ability to identify an individual with disease as positive. The more sensitive a test the more likely to detect an individual with a positive test and thus the more accurate to be used in screening. Since it is important to screen and correctly define patients with ICU-acquired muscle weakness, a lower cut-off value with high sensitivity may have utility in the assessment of post-ICU COVID-19 patients.

In CPAx, there are no items for continence, toileting, feeding, dressing, bathing or grooming, although Barthel and Katz indexes question the level of independency in these activities of daily living. CPAx mainly concentrates on the physical mobility within the bed, transfer from bed to chair, sitting, standing, and stepping. As it was developed for use in ICU and during ICU stay, the primary activity of concern is mainly transfer and mobility after breathing. ADL care is already provided by the nurses in the ICU. During the early period following discharge from the ICU, physical mobility is still one of the main parameters both for physicians and caregivers [39].

The aforementioned assessment tools mainly consider physical function, considering the potential sequalae of COVID-19 in terms of functioning [40, 41, 42, 43, 44, 45]. Patients with post-intensive care syndrome are also at high risk for mental and psychological disorders which are not questioned in any of these scales. Future scales for post-ICU discharge should consider the mental and psychological aspects as well as the swallowing domain under the physical function. Functional assessments require awakening and appropriate mental health. Possible mental impairments in the early post-ICU period may interfere with the cooperation of the patients [46]. In the present study, to prevent the impact of a possible mental problem, the patient’s awakening and response to basic commands are checked before CPAx administration. Moreover, a high prevalence of swallowing disorders was reported in COVID-19 patients with acute respiratory distress syndrome after discharge from the ICU. As mentioned before, the involvement of swallowing status in the assessment tools would be a better representation of the functional status [47]. To the best of our knowledge, CPAx tool can be strengthened by adding an item related to cognitive and swallowing impairment.

In line with previous studies, we did not detect a floor or ceiling effect of CPAx [19, 37, 48]. This was previously explained by the superiority of CPax’s measurement features, including evaluation of both physical function (whole body activities and hand grip strength) and respiratory (ventilation, oxygenation, and secretion clearance) measures [49].

COVID-19 patients who are discharged alive from intensive care units (ICUs) suffer from intensive care syndrome which is a multisystem disability that encompasses rapid acute muscle wasting and associated impaired mobility; cognitive problems relating to impaired short-term memory and executive function; depression, anxiety and post-traumatic stress-disorder; and dysphonia and dysphagia in those with and without tracheostomies. It can be hard to assess the functional status of these patients due to physical and cognitive impairments because the existing tools to assess functional status of these patients may require awakening, appropriate cognitive and conscious status. Also, several of functional assessment tests can be impractical because they require space to perform and may require management of several drips, drains, and oxygen delivery systems while the patient is walking and turning that render the tests difficult to carry out. During COVID-19 pandemic, there was a surge of patients admitting to hospital intensive care units and stepping down to other wards of hospitals due to shortage of intensive care beds. Due to all aforementioned reasons, to plan personalized rehabilitation, the tools for assessing functional status must be practical and feasible. In our sample, CPAx was the most feasible assessment tool including assessment of wide range of disability related to post-intensive care syndrome. Very few of the patients could be physically and mentally able to complete function tests such as the 5 time sit-to-stand, 30 seconds sit-to-stand tests. Also, several of the patients could not complete MRC sum score which is a frequent assessment tool to screen ICU acquired muscle weakness.

Walk tests or tools for functional status assessment of critical care survivors which include walking or shuttling items may show ceiling or floor effects if the patients have extreme (very low or very high) levels of physical performance. For patients who demonstrate high levels of physical performance, stride length and speed may limit the distance walked resulting in a ceiling effect. In contrast, after discharge from the intensive care unit, greatly disabled patients may not be able to walk at all, resulting in a “floor effect”. For example, in an observational study (n= 69), almost 40% of patients were unable to walk or needed assistance 4 days after discharge from the ICU [8]. In the present study, CPAx did not show floor and ceiling effects. We were able to calculate ceiling or floor effect of neither 5 time sit-to-stand test nor 30 seconds sit-to-stand test because only two patients were physically able to perform these tests.

A specific scale has been proposed for COVID-19 to measure functional status after hospital discharge termed the post-COVID-19 functional status scale. It mainly defines the need of assistance at work and at home and questions the presence of depression and anxiety [50], considering that long COVID patients might be also affected by a severe fatigue affecting the other clinical manifestations [51]. Therefore, this scale does not seem to be appropriate for use in critically ill patients early after discharge from the ICU.

Another specific scale which has been proposed to measure functional status of the COVID-19 patients with post-intensive care syndrome is the Post- ICU Presentation Screen (PICUPS), which was developed by the National Post-Intensive Care Rehabilitation Collaborative formed by the British Society of Rehabilitation Medicine’s (BSRM’s) and The Intensive Care Society (ICS) to address the rehabilitation needs of post-ICU COVID-19 patients as well as of all intensive care unit survivors as patients discharge from the ICU and acute hospital care [6, 7, 52]. Furthermore, by using PICUPS, to support triage and handover of patients discharging from the ICU to acute wards into rehabilitation, to inform the immediate care plan on the acute ward, to identify problems that are likely to require further detailed assessment/evaluation by members of the multi-disciplinary team, and thus, to develop a personalized rehabilitation plan. The PICUPS tool consists of four domains: medical and essential care, breathing and nutrition, physical movement and communication, cognition and behavior. PICUPS domains were developed to target to assess whole post-intensive care syndrome related impairments. The PICUPS Plus also include additional items related to ICU-acquired dysphagia, dysphonia or upper airway dysfunction that helps identifying breathing, voice and swallowing impairments thereby triggering further evaluation and intervention and referral for speech and language therapy [52]. PICUPS tool has been shown to have strong robust scaling properties including face and content validity, utility, feasibility, structural validity and responsiveness but its reliability has not been studied. Similar to CPAx, PICUPS did not show floor or ceiling effect [7]. This scale has been proposed during patient recruitment of the present study. That is why the authors of the present study were not aware of it during study protocol planning. We have chosen CPAx for the assessment of post-ICU COVID-19 patients taking into account the short time required for being administered and the relatively minimal use of equipment and space [53]. It may take more time to complete The PICUPS than CPAx. The PICUPS has the potential for use in assessing the functional status of COVID-19 patients after discharge from the intensive care unit but comparative feasibility, validity and reliability of PICUPS should be investigated in further studies.

Nowadays, the rehabilitative approach and evaluation has changed in order to cope with the COVID-19 pandemic [31, 54, 55]. However, the validity of CPAx in post-ICU COVID-19 patients might be considered valuable, considering severely ill neurological patients, early discharged from the ICU, requiring assessment tools more focused on physical early function improvements rather than ADL. From this point of view, demonstrating reliability, validity, and feasibility of CPAx compared to existing functional assessment tools in post-ICU COVID-19 patients can be viewed as the strength of the study.

The small sample size can be viewed as the limitation of the present study. We did not calculate the sample size for discriminative validity, but ROC analysis was performed and 12 or lower scores in CPAx was found to discriminate patients with muscle weakness (MRC sum score < 48); this result is suggested to be repeated with a larger sample size.

5.Conclusions

The findings showed that CPAx might be considered as a valid, reliable and feasible tool that might be used to assess the physical functional state in COVID-19 patients during the 48 hours following discharge from the ICU. It is a useful tool that can be used for supporting triage and handover of ICU patients stepping down to rehabilitation units or other hospital wards where individualized rehabilitation programs of patients are planned. Further observational studies are mandatory to improve the knowledge on the use of this instrumental tool.

Ethical approval

This study was approved by the Research Ethical Board of University of Health Sciences of Koç University Hospital (approval number: 245/2021). The study was also registered on clinicaltrials.gov.tr (NCT04762056).

Funding

This research received no external funding.

Informed consent

Not applicable.

Author contributions

Conceptualization, E.G., A.d.S., O.O.T.; methodology, E.G., A.d.S., O.O.T.; formal analysis, Z.T., M.T., A.B., C.C.; investigation, Z.T., M.T., A.B.; resources, O.O.T.; data curation, A.d.S., O.O.T.; writing – original draft preparation, E.G., A.d.S., writing – review and editing, O.O.T.; visualization, Z.T., M.T., A.B., C.C.; supervision, A.d.S, O.O.T. All authors have read and agreed to the published version of the manuscript.

Data availability statement

Due to the nature of this research, the supporting data are not available to be shared publicly due to ethical restrictions.