The quality of clinical practice guidelines for fiberoptic endoscopic evaluation of swallowing: A systematic review

2.1Research design

A systematic review of the literature was completed to identify and appraise international FEES CPGs, to establish their ability to effectively guide clinical dysphagia practice. This systematic review is reported in line with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) checklist. Ethical approval was not required as this study did not involve human or animal participants.

2.2Search strategy

A search strategy was developed to search across four bibliographic databases, including PubMed (Medline), Embase (Scopus), CINAHL and Web of Science. This search strategy was reviewed by a subject librarian and performed on 10/10/20. The timeframe was from inception of the database to October 2020. Other sources were searched on 11/10//20, including The National Institute for Health & Care Excellence (NICE) Guidance, Prospero- The International Prospective Register of Systematic Reviews, Google Search and through the manual searching of national SLT professional associations’ websites. The search of the academic databases and Prospero included 3 search strings. Search terms were customised for each database but included similar variations of ‘dysphagia’, ‘FEES’ and ‘practice guidelines’ for the controlled vocabulary and title/abstract terms of the three respective search strings. The full search strategy is presented in Supplementary Information Table 1.

Websites from the following professional associations were manually searched on 11/10/20 to identify publications of potential relevance: Royal College of Speech & Language Therapists (RCSLT); Speech Pathology Australia (SPA); American Board for Swallowing & Swallowing Disorders; American Speech-Language Hearing Association (ASHA); New Zealand Speech-Language Therapists’ Association (NZSTA); Dysphagia Research Society; Japanese Society of Dysphagia Rehabilitation (JSDR); European Society for Swallowing Disorders (ESSD) and Hong Kong Association of Speech Therapists. NICE Guidance and Google were also searched in this manner. Search terms used within these websites were ‘fiberoptic endoscopic evaluation of swallowing’ or ‘flexible endoscopic evaluation of swallowing’. An additional search was carried out on 05/09/21 to ensure that no other CPGs were published more recently.

2.3Eligibility criteria

2.4Selection process

The search results were stored in EndNote X9 and uploaded to the Covidence Systematic Review Software, where duplicates were removed and the title/abstract screen was completed. Reference lists of selected studies from the search of databases and other sources were screened to locate publications of potential relevance. Four studies were identified in this manner and were included in the full text review. Worldwide experts in the area of FEES and relevant professional associations were contacted via email regarding their knowledge of any additional CPGs available. No further publications were identified in this manner.

2.5Data extraction

Abstract screening was carried out by two independent reviewers (LMS and RB). Abstracts were discussed collectively using the eligibility criteria if any disagreements occurred. If uncertainty remained, the full text would be retrieved and reviewed to arbitrate the disagreement, but this did not occur. Full consensus was required for abstracts to be included for full text review.

Data was extracted and reviewed by two researchers (LMS and JR) using a pre-developed data extraction table on Microsoft Excel. Data extraction tables were developed to extract CPG characteristics and AGREE-II appraisal scores. The following data was extracted to identify characteristics from the included studies; professional association, year published, CPG title, development method included, systematic review included, external review included, grading system included, service user involvement, search strategy included, target group outlined and potential candidates for FEES discussed. The AGREE-II individual domain scores and rationale for scores for each appraiser were documented in a data extraction table also. All data was then combined and tabulated to facilitate the descriptive analysis of results.

2.6AGREE-II training and rating

The four CPGs underwent a quality assessment using the AGREE-II tool, which was rated by 3 independent researchers (LMS, SL and JR). The AGREE-II instrument evaluates the methodological rigour and transparency of CPG development. (Brouwers et al., 2010) The tool consists of 23 items organised into six domains: 1) Scope and Purpose, 2) Stakeholder’(s) Involvement, 3) Clarity of Presentation, 4) Rigour of Development, 5) Applicability and 6) Editorial Independence. Each item is assessed using a 7-point Likert scale (1- “strongly disagree” to 7- “strongly agree”) based on the criteria provided. This instrument was chosen as it is the most common tool to specifically evaluate CPGs. (Boaden et al., 2020)

The AGREE Enterprise website (www.agreetrust.org/), provides an AGREE training module, which was completed by the researchers and the user’s manual was examined before the employment of the instrument. Items were rated with reference to these resources and rationale for the scores issued were recorded in the comments section.

Overall scores for each domain were calculated in accordance with the algorithm discussed in the AGREE-II manual (as seen on www.agreetrust.org/). A domain score of 60% or above was deemed as good quality. This score was taken from previous research evaluating the quality of CPGs (Boaden et al., 2020; Hurkmans, Jones, Li, & Vliet Vlieland, 2011; Johnston et al., 2018; Lucendo et al., 2017). The AGREE-II does not provide specific scores to classify as adequate quality, and states that this should be decided by the user and guided by the context in which they are used. The same cut-off point was used for each domain. Two overall assessments of the CPGs were also completed within the AGREE-II checklist. These consider the overall quality score and recommendation for use in clinical practice, taking into account the criteria considered during the critical appraisal. The total score of the AGREE-II domains for each CPG was not calculated as recommended by the AGREE protocol, given that domains may not be weighted equally. Reviewers met to agree final scores if any disagreement occurred on these items.

2.7Data analysis

Data was analysed to compare common characteristics and quality appraisal scores. The level of agreement between AGREE-II appraisers across the four CPGs was evaluated to assess reliability between appraisers. This was identified by calculating the percentage agreement in appraisers’ scores of the 23 individual AGREE-II items assessed across the four CPGs. An intraclass correlation coefficient (ICC) was calculated using SPSS v.26 to measure inter-rater reliability of the three appraisers’ scores for each CPG evaluated.

3.1Characteristics of included CPGs

The international guidelines represented national SLT professional associations across four countries: Australia, United Kingdom, United States and New Zealand. Three of the CPGs were published between 2018 and 2020, with one published less recently in 2004. All of the included CPGs were written in English.

Discrepancies were established in terms of characteristics such as their development method, grading systems included and service user involvement in the development process. Additional characteristics are outlined in Table 1.

3.2CPG quality appraisal

The domain scores for each CPG are demonstrated in Table 2 and Fig. 2. Table 2 displays a summary of the scores for each AGREE-II domain, including the median and range. A wide range of scores are depicted between domains. Four out of six domains achieved a median score of below 60% and no CPG scored above the 60% threshold in more than three domains.

Fig. 2

AGREE-II domain scores across the appraised clinical practice guidelines.

The highest median domain score (66%) was achieved in domain 1 “Scope and Purpose”. Three out of four CPGs scored above 60% (range 50–89%) in this domain. A median score of 60% or less was achieved in all other domains. However, two out of four CPGs scored above 60% in domain 4 “Clarity of Presentation” (median 60%; range 33–78%). In domain 2 “Stakeholder Involvement”, one CPG scored above 60% (median 47%; range 35–78%).

No CPG achieved a score above 60% in the following domains: “Rigour of development” (median 29%; range 24–59%), “Applicability” (median 13%; range 8–26%), and “Editorial Independence” (median 0%; range 0%). All median scores in these domains were below 30%.

3.3Overall assessment

The overall assessment was rated on a Likert scale of 1–7, considering the CPGs quality over the various domains. Table 2 states whether the CPG was recommended for use in clinical practice based on these scores. A CPG was considered to be “recommended for clinical practice” if a score of 60% was achieved in at least 4 domains. CPGs were “recommended with modifications” if a score of 60% was achieved in three domains. CPGs were “not recommended for use in clinical practice” if at least four domains scored less than 60%.

3.4Appraiser’s consistency

The ICCs calculated for all CPGs were above 0.90, indicating excellent inter-rater reliability, as displayed in Table 3. These scores demonstrate strong agreement and little variation between appraisers’ scores across CPGs (O’Donnell, Allison, Melikian, & Iafrati, 2020).

4.1Main findings

To the authors’ knowledge, this is the first study to systematically appraise the quality of FEES CPGs. Study findings were consistent with the original hypothesis, conveying the scarcity of FEES CPGs worldwide and their inferior quality. The majority of CPGs omitted relevant detail within their recommendations or the development process. Based on the critical appraisal using the AGREE-II instrument, none of the included CPGs are recommended for use in clinical practice. However, one CPG by RCSLT, was recommended for use with modifications, based on the domain scores achieved. Standardisation of FEES CPGs is urgently required to provide reliable and high-quality care to service users. Authors acknowledge that other multi-disciplinary team (MDT) members such as ENT specialists, otolaryngologists, or neurologists may carry out FEES. This review was focused solely on CPGs for SLTs carrying out the procedure.

4.2Variance among CPG characteristics

Information regarding the development process, search strategies or included evidence to support recommendations was not sufficiently included in the evaluated CPGs. Details were omitted regarding grading systems used to rate their evidence, service user involvement in the development process or the inclusion of an external review by an expert team. The above characteristics are considered crucial features required to develop a document of evidence-based recommendations with a transparent methodology (Zafra-Tanaka et al., 2019).

CPG development processes varied widely. No CPG included a systematic review of the evidence or discussed any methodology by which they reached their recommendations, with the exception of RCSLT’s CPG, including a brief discussion only. Similarly, service user and MDT involvement were solely mentioned in RCSLT’s CPG, but not sufficiently discussed. Given that CPGs are developed to optimise patient care, it is indispensable that service users are consulted during the guideline development process to promote implementation of CPGs in a way that service users deem advantageous (Armstrong, Mullins, Gronseth, & Gagliardi, 2018).

Recommendations were often provided in the form of generalised statements, without a clearly stated rationale or supporting evidence, which reduces the recommendations’ applicability. Recommendations should have a key action statement, elaborated by text, evidence and strength of recommendation ratings (Institute of Medicine, 2011). Whilst only one CPG graded their practice statements, most were graded as “GP”, abbreviated for “good practice based on clinical experience and expert opinion” rather than trusted evidence, which would be graded as “A”. (SPA, 2019). No recommendation was graded as trusted evidence and 26 out of 32 recommendations made in the CPG were given the grading “GP”. This could imply that these recommendations were graded as “GP” due to the fact that there is limited evidence available to support the recommendation, and thus developers are relying more on clinical practice and expert opinion. Additionally, ambiguity in recommendations can signify a limited evidence base (Institute of Medicine, 2011). Further high-quality research in areas of dysphagia assessment may be beneficial to ensure that recommendations can be supported by appropriate evidence, in addition to expert opinions, in order for superior CPGs to be produced. (Koidou, Kollias, Sdravou, & Grouios, 2013; Plowman, Mehdizadeh, Leder, Martino, & Belafsky, 2013).

4.3Limited CPG quality

The AGREE-II critical appraisal concluded that no CPG is suitable for use in clinical practice without modifications. The most recently published CPG by RCSLT, was recommended for use in clinical practice with modifications, achieving an overall quality score of 5. However, even this CPG scored inadequately in certain domains (“Rigour of development”, “Applicability”, “Editorial Independence”).

Higher scoring domains across the appraised CPGs included “Scope and Purpose” and “Clarity of Presentation”. Some adequate scores in these domains indicates effective reporting of the objective, target population and specific health question, as well as a clear structure and format within the CPG. Only one CPG scored adequately in Domain 2 “Stakeholder Involvement”, meaning that all other CPGs did not include relevant patient, carer or professional involvement in the development process.

All median domain scores in “Rigour of Development”, “Applicability” and “Editorial Independence” were below 30%, indicating very poor quality. These scores depict an absence of systematic methods used to develop and update recommendations, as well as a critical evaluation of the evidence supporting the recommendations. These scores also reflect a lack of advice to put the recommendation into practice, discussion of the resource implications in doing so, and an omission of declarations of competing interests or statements relating to the funding bodies.

“Rigour of development” is the largest domain of the AGREE-II instrument and plausibly one of the most important domains in the assessment (Amer et al., 2019; Zafra-Tanaka et al., 2019). It is therefore concerning that three of the four CPGs scored very poorly in this domain (below 30%), due to a lack of transparent and robust methodology.

4.4Comparison to existing research

Findings relating to the quality of FEES CPGs can be compared to VFSS CPGs, researched by Boaden and colleagues (2020). Figure 3 highlights that areas of strength and downfall exist in the same domains of the AGREE-II for FEES and VFSS CPGs. Considerations can therefore be made regarding improvements in guideline formulation and quality with particular focus on the lower scoring domains displayed in Fig. 3.

Fig. 3

Trend in median domain scores of AGREE-II in FEES vs VFSS CPGs. Blue represents the median domain scores for FEES CPGs. Orange represents the median domain scores for VFSS CPGs. VFSS data adapted from Boaden et al., 2020.

The lowest scoring domains for both VFSS and FEES were the following: “Rigour of Development”, “Applicability” and “Editorial Independence” respectively, whereby no CPG scored above the 60% threshold in either study. With a lack of focus on these domains, CPGs are omitting crucial steps in formulating recommendations using systematic methods and high-quality evidence and reducing the ability for clinicians to apply the CPGs to practice. They are also failing to state competing interests or information regarding funding bodies.

Lower scores in these domains have been noted across healthcare disciplines, including CPGs for the treatment of tic disorders, depression in adults and chronic kidney disease (Boaden et al., 2020; Coronado Daza, Vernooij, Salas, Osorio, & Urrútia Cuchí, 2019; Yang et al., 2018; Zafra-Tanaka et al., 2019).

4.5Clinical implications

This study’s findings carry clinical significance as it highlights the gaps that exist in the provision of optimum patient care. Prior to this study there was limited information surrounding the nature of FEES CPGs, a resource frequently adopted to guide numerous assessment and management plans for SLTs worldwide. As the primary role of CPGs is to promote evidence-based practice, poor-quality CPGs are potentially detrimental to patient outcomes.

As the demand for FEES is growing and will continue to increase as the prevalence of dysphagia rises (Adkins et al., 2020; Madhavan, Lagorio, Crary, Dahl, & Carnaby, 2016; Smithard, 2018), these findings are crucial to allow clinicians to make decisions about their use of CPGs in practice. This research portrays the nature and quality of such resources and the need for additional CPG development.

4.6Limitations and future directions

The authors acknowledge certain limitations. Given the broad definition of CPGs employed in this study, it could be argued that some included documents should not be measured using the AGREE-II tool. However, each of the documents included were developed with the intention to guide clinical practice and are the source of guidance for SLTs carrying out FEES procedures. Additionally, similar broad definitions have been used in other CPG research (Boaden et al., 2020).

All CPGs included were published in English. Findings did not capture CPGs that were published in other languages. Quality ratings were based on authors’ reports of the completion of certain steps within the development process. For example, if CPG developers did not report the systematic methods used to formulate recommendations, a transparent methodology is not evident and a lower score will be assigned to that item.

The ‘overall guidelines assessment’ at the end of each AGREE-II assessment requires the reviewer to assign a score between one and seven of the overall CPG quality based on the scored domains. Whilst there were no large disagreements on this item between reviewers, the authors found this section somewhat subjective with no outlined criteria to designate a score. Additionally, whilst the AGREE-II provides explicit criteria allowing the authors to effectively evaluate the quality of CPGs, the involvement of an epidemiologist with expertise in CPG development would have added value to the critical analysis of the international CPGs included in this study.

In order for direct improvements to be made to FEES CPGs, particular focus needs to be placed on the following measures: including systematic methods within the development process; incorporating service users and other relevant stakeholders’ views and preferences in the development process; the provision of advice and tools to put the recommendations into practice; and statements regarding the views of funding bodies and competing interests.

Any future FEES CPG developers should be advised to closely follow the AGREE-II reporting checklist outlined by Brouwers and colleagues (2016), as one of its primary functions is to guide CPG development. By doing so, all necessary components of high-quality and standardised CPGs can be formulated efficiently. Particular focus on the aforementioned lower scoring domains (“Rigour of development”, “Applicability” and “Editorial Independence”) in FEES CPGs will assist developers to overcome common errors and omissions. Future developers should also be advised to ensure all steps within the development process are reported within the CPG. This study’s findings convey that an omission of this information often leads to an inadequate quality score in some domains.