A Systematic Review of Outcome Reporting, Definition and Measurement Heterogeneity in Non-Muscle Invasive Bladder Cancer Effectiveness Trials of Adjuvant, Prophylactic Treatment After Transurethral Resection

INTRODUCTION

METHODS

Data collection and analysis

Selection of studies

Following de-duplication, at least two review authors (DC, SM, SS, IO, EV, RC) independently screened the titles and abstracts of identified systematic reviews for eligibility. The full texts of all potentially eligible publications were retrieved and screened independently by two review authors (DC, SM, SS, IO, EV, RC) using a standardised form, linking together multiple records of the same study in the process. Any disagreements were resolved by discussion or by consulting a senior review author (RS). Once the list of systematic reviews meeting the inclusion criteria were finalised, a second screening process was initiated whereby the studies included in the systematic reviews were screened against our inclusion criteria. Where lists of studies excluded from the systematic reviews were available, we also screened these in case the studies had been excluded for not reporting on outcomes of interest. In such instances the trial may still have met inclusion criteria for our review. The study selection process is described in the PRISMA flow diagram (Fig. 1) [25].

Fig.1

Preferred Reporting Items for Systematic Reviews (PRISMA) diagram of studies. SR, systematic review; RCT, randomized controlled trial.

Data extraction and management

A standardised data extraction form was developed and piloted. One review author extracted data and a second review author checked data extractions for accuracy (DC, SM, SS, IO, EV, RC). Any disagreements were resolved by discussion or by consulting a third review author.

Data that were extracted included: the study design; countries and institutions where the data were collected; dates defining start and end of patient recruitment and follow-up; how intervention comparator groups were formed; participant demographic and clinical characteristics; eligibility criteria for participants; the numbers of participants who were included in the study, assigned to each intervention comparator group; description of interventions; study funding sources; and ethical approval. All primary and secondary effectiveness outcomes reported, their definitions, and any outcome measurement instruments used were extracted verbatim.

Assessment of risk of bias in included studies

Risk of bias assessment is not necessary for systematic reviews undertaken for COS development. Some outcomes may be at risk of detection bias depending on whether they are relatively subjective or objective. Although these aspects were extracted under the ‘definition’ or ‘measurement’ fields in the data extraction form, this is out of the scope of this phase of our project. They will be investigated in a subsequent phase whereby we will assess the psychometric properties of the various outcome measurements and seek consensus on the most appropriate and feasible definitions and measurements [26, 27].

Data synthesis

Verbatim outcome names were recoded to common names. This was done by categorising outcomes referring to the same underlying constructs under a common term. For example, “survival rates”, “overall survival”, “number of deaths at median follow up” and “mortality rate” all refer to the concept of ‘overall survival’ and were coded as such. The outcome and domain coding process was inductive and iterative. Coded outcomes were further grouped in broader domains using the standardised Williamson and Clarke Taxonomy (W/C Taxonomy) [28].

EVIDENCE SYNTHESIS

ADVERSE EVENTS

Adverse events (AEs) were heterogeneously defined. In 12 of the 44 studies (27%) reporting AEs, there was no definition of an AE, and overall 24 different definitions/instruments were used. Studies reporting AEs used unique systems to categorise the type of AE or grade the severity of the AEs, and made no reference to a standardised reporting system. Across 10 studies, 3 standardised AE reporting instruments were used, but these did not include some of the most relevant AEs for intravesical instillations:

Adverse events were further grouped in numerous ways, e.g. local or systemic toxicity, constitutional symptoms, laboratory abnormality, death, and treatment interruption due to AEs (Table 5). Detailed lists of how AEs were described and reported are provided in Supplementary Table 2.

In 25 of the 44 studies (57%) specific AEs were not listed; instead, authors reported either only local toxicities, or major/severe/more common side-effects or AEs that resulted in treatment interruption. Five of these 25studies did not report the list of individual toxicities at all; instead, authors presented only the frequency and percentage [n (%)] of any AEs which occurred. Furthermore, poor treatment compliance related to AEs was not consistently reported.

TREATMENT FAILURE/CHANGE OF TREATMENT

21/57 studies (37%) reported treatment failure and/or the need to change from instillations to a different treatment. 21 studies specified the treatment that was given after instillations were discontinued:

GLOBAL QUALITY OF LIFE

Two studies measured and reported patient experience during the instillations; Koga et al by measuring Qol, and Huang et al by evaluating instillation related pain/irritation [30, 31].

In the study by Koga et al, Qol was assessed according to the Japanese version of the European Organization for Research and Treatment of Cancer Quality of Life Questionnaire-Core 30 (EORTC QLQ-C30) v2.0. QoL was assessed before induction therapy, after the 5th instillation of induction therapy, 4 weeks after the completion of induction therapy, and 14 months after randomization [30].

Huang et al evaluated the effect of hyaluronic acid in reducing pirarubicin instillation related side-effects. A visual analog scale (VAS) was used daily to evaluate pain [31].

RESOURCE USE (HEALTH ECONOMICS)

Only one study evaluated the costs related to the treatment. Berrum-Svennung et al randomized NMIBC patients to one instillation of epirubicin or placebo after TURBT and evaluated cancer recurrences. They also calculated the cost of delivering a single instillation during the initial treatment and as first recurrences occurred [32].

DISCUSSION

This is the first study to systematically and comprehensively overview the extent of outcome reporting, measurement, and definition heterogeneity in the setting of adjuvant treatments for NMIBC.

Recurrence was frequently reported in the included RCTs; yet, some studies did not define it. In those that did, there was variability in the names that were used, the definitions and the reporting. Most concerning, however, was the variation in how progression was handled in the analysis of recurrence. In studies where progression as the first event was counted as a recurrence, the measure provided is qualitatively and quantitatively different from those where recurrence was more narrowly defined as the re-appearance of a non-muscle invasive tumour. To overlook this subtlety runs the risk of not comparing like with like across studies, or statistically pooling aggregated results in a potentially misleading way.

Progression was also frequently reported, but again the definitions were inconsistent across trials. Worsening of the disease leads to a change in treatment strategy, and that was also inconsistently reported. It is also crucial to point out whether assessment of progression has been made based on imaging (e.g. CT or MRI), TURBT or radical cystectomy. As only four studies gave a comprehensive overview of reasons to change the treatment strategy, there is a high risk of getting misleading results. If prior to progression, patients die due to an unrelated cause, or undergo cystectomy (for example due to recurrent high grade T1 disease), then the progression rates at specific time points will be different according to whether the death and cystectomy have been counted as a competing risk (cumulative incidence function) or simply as censored (Kaplan-Meier curve). Equally important is to highlight how patients are followed for the efficacy outcomes in case the treatment has been stopped due to side-effects. There may also be a difference in outcomes according to whether the results are reported in all randomized patients (intent to treat analysis) or only in eligible patients who have been treated according to the protocol (per protocol analysis).

Treatment response in patients with CIS in specific was evaluated and reported in only 40% of studies. The rest of the studies recruiting patients with CIS evaluated CIS as papillary tumors, and reported only recurrence or/and progression. However, CIS has additional diagnostic challenges and may have a very different disease course than papillary tumors do: as such, separate approaches to measure and define their outcomes should be applied [23].

The most heterogenous outcome was AEs, evident in the many categorizations and instruments used to record AEs, and in the system level subgroupings chosen by trialists. Unfortunately, many of these were not optimal for instillation-related AEs. Whilst in some instances it may be possible for systematic reviewers to recode lists of AEs (if they are provided) to a common standardized toxicity classification system, this is a poor excuse for lack of standardization in primary trials and needlessly adds time and complexity to the critical interpretation of the evidence base. Poor treatment compliance reporting is likely to confound other cancer related outcomes such as recurrence, progression and overall survival.

Perhaps the most alarming finding is that QoL is conspicuously missing. Instillation treatments are demanding for patients and it would be very important to understand all the consequences (both oncological and QoL-related) for patients before the decision about treatment is made. A recent investigation of QoL in bladder cancer patients compared to a matched sample of older adults without bladder cancer in a US population found significant declines in health-related QoL (HRQoL) scores over time in the physical, mental and social components of the SF-36 [33]. The EORTC Quality of Life Group also developed an externally validated QLQ-BLS24 questionnaire for NMIBC [34]. In a systematic review, Mason and colleagues used the COSMIN checklist to evaluate the psychometric properties of patient reported outcome measures (PROMs) used in bladder cancer populations, of which two of the 15 included PROMs were NMIBC-specific (QLQ-BLS24 and CAVICAVEMNI) [35, 36]. Of note, they found that no existing PROM stood out as the most appropriate measure of QoL in any bladder cancer population and although further validation studies are required, generic PROMs, cancer-generic PROMs and bladder cancer-specific PROMs will currently provide the most robust picture. Mason et al’s study [35] is very important for a subsequent phase of our COS development as most existing cancer COS have included QoL and it is anticipated NMIBC patients will also prioritise this, encompassing urinary, bowel and sexual function, as critically important outcome domains.

Without having included NMIBC patients in a qualitative study of their experiences of bladder cancer and its treatments, it cannot yet be known which outcomes are of most importance to them, or if they are adequately captured in current trials, but it is discouraging that so few trials routinely include PROMs.

Health economics was considered in only one RCT, which calculated costs of single instillation [32]. Bladder cancer, especially NMIBC, contributes significantly to healthcare costs due to intense surveillance strategies and its potential to recur and progress [8, 37]. This should be considered when treatments and outcomes are compared.

Kamat et al provided recommendations on NMIBC intervention trial designs, eligibility criteria, and ‘clinically meaningful’ effect size thresholds for outcomes [23]. Likewise, Lamm et al suggested a change in definition for progression in NMIBC [38]. These initiatives are important to bear in mind for subsequent phases of our project. Once the outcomes considered core by all stakeholders (e.g. patients, urologists, oncologists, nurses, payers, methodologists) are known (i.e. what to measure) [22] then we will turn attention to definitions and measurement tools (i.e. how to measure) [39] whilst again including key stakeholders. Importantly, these initiatives, in conjunction with ours, show that there is an acknowledgement of problems with the evidence base and a desire to do improvements.

LIMITATIONS

The decision to exclude phase I and II trials (phases before determining the therapeutic effect of the drug) and to exclude all non-randomised designs may have limited the chance to capture longer-term and patient reported outcomes relating to function and QoL. However, in subsequent phases of the project, such as in Delphi survey and consensus meetings, participants will have an opportunity to propose ‘new’ outcomes not already considered for prioritisation, therefore we consider that the risk of having missed outcomes is minimal, and that we have carried out a pragmatic trade-off against the resource implication of including all study designs.

CONCLUSIONS

We have shown that there is inconsistency in outcome reporting and variation in definitions in randomized trials comparing adjuvant treatments in NMIBC patients. This situation makes comparing the results of individual studies difficult, and makes their statistical combination challenging, impossible, or inappropriate; hence, providing summaries of the evidence which are, at best, unwieldy and at worst misleading, making evidence-based treatment recommendations difficult. A core outcome set, incorporating the views of a variety of stakeholders such as urologists, oncologists, methodologists and, most importantly, patients, is urgently required.