Methodological Quality of Clinical Trials in Amyotrophic Lateral Sclerosis: A Systematic Review

Search strategy

An expert reference librarian and a study author with expertise in conducting systematic reviews and in ALS clinical trial design and management (EP), developed the search strategy from the time of database inception until December 2022. We searched PubMed, EMBASE, and the Cochrane Library. The detailed search strategy is reported in Appendix 1. The study followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) reporting guidelines in terms of study selection, data collection and synthesis, and assessment of bias and quality[3]. We checked the reference list of the retrieved eligible studies to find relevant articles missed from database searching. We also searched for additional references from recent systematic reviews to avoid any potentially missed papers.

The following databases were searched only with the purpose of evaluating publication bias: The International Clinical Trials Registry Platform, the European Union Clinical Trials Register, and clinicaltrials.gov datasets.

Eligibility criteria

We included all original full-text reports related to clinical trials (any type: open label, double-blind, randomized, not-randomized, cross-over, adaptative) performed on people with ALS/MND aged 18 + years, and conducted in hospitals, nursing homes, outpatient clinics, or at community level, published in the English language before December 31, 2022. We included studies of disease progression or symptomatic pharmaceutical intervention, with efficacy as primary or secondary endpoint. We included studies of phase I/II, II and III, according to the Authors’ definition. When the phase was not stated, we categorized the study depending on the study characteristics and aims. Excluded were phase I trials, reports describing a study protocol or interim analyses or post-hoc analyses, reports, letters to the editor, book chapters, conference proceedings, dissertations, theses, and animal studies.

We considered placebo, any active drug, standard practice “for comparative studies” and “none, with before-after comparison” for non-comparative studies.

Study selection

We exported all the references retrieved from the searches into the Rayyan online tool [4]. Duplicated records were manually removed by one author (EP) prior to screening. After that, four pairs of reviewers working independently, blindly assessed each article’s title and abstract for eligibility. When each pair completed all the evaluations, the blind was open and disagreements were solved by discussion and consensus during a direct meeting between the two members of the pair. In a few cases, when the reviewers did not reach an agreement, a third evaluator (EP, ML) reviewed the study and made the final decision. The reasons for the exclusion were listed for all excluded documents. After the title and abstract screening, four pairs of independent reviewers read the full articles of the remaining studies and completed a full-text review.

Reviewers extracted study details from the full text articles using a pilot-tested form defined by three senior epidemiologists and two junior investigators. Variables were selected according to Sackett [5]. All variables were pre-defined and categorized. The main domains of data extraction were: general information, demographic characteristics (number of participants recruited and withdrawn), comorbidities, investigational medication(s) used, diagnostic criteria, inclusion/exclusion criteria, primary objective, study design, primary (with effect sizes) and secondary outcomes, statistical aspects, risk of bias. Data were extracted directly from published reports and no additional information was requested from the corresponding authors nor collected by external sources.

To ensure accurate data extraction, a training meeting was performed and any question was solved before starting the extraction. Data were extracted independently by 5 reviewers in double. After the evaluation of 3% of the eligible reports, a second meeting was arranged and the data extracted were compared to discuss any discrepancy and to ensure consistency between reviewers. A third meeting was planned and carried out after the evaluation of 30% of the full texts. After the third meeting, data were extracted by a single reviewer for the remaining publications.

The risk of bias was also assessed using the appropriate Cochrane tools for clinical trials (ROB2 for randomized trial, ROB2 for cross over trial) structured into a fixed set of domains describing different sources of bias, focusing on different aspects of trial design, conduct and reporting. The risk of bias for non-randomized trials was not assessed. The risk of bias was classified into three categories including “low”, “high” or “some concerns” according to the Cochrane manual [6]. The risk of bias was assessed by the same pairs of reviewers and was performed in double for 36% of studies (during the 3 meetings performed by reviewers, the risk of bias was also discussed). The Risk-of-bias visualization tool (ROBVIS) [7] was used to produce the plots that summarize the risk of bias assessment.

Systematic review methods were established before conducting the review by determining the search strategy, article inclusion criteria, quality assessment methods, data extraction methods, and statistical analysis plans. No protocol deviations were made.

The study protocol was published in the PROSPERO database to minimize reporting bias (CRD42022381689).

Study rationale

Potential active substances should be identified in the laboratory by several preclinical studies and their results should be fully described in the introduction of all reports. Consideration should be given to which animal models may be appropriate, whether they provide sufficient information to advance a clinical program, if doses can be extrapolated from these models, and the need to test comedications [8]. FDA recommends that a pragmatic trial should be initiated only after an appropriate preclinical study is performed in specific disease models [9].

Although rationale was declared in most trials, less than 50% reported preclinical studies, in all cases positive. Unfortunately, almost all drugs showing positive effects in animals failed in subsequent human trials, but some of these negative trials might have more properly been evaluated as inconclusive for multiple reasons, including poor quality design, underpowering and excessive quantities of missing data [10].

Choice of controls

The choice of controls in ALS trials cannot disregard the use of riluzole, since this is the gold standard therapy, available since 1994. The use of riluzole implies knowledge of its possible interaction with the tested drug. For this reason, preclinical animal studies should assess the potential conse-quences of possible drug-drug interaction [11] and the safety profile when used in add-on. This could be one reason for negative results in human studies. The risk in the use of riluzole as add-on therapy in ALS is exemplified by the results of the xaliproden study in which the potential efficacy of a new drug may have been masked or even negatively affected when Xaliproden was combined with riluzole [12].

Ideally, the use of placebo arms would be avoided in ALS because of the ethics of placebo use in a terminal illness, but there are significant challenges in such an approach.

In the present review, 5.5% of the trials used historical controls for comparison. The alternative use of an historical control group remains controversial and problematic [13], and is discouraged by FDA, because several controlled trials have demonstrated differences in rates of progression and survival among placebo cohorts [8, 13].

In line with previous observations [14], we support the idea that it is not appropriate to use historical controls for comparisons, due to a large degree of variability in disease course among individual patients, the evolution of standards of patient care (such as the availability of new treatments or changes in the frequency of use for some treatments) during the course of the year and by sites. Historical controls might be of value in the design of future trials as we develop a more comprehensive and reliable characterization of the disease course [9].

The use of matched historical controls could not be fully justified even if they share comparable standards of care and inclusion criteria as the participants. In particular, matching by propensity score allows one to obtain comparable arms. Propensity score is defined as the conditional probability of receiving a treatment given a set of patient-specific covariates [15]. In the presence of a non-randomized historical arm the propensity score is not known, but could be estimated using a known set of covariates. Propensity score matched groups will then be comparable in terms of this set of covariates [15]. This method sometimes is taken into account because it allows one to avoid the use of placebo. However, this is methodology less robust than a parallel-arm and randomized design [16].

Disease heterogeneity and inclusion criteria

The heterogeneity of ALS means, despite all the best stratification strategies, the risk of having groups that are not perfectly comparable. This challenges trialists to categorize patients in homogeneous subgroups based on phenotype characteristics, such as age, clinical type, comorbidities, respiratory function, stage of disease progression, disease management [8], and even genotype [17], all possible predictors of disease course also related to aging [18–20]. Furthermore, the stage of neurodegeneration is not the same in all neurons, and different biological manifestations might be active in different regional groups of motor neurons. Diagnostic delay is another concern, since it correlates with rate of disease progression [21].

Researchers can minimize the effects of this variability with different strategies at different stages of study design: the first is at inclusion level. Selection of the right patients for participation in clinical trials is key to the potential success of the study, and necessitates recruitment of sufficient numbers of participants and the exclusion of those not fulfilling the study aims.

The importance of the definition of eligibility criteria in the design of ALS clinical trials was observed by Torrieri et al. [22]. Researchers could minimize the effect of disease heterogeneity by enrolling for example only people with a given diagnostic delay, or with a defined progression rate (fast progressors), this strategy could markedly reduce trial heterogeneity and boost statistical power [22, 23]). The observation that only specific subsets of patients responded to experimental drugs has highlighted the importance of considering patient subgroups in designing clinical trials [22, 24, 25]. On the other hand, the probability of detecting a meaningful effect size by minimizing heterogeneity should balance with the need to ensure that any finding can be generalized to the real ALS population (external validity).

The other two strategies to account for heterogeneity are to stratify randomization based on predictors or to perform subgroup analysis.

Half the trials we reviewed did not specify in their inclusion criteria any limits for FVC/SVC, diagnostic delay, or disease duration, nor did they state whether site of onset or familiarity were considered for inclusion. The other half used extremely different inclusion criteria. For example, FVC ranged from < 50% to > 80% and disease duration at enrolment ranged from < 2 months to > 60 months, determining a large heterogeneity as regard to progression rate, including participants with different stage of disease and a diverse disease course.

Choosing to enroll patients with different levels of prognostic factors in the same trial helps the generalizability of results and allows for a more rapid enrolment but requires the analysis of data in subgroups to evaluate the treatment effect for the different categories of predictors. This requires a larger number of enrolled participants to allow for an adequate power per group. For example, in studies allowing the inclusion of participants with more than 24 months’ disease duration, the median number of enrolled participants in our review was only 44.5, thus probably not permitting subgroup analyses with adequate power.

To overcome these flaws, several solutions are possible. Based on the supposed mechanism of action of a specific drug and the trial aim, the sponsor should consider the most adequate disease durations as inclusion criteria. Patients at a relatively early disease stage should be included [14]. Given that rarely enough power is reached in post-hoc subgroup analyses, these should be considered only as exploratory analyses used to provide some additional indication that, however, must be confirmed in specific subsequent studies. Subgroups, based on genotypes, should also be considered as recommended by the FDA [9].

Study phase and related aims

As previously described [26], the choice of trial design depends on the stage of development of a given drug and which study objective trialists are interested in. Briefly, the goal of phase II studies is to evaluate the safety, tolerability, dose finding, and the effect of a treatment on a specific disease, using a superiority design. Preliminary efficacy, but not definitive testing for efficacy, could also be tested in phase II trials [9, 23, 27–29].

As described in the recommendations from FDA guidelines, clinical trialists should consider including the evaluation of appropriate biomarkers or biological targets in the design of a phase II study, to demonstrate that an investigational therapy achieves its anticipated function or biological target [9].

We identified 203 phase II studies. Of these, only 79 (38.91%) were conducted with safety, tolerability, dose finding or effect as main objective, in line with study phase requirements.

The results from phase II studies should help trialists to better design subsequent efficacy phase III studies [9]. In this context, out of 26 phase III trials only 11 had at least one previous positive phase II study investigating the same active principle. Methodological characteristics of Phase II trials that proceeded to phase III were not different from those that did not, suggesting that commercial strategies or scientific interest of the single researchers impact this decision more than methodological quality, rationale and previous trial results.

Sample size

Trials must be designed specifying a pre-defined effect size that should be detected with adequate power at a pre-specified level of significance. If the initial hypothesis on the effect size is not well set, the sample size required and defined in the design phase may not be sufficient to detect differences between treatment groups.

A small sample size can be reached in a pilot study to confirm the results coming from preclinical study and to provide preliminary data for a next confirmatory study. When conducting phase III study, a formal sample size calculation must be provided based on previous observation and the expected clinical relevant benefit.

In our review, the planned sample size was not reached in 12.8% of the studies and a sample size calculation was not reported in 59.1%. In phase III studies, 92.3% provided a sample size calculation (power≥80%).

Dropouts

Disease progression, travel difficulties and caregiver burden represent common reasons for high attrition rates [30]. Since participants who drop out are usually unrepresentative of those who are randomized at the beginning of the trial, the credibility of the result is impacted in the presence of missing data [31], especially if this differentially affects study arms, leading to biased estimates. Participants who discontinue the study are those with a higher risk of not reaching the end of the study and at higher risk of experiencing more adverse events, which will affect the results of the per protocol analyses and could undermine the interpretation of the final results.

In our review, a drop-out rate higher than 20% was detected in 30.5% of trials. Also, one trial evaluated per protocol population only and the same study had a dropout rate higher than 20%.

Under the principle of intention-to-treat, in order to have an unbiased and statistically valid analysis, it is necessary to have a complete dataset to be used for the primary analysis. It is therefore necessary to use appropriate methods to manage missing data. The most efficient solution is the ability of the Sponsor to keep missing data as low as possible, acting on more appropriate inclusion criteria and study procedures.

Provided their validity and reproducibility is evaluated, several alternatives may be considered to carry on a patient’s evaluation when they cannot reach the study site with the aim to lower the dropout rate and missing data. These include the use of telemedicine technology, even if in-person evaluation is to be preferred [32–34], home self-administration of the Revised Amyotrophic Lateral Sclerosis Functional Rating Scale (ALSFRS-R) [35–38], remote assessment of FVC and of maximum inspiratory pressure (MIP) through smart phone-based technology with real time transmission of the data [39]. During the COVID-19 pandemic restriction, this strategy was the basis for a new trial approach.

Outcomes

A primary end point should be clearly stated and easily verifiable at the end of the study in order to be able to provide a clinical interpretation of the obtained results. We found a large variety of primary end points in our review: clinical, instrumental parameters and functional scales, with different degrees of accuracy and precision over the decades.

The choice of the most appropriate primary outcome is crucial, given that it is used to calculate the sample size needed for the trial to detect a treatment effect. There is need for outcomes sufficiently sensible to changes induced by treatments, even during the restricted time limits of a clinical trial.

Phase III trials used the ALSFRS-R and survival as the most common endpoints, whereas phase II studies used a larger variety of endpoints, including also muscle parameters, adverse events, other scales and respiratory evaluation. The time of death is strongly influenced by the use of devices and palliative care available to manage disease symptoms. For this reason, the assessment of survival should be combined with an evaluation of the need for full-time (or nearly full-time) respiratory support. Thus, survival is not the best option for early phase trials to be conducted in a short timeframe. Surrogate outcomes are better suited in this context.

Notably, the primary outcome of a trial, is used to calculate the sample size of both phase II and III. In this context, survival is an inefficient primary outcome measure for Phase II trials due to the large sample size and long trial duration needed to detect adequate numbers of events [11]. If patient function is intended to be assessed by the primary outcome, mortality should be integrated as a secondary endpoint [9]. It should be noted however, that the only disease-modifying therapy approved worldwide is Riluzole, and those trials used survival as an endpoint.

The ALSFRS-R can be used instead of mortality, because it strongly predicts survival, is easy and inexpensive to administer, and minimizes dropout rates. However, the ALSFRS-R varies with time and requires moderately large sample sizes with at least 6 months duration of the trial. Combining different outcomes (eg, functioning, muscle strength, lung function, and survival) can considerably increase the efficiency of clinical trials, providing a more accurate measure of drug efficacy than the use of a single end point [11].

Although phase II clinical trials are not designed to determine clinical efficacy [9], they often include several clinical outcome measures to evaluate unexpected or large clinical impact (positive or negative). However, the inclusion of clinical outcome measures in phase II trials creates a paradox: phase II clinical trials do not have enough power to detect clinical effect [9], lacking adequate sample size but a long follow up period is required to identify small effects of the treatment [40].

Furthermore, we found that many phase II studies used more than one measure as a primary endpoint, thus providing conflicting data.

Since phase II studies are still expected to be the prominent type in the future, but they require large cohorts of patients and a long follow up period to identify effects of the treatment [40], new surrogate measures, including biomarkers, should be used in phase II studies to screen and help to select the most promising compounds to be brought into phase III clinical trials.

Huge strides in identifying biomarkers for ALS progression could lead to the optimization of already available outcomes and provide the possibility to better investigate potential therapies for ALS.

Study design

Most of the clinical trials analyzed were carried out using two study models: parallel arms, in which, after randomization, one group of patients receives the drug and the other the placebo for the entire duration of the trial, and a crossover design, in which the same person is exposed at different times, to one or more treatments, according to a random sequence. In this way all participants receive all the tested treatments.

A disease with rapid progression such as ALS, with currently available outcomes that make it difficult to detect clinically relevant changes in the earliest phases of a study, does not lend itself to a crossover study design. The rapid evolution that characterizes neurodegeneration in ALS constitutes a bias in the choice of crossover designs as evaluations made on the same person at two different consecutive times are not comparable in terms of disease progression.

Risk of bias

The reliability of the results of a randomized trial depends on the extent to which potential sources of bias have been avoided. A systematic error may lead to underestimation or overestimation of an effect. The risk-of-bias analysis carried out in this review has revealed that most published studies in ALS have biases that negatively affect the veracity of the results obtained. The most prevalent bias is represented by the presence of missing data caused by the high number of drop-outs in clinical trials on ALS. Thus, the risk of bias is expected to be directly correlated with the final results of a trial.

Publication bias

Publication bias is not only a scientific problem, but also an ethical wrongdoing. People who participate in a clinical trial will expect that the research will add information to the present knowledge. Failing to report research is a waste of time and resources and it undermines the ability to make truly evidence-informed decisions about health care.

In order to give a complete picture of clinical trials in ALS, we performed a publication bias evaluation by consulting additional sources. Only 6/92 trials (6.2%) were completed but not published and 10 were prematurely interrupted without any published report (10.3%). In addition, 33 trials were marked as completed in registers but no publications were available. Thus, the publication bias is present but arguably is not a major problem in ALS.

Final considerations

Our review shows that many substances have been evaluated in ALS, usually with phase II trials, using a large number of outcomes, but without identifying a sufficient signal for progression in most cases. Although a third of the revised trials claimed to be “positive”, only riluzole has been approved in Europe; in North America, riluzole, sodium phenylbutyrate with taurursodiol and edaravone have been approved by the FDA, all with marginal efficacy. This astounding record of failures can be partly attributed to flaws in trial design which can lead to false negative and false positive results.

Major issues to be addressed in the design and conduct of clinical trials in ALS include the requirement for standard animal toxicology and phase I safety studies, the resource-intensive nature of phase II and III studies and the always challenging need to balance homogeneity of included patients and external validity. This and previous reviews of clinical trials for ALS identify a number of issues possibly contributing to their failure. It is important that these are considered in the design and implementation of future trials in this therapeutic area [8].

The primary limitation of the current study was that we included only articles in English, possibly excluding relevant trials in other languages. We also excluded literature such as theses, websites and similar documents that may or may not have been through peer review, or published elsewhere, but our analysis of publication bias makes us quite confident that our review was comprehensive.

This review has some strengths. First, it is the first review that investigated the methodological quality of all clinical trial performed in ALS. Second, most of the activities were performed in double and in blind, allowing authors to provide a robust final evaluation. Third, the assessment of the publication bias was performed using different sources and with the involvement of the contact point of each trial. Future studies should expand upon the promising preliminary results from phase II to Phase III studies by conducting large, multicentre, randomized controlled trials that examine the impact of various treatments over a longer period to assist in elucidating superior regimens and optimal dosage parameters in this vulnerable patient population.

It is crucial to develop alternative clinical trial models or any strategy to make investigators able to move through to precision medicine in such a heterogeneous disease. For example, umbrella trials, adaptative models [41], testing different drugs, possibly with multiple subgroups, and the use of biomarkers both for characterizing phenotypes and to serve as surrogate outcomes may be used. Composite outcomes, such as progression-free survival may also be used to increase the power of the study [42].