Conducting an investigator-initiated randomized double-blinded intervention trial in acute decompensation of inborn errors of metabolism: Lessons from the N-Carbamylglutamate Consortium

1Introduction

Inborn errors of metabolism (IEMs) are rare genetic disorders caused by defects in enzymes, transporters, or other functional proteins, resulting in blocks in a biochemical pathway. The pathophysiology of these disorders typically derives from the accumulation of toxic substrates upstream of the block or deficiency of essential products beyond the block. Although the collective incidence of all IEMs is estimated to be as high as 1 in 800 births [1] virtually all individual IEMs are rare diseases, defined in the US Code of Federal Regulations (CFR) as affecting fewer than 200,000 people (approximately 1 in 1600 Americans) [2] or in Europe, fewer than 1 in 2,000 [3]. Although prevalence varies by population, the frequency of most IEMs is so low that they meet the definition of ‘ultra-rare’, a concept which has been proposed for diseases with a prevalence of less than 1:50,000 [4]. Ultra-rare disorders present unique challenges for the conduct of randomized clinical trials.

Propionic acidemia (PA), methylmalonic acidemia (MMA), ornithine transcarbamylase deficiency (OTCD), and carbamyl phosphate synthetase 1 deficiency (CPSD) are among many ultra-rare disorders associated with elevated blood ammonia levels, or hyperammonemia (HA) [5], which may result in brain edema causing encephalopathy during acute events and if untreated, could cause death. Even with current therapies for acute hyperammonemia, long-term sequelae often include developmental delays leading to intellectual and developmental disability [5].

In mammals, including humans, toxic ammonia is converted into urea via the urea cycle, a biochemical pathway of which carbamyl phosphate synthetase 1 (CPS1) catalyzes the first and rate-limiting step and ornithine transcarbamylase (OTC) catalyzes the second step [6]. For enzymatic activity, CPS1 requires an allosteric activator, N-acetylglutamate (NAG), which is produced by N-acetylglutamate synthetase (NAGS). In the absence of NAG, CPS1 is catalytically inactive. Thus, a deficiency of the NAGS, CPS1 or OTC enzyme results in reduced nitrogen flux through the urea cycle, causing HA [6]. In PA and MMA, the accumulated toxic substrates competitively inhibit the NAGS enzyme [7], thus also causing HA.

N-carbamylglutamate (NCG), a stable analogue of N-acetylglutamate, is currently approved in both the United States (US) and Europe for the treatment of NAGS deficiency. There have been retrospective reports [8–11], pre-clinical experiments [7, 12], and translational studies [13], that suggest that NCG may be effective in reducing ammonia levels during acute decompensation in PA, MMA, CPS1D, and OTCD. However, none of these reports were based on randomized, controlled trials.

Six years ago, we initiated two studies to evaluate whether the addition of NCG to standard therapy during acute HA 1) would result in improved neurocognitive outcomes; and 2) would more rapidly reduce ammonia levels, improve function and shorten hospitalization relative to standard therapy alone. The first study randomly assigned participants (all events per participant) to NCG or placebo; in the second study, each separate HA event experienced by participants is randomized to NCG or placebo. Both studies were investigator-initiated trials (IND 68,185), with no input from industry in their design or conduct and were developed to primarily answer scientific questions. The study sites included nine pediatric academic institutions in the US (Fig. 1), referred to collectively as the N-carbamylglutamate Consortium (NCGC).

Fig.1

The N-Carbamylglutamate Consortium. Date each site joined shown in parentheses.

These studies were primarily funded through an R01 award from the NIH National Institute of Child Health and Human Development (HD058567), and cost-shared with philanthropic funds. Orphan Europe, the only pharmaceutical distributor of NCG (Carbaglu) provided drug and placebo free-of-charge. Once NIH funding ended, Orphan Europe committed to funding the study to completion.

Herein, we summarize the design, methods, experiences, challenges and lessons from the NCGC-conducted trials, one of which is still in progress.

2Methods

Two randomized double-blind, placebo-controlled studies with NCG were designed and launched in 2012.

The first, “long-term neurocognitive outcome study” or LTO (Clinicaltrials.gov registry number: NCT01597440), focused on enrolling newly-diagnosed infants with PA and MMA, presenting with HA at or less than 4 weeks of age. Enrollees were randomly assigned to receive NCG or placebo, in addition to standard therapy, during all their hospitalizations for episodes of HA. Follow-up assessments were scheduled for 9, 15, 21, and 30 months of age. Each of these assessments included the Bayley Scales of Infant Development-Third Edition (all ages), Vineland Adaptive Behavior Scales (all ages), and the MacArthur Inventory of Communicative Development, Second edition (all ages), and the Brief Infant/Toddler Social Emotional Assessment at ages 15, 21, and 30 months.

The second, “short-term outcome” (STO) study (NCT01599286) focused on enrolling persons of all ages with PA, MMA, CPSD or OTCD not eligible for the LTO and randomizing them to NCG vs placebo added to standard care at hospitalizations for each separate HA event (Fig. 2). The primary endpoint of the STO study was the difference in average trajectory of ammonia decline during events receiving NCG vs. placebo. The secondary endpoints were based on change in clinical status during the hospital course and time to hospital discharge (Fig. 2). The Functional Status Scale (FSS) [14], ascertained pre-NCG and once daily thereafter, rated the patient’s functional status in 6 domains: mental status, sensory, communication, motor, feeding, and respiratory functions. Each domain was scored from normal (1) to very severe dysfunction (5) for potential score range from 6 to 30.

Fig.2

Schema of the Short-Term Outcome trial.

Drug/placebo assignments were block-randomized within study site but also controlled allocation balance within each participant, so that participants would receive approximately an equal number of assignments to either NCG or placebo over the course of multiple hospitalizations. This somewhat complex randomization allowed both balancing allocations to treatment within the site as a whole and balancing participant contributions to each treatment arm and resulting in multiple crossovers in patients with multiple episodes of hyperammonemia. A defined wash-out period prevented otherwise eligible participants from being randomized during any repeat hospitalizations falling within 7 days of the last dose of a prior episode.

For both studies, the prescribed dose was 150 mg/kg/d or 3.3 g/m²/d if >15 kg, which was continued for the earlier of 7 days or discharge. Per protocol, plasma ammonia levels were measured prior to NCG administration, and at least every 12 hours during the first 48 hours and daily thereafter. In addition, clinically-ordered ad hoc ammonia measurements were also captured, as well as any ad hoc plasma amino acid profiles. Safety investigations included a complete blood count, serum liver enzymes, and creatinine, and were obtained pre-NCG dosing and 2–4 days after the start of NCG. Electrocardiograms (ECG) were added to the safety investigations following a recommendation by the FDA 3 years after the studies had been opened for enrollment.

Children’s National was the lead site and coordinating center for the studies. Seven inaugural sites in the N-carbamylglutamate Consortium (NCGC) were chosen mostly based on their ongoing collaboration as members of the Urea Cycle Disorders Consortium (UCDC) [15]. These included Boston Children’s Hospital, Rainbow Babies & Children’s Hospital in Cleveland, Children’s Hospital Colorado, University of California Los Angeles Medical Center, Children’s Hospital of Philadelphia, Lucille Packard’s Children’s Hospital Stanford, and Children’s National Health System in Washington, DC. This approach ensured that sites had the requisite expertise, experience, patient base and established working relationship with the coordinating center at Children’s National. During study design, each site provided information based on a review of records from the recent past, of numbers of putatively eligible new patients and currently active patients as well as the expected frequency of admissions with HA of the four study disorders. Based on these data, each participant was expected to have an average of 4 HA admissions over the study period, which was used to estimate the study sample size for both trials. Two additional sites were later added to increase enrollment: Mount Sinai Health System in New York, and Children’s Hospital of Pittsburgh. Monthly calls with the principal investigators and staff at each site provided an opportunity to discuss updates, changes to the protocol and challenges in the conduct of the studies. Additionally, internal meetings of the study leadership and members of the coordinating center to discuss study progress were scheduled initially weekly, and then twice monthly, once the lead site was open for enrollment.

These trials have made use of the Children’s Hospital Research Information System (CHRIS) [16]. CHRIS supported screening, enrollment, random allocation of episodes to treatment arms, protocol management and form-based data collection. Challenges encountered with this platform forced a switch to REDCap (Research Electronic Data Capture) [17] within the first year of opening the study to enrollment.

All sites received IRB approval from their local IRB. The initial protocol submission as well as all modifications were first approved at the lead site, followed by approval at all other sites. A Data and Safety Monitoring Board (DSMB) was established that included a neurologist chair, an ethicist, two neonatologists, and a biostatistician. Per protocol, meetings of the DSMB were scheduled every six months to review participant safety as well as study progress and data completeness. Also per protocol, an interim analysis was scheduled at the projected halfway point of the study (72 episodes in the STO).

Sample and effect sizes were estimated in PASS (Hintze, J. PASS 11. NCSS, LLC. Kaysville, Utah, USA. www.ncss.com, 2011). Sample size projections were based on preliminary site reports, 2-tailed testing, 80% power, and assuming a type 1 error of 5% comparing timewise differences in the trajectory of neuropsychological test scores in the LTO trial and in ammonia levels in the STO trial. For the LTO study, site reports indicated the ability to collectively recruit and retain at least 34 PA and MMA participants. Evaluating Bayley scores assessed at 2-3 times per participant (assumed to be correlated at r = 0.8), we estimated that randomly assigning 17 participants to each arm would allow us to detect an effect size difference based on differences in the score trajectories between groups of at least 0.9 standard deviations (SD). This minimum detectable effect size was considered realistic given the severity of the projected insult in young participants and thus the potential for protection afforded by NCG. For the STO study, we considered the projected 4-episode experience within participant with random assignment to NCG+standard treatment (NCG-STD) vs. placebo+standard treatment (PLBO-STD) per episode to be most closely equivalent to a crossover design and thus developed our estimates based on this design and assumed a linear slope for the decline in ammonia. Accounting for gains in efficiency associated with an average of 4 episodes per enrollee, a sample size of 36 participants (144 episodes, 72 per group) was estimated to provide 80% power to detect a difference in ammonia level trajectories between groups of at least 0.6 SD. Based on recent experience at participating sites and on preliminary results regarding the magnitude of the NCG effect, the sample size was considered feasible and the projected effect was considered attainable if NCG proved to be effective. These numbers were also considered adequate to assess secondary outcomes of safety and functional improvement leading to earlier discharge. Sample and effect size estimates for the STO trial were also adjusted to include error spending to accommodate a single interim analysis targeted for the halfway point of that study (72 randomized episodes).

3Results

From December, 2012 through April, 2015, we enrolled only 1 participant into the Long-Term Outcome study. The study was therefore closed in May 2015, as it was determined that it was unlikely to recruit sufficient participants to reach enrollment targets and produce a meaningful conclusion.

In the Short-Term Outcome study, among all sites, median time (range) from initial submission to IRB approval was 68 days (range 25–164); from non-personnel amendment submission to approval was 27 days (range 11.8–39.1) and from submission of continuing review to approval was 24.5 days (range 16–54) (Table 1).

A timeline of the STO study is shown in Fig. 3. IRB protocol applications at all other sites began only after approval was obtained at the lead site. One site (Site 4) obtained IRB approval, but due to insufficient screening and enrollment efforts, was first placed on probation and after lack of progress, subsequently discontinued from the study. Reactivation was considered several years later, along with a change in site-PI, and the site was ultimately reactivated approximately 5 years into the study.

Fig.3

Study time line of the Short-Term Outcome trial. Initial IRB approval, study initiation visit, first enrollment and first episode is shown for each site. Site 7 was initially discontinued from the study for lack of progress, but subsequently re-activated.

As of July, 2018, 61 out of 123 patients who met full eligibility criteria were enrolled. 22 (36%) of the 61 participants have dropped from the study, primarily due to liver transplantation. Figure 4 shows cumulative enrollment of participants as well as randomized HA hospitalization episodes. Although actual enrollment has exceeded projections, the active number of participants has plateaued since 2015 due to patient dropout. Accrual of study episodes has been relatively linear, with a slight increase during winter months.

Fig.4

Cumulative Enrollment, Episodes and Dropouts of the Short-Term Outcome trial.

Time delay from initial qualifying ammonia level drawing until drug administration is shown in Fig. 5. Median time from ammonia level until randomization was 2.4 hours (range 0 –22.6). Median time from randomization to drug administration was 3 hours (range: 0.2 –31.2) (Fig. 5).

Fig.5

Time between initial ammonia draw, randomization and first administration of study drug for all episodes.

Plots of overall ammonia trajectory revealed that the decline was not linear. Instead, rates of ammonia decline varied at least by day, tending to plateau by day 3 following randomization, creating the need for piecewise regression models, which may vary by treatment group, to evaluate the hypothesized difference in the ammonia decline by treatment group. This unanticipated level of complexity made us reconsider the viability of the primary outcome.

The high variability in ammonia decline by day prompted us to choose an alternative composite primary outcome and modeling strategy., The composite outcome was time to reach an ammonia level of 50μmol/L or time to discharge, whichever occurred first and therefore the analysis approach used was a Cox regression model, which was simpler and required fewer assumptions. This change unburdens the primary outcome of serious modeling complexity while using a well-defined and more clinically-relevant outcome for comparison of treatment effects. The choice of 50μmol/L as the target for ammonia decline was based on the extensive experience of metabolic physicians that a patient reaching this level of ammonia is no longer at risk of acute hyperammonemic injury and would be a candidate for discharge in the absence of other complications. The original primary outcome of difference in ammonia trajectory was retained as a secondary outcome.

This change required re-evaluation of the adequacy of the study sample size to ensure at least 80% power to test the hazard ratio (HR) reflecting the difference by treatment group. Using the Cox Regression procedure in PASS we found that total sample sizes of 146, 114, and 93 episodes were sufficient to detect HR estimates of 1.6, 1.7, 1.8 and 1.9, respectively. The change allowed us to detect realistic and clinically meaningful differences at sample sizes smaller than our original projection resulting in a new target of 114 episodes after accounting for error spending associated with one interim analysis.

4Discussion

Studies of ultra-rare disorders face more challenges than those that merely meet the prevalence criteria for ‘rare disease’. The ultra-rare frequency of urea cycle disorders and organic acidemias impacted every aspect of the study, from recruitment, to regulatory compliance and protocol adherence, and importantly to the rate of accrual of eligible patient episodes.

In a hypothetical city with a population of 1 million, there would be approximately 650 affected with a disorder meeting the minimum CFR prevalence definition of ‘rare’. In contrast, there would only be 20 or fewer affected with a disorder meeting the proposed definition of ‘ultra-rare’ (1:50,000). The prevalence for PA, MMA, CPS1, and OTCD each meet this criterion, as their estimated birth incidences are all less than 1:100,000 live births [18–21], and many die in childhood. Additional eligibility criteria further reduce the number of prospective participants, not all of whom are willing to participate in clinical research. Each large academic institution joining the study would be expected to only add a small number of eligible participants. As a result, in our study, in 2014 when cumulative episodes continued to lag behind projections (Fig. 4), there was no choice but to add additional sites to the study in the hope of increasing enrollment and episodes. Recruitment may in fact be so challenging in the ultra-rare environment that in some studies, the number of sites exceeds the target number of enrolled participants (e.g., NCT02991144).

The extremely low incidence of ultra-rare disorders can add significant variability resulting in the study unfortunately hitting a recruitment ‘drought’. In the 3 years during which the LTO study was open, only 1 infant meeting eligibility criteria was born at all study sites. This was far fewer than historical numbers had predicted, and far fewer than have been born following closure of the LTO study. In the STO protocol, between the time that sites reported their potential subject numbers and the time that regulatory approval allowed recruitment, many of the most severely affected patients (and thus most likely to have eligible episodes of HA) either died or underwent liver transplantation. This loss of potential subjects, along with the natural variability of birth of additional eligible patients, combined to greatly slow accrual of episodes. Although neither could be considered unexpected, the time lag between conceptualization and planning of the study, and the beginning of enrollment, created a significant barrier to accrual.

Regulatory requirements in rare and more common disorders are similar, thereby creating a much higher ratio of administrative “overhead” effort-per-enrollee in rare disorders, whereas studies of common disorders benefit from economies of scale. It was also difficult to benefit from prior experience with study procedures. Hyperammonemia episodes happened so infrequently that study personnel would essentially have to re-learn the protocol details with every new episode.