Impact of allocation concealment on conclusion drawn from meta-analyses of randomized trials

IT University of Copenhagen, København, Capital Region, Denmark
International Journal of Epidemiology (Impact Factor: 9.2). 09/2007; 36(4). DOI: 10.1093/ije/dym087
Source: OAI

ABSTRACT Background Randomized trials without reported adequate allocation concealment have been shown to overestimate the benefit of experimental interventions. We investigated the robustness of conclusions drawn from meta-analyses to exclusion of such trials. Material Random sample of 38 reviews from The Cochrane Library 2003, issue 2 and 32 other reviews from PubMed accessed in 2002. Eligible reviews presented a binary effect estimate from a meta-analysis of randomized controlled trials as the first statistically significant result that supported a conclusion in favour of one of the interventions. Methods We assessed the methods sections of the trials in each included meta-analysis for adequacy of allocation concealment. We replicated each meta-analysis using the authors' methods but included only trials that had adequate allocation concealment. Conclusions were defined as not supported if our result was not statistically significant. Results Thirty-four of the 70 meta-analyses contained a mixture of trials with unclear or inadequate concealment as well as trials with adequate allocation concealment. Four meta-analyses only contained trials with adequate concealment, and 32, only trials with unclear or inadequate concealment. When only trials with adequate concealment were included, 48 of 70 conclusions (69%; 95% confidence interval: 56–79%) lost support. The loss of support mainly reflected loss of power (the total number of patients was reduced by 49%) but also a shift in the point estimate towards a less beneficial effect. Conclusion Two-thirds of conclusions in favour of one of the interventions were no longer supported if only trials with adequate allocation concealment were included.

Download full-text


Available from: Karsten Juhl Jørgensen, Jun 24, 2014
  • Source
    • "We extracted data on the effect of self-management interventions on health care utilisation and total costs. To assess study quality, we chose a dichotomous measure based on allocation concealment, as this is consistently associated with treatment effect [20, 21]. Allocation concealment was judged as adequate or inadequate according to the Cochrane risk of bias tool. "
    [Show abstract] [Hide abstract]
    ABSTRACT: There is increasing interest in the role of 'self-management' interventions to support the management of long-term conditions in health service settings. Self-management may include patient education, support for decision-making, self-monitoring and psychological and social support. Self-management support has potential to improve the efficiency of health services by reducing other forms of utilisation (such as primary care or hospital use), but a shift to self-management may lead to negative outcomes, such as patients who feel more anxious about their health, are less able to cope, or who receive worse quality of care, all of which may impact on their health and quality of life. We sought to determine which models of self-management support are associated with significant reductions in health services utilisation without compromising outcomes among patients with long-term conditions.
    BMC Health Services Research 08/2014; 14(1):356. DOI:10.1186/1472-6963-14-356 · 1.66 Impact Factor
  • Source
    • "An RCT was assessed to be at low risk of bias based on its performance across two domains – allocation concealment and blinding of outcome assessment. These domains were chosen for bias assessment as the significance of good allocation concealment and outcome assessment blinding in minimising bias and, in particular, overestimation of treatment effect is well supported by empirical evidence [24], [25]. While empirical evidence also exists to support the significance of adequate blinding of participants in reducing exaggeration of estimated treatment effects [25] the inherent difficulty of blinding participants in surgical RCTs necessitated the exclusion of this domain in our assessment. "
    [Show abstract] [Hide abstract]
    ABSTRACT: We investigated the proportion of orthopaedic procedures supported by evidence from randomised controlled trials comparing operative procedures to a non-operative alternative. Orthopaedic procedures conducted in 2009, 2010 and 2011 across three metropolitan teaching hospitals were identified, grouped and ranked according to frequency. Searches of the Cochrane Central Register of Controlled Trials (CENTRAL), the Cochrane Database of Systematic Reviews (CDSR) and the Database of Abstracts of Reviews of Effects (DARE) were performed to identify RCTs evaluating the most commonly performed orthopaedic procedures. Included studies were categorised as "supportive" or "not supportive" of operative treatment. A risk of bias analysis was conducted for included studies using the Cochrane Collaboration's Risk of Bias tool. A total of 9,392 orthopaedic procedures were performed across the index period. 94.6% (8886 procedures) of the total volume, representing the 32 most common operative procedure categories, were used for this analysis. Of the 83 included RCTs, 22.9% (19/83) were classified as supportive of operative intervention. 36.9% (3279/8886) of the total volume of procedures performed were supported by at least one RCT showing surgery to be superior to a non-operative alternative. 19.6% (1743/8886) of the total volume of procedures performed were supported by at least one low risk of bias RCT showing surgery to be superior to a non-operative alternative. The level of RCT support for common orthopaedic procedures compares unfavourably with other fields of medicine.
    PLoS ONE 06/2014; 9(6):e96745. DOI:10.1371/journal.pone.0096745 · 3.23 Impact Factor
  • Source
    • "[43], [44] Similarly, effect sizes from trials that excluded dropouts in the analysis or considered a modified intention to treat (ITT) approach were more likely to show a beneficial effect than trials without exclusions, demonstrating that the ITT principle is important to preserve the benefits of randomization and keep unbiased estimates [45]–[47]. Over-estimates of treatment effects, or bias, at the trial level, can lead to biased or inaccurate results and conclusions in systematic reviews and meta-analyses [40], [41], [48]–[50]. In addition, our analyses showed no agreement between decisions made based on RoB assessments at the level of meta-analysis. "
    [Show abstract] [Hide abstract]
    ABSTRACT: To test the inter-rater reliability of the RoB tool applied to Physical Therapy (PT) trials by comparing ratings from Cochrane review authors with those of blinded external reviewers. Randomized controlled trials (RCTs) in PT were identified by searching the Cochrane Database of Systematic Reviews for meta-analysis of PT interventions. RoB assessments were conducted independently by 2 reviewers blinded to the RoB ratings reported in the Cochrane reviews. Data on RoB assessments from Cochrane reviews and other characteristics of reviews and trials were extracted. Consensus assessments between the two reviewers were then compared with the RoB ratings from the Cochrane reviews. Agreement between Cochrane and blinded external reviewers was assessed using weighted kappa (κ). In total, 109 trials included in 17 Cochrane reviews were assessed. Inter-rater reliability on the overall RoB assessment between Cochrane review authors and blinded external reviewers was poor (κ = 0.02, 95%CI: -0.06, 0.06]). Inter-rater reliability on individual domains of the RoB tool was poor (median κ = 0.19), ranging from κ = -0.04 ("Other bias") to κ = 0.62 ("Sequence generation"). There was also no agreement (κ = -0.29, 95%CI: -0.81, 0.35]) in the overall RoB assessment at the meta-analysis level. Risk of bias assessments of RCTs using the RoB tool are not consistent across different research groups. Poor agreement was not only demonstrated at the trial level but also at the meta-analysis level. Results have implications for decision making since different recommendations can be reached depending on the group analyzing the evidence. Improved guidelines to consistently apply the RoB tool and revisions to the tool for different health areas are needed.
    PLoS ONE 05/2014; 9(5):e96920. DOI:10.1371/journal.pone.0096920 · 3.23 Impact Factor
Show more