Song F, Loke YK, Walsh T, Glenny AM, Eastwood AJ, Altman DG. Methodological problems in the use of indirect comparisons for evaluating healthcare interventions: survey of published systematic reviews. BMJ 338: b1147

Faculty of Health, University of East Anglia, Norwich NR4 7TJ.
BMJ (online) (Impact Factor: 17.45). 02/2009; 338(apr03 1):b1147. DOI: 10.1136/bmj.b1147
Source: PubMed


To investigate basic assumptions and other methodological problems in the application of indirect comparison in systematic reviews of competing healthcare interventions.
Survey of published systematic reviews. Inclusion criteria Systematic reviews published between 2000 and 2007 in which an indirect approach had been explicitly used.
Identified reviews were assessed for comprehensiveness of the literature search, method for indirect comparison, and whether assumptions about similarity and consistency were explicitly mentioned.
The survey included 88 review reports. In 13 reviews, indirect comparison was informal. Results from different trials were naively compared without using a common control in six reviews. Adjusted indirect comparison was usually done using classic frequentist methods (n=49) or more complex methods (n=18). The key assumption of trial similarity was explicitly mentioned in only 40 of the 88 reviews. The consistency assumption was not explicit in most cases where direct and indirect evidence were compared or combined (18/30). Evidence from head to head comparison trials was not systematically searched for or not included in nine cases.
Identified methodological problems were an unclear understanding of underlying assumptions, inappropriate search and selection of relevant trials, use of inappropriate or flawed methods, lack of objective and validated methods to assess or improve trial similarity, and inadequate comparison or inappropriate combination of direct and indirect evidence. Adequate understanding of basic assumptions underlying indirect and mixed treatment comparison is crucial to resolve these methodological problems. APPENDIX 1: PubMed search strategy. APPENDIX 2: Characteristics of identified reports. APPENDIX 3: Identified studies. References of included studies.

Download full-text


Available from: Fujian Song,
  • Source
    • "The analysis is reported according to the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) [11] and supplied with an analysis of consistency between indirect and direct evidence [12]. The first version of a protocol for the present study was performed on October 12, 2010 and was based on our previous meta-analysis [1]. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Background Despite significant cost differences, the comparative effect of combination treatments of disease modifying anti-rheumatic drugs (DMARDs) with and without biologic agents has rarely been examined. Thus we performed a network meta-analysis on the effect of combination therapies on progression of radiographic joint erosions in patients with rheumatoid arthritis (RA). Methods and Findings The following combination drug therapies compared versus single DMARD were investigated: Double DMARD: 2 DMARDs (methotrexate, sulfasalazine, leflunomide, injectable gold, cyclosporine, chloroquine, azathioprin, penicillamin) or 1 DMARD plus low dose glucocorticoid (LDGC); triple DMARD: 3 DMARDs or 2 DMARDs plus LDGC; biologic combination: 1 DMARD plus biologic agent (tumor necrosis factor α inhibitor (TNFi) or abatacept or tocilizumab or CD20 inhibitor (CD20i)). Randomized controlled trials were identified in a search of electronic archives of biomedical literature and included in a star-shaped network meta-analysis and reported according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement protocol. Effects are reported as standardized mean differences (SMD). The effects of data from 39 trials published in the period 1989–2012 were as follows: Double DMARD: −0.32 SMD (CI: −0.42, −0.22); triple DMARD: −0.46 SMD (CI: −0.60, −0.31); 1 DMARD plus TNFi: −0.30 SMD (CI: −0.36, −0.25); 1 DMARD plus abatacept: −0.20 SMD (CI: −0.33, −0.07); 1 DMARD plus tocilizumab: −0.34 SMD (CI: −0.48, −0.20); 1 DMARD plus CD20i: −0.32 SMD (CI: −0.40, −0.24). The indirect comparisons showed similar effects between combination treatments apart from triple DMARD being significantly better than abatacept plus methotrexate (−0.26 SMD (CI: −0.45, −0.07)) and TNFi plus methotrexate (−0.16 SMD (CI: −0.31, −0.01)). Conclusion Combination treatment of a biologic agent with 1 DMARD is not superior to 2–3 DMARDs including or excluding LDGC in preventing structural joint damage. Future randomized studies of biologic agents should be compared versus a combination of DMARDs.
    PLoS ONE 09/2014; 9(9-9):e106408-. DOI:10.1371/journal.pone.0106408 · 3.23 Impact Factor
  • Source
    • "We included studies, in full text or abstract form, which assessed one or more aspects of the quality of reporting of network meta-analyses or indirect treatment comparisons. Relevant aspects of reporting included (but were not limited to) the following elements: completeness of literature search description; completeness and adequacy of reporting of statistical methods used for analysis; statement and description of assumptions made for network meta-analysis (common terminology including the words similarity, homogeneity, consistency, and transitivity [15]); adequacy of describing evidence included in the network of treatments; and completeness and adequacy of reporting of results from analysis (including summary estimates and related uncertainty, treatment rankings, probability-oriented parameters of interest, and strategies for presenting complete findings). "
    [Show abstract] [Hide abstract]
    ABSTRACT: Some have suggested the quality of reporting of network meta-analyses (a technique used to synthesize information to compare multiple interventions) is sub-optimal. We sought to review information addressing this claim. To conduct an overview of existing evaluations of quality of reporting in network meta-analyses and indirect treatment comparisons, and to compile a list of topics which may require detailed reporting guidance to enhance future reporting quality. An electronic search of Medline and the Cochrane Registry of methodologic studies (January 2004-August 2013) was performed by an information specialist. Studies describing findings from quality of reporting assessments were sought. Screening of abstracts and full texts was performed by two team members. Descriptors related to all aspects of reporting a network meta-analysis were summarized. We included eight reports exploring the quality of reporting of network meta-analyses. From past reviews, authors found several aspects of network meta-analyses were inadequately reported, including primary information about literature searching, study selection, and risk of bias evaluations; statement of the underlying assumptions for network meta-analysis, as well as efforts to verify their validity; details of statistical models used for analyses (including information for both Bayesian and Frequentist approaches); completeness of reporting of findings; and approaches for summarizing probability measures as additional important considerations. While few studies were identified, several deficiencies in the current reporting of network meta-analyses were observed. These findings reinforce the need to develop reporting guidance for network meta-analyses. Findings from this review will be used to guide next steps in the development of reporting guidance for network meta-analysis in the format of an extension of the PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analysis) Statement.
    PLoS ONE 03/2014; 9(3):e92508. DOI:10.1371/journal.pone.0092508 · 3.23 Impact Factor
  • Source
    • "An important assumption with NMAs is that the studies used are sufficiently similar in terms of relative treatment effect modifiers [7] - that is, study-level factors that may influence the size of the treatment effect seen with a particular pair-wise intervention. These include patient characteristics, outcomes measured, and study design. "
    [Show abstract] [Hide abstract]
    ABSTRACT: To date no network meta-analysis (NMA) has accounted for baseline variations in viral load when assessing the relative efficacy of interventions for chronic hepatitis B (CHB). We undertook baseline-adjusted and unadjusted analyses using the same data to explore the impact of baseline viral load (BVL) on CHB treatment response. We searched Embase, Medline, Medline in Process and the Cochrane CENTRAL databases for randomised clinical trials (RCTs) of monotherapy interventions at licensed doses for use in CHB. Search strategies comprised CHB disease and drug terms (a combination of controlled vocabulary and free text terms) and also a bespoke RCT filter.The NMA was undertaken in WinBUGs using fixed and random effects methods, using data obtained from a systematic review. Individual patient data (IPD) from an entecavir clinical trial were used to quantify the impact of different baseline characteristics (in particular undetectable viral load (UVL) at 1 year) on relative treatment effect. Study level mean baseline values from all identified studies were used. Results were generated for UVL and presented as relative risks (RRs) and 95% credible intervals (CrIs) using entecavir as reference treatment. Overall, for all eight relevant interventions we identified 3,000 abstracts. Following full text review a total of 35 (including the contents of six clinical study reports) met the inclusion critera; 19 were in hepatitis B e antigen (HBeAg)-positive patients and 14 of the 19 contained outcome information of relevance to the NMA.Entecavir and tenofovir studies had heterogeneous patient populations in terms of BVL (mean values 9.29 and 8.65 log10 copies/ml respectively). After adjusting UVL for BVL using an informative prior based on the IPD analysis, the difference between entecavir and tenofovir was not statistically significant (RR 1.27, 95% CrI 0.96 to 1.47 - fixed effects). A similar conclusion was found in all sensitivity analyses. Adjusted tenofovir results were more consistent with observed clinical trial response rates. This study demonstrates the importance of adjusting for BVL when assessing the relative efficacy of CHB interventions in achieving UVL. This has implications for both clinical and economic decision making.
    Systematic Reviews 03/2014; 3(1):21. DOI:10.1186/2046-4053-3-21
Show more