Meta-analyses of Adverse Effects Data Derived from Randomised Controlled Trials as Compared to Observational Studies: Methodological Overview

Centre for Reviews and Dissemination, University of York, York, United Kingdom.
PLoS Medicine (Impact Factor: 14.43). 05/2011; 8(5):e1001026. DOI: 10.1371/journal.pmed.1001026
Source: PubMed


There is considerable debate as to the relative merits of using randomised controlled trial (RCT) data as opposed to observational data in systematic reviews of adverse effects. This meta-analysis of meta-analyses aimed to assess the level of agreement or disagreement in the estimates of harm derived from meta-analysis of RCTs as compared to meta-analysis of observational studies.
Searches were carried out in ten databases in addition to reference checking, contacting experts, citation searches, and hand-searching key journals, conference proceedings, and Web sites. Studies were included where a pooled relative measure of an adverse effect (odds ratio or risk ratio) from RCTs could be directly compared, using the ratio of odds ratios, with the pooled estimate for the same adverse effect arising from observational studies. Nineteen studies, yielding 58 meta-analyses, were identified for inclusion. The pooled ratio of odds ratios of RCTs compared to observational studies was estimated to be 1.03 (95% confidence interval 0.93-1.15). There was less discrepancy with larger studies. The symmetric funnel plot suggests that there is no consistent difference between risk estimates from meta-analysis of RCT data and those from meta-analysis of observational studies. In almost all instances, the estimates of harm from meta-analyses of the different study designs had 95% confidence intervals that overlapped (54/58, 93%). In terms of statistical significance, in nearly two-thirds (37/58, 64%), the results agreed (both studies showing a significant increase or significant decrease or both showing no significant difference). In only one meta-analysis about one adverse effect was there opposing statistical significance.
Empirical evidence from this overview indicates that there is no difference on average in the risk estimate of adverse effects of an intervention derived from meta-analyses of RCTs and meta-analyses of observational studies. This suggests that systematic reviews of adverse effects should not be restricted to specific study types. Please see later in the article for the Editors' Summary.

Download full-text


Available from: Su Golder
  • Source
    • "8 Much of the evidence for harms comes from anecdotal reports, case series, or survey data. The role of this ''lower-level'' evidence is increasingly acknowledged to be a valid source of information that contributes to assessing the risk profile of medications on theoretical[26,33,34]or empirical grounds787980, but current practices have difficulty in assigning a precise epistemic status to this type of evidence and integrating it with more standard methods of hypothesis testing. A recent methodological review[9]authored by the Drug Information Association Bayesian Scientific Working Group, points to the unique challenges faced by safety evaluation in drug development and surveillance: ''unlike efficacy assessments that are primarily driven by hypotheses , safety assessments involve a wide range of safety measures (such as adverse events, laboratory, etc.), which need to be studied together to make an overall safety conclusion'' ([9], 15). "
    [Show abstract] [Hide abstract]
    ABSTRACT: It is increasingly acknowledged both among epidemiologists and regulators that the assessment of pharmaceutical harm requires specific methodological approaches that cannot simply duplicate those developed for testing efficacy. However, this intuition lacks sound epistemic bases and delivers ad hoc advice. This paper explains why the same methods of scientific inference do not fare equally well for efficacy and safety assessment by tracing them back to their epistemic foundations. To illustrate this, Cartwright’s distinction into clinching and vouching methods is adopted and a series of reasons is provided for preferring the latter to the former: (1) the need to take into account all available knowledge and integrate it with incoming data; (2) the awareness that a latent unknown risk may always change the safety profile of a given drug (precautionary principle); (3) cumulative learning over time; (4) requirement of probabilistic causal assessment to allow decision under uncertainty; (5) impartiality; and (6) limited and local information provided by randomised controlled trials. Subsequently, the clinchers/vouchers distinction is applied to a case study concerning the debated causal association between paracetamol and asthma. This study illustrates the tension between implicit epistemologies adopted in evaluating evidence and causality; furthermore, it also shows that discounting causal evidence may be a result of unacknowledged low priors or lack of valid alternative options. We conclude with a presentation of the changing landscape in pharmacology and the trend towards an increased use of Bayesian tools for assessment of harms.
    Full-text · Article · Dec 2014 · Drug Safety
  • Source
    • "Empirical evidence indicates that flaws in the design, conduct, and analysis of trials can lead to bias and distort their effects. Previous meta-epidemiologic studies have assessed the influence of various study characteristics on their effects, including among others indexing in MEDLINE [1], language [2] [3], design [4] [5], methodological characteristics [6], sample size [7e10], and others with most focus on randomized trials. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Objectives To examine the influence of the following study characteristics on their study effect estimates: (1) indexing in MEDLINE, (2) language, and (3) design. For randomized trials, (4) trial size and (5) unequal randomization were also assessed. Study Design and Setting The CAtegorical Dental and Maxillofacial Outcome Syntheses meta-epidemiologic study was conducted. Eight databases/registers were searched up to September 2012 for meta-analyses of binary outcomes with at least five studies in the field of dental and maxillofacial medicine. The previously mentioned five study characteristics were investigated. The ratio of odds ratios (ROR) according to each characteristic was calculated with random-effects meta-regression and then pooled across meta-analyses. Results A total of 281 meta-analyses were identified and used to assess the influence of the following factors: non-MEDLINE indexing vs. MEDLINE indexing (n = 78; ROR, 1.12; 95% confidence interval [CI]: 1.05, 1.19; P = 0.001), language (n = 61; P = 0.546), design (n = 24; P = 0.576), small trials (<200 patients) vs. large trials (≥200 patients) (n = 80; ROR, 0.92; 95% CI: 0.87, 0.98; P = 0.009) and unequal randomization (n = 36; P = 0.828). Conclusion Studies indexed in MEDLINE might present greater effects than non-indexed ones. Small randomized trials might present greater effects than large ones.
    Full-text · Article · Sep 2014 · Journal of clinical epidemiology
  • Source
    • "Second, most observational studies are cohort studies and we only included those with adjusted risk estimates controlled for potential confounders such as age, sex, BMI, HbA1C, smoking and so on. A cohort study can provide strong evidence in assessing latent or rare outcomes such as lung cancer incidence [54]. Third, we did a multiple subgroup analysis according to study design, adjusting variables such as smoking and other glucose-lowering drugs. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Background Accumulating evidence suggests that hypoglycaemic agents influence lung cancer risk in patients with diabetes. It remains to be fully elucidated whether conventional hypoglycaemic agents (metformin, sulfonylureas, thiazolidinediones [TZDs] or insulin) affect lung cancer incidence in patients with diabetes. Methods We performed a meta-analysis using EMBASE, MEDLINE and Web of Science to search randomised controlled trials (RCTs), cohort studies, and case-control studies published up to October 2013 that assessed the effects of metformin, sulfonylurea, TZDs or insulin on lung cancer risk in subjects with diabetes. Fixed and random effects meta-analysis models were used, and the effect size was expressed as a summary odds ratio (OR) with 95% confidence intervals (CI). The Grades of Research, Assessment, Development and Evaluation (GRADE) approach was applied to define the quality of the evidence. Results Analysis of 15 studies (11 cohort studies, 2 case-control studies, and 2 RCTs) showed that metformin use was associated with a 15% reduction in risk of lung cancer (OR 0.85, 95% CI 0.77 to 0.92), but this finding was not supported by sub-analysis of smoking-adjusted studies (OR 0.84, 95% CI 0.61 to 1.06). Moreover, sulfonylurea or TZDs use was not associated with increased or decreased lung cancer risk, respectively (OR 1.10, 95% CI 0.93 to 1.26), (OR 0.86, 95% CI 0.70 to 1.02). Higher lung cancer risk was related to insulin (OR 1.23, 95% CI 1.10 to 1.35). However, all data from RCTs failed to demonstrate a statistically significant effect. Conclusions This analysis demonstrated that metformin use may reduce lung cancer risk in patients with diabetes but not in a smoking-adjusted subgroup and that insulin use may be associated with an increased lung cancer risk in subjects with diabetes.
    Full-text · Article · Jun 2014 · PLoS ONE
Show more

Similar Publications