Meta-analyses of Adverse Effects Data Derived from Randomised Controlled Trials as Compared to Observational Studies: Methodological Overview

Centre for Reviews and Dissemination, University of York, York, United Kingdom.
PLoS Medicine (Impact Factor: 14.43). 05/2011; 8(5):e1001026. DOI: 10.1371/journal.pmed.1001026
Source: PubMed


There is considerable debate as to the relative merits of using randomised controlled trial (RCT) data as opposed to observational data in systematic reviews of adverse effects. This meta-analysis of meta-analyses aimed to assess the level of agreement or disagreement in the estimates of harm derived from meta-analysis of RCTs as compared to meta-analysis of observational studies.
Searches were carried out in ten databases in addition to reference checking, contacting experts, citation searches, and hand-searching key journals, conference proceedings, and Web sites. Studies were included where a pooled relative measure of an adverse effect (odds ratio or risk ratio) from RCTs could be directly compared, using the ratio of odds ratios, with the pooled estimate for the same adverse effect arising from observational studies. Nineteen studies, yielding 58 meta-analyses, were identified for inclusion. The pooled ratio of odds ratios of RCTs compared to observational studies was estimated to be 1.03 (95% confidence interval 0.93-1.15). There was less discrepancy with larger studies. The symmetric funnel plot suggests that there is no consistent difference between risk estimates from meta-analysis of RCT data and those from meta-analysis of observational studies. In almost all instances, the estimates of harm from meta-analyses of the different study designs had 95% confidence intervals that overlapped (54/58, 93%). In terms of statistical significance, in nearly two-thirds (37/58, 64%), the results agreed (both studies showing a significant increase or significant decrease or both showing no significant difference). In only one meta-analysis about one adverse effect was there opposing statistical significance.
Empirical evidence from this overview indicates that there is no difference on average in the risk estimate of adverse effects of an intervention derived from meta-analyses of RCTs and meta-analyses of observational studies. This suggests that systematic reviews of adverse effects should not be restricted to specific study types. Please see later in the article for the Editors' Summary.


Available from: Su Golder
  • Source
    • "Empirical evidence indicates that flaws in the design, conduct, and analysis of trials can lead to bias and distort their effects. Previous meta-epidemiologic studies have assessed the influence of various study characteristics on their effects, including among others indexing in MEDLINE [1], language [2] [3], design [4] [5], methodological characteristics [6], sample size [7e10], and others with most focus on randomized trials. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Objectives To examine the influence of the following study characteristics on their study effect estimates: (1) indexing in MEDLINE, (2) language, and (3) design. For randomized trials, (4) trial size and (5) unequal randomization were also assessed. Study Design and Setting The CAtegorical Dental and Maxillofacial Outcome Syntheses meta-epidemiologic study was conducted. Eight databases/registers were searched up to September 2012 for meta-analyses of binary outcomes with at least five studies in the field of dental and maxillofacial medicine. The previously mentioned five study characteristics were investigated. The ratio of odds ratios (ROR) according to each characteristic was calculated with random-effects meta-regression and then pooled across meta-analyses. Results A total of 281 meta-analyses were identified and used to assess the influence of the following factors: non-MEDLINE indexing vs. MEDLINE indexing (n = 78; ROR, 1.12; 95% confidence interval [CI]: 1.05, 1.19; P = 0.001), language (n = 61; P = 0.546), design (n = 24; P = 0.576), small trials (<200 patients) vs. large trials (≥200 patients) (n = 80; ROR, 0.92; 95% CI: 0.87, 0.98; P = 0.009) and unequal randomization (n = 36; P = 0.828). Conclusion Studies indexed in MEDLINE might present greater effects than non-indexed ones. Small randomized trials might present greater effects than large ones.
    Journal of clinical epidemiology 09/2014; 67(9). DOI:10.1016/j.jclinepi.2014.04.002 · 3.42 Impact Factor
  • Source
    • "Second, most observational studies are cohort studies and we only included those with adjusted risk estimates controlled for potential confounders such as age, sex, BMI, HbA1C, smoking and so on. A cohort study can provide strong evidence in assessing latent or rare outcomes such as lung cancer incidence [54]. Third, we did a multiple subgroup analysis according to study design, adjusting variables such as smoking and other glucose-lowering drugs. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Background Accumulating evidence suggests that hypoglycaemic agents influence lung cancer risk in patients with diabetes. It remains to be fully elucidated whether conventional hypoglycaemic agents (metformin, sulfonylureas, thiazolidinediones [TZDs] or insulin) affect lung cancer incidence in patients with diabetes. Methods We performed a meta-analysis using EMBASE, MEDLINE and Web of Science to search randomised controlled trials (RCTs), cohort studies, and case-control studies published up to October 2013 that assessed the effects of metformin, sulfonylurea, TZDs or insulin on lung cancer risk in subjects with diabetes. Fixed and random effects meta-analysis models were used, and the effect size was expressed as a summary odds ratio (OR) with 95% confidence intervals (CI). The Grades of Research, Assessment, Development and Evaluation (GRADE) approach was applied to define the quality of the evidence. Results Analysis of 15 studies (11 cohort studies, 2 case-control studies, and 2 RCTs) showed that metformin use was associated with a 15% reduction in risk of lung cancer (OR 0.85, 95% CI 0.77 to 0.92), but this finding was not supported by sub-analysis of smoking-adjusted studies (OR 0.84, 95% CI 0.61 to 1.06). Moreover, sulfonylurea or TZDs use was not associated with increased or decreased lung cancer risk, respectively (OR 1.10, 95% CI 0.93 to 1.26), (OR 0.86, 95% CI 0.70 to 1.02). Higher lung cancer risk was related to insulin (OR 1.23, 95% CI 1.10 to 1.35). However, all data from RCTs failed to demonstrate a statistically significant effect. Conclusions This analysis demonstrated that metformin use may reduce lung cancer risk in patients with diabetes but not in a smoking-adjusted subgroup and that insulin use may be associated with an increased lung cancer risk in subjects with diabetes.
    PLoS ONE 06/2014; 9(6):e99577. DOI:10.1371/journal.pone.0099577 · 3.23 Impact Factor
  • Source
    • "There is a common perception within the scientific community that observational studies overestimate treatment effects and as such, the validity of studies is often widely disputed [1, 2]. However, with the advent of new healthcare practices, such as the use of electronic health records (EHRs), observational studies with novel statistics conducted in much larger study populations will be possible and will allow conclusions to be drawn from observed trends [3]. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Post-marketing observational studies are valuable for establishing the real-world effectiveness of treatment regimens in routine clinical practice as they typically monitor a diverse population of patients over many months. This article reviews recent observational studies of angiotensin receptor blockers (ARBs) for the management of hypertension: the 6-month eprosartan POWER study (n~29,400), the 3-month valsartan translational research programme (n~19,500), the 9-month irbesartan Treat to Target study (n=14,200), the 6-month irbesartan DO-IT survey (n~3300) and the 12-week candesartan CHILI survey programme (n=4600). Reduction in blood pressure with ARBs reported across these studies appears to be comparable for the different agents, although direct comparisons between studies cannot be made owing to different treatment durations and baseline patient demographics. Of these studies, the eprosartan POWER study, 2 of the 7 studies in the valsartan translational research programme, and the candesartan CHILI Triple T study measured total cardiovascular risk, as recommended in the 2013 European Society of Cardiology-European Society of Hypertension guidelines. The POWER study confirmed the value of the Systemic Coronary Risk Evaluation (SCORE) to accurately assess total cardiovascular risk. With the advent of new healthcare practices, such as the use of electronic health records (EHRs), observational studies in larger patient populations will become possible. In the future, algorithms embedded in EHR systems could evolve as decision support tools to inform on patient care.
    The Open Cardiovascular Medicine Journal 04/2014; 8(1):35-42. DOI:10.2174/1874192401408010035
Show more