Issues in Comparisons Between Meta-analyses and Large Trials

Clinical Research Division, Tufts University, Бостон, Georgia, United States
JAMA The Journal of the American Medical Association (Impact Factor: 35.29). 04/1998; 279(14):1089-93. DOI: 10.1001/jama.279.14.1089
Source: PubMed


The extent of concordance between meta-analyses and large trials on the same topic has been investigated with different protocols. Inconsistent conclusions created confusion regarding the validity of these major tools of clinical evidence.
To evaluate protocols comparing meta-analyses and large trials in order to understand if and why they disagree on the concordance of these 2 clinical research methods.
Systematic comparison of protocol designs, study selection, definitions of agreement, analysis methods, and reported discrepancies between large trials and meta-analyses.
More discrepancies were claimed when large trials were selected from influential journals (which may prefer trials disagreeing with prior evidence) than from already performed meta-analyses (which may target homogeneous trials) and when both primary and secondary (rather than only primary) end points were considered. Depending on how agreement was defined, kappa coefficients varied from 0.22 (low agreement) to 0.72 (excellent agreement). The correlation of treatment effects between large trials and meta-analyses varied from -0.12 to 0.76, but was more similar (0.50-0.76) when only primary end points were considered. When both the magnitude and uncertainty of treatment effects were considered, large trials disagreed with meta-analyses 10% to 23% of the time. Discrepancies were attributed to different disease risks, variable protocols, quality, and publication bias.
Comparisons of large trials with meta-analyses may reach different conclusions depending on how trials and meta-analyses are selected and how end points and agreement are defined. Scrutiny of these 2 major research methods can enhance our appreciation of both for guiding medical practice.

Download full-text


Available from: Joseph C Cappelleri, Jan 20, 2015
  • Source
    • "This review excluded studies conducted exclusively in pure palliative care settings or inpatients, as this was not the focus of this review. It also excluded studies with fewer than 50 patients because of the high risk of publication bias associated with very small studies [13]. The search included publications between January 1, 1990 till December 31, 2012. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Purpose: Increasing interest in the Quality of Life outcomes in cancer patients led to increase implementation of their use in routine clinical practice. The aim of this systemic review is to review the scientific evidence behind recommending the use of quality of life (QoL) scales routinely in outpatient evaluation. Methods: Systematic review for all published randomized controlled trials in English language between January 1, 1990 till December 31, 2012. Out of 487 articles (476 identified by electronic search+11 articles identified by manual search), six trials satisfied the eligibility criteria: (1) the study was a randomized controlled trial (RCT) with randomization of patients or health care providers; (2) the findings of the administered questionnaire or scale (the intervention) were given to health care provider, and compared to standard care with no questionnaire administered (the control); (3) study was conducted in outpatient oncology clinics; and (4) an outcome was measured that related to (i) QoL improvement, (ii) reduction in morbidity, (iii) reduction in stress for the patients, (iv) improvement in communication between patients and health care provider, or (v) improved patient satisfaction. Assessment for the quality of the study was done using the GRADE methodology. Results: Serious methodological issues were affecting most of the trials. Overall the evaluation of the quality of the evidence from these identified trials suggests that there is a weak recommendation to use QoL scales in routine oncology practice to improve communication between physicians and patients. Conclusion: The routine use of such tools in the outpatient settings at improving the patient outcome or satisfaction cannot be recommended based on the available evidence. The potential harm with the excess use of resources needed to implement, collect, store, analyse, and present such data to health care providers should be also considered. Further research and better designed trials is required using recent methodological techniques such as item-response theory based questionnaire and cluster randomization might help to reach an answer to this question.
    Preview · Article · Jun 2013 · Journal of the Egyptian National Cancer Institute
  • Source
    • "These publications have been used to inform clinical practice, to aid teaching, direct health policy, guide future research and to serve as a foundation for practice guidelines.[5] Accordingly, systemic reviews and meta-analyses of high-quality primary studies are the highest level of evidence for issues of prevention and treatment in evidence-based medicine.[6] Indeed, the Oxford Centre for Evidence-Based Medicine ranks meta-analyses as level 1a evidence. "
    [Show abstract] [Hide abstract]
    ABSTRACT: The purpose of this study is to compare the difference of incidence of post-operative complications, operative time, length of stay and recurrence of patients undergoing laparoscopic or open repair of their ventral/incisional hernia a meta-analytic technique for observational studies. A literature search was performed using Medline, PubMed, Embase and Cochrane databases for studies reported between 1998 and 2009 comparing laparoscopic and open surgery for the treatment of ventral (incisional) hernia. This meta-analysis of all the observational studies compared the post-operative complications recurrence rate and length of stay. The random effects model was used. Sensitivity and heterogeneity were analysed. Analysis of 15 observational studies comprising 2452 patients qualified for meta-analysis according to the study's inclusion criteria. Laparoscopic surgery was attempted in 1067 out of 2452. The results showed that the length of stay (odds ratio [OR], - 1.00; 95% confidence interval [CI], - 1.09 to - 0.91; P < 0.00001) and operative time (OR, 59.33; 95% CI, 58.55 to 60.11; P < 0.00001) was significantly lower in the laparoscopic group. The results also showed that there was a significant reduction in the formation of abscesses (OR, 0.38; 95% CI, 0.16 to 0.92; P = 0.03) and wound infections (OR, 0.49; 95% CI, 0.29 to 0.82; P = 0.007) post-operatively. There is a trend which indicates that the recurrence of the hernia using laparoscopic repair versus open repair was overall lower with the laparoscopic repair (OR, 0.48; 95% CI, 0.22 to 1.04; P = 0.06), however, this was not significant. Laparoscopic incisional hernia repair was associated with a reduced length of stay, operative time and lower incidence of abscess and wound infection post-operatively. This study also highlights the benefit of using observational studies as a form of research and its value as a tool in answering questions where large sample sizes of patient groups would be impossible to accumulate in a reasonable length of time.
    Full-text · Article · Oct 2012 · Journal of Minimal Access Surgery
  • Source
    • "Our second hypothesis suggests that trials with larger sample sizes will have higher descriptive validity. In the medical literature, descriptive validity tends to be greater in trials with larger sample sizes (Ioannidis et al. 1998; Kjaergard et al. 2001). One of the reasons behind this argument is that larger trials are generally backed by external funders and can require more rigorous methods of delivery, peerreview processes and the funding of studies allows for careful construction of the randomization design. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Descriptive validity is an important factor in assessing the transparent reporting of randomized controlled trials (RCTs). Measures of validity in crime and justice have reported on this issue but there has been a lack of standardization in comparison to other discipline areas (e.g., healthcare) where tools such as the Consolidated Standards of Reporting Trials (CONSORT) Statement have improved reporting standards. In this study, we evaluate crime and justice trials from five different settings (community, prevention, policing, correctional, and court) and assess the extent to which they transparently report information using the CONSORT Statement as a guide. Overall, the findings suggest that crime and justice studies have low descriptive validity. Reporting was poor on methods of randomization, outcome measures, statistical analyses, and study findings, though much better in regard to reporting of background and participant details. We found little evidence of improvement in reporting over time and no significant relationship between the number of CONSORT items reported and size of the trial sample. In conclusion, we argue that the state of descriptive validity in crime and justice is inadequate, and must change if we are to develop higher-quality studies that can be assessed systematically. We suggest the adoption of a modified CONSORT Statement for crime and justice research. KeywordsCONSORT-Criminal justice-Randomized controlled trials-Descriptive validity
    Full-text · Article · Sep 2010 · Journal of Experimental Criminology
Show more