Article

Issues in comparisons between meta-analyses and large trials

Clinical Research Division, Tufts University, Бостон, Georgia, United States
JAMA The Journal of the American Medical Association (Impact Factor: 30.39). 04/1998; 279(14):1089-93. DOI: 10.1001/jama.279.14.1089
Source: PubMed

ABSTRACT The extent of concordance between meta-analyses and large trials on the same topic has been investigated with different protocols. Inconsistent conclusions created confusion regarding the validity of these major tools of clinical evidence.
To evaluate protocols comparing meta-analyses and large trials in order to understand if and why they disagree on the concordance of these 2 clinical research methods.
Systematic comparison of protocol designs, study selection, definitions of agreement, analysis methods, and reported discrepancies between large trials and meta-analyses.
More discrepancies were claimed when large trials were selected from influential journals (which may prefer trials disagreeing with prior evidence) than from already performed meta-analyses (which may target homogeneous trials) and when both primary and secondary (rather than only primary) end points were considered. Depending on how agreement was defined, kappa coefficients varied from 0.22 (low agreement) to 0.72 (excellent agreement). The correlation of treatment effects between large trials and meta-analyses varied from -0.12 to 0.76, but was more similar (0.50-0.76) when only primary end points were considered. When both the magnitude and uncertainty of treatment effects were considered, large trials disagreed with meta-analyses 10% to 23% of the time. Discrepancies were attributed to different disease risks, variable protocols, quality, and publication bias.
Comparisons of large trials with meta-analyses may reach different conclusions depending on how trials and meta-analyses are selected and how end points and agreement are defined. Scrutiny of these 2 major research methods can enhance our appreciation of both for guiding medical practice.

Download full-text

Full-text

Available from: Joseph C Cappelleri, Jan 20, 2015
0 Followers
 · 
77 Views
  • Source
    • "This review excluded studies conducted exclusively in pure palliative care settings or inpatients, as this was not the focus of this review. It also excluded studies with fewer than 50 patients because of the high risk of publication bias associated with very small studies [13]. The search included publications between January 1, 1990 till December 31, 2012. "
    [Show abstract] [Hide abstract]
    ABSTRACT: PURPOSE: Increasing interest in the Quality of Life outcomes in cancer patients led to increase implementation of their use in routine clinical practice. The aim of this systemic review is to review the scientific evidence behind recommending the use of quality of life (QoL) scales routinely in outpatient evaluation. METHODS: Systematic review for all published randomized controlled trials in English language between January 1, 1990 till December 31, 2012. Out of 487 articles (476 identified by electronic search+11 articles identified by manual search), six trials satisfied the eligibility criteria: (1) the study was a randomized controlled trial (RCT) with randomization of patients or health care providers; (2) the findings of the administered questionnaire or scale (the intervention) were given to health care provider, and compared to standard care with no questionnaire administered (the control); (3) study was conducted in outpatient oncology clinics; and (4) an outcome was measured that related to (i) QoL improvement, (ii) reduction in morbidity, (iii) reduction in stress for the patients, (iv) improvement in communication between patients and health care provider, or (v) improved patient satisfaction. Assessment for the quality of the study was done using the GRADE methodology. RESULTS: Serious methodological issues were affecting most of the trials. Overall the evaluation of the quality of the evidence from these identified trials suggests that there is a weak recommendation to use QoL scales in routine oncology practice to improve communication between physicians and patients. CONCLUSION: The routine use of such tools in the outpatient settings at improving the patient outcome or satisfaction cannot be recommended based on the available evidence. The potential harm with the excess use of resources needed to implement, collect, store, analyse, and present such data to health care providers should be also considered. Further research and better designed trials is required using recent methodological techniques such as item-response theory based questionnaire and cluster randomization might help to reach an answer to this question.
    Journal of the Egyptian National Cancer Institute 06/2013; 25(2):63-70. DOI:10.1016/j.jnci.2013.03.001
  • Source
    • "Our second hypothesis suggests that trials with larger sample sizes will have higher descriptive validity. In the medical literature, descriptive validity tends to be greater in trials with larger sample sizes (Ioannidis et al. 1998; Kjaergard et al. 2001). One of the reasons behind this argument is that larger trials are generally backed by external funders and can require more rigorous methods of delivery, peerreview processes and the funding of studies allows for careful construction of the randomization design. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Descriptive validity is an important factor in assessing the transparent reporting of randomized controlled trials (RCTs). Measures of validity in crime and justice have reported on this issue but there has been a lack of standardization in comparison to other discipline areas (e.g., healthcare) where tools such as the Consolidated Standards of Reporting Trials (CONSORT) Statement have improved reporting standards. In this study, we evaluate crime and justice trials from five different settings (community, prevention, policing, correctional, and court) and assess the extent to which they transparently report information using the CONSORT Statement as a guide. Overall, the findings suggest that crime and justice studies have low descriptive validity. Reporting was poor on methods of randomization, outcome measures, statistical analyses, and study findings, though much better in regard to reporting of background and participant details. We found little evidence of improvement in reporting over time and no significant relationship between the number of CONSORT items reported and size of the trial sample. In conclusion, we argue that the state of descriptive validity in crime and justice is inadequate, and must change if we are to develop higher-quality studies that can be assessed systematically. We suggest the adoption of a modified CONSORT Statement for crime and justice research. KeywordsCONSORT-Criminal justice-Randomized controlled trials-Descriptive validity
    Journal of Experimental Criminology 09/2010; 6(3):245-262. DOI:10.1007/s11292-010-9099-z · 1.17 Impact Factor
  • Source
    • "Our second hypothesis suggests that trials with larger sample sizes will have higher descriptive validity. In the medical literature, descriptive validity tends to be greater in trials with larger sample sizes (Ioannidis et al. 1998; Kjaergard et al. 2001). One of the reasons behind this argument is that larger trials are generally backed by external funders and can require more rigorous methods of delivery, peerreview processes and the funding of studies allows for careful construction of the randomization design. "
    [Show abstract] [Hide abstract]
    ABSTRACT: While the concept of validity has been around for a number of decades, descriptive validity has been considered to a lesser extent in criminal justice research. In this chapter, we illustrate and evaluate several examples of descriptive validity in randomized controlled trials conducted in criminal justice. In particular, we introduce the idea of the CONSORT Statement, which was originally developed in healthcare to assess the descriptive validity of trials. This Statement was developed originally over concerns about the appropriateness of synthesising trials together in meta-analyses with differing levels of descriptive validity. We then go on to demonstrate the use of descriptive validity, using two recent studies. We examine examples of descriptive validity which report poorly in criminal justice trials and provide examples of good practice.Throughout this chapter, we emphasize the importance of descriptive validity to criminal justice researchers in upholding the quality of research within the field. A comparison and discussion about the applicability of the CONSORT Statement with criminal justice and healthcare trials suggests that some amendments may need to be made to make relevance to trials conducted in criminal justice. Generally adoption of a CONSORT-like Statement in criminal justice is thought to improve the reporting of descriptive validity in future research.
    12/2009: pages 333-352;
Show more