Discrepancies in sample size calculations and data analyses reported in randomized trials: Comparison of publications with protocols

Mayo Clinic, Rochester, USA.
BMJ (online) (Impact Factor: 17.45). 02/2008; 337(dec04 1):a2299. DOI: 10.1136/bmj.a2299
Source: PubMed


To evaluate how often sample size calculations and methods of statistical analysis are pre-specified or changed in randomised trials.
Retrospective cohort study. Data source Protocols and journal publications of published randomised parallel group trials initially approved in 1994-5 by the scientific-ethics committees for Copenhagen and Frederiksberg, Denmark (n=70).
Proportion of protocols and publications that did not provide key information about sample size calculations and statistical methods; proportion of trials with discrepancies between information presented in the protocol and the publication.
Only 11/62 trials described existing sample size calculations fully and consistently in both the protocol and the publication. The method of handling protocol deviations was described in 37 protocols and 43 publications. The method of handling missing data was described in 16 protocols and 49 publications. 39/49 protocols and 42/43 publications reported the statistical test used to analyse primary outcome measures. Unacknowledged discrepancies between protocols and publications were found for sample size calculations (18/34 trials), methods of handling protocol deviations (19/43) and missing data (39/49), primary outcome analyses (25/42), subgroup analyses (25/25), and adjusted analyses (23/28). Interim analyses were described in 13 protocols but mentioned in only five corresponding publications.
When reported in publications, sample size calculations and statistical methods were often explicitly discrepant with the protocol or not pre-specified. Such amendments were rarely acknowledged in the trial publication. The reliability of trial reports cannot be assessed without having access to the full protocols.

Download full-text


Available from: Karsten Juhl Jørgensen
  • Source
    • "There is published research addressing the accuracy and quality of sample size calculations and their reporting in clinical trials [22,26,27]. These papers reported several discrepancies between protocols and reports [26] but also inadequate reporting and inaccuracies in general [27]. "
    [Show abstract] [Hide abstract]
    ABSTRACT: At the design stage of a clinical trial, several assumptions have to be made. These usually include guesses about parameters that are not of direct interest but must be accounted for in the analysis of the treatment effect and also in the sample size calculation (nuisance parameters, e.g. the standard deviation or the control group event rate). We conducted a systematic review to investigate the impact of misspecification of nuisance parameters in pediatric randomized controlled trials conducted in intensive care units. We searched MEDLINE through PubMed. We included all publications concerning two-arm RCTs where efficacy assessment was the main objective. We included trials with pharmacological interventions. Only trials with a dichotomous or a continuous outcome were included. This led to the inclusion of 70 articles describing 71 trials. In 49 trial reports a sample size calculation was reported. Relative misspecification could be calculated for 28 trials, 22 with a dichotomous and 6 with a continuous primary outcome. The median [inter-quartile range (IQR)] overestimation was 6.9 [-12.1, 57.8] % for the control group event rate in trials with dichotomous outcomes and -1.5 [-15.3, 5.1] % for the standard deviation in trials with continuous outcomes. Our results show that there is room for improvement in the clear reporting of sample size calculations in pediatric clinical trials conducted in ICUs. Researchers should be aware of the importance of nuisance parameters in study design and in the interpretation of the results.
    Full-text · Article · Jul 2014 · Trials
  • Source
    • "Unfortunately, a high proportion of trial protocols do not adequately describe important methodological details, decreasing their utility for trial implementation and critical appraisal of trials. For example, protocols often fail to designate primary outcomes [11] or detail allocation concealment [12], sample size calculations [13] and sponsor and investigator roles in trial conduct [14], all of which have been associated with biased trial results and conclusions. Additionally, comparisons of trial protocols with corresponding journal publications have consistently shown important, unacknowledged discrepancies, including discrepancies in primary outcomes [5] and statistical methods [13,15]. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Background All randomized clinical trials (RCTs) require a protocol; however, numerous studies have highlighted protocol deficiencies. Reporting guidelines may improve the content of research reports and, if developed using robust methods, may increase the utility of reports to stakeholders. The objective of this study was to systematically identify and review RCT protocol guidelines, to assess their characteristics and methods of development, and to compare recommendations. Methods We conducted a systematic review of indexed literature (MEDLINE, EMBASE and the Cochrane Methodology Register from inception to September 2010; reference lists; related article features; forward citation searching) and a targeted search of supplementary sources, including a survey of major trial funding agencies in six countries. Records were eligible if they described a content guideline in English or French relevant to RCT protocols. Guidelines were excluded if they specified content for protocols for trials of specific procedures or conditions or were intended to assess trial quality. We extracted guideline characteristics and methods. Content was mapped for a subset of guidelines that described development methods or had institutional endorsement. Results Forty guidelines published in journals, books and institutional reports were included in the review; seven were specific to RCT protocols. Only eight (20%) described development methods which included informal consensus methods, pilot testing and formal validation; no guideline described all of these methods. No guideline described formal consensus methods or a systematic retrieval of empirical evidence to inform its development. The guidelines included a median of 23 concepts per guideline (interquartile range (IQR) = 14 to 34; range = 7 to 109). Among the subset of guidelines (n = 23) for which content was mapped, approximately 380 concepts were explicitly addressed (median concepts per guideline IQR = 31 (24,80); range = 16 to 150); most concepts were addressed in a minority of guidelines. Conclusions Existing guidelines for RCT protocol content varied substantially in their recommendations. Few reports described the methods of guideline development, limiting comparisons of guideline validity. Given the importance of protocols to diverse stakeholders, we believe a systematically developed, evidence-informed guideline for clinical trial protocols is needed.
    Full-text · Article · Sep 2012 · Systematic Reviews
  • Source
    • "Unfortunately, many protocols do not adequately describe important methodological details such as allocation concealment (59%) [9], primary outcomes (25%) [1], power calculations (27%) [3] and sponsor and investigators’ roles in aspects of trial conduct [10] - all of which have been associated with exaggerated effect sizes and potential bias in trials. The lack of transparency and incomplete description of methods makes critical assessment of trials difficult. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Recent evidence has highlighted deficiencies in clinical trial protocols, having implications for many groups. Existing guidelines for randomized clinical trial (RCT) protocol content vary substantially and most do not describe systematic methodology for their development. As one of three prespecified steps for the systematic development of a guideline for trial protocol content, the objective of this study was to conduct a three-round Delphi consensus survey to develop and refine minimum content for RCT protocols. Panellists were identified using a multistep iterative approach, met prespecified minimum criteria and represented key stakeholders who develop or use clinical trial protocols. They were asked to rate concepts for importance in a minimum set of items for RCT protocols. The main outcome measures were degree of importance (scale of 1 to 10; higher scores indicating higher importance) and level of consensus for items. Results were presented as medians, interquartile ranges, counts and percentages. Ninety-six expert panellists participated in the Delphi consensus survey including trial investigators, methodologists, research ethics board members, funders, industry, regulators and journal editors. Response rates were between 88 and 93% per round. Overall, panellists rated 63 of 88 concepts of high importance (of which 50 had a 25th percentile rating of 8 or greater), 13 of moderate importance (median 6 or 7) and 12 of low importance (median less than or equal to 5) for minimum trial protocol content. General and item-specific comments and subgroup results provided valuable insight for further discussions. This Delphi process achieved consensus from a large panel of experts from diverse stakeholder groups on essential content for RCT protocols. It also highlights areas of divergence. These results, complemented by other empirical research and consensus meetings, are helping guide the development of a guideline for protocol content.
    Full-text · Article · Sep 2012 · Trials
Show more