Misrepresentation of Randomized Controlled Trials in Press Releases and News Coverage: A Cohort Study

INSERM, U738, Paris, France
PLoS Medicine (Impact Factor: 14.43). 09/2012; 9(9):e1001308. DOI: 10.1371/journal.pmed.1001308
Source: PubMed


Previous studies indicate that in published reports, trial results can be distorted by the use of "spin" (specific reporting strategies, intentional or unintentional, emphasizing the beneficial effect of the experimental treatment). We aimed to (1) evaluate the presence of "spin" in press releases and associated media coverage; and (2) evaluate whether findings of randomized controlled trials (RCTs) based on press releases and media coverage are misinterpreted.
We systematically searched for all press releases indexed in the EurekAlert! database between December 2009 and March 2010. Of the 498 press releases retrieved and screened, we included press releases for all two-arm, parallel-group RCTs (n = 70). We obtained a copy of the scientific article to which the press release related and we systematically searched for related news items using Lexis Nexis. "Spin," defined as specific reporting strategies (intentional or unintentional) emphasizing the beneficial effect of the experimental treatment, was identified in 28 (40%) scientific article abstract conclusions and in 33 (47%) press releases. From bivariate and multivariable analysis assessing the journal type, funding source, sample size, type of treatment (drug or other), results of the primary outcomes (all nonstatistically significant versus other), author of the press release, and the presence of "spin" in the abstract conclusion, the only factor associated, with "spin" in the press release was "spin" in the article abstract conclusions (relative risk [RR] 5.6, [95% CI 2.8-11.1], p<0.001). Findings of RCTs based on press releases were overestimated for 19 (27%) reports. News items were identified for 41 RCTs; 21 (51%) were reported with "spin," mainly the same type of "spin" as those identified in the press release and article abstract conclusion. Findings of RCTs based on the news item was overestimated for ten (24%) reports.
"Spin" was identified in about half of press releases and media coverage. In multivariable analysis, the main factor associated with "spin" in press releases was the presence of "spin" in the article abstract conclusion.

Download full-text


Available from: Philippe Ravaud,
1 Follower
59 Reads
  • Source
    • "Confirmatory bias and the spinning of weak and null findings generally start in abstracts (Yavchitz et al., 2012). Although CONSORT has specific recommendations for what should be reported in abstracts (Hopewell et al., 2008), these standards are largely ignored in reports of its psychotherapy interventions. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Recognition of the unreliability of findings in the biomedical literature, and especially pharmaceutical trials has led to a number of reforms. These include reporting preregistration of protocols for clinical trials and meta-analysis, reporting standards and making data available to others for all clinical trials, as well as recognition of the influence of conflicts of interest. These improvements are only partially and inconsistently reflected in the literature evaluating psychotherapies. The psychotherapy literature is currently of too poor quality to provide a reliable guide to clinicians, consumers, and policymakers. The literature is dominated by underpowered trials with high risk of bias producing positive effects at a statistically improbable rate. Meta-analyses that are poorly conducted with undisclosed conflicts of interest compound these problems. A number of reforms are proposed. These include accelerating adoption of those already occurring in the pharmaceutical literature. Additionally, psychotherapy research should parallel the orderly sequence of treatment development seen in the pharmaceutical literature. Phase III trials providing the effect sizes of treatments should not be conducted until the acceptability of treatment and the feasibility of accruing sufficient numbers of patients are established. The role of investigator allegiance as a potential and potent source of conflict of interest needs to be recognized. Yet, enforcement of existing standards could counter many of the deficiencies of the current literature, but such enforcement may only come with pressures emanating from outside the field of psychotherapy.
  • Source
    • "Common types of spin included emphasizing within-group assessment, claiming efficacy regardless of non-significant findings, and addressing only significant findings. Yavchitz et al. (2012) also found spin present in 40% of abstracts from RCTs that they examined. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Valid, scientific research is critical for ascertaining the effects of instructional techniques on learners with disabilities and for guiding effective special education practice and policy. Researchers in fields such as psychology and medicine have identified serious and widespread shortcomings in their research literatures related to replication and bias, leading many to question the validity of research findings in these fields. Replication and bias have not been systematically examined in special education research. In this article, I explore research on replication and bias conducted in other fields, discuss the likelihood that problems in replication and bias may exist in the special education research base, propose a research agenda for investigating replication and bias in special education research, and make initial recommendations for increasing replication and reducing bias in intervention studies in special education.
    Remedial and Special Education 07/2014; 35(4):233-246. DOI:10.1177/0741932514528995 · 0.68 Impact Factor
  • Source
    • "The value of science is dependent on complete and transparent reporting of scientific investigations, but recent examples highlight the existence of incomplete or distorted research reporting [5], [6]. Here, we have demonstrated that the careful use of crowdsourcing can be an economical and timesaving means to evaluate large bodies of published literature. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Systematically evaluating scientific literature is a time consuming endeavor that requires hours of coding and rating. Here, we describe a method to distribute these tasks across a large group through online crowdsourcing. Using Amazon's Mechanical Turk, crowdsourced workers (microworkers) completed four groups of tasks to evaluate the question, "Do nutrition-obesity studies with conclusions concordant with popular opinion receive more attention in the scientific community than do those that are discordant?" 1) Microworkers who passed a qualification test (19% passed) evaluated abstracts to determine if they were about human studies investigating nutrition and obesity. Agreement between the first two raters' conclusions was moderate (κ = 0.586), with consensus being reached in 96% of abstracts. 2) Microworkers iteratively synthesized free-text answers describing the studied foods into one coherent term. Approximately 84% of foods were agreed upon, with only 4 and 8% of ratings failing manual review in different steps. 3) Microworkers were asked to rate the perceived obesogenicity of the synthesized food terms. Over 99% of responses were complete and usable, and opinions of the microworkers qualitatively matched the authors' expert expectations (e.g., sugar-sweetened beverages were thought to cause obesity and fruits and vegetables were thought to prevent obesity). 4) Microworkers extracted citation counts for each paper through Google Scholar. Microworkers reached consensus or unanimous agreement for all successful searches. To answer the example question, data were aggregated and analyzed, and showed no significant association between popular opinion and attention the paper received as measured by Scimago Journal Rank and citation counts. Direct microworker costs totaled $221.75, (estimated cost at minimum wage: $312.61). We discuss important points to consider to ensure good quality control and appropriate pay for microworkers. With good reliability and low cost, crowdsourcing has potential to evaluate published literature in a cost-effective, quick, and reliable manner using existing, easily accessible resources.
    PLoS ONE 07/2014; 9(7):e100647. DOI:10.1371/journal.pone.0100647 · 3.23 Impact Factor
Show more