Likelihood of False-Positive Results in High-Impact Journals Publishing Groundbreaking Research

Address correspondence to Anurag Agrawal, .
Infection and immunity (Impact Factor: 3.73). 03/2012; 80(3):1300; author reply 1301. DOI: 10.1128/IAI.06233-11
Source: PubMed
Download full-text


Available from: Anurag Agrawal, Mar 18, 2014
26 Reads
  • Source
    [Show abstract] [Hide abstract]
    ABSTRACT: Gene-by-environment interaction (G×E) studies in psychiatry have typically been conducted using a candidate G×E (cG×E) approach, analogous to the candidate gene association approach used to test genetic main effects. Such cG×E research has received widespread attention and acclaim, yet cG×E findings remain controversial. The authors examined whether the many positive cG×E findings reported in the psychiatric literature were robust or if, in aggregate, cG×E findings were consistent with the existence of publication bias, low statistical power, and a high false discovery rate. The authors conducted analyses on data extracted from all published studies (103 studies) from the first decade (2000-2009) of cG×E research in psychiatry. Ninety-six percent of novel cG×E studies were significant compared with 27% of replication attempts. These findings are consistent with the existence of publication bias among novel cG×E studies, making cG×E hypotheses appear more robust than they actually are. There also appears to be publication bias among replication attempts because positive replication attempts had smaller average sample sizes than negative ones. Power calculations using observed sample sizes suggest that cG×E studies are underpowered. Low power along with the likely low prior probability of a given cG×E hypothesis being true suggests that most or even all positive cG×E findings represent type I errors. In this new era of big data and small effects, a recalibration of views about groundbreaking findings is necessary. Well-powered direct replications deserve more attention than novel cG×E findings and indirect replications.
    American Journal of Psychiatry 09/2011; 168(10):1041-9. DOI:10.1176/appi.ajp.2011.11020191 · 12.30 Impact Factor
  • Source
    [Show abstract] [Hide abstract]
    ABSTRACT: There is increasing concern that most current published research findings are false. The probability that a research claim is true may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientific field. In this framework, a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance. Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias. In this essay, I discuss the implications of these problems for the conduct and interpretation of research.
    PLoS Medicine 09/2005; 2(8):e124. DOI:10.1371/journal.pmed.0020124 · 14.43 Impact Factor
  • Source
    [Show abstract] [Hide abstract]
    ABSTRACT: Articles may be retracted when their findings are no longer considered trustworthy due to scientific misconduct or error, they plagiarize previously published work, or they are found to violate ethical guidelines. Using a novel measure that we call the "retraction index," we found that the frequency of retraction varies among journals and shows a strong correlation with the journal impact factor. Although retractions are relatively rare, the retraction process is essential for correcting the literature and maintaining trust in the scientific process.
    Infection and immunity 08/2011; 79(10):3855-9. DOI:10.1128/IAI.05661-11 · 3.73 Impact Factor