High Impact = High Statistical Standards? Not Necessarily So

Dipartimento di Psicologia Generale, Università di Padova, Padova, Italy.
PLoS ONE (Impact Factor: 3.53). 02/2013; 8(2):e56180. DOI: 10.1371/journal.pone.0056180
Source: PubMed

ABSTRACT What are the statistical practices of articles published in journals with a high impact factor? Are there differences compared with articles published in journals with a somewhat lower impact factor that have adopted editorial policies to reduce the impact of limitations of Null Hypothesis Significance Testing? To investigate these questions, the current study analyzed all articles related to psychological, neuropsychological and medical issues, published in 2011 in four journals with high impact factors: Science, Nature, The New England Journal of Medicine and The Lancet, and three journals with relatively lower impact factors: Neuropsychology, Journal of Experimental Psychology-Applied and the American Journal of Public Health. Results show that Null Hypothesis Significance Testing without any use of confidence intervals, effect size, prospective power and model estimation, is the prevalent statistical practice used in articles published in Nature, 89%, followed by articles published in Science, 42%. By contrast, in all other journals, both with high and lower impact factors, most articles report confidence intervals and/or effect size measures. We interpreted these differences as consequences of the editorial policies adopted by the journal editors, which are probably the most effective means to improve the statistical practices in journals with high or low impact factors.

Download full-text


Available from: Francesco Sella, Aug 13, 2015
1 Follower
  • Source
    [Show abstract] [Hide abstract]
    ABSTRACT: The possibility of predicting random future events before any sensory clues by using human physiology as a dependent variable has been supported by the meta-analysis of Mossbridge et al.1 and by recent findings by Tressoldi et al.2,3. Mossbridge et al.4 defined this phenomenon predictive anticipatory activity (PAA).From a theoretical point of view, one interesting question is whether PAA is related to the effective, real future presentation of these stimuli or whether it is related only to the probability of their presentation.This hypothesis was tested with four experiments two using heart rate and two using pupil dilation as dependent variables.In all four experiments, both a neutral and a potentially threatening stimulus were predicted 7 to 10% above chance, independently from whether the predicted threatening stimulus was presented or not.These findings are discussed with reference to the “grandfather paradox” and some candidate explanations for this phenomena are presented.
    SSRN Electronic Journal 01/2013; DOI:10.2139/ssrn.2371577
  • Source
    [Show abstract] [Hide abstract]
    ABSTRACT: Most researchers acknowledge an intrinsic hierarchy in the scholarly journals ("journal rank") that they submit their work to, and adjust not only their submission but also their reading strategies accordingly. On the other hand, much has been written about the negative effects of institutionalizing journal rank as an impact measure. So far, contributions to the debate concerning the limitations of journal rank as a scientific impact assessment tool have either lacked data, or relied on only a few studies. In this review, we present the most recent and pertinent data on the consequences of our current scholarly communication system with respect to various measures of scientific quality (such as utility/citations, methodological soundness, expert ratings or retractions). These data corroborate previous hypotheses: using journal rank as an assessment tool is bad scientific practice. Moreover, the data lead us to argue that any journal rank (not only the currently-favored Impact Factor) would have this negative impact. Therefore, we suggest that abandoning journals altogether, in favor of a library-based scholarly communication system, will ultimately be necessary. This new system will use modern information technology to vastly improve the filter, sort and discovery functions of the current journal system.
    Frontiers in Human Neuroscience 06/2013; 7:291. DOI:10.3389/fnhum.2013.00291 · 2.90 Impact Factor
  • Nature Reviews Neuroscience 07/2013; 14. DOI:10.1038/nrn3475-c4 · 31.38 Impact Factor
Show more