High Impact = High Statistical Standards? Not Necessarily So

Dipartimento di Psicologia Generale, Università di Padova, Padova, Italy.
PLoS ONE (Impact Factor: 3.23). 02/2013; 8(2):e56180. DOI: 10.1371/journal.pone.0056180
Source: PubMed


What are the statistical practices of articles published in journals with a high impact factor? Are there differences compared with articles published in journals with a somewhat lower impact factor that have adopted editorial policies to reduce the impact of limitations of Null Hypothesis Significance Testing? To investigate these questions, the current study analyzed all articles related to psychological, neuropsychological and medical issues, published in 2011 in four journals with high impact factors: Science, Nature, The New England Journal of Medicine and The Lancet, and three journals with relatively lower impact factors: Neuropsychology, Journal of Experimental Psychology-Applied and the American Journal of Public Health. Results show that Null Hypothesis Significance Testing without any use of confidence intervals, effect size, prospective power and model estimation, is the prevalent statistical practice used in articles published in Nature, 89%, followed by articles published in Science, 42%. By contrast, in all other journals, both with high and lower impact factors, most articles report confidence intervals and/or effect size measures. We interpreted these differences as consequences of the editorial policies adopted by the journal editors, which are probably the most effective means to improve the statistical practices in journals with high or low impact factors.

Download full-text


Available from: Francesco Sella
  • Source
    • "Note that this is not only a problem of obscure niche research. On the contrary, there is correlation between impact factor of the journal a paper is published in and the probability that the paper will have to be retracted (Brembs, Button et al. 2013), quite possibly due to the fact that power considerations are even rare for papers published in flagship journals such as Nature or Science (Tressoldi, Giofré et al. 2013). The fact that the problem seems to be most manifest in the most prominent journals that are read by the most people and have the most potential impact only amplifies these problems. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Participant needs to achieve a given power are frequently underestimated. This is particularly problematic when effect sizes are small, such as is common in neuroscience and psychology. We provide tools to make these demands immediately obvious in the form of a powerscape visualization.
    Preview · Article · Dec 2015
  • Source
    • "do report SE or their confidence intervals; the foregoing has excited the question from some authors as to whether high impact is equal to high statistical standards in scientific journals, and the answer is no (Tressoldi et al., 2013). "
    [Show abstract] [Hide abstract]
    ABSTRACT: Quantitative psychological research is focused on detecting the occurrence of certain population phenomena by analyzing data from a sample, and statistics is a particularly helpful mathematical tool that is used by researchers to evaluate hypotheses and make decisions to accept or reject such hypotheses. In this paper, the various statistical tools in psychological research are reviewed. The limitations of null hypothesis significance testing (NHST) and the advantages of using effect size and its respective confidence intervals are explained, as the latter two measurements can provide important information about the results of a study. These measurements also can facilitate data interpretation and easily detect trivial effects, enabling researchers to make decisions in a more clinically relevant fashion. Moreover, it is recommended to establish an appropriate sample size by calculating the optimum statistical power at the moment that the research is designed. Psychological journal editors are encouraged to follow APA recommendations strictly and ask authors of original research studies to report the effect size, its confidence intervals, statistical power and, when required, any measure of clinical significance. Additionally, we must account for the teaching of statistics at the graduate level. At that level, students do not receive sufficient information concerning the importance of using different types of effect sizes and their confidence intervals according to the different types of research designs; instead, most of the information is focused on the various tools of NHST. 1 A brighter day is dawning in which researchers will ask not only if a sample result is likely but also if an effect is practically noteworthy or replicable (Thompson, 2002).
    Full-text · Article · Jan 2015 · Psychology in Russia: State of the Art
  • Source
    • "For that reason, Cumming (2014) has proposed that we consider the so-called " new statistics, " the transition from dichotomous thinking to estimation thinking, by using ES, confidence intervals and meta-analysis. Some authors (Cumming, 2014; Schmidt and Hunter, 2004; Tressoldi, Giofre, Sella, & Cumming, 2013) summarize the difficulties of NHST as follows: 1. NHST is centered on null hypothesis rejection at a level that was previously chosen, usually .05; thus, researchers shall obtain only an answer to " if there is or is not a change that is different from zero " . "
    [Show abstract] [Hide abstract]
    ABSTRACT: This is the second in a series of four articles scheduled for publication in this journal. In the previous article I proposed a description of a new so-called existential criterion of normal and abnormal personality that is implicitly present in the works of Erich Fromm. According to this criterion, normal and abnormal personalities are determined, first, by special features of the content of their position regarding existential dichotomies that are natural to human beings and, second, by particular aspects of the formation of this position. Such dichotomies, entitatively existent in all human life, are inherent, two-alternative contradictions. The position of a normal personalityin its content orients one toward a contradictious predetermination of life in the form of existential dichotomies and the necessity of searching for compromise in resolving these dichotomies. This position is created on a rational basis with the person's active participation. The position of an abnormal personality in its content subjectively denies a contradictious predetermination of life in the form of existential dichotomies and orients one toward a consistent, noncompetitive, and, as a consequence, onesided way of life that doesn't include self-determination. This position is imposed by other people on an irrational basis. Abnormality of personality interpreted like that is one of the most important factors influencing the development of various kinds of psychological problems and mental disorders - primarily, neurosis. In this article I show that this criterion is implicitly present in the personality theories of Sigmund Freud and Alfred Adler, although in more special cases. In the following articles I will show that this criterion is also implicitly present in the personality theories of Carl Jung, Carl Rogers, and Viktor Frankl. © Lomonosov Moscow State University, 2015, Russian Psychological Society, 2015.
    Preview · Article · Jan 2015 · Psychology in Russia: State of the Art
Show more

We use cookies to give you the best possible experience on ResearchGate. Read our cookies policy to learn more.