Jelte Wicherts

Jelte Wicherts
Tilburg University | UVT · Department of Methodology and Statistics

PhD

About

202
Publications
131,094
Reads
How we measure 'reads'
A 'read' is counted each time someone views a publication summary (such as the title, abstract, and list of authors), clicks on a figure, or views or downloads the full-text. Learn more
9,714
Citations
Additional affiliations
April 2017 - present
Tilburg University
Position
  • Professor
May 2012 - March 2017
Tilburg University
Position
  • Professor
March 2007 - April 2012
University of Amsterdam
Position
  • Professor (Assistant)

Publications

Publications (202)
Article
Stereotype threat theory states that female and minority test-takers underperform on cognitive tests because they experience pressure by negative stereotypes about their group's performance. The theory hypothesizes that this effect is larger for test-takers who strongly identify with an academic domain, and for whom the test is the most difficult....
Preprint
The meta-plot is a descriptive visual tool for meta-analysis that provides information on the primary studies in the meta-analysis and the results of the meta-analysis. More precisely, the meta-plot portrays (i) the precision and statistical power of the primary studies in the meta-analysis, (ii) the estimate and confidence interval of a random-eff...
Article
Background: Traditionally, research integrity studies have focused on research misbehaviors and their explanations. Over time, attention has shifted towards preventing questionable research practices and promoting responsible ones. However, data on the prevalence of responsible research practices, especially open methods, open codes and open data a...
Preprint
Given the many benefits of sharing data, an increasing number of psychological researchers publicly share the data underlying their research via online repositories. While undoubtedly a positive scientific development that enables greater verification and data re-use, it is important to protect the interests and confidentiality of research particip...
Article
Full-text available
Prevalence of research misconduct, questionable research practices (QRPs) and their associations with a range of explanatory factors has not been studied sufficiently among academic researchers. The National Survey on Research Integrity targeted all disciplinary fields and academic ranks in the Netherlands. It included questions about engagement in...
Preprint
Full-text available
When cognitive and educational tests are administered under time limits, tests may become speeded and this may affect reliability and validity of the resulting test scores. Prior research has shown that time limits may create or enlarge gender gaps in cognitive and academic testing. On average, women complete fewer items than men when a test is adm...
Article
Full-text available
Effect misestimations plague Psychological Science, but advances in the identification of dissemination biases in general and publication bias in particular have helped in dealing with biased effects in the literature. However, the application of publication bias detection methods appears to be not equally prevalent across subdisciplines. It has be...
Article
Full-text available
Any large dataset can be analyzed in a number of ways, and it is possible that the use of different analysis strategies will lead to different results and conclusions. One way to assess whether the results obtained depend on the analysis strategy chosen is to employ multiple analysts and leave each of them free to follow their own approach. Here, w...
Preprint
Full-text available
Meta-researchers increasingly study biases in quantitative study outcomes (effect sizes) that emerge from questionable research practices (QRPs) in designing, running, analyzing, and reporting studies. Here, we introduce an extensible and modular C++ simulation framework called SAM (Science Abstract Model) that enables systematic study of the effec...
Preprint
Full-text available
Inflated or outright false effects plague Psychological Science, but advances in the identification of dissemination biases in general and publication bias in particular have helped in dealing with biased effects in the literature. However, the application of publication bias detection methods appears to be not equally prevalent across subdisciplin...
Article
Grass-roots action against bad behaviour has spurred reform — and should keep going. Grass-roots action against bad behaviour has spurred reform — and should keep going. “We need a culture in which top research is trustworthy as well as exciting.” “We need a culture in which top research is trustworthy as well as exciting.”
Preprint
Stereotype threat theory states that female and minority test-takers underperform on cognitive tests because they experience pressure by negative stereotypes about their group’s performance. The theory implicates larger effects for test-takers who strongly identify with an academic domain, and for whom the test is most difficult. These moderators c...
Preprint
Full-text available
Background Prevalence of research misconduct, questionable research practices (QRPs) and their associations with a range of explanatory factors has not been studied sufficiently among academic researchers.Methods The National Survey on Research Integrity was aimed at all disciplinary fields and academic ranks in the Netherlands. The survey enquired...
Preprint
Full-text available
Background While substantial attention has been paid to research misbehaviors, responsible research practices (RRPs) and their potential explanatory factors have not been studied extensively.Methods The National Survey on Research Integrity (NSRI) is an online survey targeting all disciplines and academic ranks in The Netherlands. Data was collecte...
Preprint
There are arbitrary decisions to be made (i.e., researcher degrees of freedom) in the execution and reporting of most research. These decisions allow for many possible outcomes from a single study. Selective reporting of results from this ‘multiverse’ of outcomes, whether intentional (_p_-hacking) or not, can lead to inflated effect size estimates...
Preprint
Full-text available
We present consensus-based guidance for conducting and documenting multi-analyst studies. We discuss why broader adoption of the multi-analyst approach will strengthen the robustness of results and conclusions in empirical sciences.
Article
Full-text available
Introduction Type D personality, operationalized as high scores on negative affectivity (NA) and social inhibition (SI), has been associated with various medical and psychosocial outcomes. The recent failure to replicate earlier findings could result from the various methods used to assess the Type D effect. Despite recommendations to analyze the c...
Preprint
Outcome reporting bias (ORB) refers to the biasing effect caused by researchers selectively reporting outcomes based on their statistical significance. ORB leads to inflated average effect size estimates in a meta-analysis if only the outcome with the largest effect size is reported due to ORB. We propose a new method (CORB) to correct for ORB that...
Chapter
Full-text available
Since the onset of the SARS-COV-2 pandemic in late 2019, the scientific literature on the SARS-COV-2 virus and the disease COVID-19 has a growth rate that resembles the growth in confirmed COVID-19 cases that continue to make media headlines all across the globe. Biomedical coronavirus research started slowly but increased to hundreds of articles p...
Article
Full-text available
Researchers face many, often seemingly arbitrary, choices in formulating hypotheses, designing protocols, collecting data, analyzing data, and reporting results. Opportunistic use of “researcher degrees of freedom” aimed at obtaining statistical significance increases the likelihood of obtaining and publishing false-positive results and overestimat...
Article
Full-text available
In this newly published study me and my colleagues focused on three popular methods to model interactions between two constructs containing measurement error in predicting an observed binary outcome: logistic regression using (1) observed scores, (2) factor scores, and (3) Structural Equation Modeling (SEM). It is still unclear how they compare wit...
Article
Full-text available
We present a meta-analytic review of the literature on sex differences in the trust game (174 effect sizes) and the related gift-exchange game (35 effect sizes). Based on parental investment theory and social role theory we expected men to be more trusting and women to be more trustworthy. Indeed, men were more trusting in the trust game (g = 0.22)...
Preprint
So-called “open science practices” seek to improve research transparency and methodological rigor. What do emotion researchers think about these practices? To address this question, we surveyed active emotion researchers (N= 144) in October 2019 about their attitudes toward several open science practices. Overall, the majority of emotion researcher...
Article
Full-text available
In this meta-study, we analyzed 2442 effect sizes from 131 meta-analyses in intelligence research, published from 1984 to 2014, to estimate the average effect size, median power, and evidence for bias. We found that the average effect size in intelligence research was a Pearson's correlation of 0.26, and the median sample size was 60. Furthermore,...
Chapter
This article looks at impacts of questionable research practices (QRP’s), specifically in statistics, and how they impact result reporting in psychological sciences. Previous studies provided mixed findings on peculiarities in p-value distributions in psychology. The paper reports the results of examining 258,050 test results across 30,710 articles...
Article
Full-text available
In this preregistered study, we investigated whether the statistical power of a study is higher when researchers are asked to make a formal power analysis before collecting data. We compared the sample size descriptions from two sources: (i) a sample of pre-registrations created according to the guidelines for the Center for Open Science Preregistr...
Article
Full-text available
We examined the evidence for heterogeneity (of effect sizes) when only minor changes to sample population and settings were made between studies and explored the association between heterogeneity and average effect size in a sample of 68 meta-analyses from 13 preregistered multilab direct replication projects in social and cognitive psychology. Amo...
Preprint
We examined the evidence for heterogeneity (of effect sizes) when only minor changes to sample population and settings were made between studies and explored the association between heterogeneity and average effect size in a sample of 68 meta-analyses from thirteen pre-registered multi-lab direct replication projects in social and cognitive psychol...
Article
Full-text available
To determine the reproducibility of psychological meta-analyses, we investigated whether we could reproduce 500 primary study effect sizes drawn from 33 published meta-analyses based on the information given in the meta-analyses, and whether recomputations of primary study effect sizes altered the overall results of the meta-analysis. Results showe...
Article
Full-text available
An amendment to this paper has been published and can be accessed via a link at the top of the paper.
Article
Full-text available
We present a consensus-based checklist to improve and document the transparency of research reports in social and behavioural research. An accompanying online application allows users to complete the form and generate a report that they can submit with their manuscript or post to a public repository.
Article
Leading scholars and publishers from ten countries have agreed a definition of predatory publishing that can protect scholarship. It took 12 hours of discussion, 18 questions and 3 rounds to reach. Leading scholars and publishers from ten countries have agreed a definition of predatory publishing that can protect scholarship. It took 12 hours of di...
Preprint
To determine the reproducibility of psychological meta-analyses, we investigated whether we could reproduce 500 primary study effect sizes drawn from 33 published meta-analyses based on the information given in the meta-analyses, and whether recomputations of primary study effect sizes altered the overall results of the meta-analysis.
Preprint
This is our commentary on the "What is IQ? Life Beyond "General Intelligence"" paper by Kovacs and Conway (2019- https://doi.org/10.1177/0963721419827275). We advocate for a latent variable approach in experimental research, and illustrate both conceptual and statistical benefits of using this approach in context of research on cognition. Benefits...
Preprint
Scientific misconduct potentially invalidates findings in many scientific fields. Improved detection of unethical practices like data fabrication is considered to deter such practices. In two studies, we investigated the diagnostic performance of various statistical methods to detect fabricated quantitative data from psychological research. In Stud...
Preprint
Unclear regulations on conflicts of interest (COI) might lead to nepotism in the handing out of academic prizes and awards awarded by professional societies and associations. To assess to availability of statements on COIs, we systematically searched the websites of 58 psychological societies. We found that 27.6% had no COI statement available on t...
Preprint
In this preregistered study, we investigated whether the statistical power of a study is higher when researchers are asked to make a formal power analysis before collecting data. We compared the sample size descriptions from two sources: (i) a sample of preregistrations created according to the guidelines for the Center for Open Science Preregistra...
Article
Full-text available
Several approaches exist to model interactions between latent variables. However, it is unclear how these perform when item scores are skewed and ordinal. Research on Type D personality serves as a good case study for that matter. In Study 1, we fitted a multivariate interaction model to predict depression and anxiety with Type D personality, opera...
Article
Full-text available
Publication bias is a substantial problem for the credibility of research in general and of meta-analyses in particular, as it yields overestimated effects and may suggest the existence of non-existing effects. Although there is consensus that publication bias exists, how strongly it affects different scientific literatures is currently less well-k...
Data
Results of meta-meta regression with a random effect to take into account that the subsets were nested in meta-analyses. The dependent variable is the absolute value of p-uniform’s effect size estimate with predictors discipline, I2-statistic, harmonic mean of the standard error (standard error), proportion of statistically significant effect sizes...
Data
Results of logistic regression predicting statistical significance of p-uniform’s publication bias test with discipline and control variable number of statistically significant effect sizes in a subset. (DOCX)
Data
Results of multilevel logistic regression predicting statistical significance of Egger’s regression test with discipline and control variable number of effect sizes in a subset. (DOCX)
Data
Results of multilevel logistic regression predicting statistical significance of rank-correlation test with discipline and control variable number of effect sizes in a subset. (DOCX)
Data
Results of quantile regression with the median of effect size overestimation in random-effects meta-analysis when compared to p-uniform (Y) and predictors discipline, I2-statistic, harmonic mean of the standard error (standard error), proportion of statistically significant effect sizes in a subset (Prop. sig. effect sizes), and number of effect si...
Data
Results of logistic regression predicting statistical significance of rank-correlation test with discipline and control variable number of effect sizes in a subset. (DOCX)
Data
Results of logistic regression predicting statistical significance of test of excess significance with discipline and control variable number of effect sizes in a subset. (DOCX)
Data
Results of multilevel logistic regression predicting statistical significance of test of excess significance with discipline and control variable number of effect sizes in a subset. (DOCX)
Data
Results of meta-meta regression with a random effect to take into account that the subsets were nested in meta-analyses. The dependent variable is the absolute value of the random-effects meta-analysis effect size estimate with predictors discipline, I2-statistic, harmonic mean of the standard error (standard error), proportion of statistically sig...
Data
Results of meta-meta-regression with a random effect to take into account that the subsets were nested in meta-analyses. The dependent variable is the effect size overestimation in random-effects meta-analysis when compared to p-uniform (Y) and predictors discipline, I2-statistic, harmonic mean of the standard error (standard error), proportion of...
Data
Results of logistic regression predicting statistical significance of Egger’s regression test with discipline and control variable number of effect sizes in a subset. (DOCX)
Data
Results of multilevel logistic regression predicting statistical significance of p-uniform’s publication bias test with discipline and control variable number of statistically significant effect sizes in a subset. (DOCX)
Data
Results of quantile regression with the median of p-uniform’s effect size estimates and predictors discipline, I2-statistic, harmonic mean of the standard error (standard error), proportion of statistically significant effect sizes in a subset (Prop. sig. effect sizes), and number of effect sizes in a subset. (DOCX)
Data
List of references of meta-analyses where the data of the primary studies were obtained after contacting the corresponding author. (DOCX)
Article
Full-text available
Research on money priming typically investigates whether exposure to money-related stimuli can affect people's thoughts, feelings, motivations, and behaviors (for a review, see Vohs, 2015). Our study answers the call for a comprehensive meta-analysis examining the available evidence on money priming (Vadillo, Hardwicke, & Shanks, 2016). By conducti...
Article
Full-text available
The effects of gender stereotype threat on mathematical test performance in the classroom have been extensively studied in several cultural contexts. Theory predicts that stereotype threat lowers girls’ performance on mathematics tests, while leaving boys’ math performance unaffected. We conducted a large-scale stereotype threat experiment in Dutch...
Preprint
We studied how academics assess the results of a set of four experiments that all test a given theory. We found that participants’ belief in the theory increases with the number of significant results, and that direct replications were considered to be more important than conceptual replications. We found no difference between authors and reviewers...
Article
Full-text available
Background: Many psychosocial and behavioral interventions have been developed for informal dementia caregivers. Because existing meta-analyses only focused on a limited number of interventions and outcomes, how effective these interventions are overall and which interventions components are associated with larger effects has yet to be explored....
Article
In determining the need to directly replicate, it is crucial to first verify the original results through independent reanalysis of the data. Original results that appear erroneous and that cannot be reproduced by reanalysis offer little evidence to begin with, thereby diminishing the need to replicate. Sharing data and scripts is essential to ensu...
Article
Full-text available
In a recent study, te Nijenhuis et al . (2017) used a version of Jensen’s method of correlated vectors to study the nature of ethnic group differences on Raven’s Progressive Matrices test. In this comment, the author points out that this method has been shown to be psychometrically inappropriate in studying group differences in performance on dicho...
Article
Full-text available
In their response to my criticism of their recent article in Journal of Biosocial Science (te Nijenhuis et al ., 2017), te Nijenhuis and van den Hoek (2018) raise four points none of which concerns my main point that the method of correlated vectors (MCV) applied to item-level data represents a flawed method. Here, I discuss te Nijenhuis and van de...
Preprint
We analyzed 2,439 effect sizes from 131 meta-analyses in intelligence research to estimate the average effect size, median power, and evidence for bias in this field. We found that the typical effect size in this field was a Pearson’s correlation of .26, and the median sample size was 60. We calculated the power of each primary study by using the c...
Preprint
In determining the need to directly replicate, it is crucial to first verify the original results through independent reanalysis of the data. Original results that appear erroneous and that cannot be reproduced by reanalysis offer little evidence to begin with, thereby diminishing the need to replicate. Sharing data and scripts is essential to ensu...
Article
Full-text available
In this paper, we present three retrospective observational studies that investigate the relation between data sharing and statistical reporting inconsistencies. Previous research found that reluctance to share data was related to a higher prevalence of statistical errors, often in the direction of statistical significance (Wicherts, Bakker, & Mole...
Article
Full-text available
In this review, the author discusses several of the weak spots in contemporary science, including scientific misconduct, the problems of post hoc hypothesizing (HARKing), outcome switching, theoretical bloopers in formulating research questions and hypotheses, selective reading of the literature, selective citing of previous results, improper blind...
Article
Full-text available
The syntax or codes used to fit Structural Equation Models (SEMs) convey valuable information on model specifications and the manner in which SEMs are estimated. We requested SEM syntaxes from a random sample of 229 articles (published in 1998–2013) that ran SEMs using LISREL, AMOS, or Mplus. After exchanging over 500 emails, we ended up obtaining...
Preprint
The R package “statcheck” (Epskamp & Nuijten, 2016) is a tool to extract statistical results from articles and check whether the reported p-value matches the accompanying test statistic and degrees of freedom. A previous study showed high interrater reliabilities (between .76 and .89) between statcheck and manual coding of inconsistencies (.76 - .8...
Preprint
The syntax or codes used to fit Structural Equation Models (SEMs) convey valuable information on model specifications and the manner in which SEMs are estimated. We requested SEM syntaxes from a random sample of 229 articles (published in 1998-2013) that ran SEMs using LISREL, AMOS, or Mplus. After exchanging over 500 emails, we ended up obtaining...
Preprint
In this paper, we present three retrospective observational studies that investigate the relation between data sharing and statistical reporting inconsistencies. Previous research found that reluctance to share data was related to a higher prevalence of statistical errors, often in the direction of statistical significance (Wicherts, Bakker, & Mole...
Preprint
Publication bias is a substantial problem for the credibility of research in general and of meta-analyses in particular, as it yields overestimated effects and may suggest the existence of non-existing effects. Although there is consensus that publication bias is widespread, how strongly it affects different scientific literatures is currently less...
Article
Full-text available
Due to its probabilistic nature, Null Hypothesis Significance Testing (NHST) is subject to decision errors. The concern for false positives has overshadowed the concern for false negatives in the recent debates in psychology. This might be unwarranted, since reported statistically nonsignificant findings may just be 'too good to be false'. We exami...
Article
Full-text available
A survey in the United States revealed that an alarmingly large percentage of university psychologists admitted having used questionable research practices that can contaminate the research literature with false positive and biased findings. We conducted a replication of this study among Italian research psychologists to investigate whether these f...
Chapter