Marcel A.L.M. van Assen

Tilburg University | UVT

This article is about study preregistration, the process of publicly sharing the design, the hypotheses, and the analysis plans of a scientific study before conducting that study. This is said to be important because of two reasons. First, researchers are prone to confirmation bias, the phenomenon that people interpret new information as confirmati...

Self-report scales are widely used in psychology to compare means in latent constructs across groups, experimental conditions, or time points. However, for these comparisons to be meaningful and unbiased, the scales must demonstrate measurement invariance (MI) across compared time points or (experimental) groups. MI testing determines whether the l...

Chronic diseases had the strongest associations with total and physical frailty. • The effects of chronic diseases on frailty differed strongly across diseases. • Urinary incontinence and severe back disorder impaired frailty most. • Cancer and hypertension had the weakest associations with frailty. • Different weight should be given to individual...

Preregistration has gained traction as one of the most promising solutions to improve the replicability of scientific effects. In this project, we compared 193 psychology studies that earned a Preregistration Challenge prize or preregistration badge to 193 related studies that were not preregistered. In contrast to our theoretical expectations and...

In this study, we assessed the extent of selective hypothesis reporting in psychological research by comparing the hypotheses found in a set of 459 preregistrations with the hypotheses found in the corresponding articles. We found that more than half of the preregistered studies we assessed contained omitted hypotheses ( N = 224; 52%) or added hypo...

Preregistration has gained traction as one of the most promising solutions to improve the replicability of scientific effects. In this project, we compared 193 psychology studies that earned a Preregistration Challenge prize or Preregistration Badge to 193 related studies that were not preregistered. In contrast with our theoretical expectations an...

Study preregistration has become increasingly popular in psychology, but its effectiveness in restricting potentially biasing researcher degrees of freedom remains unclear. We used an extensive protocol to assess the strictness of preregistrations and the consistency between preregistration and publications of 300 preregistered psychology studies....

Employing two vignette studies, we examined how psychology researchers interpret the results of a set of four experiments that all test a given theory. In both studies, we found that participants’ belief in the theory increased with the number of statistically significant results, and that the result of a direct replication had a stronger effect on...

We report a vignette study and a survey to investigate which study characteristics influence quality ratings academics give of articles submitted for publication, and academics and students give of students’ theses. In the vignette study, 800 respondents evaluated the quality of an abstract of studies with small or large sample sizes, showing stati...

The meta-plot is a descriptive visual tool for meta-analysis that provides information on the primary studies in the meta-analysis and the results of the meta-analysis. More precisely, the meta-plot portrays (1) the precision and statistical power of the primary studies in the meta-analysis, (2) the estimate and confidence interval of a random-effe...

This study assesses the extent of selective hypothesis reporting in psychological research by comparing the hypotheses found in a set of 459 preregistrations to the hypotheses found in the corresponding papers. We found that more than half of the preregistered studies we assessed contain omitted hypotheses (N = 224; 52.2%) or added hypotheses (N =...

In psychological science, self-report scales are widely used to compare means in targeted latent constructs across time points, groups, or experimental conditions. For these scale mean comparisons (SMC) to be meaningful and unbiased, the scales should be measurement invariant across the compared time points or (experimental) groups. Measurement inv...

The meta-plot is a descriptive visual tool for meta-analysis that provides information on the primary studies in the meta-analysis and the results of the meta-analysis. More precisely, the meta-plot portrays (i) the precision and statistical power of the primary studies in the meta-analysis, (ii) the estimate and confidence interval of a random-eff...

Background Multidimensional frailty, including physical, psychological, and social components, is associated to disability, lower quality of life, increased healthcare utilization, and mortality. In order to prevent or delay frailty, more knowledge of its determinants is necessary; one of these determinants is lifestyle. The aim of this study is to...

In this crowdsourced initiative, independent analysts used the same dataset to test two hypotheses regarding the effects of scientists' gender and professional status on verbosity during group meetings. Not only the analytic approach but also the operationalizations of key variables were left unconstrained and up to individual analysts. For instanc...

We present consensus-based guidance for conducting and documenting multi-analyst studies. We discuss why broader adoption of the multi-analyst approach will strengthen the robustness of results and conclusions in empirical sciences.

Publication bias remains to be a great challenge when conducting a meta-analysis. It may result in overestimated effect sizes, increased frequency of false positives, and over- or underestimation of the effect size heterogeneity parameter. A new method is introduced, Bayesian Meta-Analytic Snapshot (BMAS), which evaluates both effect size and its h...

Researchers face many, often seemingly arbitrary, choices in formulating hypotheses, designing protocols, collecting data, analyzing data, and reporting results. Opportunistic use of “researcher degrees of freedom” aimed at obtaining statistical significance increases the likelihood of obtaining and publishing false-positive results and overestimat...

We present a meta-analytic review of the literature on sex differences in the trust game (174 effect sizes) and the related gift-exchange game (35 effect sizes). Based on parental investment theory and social role theory we expected men to be more trusting and women to be more trustworthy. Indeed, men were more trusting in the trust game (g = 0.22)...

In this meta-study, we analyzed 2442 effect sizes from 131 meta-analyses in intelligence research, published from 1984 to 2014, to estimate the average effect size, median power, and evidence for bias. We found that the average effect size in intelligence research was a Pearson's correlation of 0.26, and the median sample size was 60. Furthermore,...

This article looks at impacts of questionable research practices (QRP’s), specifically in statistics, and how they impact result reporting in psychological sciences. Previous studies provided mixed findings on peculiarities in p-value distributions in psychology. The paper reports the results of examining 258,050 test results across 30,710 articles...

When examining a social phenomenon, theoretical and empirical sociologists require a model. Although models are by definition simplified representations of theories of reality, sociologists typically argue that the micro-level model of individual behavior should be simplified, but not the model of the macro-level system (including macro-micro and m...

Objective To predict mortality with the Tilburg Frailty Indicator (TFI) in a sample of community-dwelling older people, using a follow-up of 7 years. Design Longitudinal. Setting and Participants 479 Dutch community-dwelling people aged 75 years or older. Measurements The TFI, a self-report questionnaire, was used to collect data about total, ph...

In this preregistered study, we investigated whether the statistical power of a study is higher when researchers are asked to make a formal power analysis before collecting data. We compared the sample size descriptions from two sources: (i) a sample of pre-registrations created according to the guidelines for the Center for Open Science Preregistr...

We examined the evidence for heterogeneity (of effect sizes) when only minor changes to sample population and settings were made between studies and explored the association between heterogeneity and average effect size in a sample of 68 meta-analyses from 13 preregistered multilab direct replication projects in social and cognitive psychology. Amo...

We examined the evidence for heterogeneity (of effect sizes) when only minor changes to sample population and settings were made between studies and explored the association between heterogeneity and average effect size in a sample of 68 meta-analyses from thirteen pre-registered multi-lab direct replication projects in social and cognitive psychol...

To determine the reproducibility of psychological meta-analyses, we investigated whether we could reproduce 500 primary study effect sizes drawn from 33 published meta-analyses based on the information given in the meta-analyses, and whether recomputations of primary study effect sizes altered the overall results of the meta-analysis. Results showe...

To determine the reproducibility of psychological meta-analyses, we investigated whether we could reproduce 500 primary study effect sizes drawn from 33 published meta-analyses based on the information given in the meta-analyses, and whether recomputations of primary study effect sizes altered the overall results of the meta-analysis.

Scientific misconduct potentially invalidates findings in many scientific fields. Improved detection of unethical practices like data fabrication is considered to deter such practices. In two studies, we investigated the diagnostic performance of various statistical methods to detect fabricated quantitative data from psychological research. In Stud...

In this preregistered study, we investigated whether the statistical power of a study is higher when researchers are asked to make a formal power analysis before collecting data. We compared the sample size descriptions from two sources: (i) a sample of preregistrations created according to the guidelines for the Center for Open Science Preregistra...

Publication bias is a substantial problem for the credibility of research in general and of meta-analyses in particular, as it yields overestimated effects and may suggest the existence of non-existing effects. Although there is consensus that publication bias exists, how strongly it affects different scientific literatures is currently less well-k...

Results of meta-meta regression with a random effect to take into account that the subsets were nested in meta-analyses. The dependent variable is the absolute value of p-uniform’s effect size estimate with predictors discipline, I2-statistic, harmonic mean of the standard error (standard error), proportion of statistically significant effect sizes...

Results of logistic regression predicting statistical significance of p-uniform’s publication bias test with discipline and control variable number of statistically significant effect sizes in a subset. (DOCX)

Results of multilevel logistic regression predicting statistical significance of Egger’s regression test with discipline and control variable number of effect sizes in a subset. (DOCX)

Results of multilevel logistic regression predicting statistical significance of rank-correlation test with discipline and control variable number of effect sizes in a subset. (DOCX)

Results of quantile regression with the median of effect size overestimation in random-effects meta-analysis when compared to p-uniform (Y) and predictors discipline, I2-statistic, harmonic mean of the standard error (standard error), proportion of statistically significant effect sizes in a subset (Prop. sig. effect sizes), and number of effect si...

Results of logistic regression predicting statistical significance of rank-correlation test with discipline and control variable number of effect sizes in a subset. (DOCX)

Results of logistic regression predicting statistical significance of test of excess significance with discipline and control variable number of effect sizes in a subset. (DOCX)

Results of multilevel logistic regression predicting statistical significance of test of excess significance with discipline and control variable number of effect sizes in a subset. (DOCX)

Results of meta-meta regression with a random effect to take into account that the subsets were nested in meta-analyses. The dependent variable is the absolute value of the random-effects meta-analysis effect size estimate with predictors discipline, I2-statistic, harmonic mean of the standard error (standard error), proportion of statistically sig...

Results of meta-meta-regression with a random effect to take into account that the subsets were nested in meta-analyses. The dependent variable is the effect size overestimation in random-effects meta-analysis when compared to p-uniform (Y) and predictors discipline, I2-statistic, harmonic mean of the standard error (standard error), proportion of...

Results of logistic regression predicting statistical significance of Egger’s regression test with discipline and control variable number of effect sizes in a subset. (DOCX)

Results of multilevel logistic regression predicting statistical significance of p-uniform’s publication bias test with discipline and control variable number of statistically significant effect sizes in a subset. (DOCX)

Results of quantile regression with the median of p-uniform’s effect size estimates and predictors discipline, I2-statistic, harmonic mean of the standard error (standard error), proportion of statistically significant effect sizes in a subset (Prop. sig. effect sizes), and number of effect sizes in a subset. (DOCX)

List of references of meta-analyses where the data of the primary studies were obtained after contacting the corresponding author. (DOCX)

We examined the percentage of p values (.05 < p ≤ .10) reported as marginally significant in 44,200 articles, across nine psychology disciplines, published in 70 journals belonging to the American Psychological Association between 1985 and 2016. Using regular expressions, we extracted 42,504 p values between .05 and .10. Almost 40% of p values in t...

We studied how academics assess the results of a set of four experiments that all test a given theory. We found that participants’ belief in the theory increases with the number of significant results, and that direct replications were considered to be more important than conceptual replications. We found no difference between authors and reviewers...

One of the main goals of meta-analysis is to test for and estimate the heterogeneity of effect sizes. We examined the effect of publication bias on the Q test and assessments of heterogeneity as a function of true heterogeneity, publication bias, true effect size, number of studies, and variation of sample sizes. The present study has two main cont...

Publication bias is a major threat to the validity of a meta-analysis resulting in overestimated effect sizes. P-uniform is a meta-analysis method that corrects estimates for publication bias, but the method overestimates average effect size in the presence of heterogeneity in true effect sizes (i.e., between-study variance). We propose an extensio...

We analyzed 2,439 effect sizes from 131 meta-analyses in intelligence research to estimate the average effect size, median power, and evidence for bias in this field. We found that the typical effect size in this field was a Pearson’s correlation of .26, and the median sample size was 60. We calculated the power of each primary study by using the c...

This study examined the effects of secrecy on quality of life in a sample consisting of older adults (>50 years; N = 301). Three key components of secrecy were examined with the Tilburg Secrecy Scale-25 (TSS25; possession of a secret, self-concealment, and cognitive preoccupation). The TSS25 distinguishes between the tendency to conceal personal in...

In response to recommendations to redefine statistical significance to P ≤ 0.005, we propose that researchers should transparently report and justify all choices they make when designing a study, including the alpha level.

Purpose: This study aimed to determine the predictive value of the Brazilian Tilburg Frailty Indicator (TFI) for adverse health outcomes (falls, hospitalization, disability and death), in a follow-up period of twelve months. Methods: This longitudinal study was carried out with a sample of people using primary health care services in Rio de Jane...

In this paper, we present three retrospective observational studies that investigate the relation between data sharing and statistical reporting inconsistencies. Previous research found that reluctance to share data was related to a higher prevalence of statistical errors, often in the direction of statistical significance (Wicherts, Bakker, & Mole...

The R package “statcheck” (Epskamp & Nuijten, 2016) is a tool to extract statistical results from articles and check whether the reported p-value matches the accompanying test statistic and degrees of freedom. A previous study showed high interrater reliabilities (between .76 and .89) between statcheck and manual coding of inconsistencies (.76 - .8...

We examined the proportion of p-values (.05<𝑝≤.1) reported as marginally significant in 44,200 articles across 9 psychology disciplines, published in 70 journals belonging to the American Psychological Association (APA) between 1985 and 2016. Using regular expressions we extracted 42,504 p-values between .05 and .1. Almost 40% of p-values between ....

Due to rapidly aging human populations, frailty has become an essential concept, as it identifies older people who have higher risk of adverse outcomes, such as disability, institutionalization, lower quality of life, and premature death. The Tilburg Frailty Indicator (TFI) is a user-friendly questionnaire based on a multidimensional approach to fr...

The unrealistically high rate of positive results within psychology has increased the attention to replication research. However, researchers who conduct a replication and want to statistically combine the results of their replication with a statistically significant original study encounter problems when using traditional meta-analysis techniques....

In response to recommendations to redefine statistical significance to p ≤ .005, we propose that researchers should transparently report and justify all choices they make when designing a study, including the alpha level.

Objectives This study aimed to examine if autonomy-connectedness, capacity for self-governance under the condition of connectedness, would mediate sex differences in symptoms of various mental disorders (depression, anxiety, eating disorders, antisocial personality disorder). Method Participants (N = 5,525) from a representative community sample i...

Questionnaire instructions and items. (DOCX)

In this paper, we present three retrospective observational studies that investigate the relation between data sharing and statistical reporting inconsistencies. Previous research found that reluctance to share data was related to a higher prevalence of statistical errors, often in the direction of statistical significance (Wicherts, Bakker, & Mole...

Purpose: To examine the associations between components of physical, psychological and social frailty with quality of life among older people. Methods: This cross-sectional study was carried out in a sample of Dutch citizens. A total of 671 people aged 70 years or older completed a web-based questionnaire ('the Senioren Barometer'). This questio...

One of the main goals of meta-analysis is to test and estimate the heterogeneity of effect size. We examined the effect of publication bias on the Q-test and assessments of heterogeneity, as a function of true heterogeneity, publication bias, true effect size, number of studies, and variation of sample sizes. The expected values of heterogeneity me...

Research has shown that independent groups often differ not only in their means, but also in their variances. Comparing and testing variances is therefore of crucial importance to understand the effect of a grouping variable on an outcome variable. Researchers may have specific expectations concerning the relations between the variances of multiple...

This study examines how leadership self-efficacy (LSE) influences leaders’ effectiveness. Furthermore, we examine whether the relationship between self-efficacy and leadership effectiveness gets stronger when task complexity decreases. Multisource data were collected from 128 banking leaders (who responded about different aspects of leadership self...

Purpose of the study: Environmental factors play an important role in the quality of life of older people who often have difficulty maintaining physical, psychological, and social functioning. In this study, we aimed at (a) developing a measurement instrument assessing the factors of older adults' perceptions of their environment, (b) examining th...

Purpose The purpose of this paper is to examine how the quality of change information influences employees’ attitude toward organizational change and turnover intention. Additionally, the role of engagement, psychological contract fulfillment and trust in the relationship between change information and attitude toward change is assessed. Design/me...

The vast majority of published results in the literature is statistically significant, which raises concerns about their reliability. The Reproducibility Project Psychology (RPP) and Experimental Economics Replication Project (EE-RP) both replicated a large number of published studies in psychology and economics. The original study and replication...

Correlation coefficients (r), sample sizes (n), and p-values (p) of the original study and replication of the study-pairs in RRP and results of the snapshot hybrid and snapshot naïve at four different snapshots (ρS = 0; 0.1; 0.3; 0.5). (DOCX)

Correlation coefficients (r), sample sizes (n), and p-values (p) of the original study and replication of the study-pairs in EE-RP and results of the snapshot hybrid and snapshot naïve at four different snapshots (ρS = 0; 0.1; 0.3; 0.5). (DOCX)

Meta-analyses are an important tool to evaluate the literature. It is essential that meta-analyses can easily be reproduced to allow researchers to evaluate the impact of subjective choices on meta-analytic effect sizes, but also to update meta-analyses as new data comes in, or as novel statistical techniques (for example to correct for publication...

Publication bias is a substantial problem for the credibility of research in general and of meta-analyses in particular, as it yields overestimated effects and may suggest the existence of non-existing effects. Although there is consensus that publication bias is widespread, how strongly it affects different scientific literatures is currently less...

Due to its probabilistic nature, Null Hypothesis Significance Testing (NHST) is subject to decision errors. The concern for false positives has overshadowed the concern for false negatives in the recent debates in psychology. This might be unwarranted, since reported statistically nonsignificant findings may just be 'too good to be false'. We exami...

Volgens het Sociaal en Cultureel Planbureau (SCP) neemt het aantal kwetsbare ouderen van 65 jaar en ouder in Nederland tussen 2010 en 2030 toe van bijna 700.000 naar meer dan 1 miljoen (Van Campen, 2011). Naarmate mensen ouder worden en kwetsbaarheid toeneemt, brengen zij meer tijd door in huis en in de directe leefomgeving. Voor hen is een geschik...

The unrealistic high rate of positive results within psychology increased the attention for replication research. Researchers who conduct a replication and want to statistically combine the results of their replication with a statistically significant original study encounter problems when using traditional meta-analysis techniques. The original st...

The vast majority of published results in the literature is statistically significant, which raises concerns about their reliability. The Reproducibility Project Psychology (RPP) and Experimental Economics Replication Project (EE-RP) both replicated a large number of published studies in psychology and economics. The original study and replication...

Do lay people and scientists themselves recognize that scientists are human and therefore prone to human fallibilities such as error, bias, and even dishonesty? In a series of three experimental studies and one correlational study (total N = 3,278) we found that the 'storybook image of the scientist' is pervasive: American lay people and scientists...

Do lay people and scientists themselves recognize that scientists are human and therefore prone to human fallibilities such as error, bias, and even dishonesty? In a series of three experimental studies and one correlational study (total N = 3,278) we found that the ‘storybook image of the scientist’ is pervasive: American lay people and scientists...

The designing, collecting, analyzing, and reporting of psychological studies entail many choices that are often arbitrary. The opportunistic use of these so-called researcher degrees of freedom aimed at obtaining statistically significant results is problematic because it enhances the chances of false positive results and may inflate effect size es...

The designing, collecting, analyzing, and reporting of psychological studies entail many choices that are often arbitrary. The opportunistic use of these so-called researcher degrees of freedom aimed at obtaining statistically significant results is problematic because it enhances the chances of false positive results and may inflate effect size es...

Due to its probabilistic nature, Null Hypothesis Significance Testing (NHST) is subject to decision errors. The concern for false positives has overshadowed the concern for false negatives in the recent debates in psychology. This is unwarranted, since reported statistically nonsignificant findings may just be 'too good to be false'. We examined ev...

Explaining cooperation in social dilemmas is a central issue in behavioral science, and the prisoner?s dilemma (PD) is the most frequently employed model. Theories assuming rationality and selfishness predict no cooperation in PDs of finite duration, but cooperation is frequently observed. We therefore build a model of how individuals in a finitely...

Because of overwhelming evidence of publication bias in psychology, techniques to correct meta-analytic estimates for such bias are greatly needed. The methodology on which the p-uniform and p-curve methods are based has great promise for providing accurate meta-analytic estimates in the presence of publication bias. However, in this article, we sh...