Marjan Bakker

• Tilburg University

For a replication to be informative, measurement should be reliable and valid in bothoriginal and replication studies. Recent studies have identified problems with measurementand measurement reporting in both original and replication studies, although theassociation between measurement and replicability remains unclear. We investigated thereliabili...

This article is about study preregistration, the process of publicly sharing the design, the hypotheses, and the analysis plans of a scientific study before conducting that study. This is said to be important because of two reasons. First, researchers are prone to confirmation bias, the phenomenon that people interpret new information as confirmati...

Study preregistration is one of several “open science” practices (e.g., open data, preprints) that researchers use to improve the transparency and rigour of their research. As more researchers adopt preregistration as a regular practice, examining the nature and content of preregistrations can help identify the strengths and weaknesses of current p...

Objective. Surviving physical trauma can have a large impact on one’s daily life. Patients are at increased risk for poor physical health, psychological complaints, and problems in role functioning – which is often experienced simultaneously. The present study explores the interconnectedness of physical, psychological, and role functioning during t...

Preregistration has gained traction as one of the most promising solutions to improve the replicability of scientific effects. In this project, we compared 193 psychology studies that earned a Preregistration Challenge prize or preregistration badge to 193 related studies that were not preregistered. In contrast to our theoretical expectations and...

In this study, we assessed the extent of selective hypothesis reporting in psychological research by comparing the hypotheses found in a set of 459 preregistrations with the hypotheses found in the corresponding articles. We found that more than half of the preregistered studies we assessed contained omitted hypotheses ( N = 224; 52%) or added hypo...

Theoretical arguments and empirical investigations indicate that a high proportion of published findings do not replicate and are likely false. The current position paper provides a broad perspective on scientific error, which may lead to replication failures. This broad perspective focuses on reform history and on opportunities for future reform....

Introduction Worldwide, annually more than 800,000 suicides occur. In the Netherlands, suicide rates rose from 8.6 per 100,000 in 2007 to 11.4 per 100,000 in 2016. Rates in the province of Noord-Brabant were consistently higher than the national average. Noord-Brabant is a province in the south of the Netherlands covering an area of over 4700 km2 w...

Purpose A barrier for reemployment of people with mental health issues/mental illness (MHI) is workplace stigma and discrimination. In this RCT the effectiveness of a stigma-awareness intervention addressing finding work, retaining work and decisional stress were evaluated. Methods A cluster RCT was conducted in 8 Dutch municipal practices. Random...

Introduction Workplace mental health stigma is a major problem as it can lead to adverse occupational outcomes and reduced well-being. Although workplace climate is largely determined by managers and co-workers, the role of co-workers in workplace stigma is understudied. Therefore, the aims are: (1) to examine knowledge and attitudes towards having...

Measurement error (imperfect reliability) is present in any empirical effect size estimate and systematically attenuates observed effect sizes compared to true underlying effect sizes. Yet there exist broad concerns that proper measurement tends to be neglected in much of psychological research. We examined how measurement error in primary studies...

Samenvatting Dit artikel geeft een overzicht van effectieve suïcidepreventie-interventies en hun betekenis voor het Nederlandse werkveld en de Landelijke Agenda Suïcidepreventie. We bespreken een in 2020 gepubliceerde systematische review die het effect van preventieve interventies op suïcides en suïcidepogingen evalueerde, en geven een update van...

Objective: We evaluated the effect of a digitally supported systems intervention for suicide prevention (SUPREMOCOL) in Noord-Brabant, the Netherlands. Method: Non-randomized stepped wedge trial design (SWTD). Stepwise implementation in the five subregions of the systems intervention. Pre-post analysis for the whole province (Exact Rate Ratio Te...

Preregistration has gained traction as one of the most promising solutions to improve the replicability of scientific effects. In this project, we compared 193 psychology studies that earned a Preregistration Challenge prize or Preregistration Badge to 193 related studies that were not preregistered. In contrast with our theoretical expectations an...

Researcher degrees of freedom refer to arbitrary decisions in the execution and reporting of hypothesis-testing research that allow for many possible outcomes from a single study. Selective reporting of results (p-hacking) from this "multiverse" of outcomes can inflate effect size estimates and false positive rates. We studied the effects of resear...

Study preregistration has become increasingly popular in psychology, but its effectiveness in restricting potentially biasing researcher degrees of freedom remains unclear. We used an extensive protocol to assess the strictness of preregistrations and the consistency between preregistration and publications of 300 preregistered psychology studies....

Employing two vignette studies, we examined how psychology researchers interpret the results of a set of four experiments that all test a given theory. In both studies, we found that participants’ belief in the theory increased with the number of statistically significant results, and that the result of a direct replication had a stronger effect on...

The meta-plot is a descriptive visual tool for meta-analysis that provides information on the primary studies in the meta-analysis and the results of the meta-analysis. More precisely, the meta-plot portrays (1) the precision and statistical power of the primary studies in the meta-analysis, (2) the estimate and confidence interval of a random-effe...

Background Internalized stigma can have numerous negative effects on the well-being and employment of people with mental illness. Brief, valid, and reliable measures are needed to get a better understanding of self-stigmatization. The aim of this study is to translate the brief version of the Internalized Stigma of Mental Illness (ISMI-10) scale in...

This study assesses the extent of selective hypothesis reporting in psychological research by comparing the hypotheses found in a set of 459 preregistrations to the hypotheses found in the corresponding papers. We found that more than half of the preregistered studies we assessed contain omitted hypotheses (N = 224; 52.2%) or added hypotheses (N =...

Theoretical arguments and empirical investigations indicate that a high proportion of published findings are false or do not replicate. The current position paper provides a broad perspective on this scientific error, focusing both on reform history and on opportunities for future reform. Talking points are organised along four main themes: methodo...

The meta-plot is a descriptive visual tool for meta-analysis that provides information on the primary studies in the meta-analysis and the results of the meta-analysis. More precisely, the meta-plot portrays (i) the precision and statistical power of the primary studies in the meta-analysis, (ii) the estimate and confidence interval of a random-eff...

A preregistration of a study in which we update the study by Marszalek et al. (2011). We look at the development of sample sizes over time and the effect of the reproducibility crisis and journal level policies. The preregistration can be found at: https://osf.io/zsjf4

Objective To obtain more insight into the patterns of co-occurring symptoms, biomarkers and predictors in Somatic Symptom Disorders and Related Disorders (SSRD) and to identify subgroups with profiles that might allow for personalised treatment. Methods Cross-sectional study design with Latent class analysis (LCA) to determine different subgroups...

Purpose The decision whether to disclose mental illness at work can have important positive and negative consequences for sustainable employment and well-being . The aim of the study is (1) to examine workers’ expectations of outcomes of mental illness disclosure in the workplace and to evaluate their expectations regarding which factors are of inf...

Preregistration has been lauded as one of the solutions to the so-called ‘crisis of confidence’ in the social sciences and has therefore gained popularity in recent years. However, the current guidelines for preregistration have been developed primarily for studies where new data will be collected. Yet, preregistering secondary data analyses--- whe...

Purpose Studies are increasingly showing that health related stigma is a barrier to employment, but it is not known how. The aim of this systematic review is to identify, appraise and analyse studies that have directly or indirectly addressed ways in which stigma affects sustainable employment and well-being at work of people with disabilities. Me...

Study preregistration is one of several “open science” practices (e.g., open data, preprints) that researchers use to improve the transparency and rigour of their research. As more researchers adopt preregistration as a regular research practice, examining the nature and content of preregistrations can help identify strengths and weaknesses of curr...

Researcher degrees of freedom refer to arbitrary decisions in the execution and reporting of hypothesis-testing research that many possible outcomes from a single study. Selective reporting of results (p-hacking) from this ‘multiverse’ of outcomes can inflate effect size estimates and false positive rates. We studied the effects of researcher degre...

Aims Stereotactic radiosurgery is increasingly used to treat multiple (four or more) brain metastases. Preserving cognitive functions is a highly relevant treatment goal because cognitive deteriorations may negatively affect a patient's quality of life. The aim of this study was to assess cognitive change, at the group and individual level, in pati...

Background Recent studies suggest a relationship between the APOE ε4 allele and cognitive outcome in patients treated for malignant brain tumors. Still, longitudinal investigations that include a pre‐treatment cognitive assessment are lacking and APOE’s effects in patients with benign tumors are understudied. This study investigated pre‐surgical co...

Researchers face many, often seemingly arbitrary, choices in formulating hypotheses, designing protocols, collecting data, analyzing data, and reporting results. Opportunistic use of “researcher degrees of freedom” aimed at obtaining statistical significance increases the likelihood of obtaining and publishing false-positive results and overestimat...

Background Interleukin 6 (IL-6) and high-sensitivity C-reactive protein (hsCRP) are biomarkers of systemic low-grade inflammation (SLI) in depression and anxiety. The question if SLI in those conditions is related to comorbid chronic medical conditions has not been resolved. DSM-5 Somatic symptom disorders and related disorders (SSRD) are condition...

Objective To examine the association between the publication and content of suicide-related media reports and actual suicide in Noord Brabant, a province of the Netherlands. Method Between April 2017 and March 2018, a retrospective cross-sectional observational study was conducted on suicide-related media reports and incident data regarding suicid...

In this preregistered study, we investigated whether the statistical power of a study is higher when researchers are asked to make a formal power analysis before collecting data. We compared the sample size descriptions from two sources: (i) a sample of pre-registrations created according to the guidelines for the Center for Open Science Preregistr...

Preregistration has been lauded as one of the solutions to the so-called ‘crisis of confidence’ in the social sciences and has therefore gained in popularity in recent years. However, despite its growing popularity, preregistration is still in its infancy and preregistration practices are far from optimal. Moreover, the current infrastructure for p...

Background: In the Netherlands, suicide rates showed a sharp incline and this pertains particularly to the province of Noord-Brabant, one of the southern provinces in the Netherlands. This calls for a regional suicide prevention effort. Methods/design: Study protocol. A regional suicide prevention systems intervention is implemented and evaluate...

Purpose Fatigue is a common and distressing symptom in cancer patients which negatively affects patients’ daily functioning and health-related quality of life. The aim of this study was to assess multidimensional fatigue in patients with brain metastases (BM) before, and after Gamma Knife radiosurgery (GKRS). Methods Patients with BM, an expected...

In this preregistered study, we investigated whether the statistical power of a study is higher when researchers are asked to make a formal power analysis before collecting data. We compared the sample size descriptions from two sources: (i) a sample of preregistrations created according to the guidelines for the Center for Open Science Preregistra...

Objective: This study provides an estimate of the effect size of suicide prevention interventions and evaluates the possible synergistic effects of multilevel interventions. Method: A systematic review and meta-analysis were conducted of controlled studies evaluating suicide prevention interventions versus control published between 2011 and 2017...

Background Meningioma patients are known to face cognitive deficits before and after surgery. We examined individual changes in cognitive performance over time and identified preoperative predictors of cognitive functioning 12 months after surgery in a large sample of meningioma patients. Methods Patients underwent neuropsychological assessment (NP...

Background: Meningioma patients are known to face cognitive deficits before and after surgery. We examined individual changes in cognitive performance over time and identified preoperative predictors of cognitive functioning 12 months after surgery in a large sample of meningioma patients. Methods: Patients underwent neuropsychological assessmen...

We studied how academics assess the results of a set of four experiments that all test a given theory. We found that participants’ belief in the theory increases with the number of significant results, and that direct replications were considered to be more important than conceptual replications. We found no difference between authors and reviewers...

Background Meningioma patients are faced with deficits in several cognitive domains before and after surgery. We examined individual changes in cognitive performance over time and identified pre-operative predictors of cognitive functioning 12 months after surgery in a large sample of meningioma patients. Material and Methods Patients underwent ne...

In determining the need to directly replicate, it is crucial to first verify the original results through independent reanalysis of the data. Original results that appear erroneous and that cannot be reproduced by reanalysis offer little evidence to begin with, thereby diminishing the need to replicate. Sharing data and scripts is essential to ensu...

Background Time trends are one of the most studied phenomena in suicide research; however, evidence for time trends in the Dutch population remains understudied. Insight into time trends can contribute to the development of effective suicide prevention strategies. Methods Time trends in national daily and monthly data of 33,224 suicide events that...

In determining the need to directly replicate, it is crucial to first verify the original results through independent reanalysis of the data. Original results that appear erroneous and that cannot be reproduced by reanalysis offer little evidence to begin with, thereby diminishing the need to replicate. Sharing data and scripts is essential to ensu...

Publication bias and questionable research practices have long been known to corrupt the published record. One method to assess the extent of this corruption is to examine the meta-analytic collection of significant p values, the so-called p-curve (Simonsohn, Nelson, & Simmons, 2014a). Inspired by statistical research on false-discovery rates, we p...

The designing, collecting, analyzing, and reporting of psychological studies entail many choices that are often arbitrary. The opportunistic use of these so-called researcher degrees of freedom aimed at obtaining statistically significant results is problematic because it enhances the chances of false positive results and may inflate effect size es...

The designing, collecting, analyzing, and reporting of psychological studies entail many choices that are often arbitrary. The opportunistic use of these so-called researcher degrees of freedom aimed at obtaining statistically significant results is problematic because it enhances the chances of false positive results and may inflate effect size es...

Many psychology studies are statistically underpowered. In part, this may be because many researchers rely on intuition, rules of thumb, and prior practice (along with practical considerations) to determine the number of subjects to test. In Study 1, we surveyed 291 published research psychologists and found large discrepancies between their report...

The amount of data available in the context of educational measurement has vastly increased in recent years. Such data are often incomplete, involve tests administered at different time points and during the course of many years, and can therefore be quite challenging to model. In addition, intermediate results like grades or report cards being ava...

The power fallacy refers to the misconception that what holds on average -across an ensemble of hypothetical experiments- also holds for each case individually. According to the fallacy, high-power experiments always yield more informative data than do low-power experiments. Here we expose the fallacy with concrete examples, demonstrating that a pa...

Background The removal of outliers to acquire a significant result is a questionable research practice that appears to be commonly used in psychology. In this study, we investigated whether the removal of outliers in psychology papers is related to weaker evidence (against the null hypothesis of no effect), a higher prevalence of reporting errors,...

In psychology, outliers are often excluded before running an independent samples t test, and data are often nonnormal because of the use of sum scores based on tests and questionnaires. This article concerns the handling of outliers in the context of independent samples t tests applied to nonnormal sum scores. After reviewing common practice, we pr...

Broken windows theory states that cues of inappropriate behavior like litter or graffiti amplify norm-violating behavior. In a series of quasi-experiments, Keizer, Lindenberg, and Steg altered cues of inappropriate behavior in public places and observed how many passersby subsequently violated norms. They concluded that particular norm violations t...

We welcome the recommendations suggested by Asendorpf et al. Their proposed changes will undoubtedly improve psychology as an academic discipline. However, our current knowledge is based on past research. We therefore have an obligation to 'dwell on the past'; that is, to investigate the veracity of previously published findings-particularly those...

If science were a game, a dominant rule would probably be to collect results that are statistically significant. Several reviews of the psychological literature have shown that around 96% of papers involving the use of null hypothesis significance testing report significant outcomes for their main results but that the typical studies are insufficie...

With the emergence of online publishing, opportunities to maximize transparency of scientific research have grown considerably. However, these possibilities are still only marginally used. We argue for the implementation of (1) peer-reviewed peer review, (2) transparent editorial hierarchies, and (3) online data publication. First, peer-reviewed pe...

The authors argue that upon publication of a paper, the data should be made available through online archives or repositories. Reasons for not sharing data are discussed and contrasted with advantages of sharing, which include abiding by the scientific principle of openness, keeping the data for posterity, increasing one's impact, facilitation of s...

E-mail sent in 2005 to request data from published papers. (DOC)

Consideration of potential differences in statistical power between the two types of papers. (DOC)

The widespread reluctance to share published research data is often hypothesized to be due to the authors' fear that reanalysis may expose errors in their work or may produce conclusions that contradict their own. However, these hypotheses have not previously been studied systematically. We related the reluctance to share research data for reanalys...

In order to study the prevalence, nature (direction), and causes of reporting errors in psychology, we checked the consistency of reported test statistics, degrees of freedom, and p values in a random sample of high- and low-impact psychology journals. In a second study, we established the generality of reporting errors in a random sample of recent...

In this paper we elaborate on the potential of the lmer function from the lme4 package in R for item response (IRT) modeling. In line with the package, an IRT framework is described based on generalized linear mixed modeling. The aspects of the framework refer to (a) the kind of covariates -- their mode (person, item, person-by-item), and their bei...