About
300
Publications
271,244
Reads
How we measure 'reads'
A 'read' is counted each time someone views a publication summary (such as the title, abstract, and list of authors), clicks on a figure, or views or downloads the full-text. Learn more
69,124
Citations
Publications
Publications (300)
Here we test the feasibility of using decision markets to select studies for replication and provide evidence about the replicability of online experiments. Social scientists (n = 162) traded on the outcome of close replications of 41 systematically selected MTurk social science experiments published in PNAS 2015–2018, knowing that the 12 studies w...
Open science remains vital to the progress and functioning of the global research enterprise. Published in 2015, the Transparency and Openness Promotion Guidelines (TOP 2015) was developed as a policy framework to enhance the verifiability of empirical research claims in journal articles. It has been widely used and adopted by publishers and academ...
Failures to replicate evidence of new discoveries have forced scientists to ask whether this unreliability is due to suboptimal implementation of methods or whether presumptively optimal methods are not, in fact, optimal. This paper reports an investigation by four coordinated laboratories of the prospective replicability of 16 novel experimental f...
Replication is an important “credibility control” mechanism for clarifying the reliability of published findings. However, replication is costly, and it is infeasible to replicate everything. Accurate, fast, lower cost alternatives such as eliciting predictions from experts or novices could accelerate credibility assessment and improve allocation o...
Interpreting a failure to replicate is complicated by the fact that the failure could be due to the original finding being a false positive, unrecognized moderating influences between the original and replication procedures, or faulty implementation of the procedures in the replication. One strategy to maximize replication quality is involving the...
Although replication is a central tenet of science, direct replications are rare in psychology. This research tested variation in the replicability of thirteen classic and contemporary effects across 36 independent samples totaling 6,344 participants. In the aggregate, ten effects replicated consistently. One effect – imagined contact reducing prej...
This manuscript contains our responses to several commentaries about the Many Labs Project (Klein et al., 2014).
This dataset is from the Many Labs Replication Project [1] in which 13 effects were replicated across 36 samples and over 6,000 participants. Data from the replications are included, along with demographic variables about the participants and contextual information about the environment in which the replication was conducted. Data were collected in...
Replication—an important, uncommon, and misunderstood practice—is gaining appreciation in psychology. Achieving replicability is important for making research progress. If findings are not replicable, then prediction and theory development are stifled. If findings are replicable, then interrogation of their meaning and validity can advance knowledg...
We conducted the Reproducibility Project: Cancer Biology to investigate the replicability of preclinical research in cancer biology. The initial aim of the project was to repeat 193 experiments from 53 high-impact papers, using an approach in which the experimental protocols and plans for data analysis had to be peer reviewed and accepted for publi...
Replicability is an important feature of scientific research, but aspects of contemporary research culture, such as an emphasis on novelty, can make replicability seem less important than it should be. The Reproducibility Project: Cancer Biology was set up to provide evidence about the replicability of preclinical research in cancer biology by repe...
Any large dataset can be analyzed in a number of ways, and it is possible that the use of different analysis strategies will lead to different results and conclusions. One way to assess whether the results obtained depend on the analysis strategy chosen is to employ multiple analysts and leave each of them free to follow their own approach. Here, w...
In registered reports (RRs), initial peer review and in-principle acceptance occur before knowing the research outcomes. This combats publication bias and distinguishes planned from unplanned research. How RRs could improve the credibility of research findings is straightforward, but there is little empirical evidence. Also, there could be unintend...
There is evidence that prediction markets are useful tools to aggregate information on researchers' beliefs about scientific results including the outcome of replications. In this study, we use prediction markets to forecast the results of novel experimental designs that test established theories. We set up prediction markets for hypotheses tested...
Assessing the credibility of research claims is a central, continuous, and laborious part of the scientific process. Credibility assessment strategies range from expert judgment to aggregating existing evidence to systematic replication efforts. Such assessments can require substantial time and effort. Research progress could be accelerated if ther...
We present consensus-based guidance for conducting and documenting multi-analyst studies. We discuss why broader adoption of the multi-analyst approach will strengthen the robustness of results and conclusions in empirical sciences.
Replication, an important, uncommon, and misunderstood practice, is making a comeback in psychology. Achieving replicability is a necessary but not sufficient condition for making research progress. If findings are not replicable, then prediction and theory development are stifled. If findings are replicable, then interrogation of their meaning and...
Researchers face many, often seemingly arbitrary, choices in formulating hypotheses, designing protocols, collecting data, analyzing data, and reporting results. Opportunistic use of “researcher degrees of freedom” aimed at obtaining statistical significance increases the likelihood of obtaining and publishing false-positive results and overestimat...
Registered Reports (RRs) is a publishing model in which initial peer review happens before the research is completed. In-principle acceptance before knowing outcomes combats publication bias and provides a clear distinction between confirmatory and exploratory research. The theoretical case for how RRs would improve the credibility of research find...
Replications in psychological science sometimes fail to reproduce prior findings. If replications use methods that are unfaithful to the original study or ineffective in eliciting the phenomenon of interest, then a failure to replicate may be a failure of the protocol rather than a challenge to the original finding. Formal pre-data collection peer...
Preprints increase accessibility and can speed scholarly communication if researchers view them as credible enough to read and use. Preprint services do not provide the heuristic cues of a journal's reputation, selection, and peer-review processes that, regardless of their flaws, are often used as a guide for deciding what to read. We conducted a s...
Failures to replicate evidence of new discoveries have forced scientists to ask whether this unreliability is due to suboptimal implementation of optimal methods or whether presumptively optimal methods are not, in fact, optimal. This paper reports an investigation by four coordinated laboratories of the prospective replicability of 16 novel experi...
To avoid stalemates and provide lessons, replicators and original researchers must reach agreement on a study design and set out expectations ahead of time. To avoid stalemates and provide lessons, replicators and original researchers must reach agreement on a study design and set out expectations ahead of time.
This meta-analysis evaluated theoretical predictions from balanced identity theory (BIT) and evaluated the validity of zero points of Implicit Association Test (IAT) and self-report measures used to test these predictions. Twenty-one researchers contributed individual subject data from 36 experiments (total N = 12,773) that used both explicit and i...
Credibility of scientific claims is established with evidence for their replicability using new data. According to common understanding, replication is repeating a study’s procedure and observing whether the prior finding recurs. This definition is intuitive, easy to apply, and incorrect. We propose that replication is a study for which any outcome...
Preprints increase accessibility and can speed scholarly communication if researchers view them as credible enough to read and use. Preprint services, though, do not provide the heuristic cues of a journal’s reputation, selection, and peer review processes that are often used as a guide for deciding what to read. We conducted a survey of 3,759 rese...
of post-event evaluation (survey and participant interviews) of Metascience 2019 Symposium (https://www.metascience2019.org/), presented by Fetzer Franklin Fund (https://www.fetzer-franklin-fund.org/) September 5th-8th, 2019 at Stanford University.
With funding from the National Science Foundation, the Center for Open Science (COS) and Indiana University will create a dynamic, distributed, and heterogeneous data source for the advancement of science of science research. This will be achieved by using, enhancing, and combining the capabilities of the Open Science Framework (OSF) and the Collab...
We have developed and implemented normative interventions called “Rigor and Transparency Initiatives”(RTIs) to shift attitudes, norms, and behaviors in research communities. RTIs are usually a collaboration between one or more journals and one or more funders involving a funding call for a special section/issue of journals to promote new behaviors....
Preprints (early, complete versions of manuscripts made available online before journal-organized peer review) are shifting the scholarly publishing model by accelerating open access and, potentially, open review. Making these tools interoperable with preprint infrastructure will increase the confidence of the scientific community in preprints. In...
In theory, Registered Reports eliminate publication bias against negative results because publication decisions are made without knowledge of the results; increase clarity between planned (hypothesis testing; confirmatory) and unplanned (hypothesis generating; exploratory) analyses, thereby increasing the diagnosticity of statistical inferences; an...
The Community of Open Scholarship Grassroots Networks (COSGN), includes 120 grassroots networks, representing virtually every region of the world and every research discipline. These networks communicate and coordinate on topics of common interest. We propose, using an NSF 19-501 Full-Scale implementation grant, to formalize governance and coordina...
Preregistration is the act of submitting a study plan, ideally also with analytical plan, to a registry prior to conducting the work. Preregistration increases the discoverability of research even if it does not get published further. Adding specific analysis plans can clarify the distinction between planned, confirmatory tests and unplanned, explo...
We report a randomized trial of a research ethics training intervention designed to enhance ethics communication in university science and engineering laboratories, focusing specifically on authorship and data management. The intervention is a project-based research ethics curriculum that was designed to enhance the ability of science and engineeri...
Interpreting a failure to replicate is complicated by the fact that the failure could be due to the original finding being a false positive, unrecognized moderating influences between the original and replication procedures, or faulty implementation of the procedures in the replication. One strategy to maximize replication quality is involving the...
Serious concerns about the way research is organized collectively are increasingly being raised. They include the escalating costs of research and lower research productivity, low public trust in researchers to report the truth, lack of diversity, poor community engagement, ethical concerns over research practices, and irreproducibility. Open scien...
There is shared support by Riley et al., (RRL1) and Wolfe & Kanwisher (WK2) for the principles to “increase transparency, rigor, and reproducibility of science” and “[fulfill] an inherent commitment to study participants and the public”1. These principles are motivating the expansion of NIH Guidelines requiring study registration and outcome report...
Credibility of scientific claims is established with evidence for their replicability using new data. According to common understanding, replication is repeating a study’s procedure and observing whether the prior finding recurs. This definition is intuitive, easy to apply, and incorrect. We propose that replication is a study for which any outcome...
Preregistration clarifies the distinction between planned and unplanned research by reducing unnoticed flexibility. This improves credibility of findings and calibration of uncertainty. However, making decisions before conducting analyses requires practice. During report writing, respecting both what was planned and what actually happened requires...
Preregistration clarifies the distinction between planned and unplanned research by reducing unnoticed flexibility. This improves credibility of findings and calibration of uncertainty. However, making decisions before conducting analyses requires practice. During report writing, respecting both what was planned and what actually happened requires...
Most scientific research is conducted by small teams of investigators who together formulate hypotheses, collect data, conduct analyses, and report novel findings. These teams operate independently as vertically integrated silos. Here we argue that scientific research that is horizontally distributed can provide substantial complementary value, aim...
Using a novel technique known as network meta-analysis, we synthesized evidence from 492 studies (87,418 participants) to investigate the effectiveness of procedures in changing implicit measures, which we define as response biases on implicit tasks. We also evaluated these procedures’ effects on explicit and behavioral measures. We found that impl...
The Center for Open Science (COS) will create an ECR Data Resource Hub to facilitate rigorous and reproducible research practices such as data sharing and study registration. The Hub will integrate training materials, infrastructure, community engagement, and innovation in research to advance rigorous research skills and behavior across the STEM ed...
The Center for Open Science (COS) proposes to undertake a Mid-scale RI-1 Implementation Project entitled, Expanding open infrastructure for robust, open, reproducible research, primarily in association with the SBE Directorate. Reproducibility is the ability to obtain independent evidence supporting scientific findings. Recent investigations sugges...
Serious concerns about the way research is organized collectively are increasingly being raised. They include the escalating costs of research and lower research productivity, low public trust in researchers to report the truth, lack of diversity, poor community engagement, ethical concerns over research practices, and irreproducibility. Open scien...
Beliefs shape how people interpret information and may impair how people engage in logical reasoning. In three studies, we show how ideological beliefs impair people’s ability to (1) recognize logical validity in arguments that oppose their political beliefs and (2) recognize the lack of logical validity in arguments that support their political be...
In this manuscript, we (a) briefly describe proposed open-science practices to increase transparency of research in special education and related disciplines, and (b) provide recommendations for research funders, professional societies, journal editors and publishers, and individual researchers to support awareness, exploration, and adoption of ope...
In this manuscript, we (a) briefly describe proposed open-science practices to increase transparency of research in special education and related disciplines, and (b) provide recommendations for research funders, professional societies, journal editors and publishers, and individual researchers to support awareness, exploration, and adoption of ope...
Beliefs shape how people interpret information and may impair how people engage in logical reasoning. In 3 studies, we show how ideological beliefs impair people's ability to: (1) recognize logical validity in arguments that oppose their political beliefs, and, (2) recognize the lack of logical validity in arguments that support their political bel...
Social judgment is shaped by multiple biases operating simultaneously, but most bias-reduction interventions target only a single social category. In seven preregistered studies (total N > 7,000), we investigated whether asking participants to avoid one social bias affected that and other social biases. Participants selected honor society applicant...
Understanding and improving reproducibility is crucial for scientific progress. Prediction markets and related methods of eliciting peer beliefs are promising tools to predict replication outcomes. We invited researchers in the field of psychology to judge the replicability of 24 studies replicated in the large scale Many Labs 2 project. We elicite...
This letter is a reply to Ledgerwood's "The Preregistration Revolution Needs to Distinguish between Predictions and Analyses", which commented on our earlier paper, "The Preregistration Revolution" (https://osf.io/2dxu5/).
Understanding and improving reproducibility is crucial for scientific progress. Prediction markets and related methods of eliciting peer beliefs are promising tools to predict replication outcomes. We invited researchers in the field of psychology to judge the replicability of 24 studies replicated in the large scale Many Labs 2 project. We elicite...
Being able to replicate scientific findings is crucial for scientific progress. We replicate 21 systematically selected experimental studies in the social sciences published in Nature and Science between 2010 and 2015. The replications follow analysis plans reviewed by the original authors and pre-registered prior to the replications. The replicati...
Being able to replicate scientific findings is crucial for scientific progress. We replicate 21 systematically selected experimental studies in the social sciences published in Nature and Science between 2010 and 2015. The replications follow analysis plans reviewed by the original authors and pre-registered prior to the replications. The replicati...
Twenty-nine teams involving 61 analysts used the same data set to address the same research question: whether soccer referees are more likely to give red cards to dark-skin-toned players than to light-skin-toned players. Analytic approaches varied widely across the teams, and the estimated effect sizes ranged from 0.89 to 2.93 (Mdn = 1.31) in odds-...
Scientific evidence should guide the selection of practice for individuals with disabilities. Scientific evidence, however, must be trustworthy to move special education toward greater empirical certainty and more effective policies and practices. Transparency, openness, and reproducibility increase the trustworthiness of evidence. We propose that...
Most scientific research is conducted by small teams of researchers, who together formulate hypotheses, collect data, conduct analyses, and report novel findings. These teams are rather closed and operate as vertically integrated silos. Here we argue that scientific research that is horizontally distributed provides substantial complementary value...
Scientific evidence should guide the selection of practice for individuals with disabilities. Scientific evidence, however, must be trustworthy to move special education toward greater empirical certainty and more effective policies and practices. Transparency, openness, and reproducibility increase the trustworthiness of evidence. We propose that...
We examined how felt age and desired age differed from chronological age across the age span. With each passing Earth year, felt and desired age do grow older, it just takes longer for the year to go by. Past age 25 or so, subjective aging appears to occur on Mars, where one Earth decade equals only 5.3 Martian years. In some sense, our minds age m...
Individual differences in general speed lead to a positive correlation between the mean and standard deviation of mean latency. This “coarse” scaling effect causes the mean latency difference (MLD) to be spuriously correlated with general speed. Within individuals, the correlation between the mean and standard deviation of trial latencies leads con...
Progress in science relies in part on generating hypotheses with existing observations and testing hypotheses with new observations. This distinction between postdiction and prediction is appreciated conceptually but is not respected in practice. Mistaking generation of postdictions with testing of predictions reduces the credibility of research fi...
Many areas of social psychological research investigate how social information may bias judgment. However, most measures of social judgment biases are (1) low in reliability because they use a single response, (2) not indicative of individual differences in bias because they use between-subjects designs, (3) inflexible because they are designed for...
In order to increase the replicability of scientific work, the scientific community has called for practices designed to increase the transparency of research (McNutt, 2014; Nosek et al., 2015). The validity of a scientific claim depends not on the reputation of those making the claim, the venue in which the claim is made, or the novelty of the res...
Social judgment is shaped by multiple biases operating simultaneously, but most bias-reduction interventions target only a single social category. In four pre-registered studies (Total N > 4,800), we investigated whether raising awareness of one social bias impacted that and other social biases. Participants selected honor society applicants based...
Registered Reports (RRs) is a publishing model in which initial peer review is conducted prior to knowing the outcomes of the research. In-principle acceptance of papers at this review stage combats publication bias, and provides a clear distinction between confirmatory and exploratory research. Some editors raise a practical concern about adopting...
We propose to change the default P-value threshold for statistical significance for claims of new discoveries from 0.05 to 0.005.
Implicit prejudice are social preferences that exist outside of conscious awareness or conscious control. We summarize evidence for three mechanisms that influence the expression of implicit prejudice: associative change, contextual change, and change in control over implicit prejudice. We then review the evidence (or lack thereof) for five open is...
"We propose to change the default P-value threshold forstatistical significance for claims of new discoveries from 0.05 to 0.005."
Here, we advance the ideological migration hypothesis — individuals choose to live in communities with ideologies similar to their own to satisfy their need to belong. In Study 1, incongruity between personal and community ideology predicted greater residential mobility and attraction to more ideologically-congruent communities. In Study 2, partici...
This proposal is a response to NIH's call for creation of a Data Commons (RM-17-026). The Commons must support use cases of many stakeholders who need access to scholarly process, content, and outcomes in pursuit of knowledge. Moreover, the Commons must be flexible enough to respect researchers’ idiosyncratic workflows, yet specific enough to solve...
Progress in science relies on generating hypotheses with existing observations and testing hypotheses with new observations. This distinction between postdiction and prediction is appreciated conceptually, but is not respected in practice. Mistaking generation of postdictions with testing of predictions reduces the credibility of research findings....
In their review of validity of Implicit Association Test and affective priming, De Houwer, Teige-Mocigemba, Spruyt, and Moors (2009) identified validity with establishment of “basic theoretical understanding” of the measures. We agree that theoretical understanding has an important role in making measures more valid and useful. Nevertheless, we con...
Co-occurrence of an object and affective stimuli does not always mean that the object and the stimuli are the same valence (e.g., false accusations that Richard is a crook). Contemporary theory posits that information about the (in)validity of co-occurrence has stronger influence on deliberate than automatic evaluation. However, available evidence...
Shifting the current research paradigm in ecology and evolutionary biology (EVE) to one that values replication of previously published studies requires both educating the EVE community about their importance and providing the incentives to carry them out. Academic societies are uniquely poised to do both, which is why we propose that the Associati...
The Center for Open Science (COS) and technical partners propose to extend its open infrastructure to support ASAPbio's call for a central service. COS's approach is community based and is accompanied by letters of support from 12 preprint services (e.g., arXiv, bioRxiv, SocArXiv), 15 data repositories (e.g., Dryad, Protein Data Bank, figshare), an...
Twenty-nine teams involving 61 analysts used the same dataset to address the same research question: whether soccer referees are more likely to give red cards to dark skin toned players than light skin toned players. Analytic approaches varied widely across teams, and estimated effect sizes ranged from 0.89 to 2.93 in odds ratio units, with a media...
About 70% of more than half a million Implicit Association Tests completed by citizens of 34 countries revealed expected implicit stereotypes associating science with males more than with females. We discovered that nation-level implicit stereotypes predicted nation-level sex differences in 8th-grade science and mathematics achievement. Self-report...
Many Labs 3 (Ebersole et al., 2016) failed to replicate a classic finding from the Elaboration Likelihood Model of persuasion (Cacioppo, Petty, & Morris, 1983; Study 1). Petty and Cacioppo (2016) noted possible limitations of the Many Labs 3 replication (Ebersole et al., 2016) based on the cumulative literature. Luttrell, Petty, and Xu (2017) subje...