Brian Nosek

Brian Nosek
University of Virginia | UVa · Department of Psychology

About

292
Publications
215,868
Reads
How we measure 'reads'
A 'read' is counted each time someone views a publication summary (such as the title, abstract, and list of authors), clicks on a figure, or views or downloads the full-text. Learn more
52,987
Citations

Publications

Publications (292)
Article
Interpreting a failure to replicate is complicated by the fact that the failure could be due to the original finding being a false positive, unrecognized moderating influences between the original and replication procedures, or faulty implementation of the procedures in the replication. One strategy to maximize replication quality is involving the...
Preprint
Although replication is a central tenet of science, direct replications are rare in psychology. This research tested variation in the replicability of thirteen classic and contemporary effects across 36 independent samples totaling 6,344 participants. In the aggregate, ten effects replicated consistently. One effect – imagined contact reducing prej...
Preprint
This manuscript contains our responses to several commentaries about the Many Labs Project (Klein et al., 2014).
Preprint
This dataset is from the Many Labs Replication Project [1] in which 13 effects were replicated across 36 samples and over 6,000 participants. Data from the replications are included, along with demographic variables about the participants and contextual information about the environment in which the replication was conducted. Data were collected in...
Article
Replication—an important, uncommon, and misunderstood practice—is gaining appreciation in psychology. Achieving replicability is important for making research progress. If findings are not replicable, then prediction and theory development are stifled. If findings are replicable, then interrogation of their meaning and validity can advance knowledg...
Article
Full-text available
We conducted the Reproducibility Project: Cancer Biology to investigate the replicability of preclinical research in cancer biology. The initial aim of the project was to repeat 193 experiments from 53 high-impact papers, using an approach in which the experimental protocols and plans for data analysis had to be peer reviewed and accepted for publi...
Article
Full-text available
Replicability is an important feature of scientific research, but aspects of contemporary research culture, such as an emphasis on novelty, can make replicability seem less important than it should be. The Reproducibility Project: Cancer Biology was set up to provide evidence about the replicability of preclinical research in cancer biology by repe...
Article
Full-text available
Any large dataset can be analyzed in a number of ways, and it is possible that the use of different analysis strategies will lead to different results and conclusions. One way to assess whether the results obtained depend on the analysis strategy chosen is to employ multiple analysts and leave each of them free to follow their own approach. Here, w...
Article
Full-text available
In registered reports (RRs), initial peer review and in-principle acceptance occur before knowing the research outcomes. This combats publication bias and distinguishes planned from unplanned research. How RRs could improve the credibility of research findings is straightforward, but there is little empirical evidence. Also, there could be unintend...
Article
Full-text available
There is evidence that prediction markets are useful tools to aggregate information on researchers' beliefs about scientific results including the outcome of replications. In this study, we use prediction markets to forecast the results of novel experimental designs that test established theories. We set up prediction markets for hypotheses tested...
Preprint
Full-text available
Assessing the credibility of research claims is a central, continuous, and laborious part of the scientific process. Credibility assessment strategies range from expert judgment to aggregating existing evidence to systematic replication efforts. Such assessments can require substantial time and effort. Research progress could be accelerated if ther...
Preprint
Full-text available
We present consensus-based guidance for conducting and documenting multi-analyst studies. We discuss why broader adoption of the multi-analyst approach will strengthen the robustness of results and conclusions in empirical sciences.
Preprint
Replication, an important, uncommon, and misunderstood practice, is making a comeback in psychology. Achieving replicability is a necessary but not sufficient condition for making research progress. If findings are not replicable, then prediction and theory development are stifled. If findings are replicable, then interrogation of their meaning and...
Article
Full-text available
Researchers face many, often seemingly arbitrary, choices in formulating hypotheses, designing protocols, collecting data, analyzing data, and reporting results. Opportunistic use of “researcher degrees of freedom” aimed at obtaining statistical significance increases the likelihood of obtaining and publishing false-positive results and overestimat...
Preprint
Registered Reports (RRs) is a publishing model in which initial peer review happens before the research is completed. In-principle acceptance before knowing outcomes combats publication bias and provides a clear distinction between confirmatory and exploratory research. The theoretical case for how RRs would improve the credibility of research find...
Preprint
Full-text available
Replications in psychological science sometimes fail to reproduce prior findings. If replications use methods that are unfaithful to the original study or ineffective in eliciting the phenomenon of interest, then a failure to replicate may be a failure of the protocol rather than a challenge to the original finding. Formal pre-data collection peer...
Article
Full-text available
Preprints increase accessibility and can speed scholarly communication if researchers view them as credible enough to read and use. Preprint services do not provide the heuristic cues of a journal's reputation, selection, and peer-review processes that, regardless of their flaws, are often used as a guide for deciding what to read. We conducted a s...
Preprint
Failures to replicate evidence of new discoveries have forced scientists to ask whether this unreliability is due to suboptimal implementation of optimal methods or whether presumptively optimal methods are not, in fact, optimal. This paper reports an investigation by four coordinated laboratories of the prospective replicability of 16 novel experi...
Article
To avoid stalemates and provide lessons, replicators and original researchers must reach agreement on a study design and set out expectations ahead of time. To avoid stalemates and provide lessons, replicators and original researchers must reach agreement on a study design and set out expectations ahead of time.
Article
This meta-analysis evaluated theoretical predictions from balanced identity theory (BIT) and evaluated the validity of zero points of Implicit Association Test (IAT) and self-report measures used to test these predictions. Twenty-one researchers contributed individual subject data from 36 experiments (total N = 12,773) that used both explicit and i...
Article
Full-text available
Credibility of scientific claims is established with evidence for their replicability using new data. According to common understanding, replication is repeating a study’s procedure and observing whether the prior finding recurs. This definition is intuitive, easy to apply, and incorrect. We propose that replication is a study for which any outcome...
Preprint
Preprints increase accessibility and can speed scholarly communication if researchers view them as credible enough to read and use. Preprint services, though, do not provide the heuristic cues of a journal’s reputation, selection, and peer review processes that are often used as a guide for deciding what to read. We conducted a survey of 3,759 rese...
Preprint
of post-event evaluation (survey and participant interviews) of Metascience 2019 Symposium (https://www.metascience2019.org/), presented by Fetzer Franklin Fund (https://www.fetzer-franklin-fund.org/) September 5th-8th, 2019 at Stanford University.
Preprint
With funding from the National Science Foundation, the Center for Open Science (COS) and Indiana University will create a dynamic, distributed, and heterogeneous data source for the advancement of science of science research. This will be achieved by using, enhancing, and combining the capabilities of the Open Science Framework (OSF) and the Collab...
Preprint
We have developed and implemented normative interventions called “Rigor and Transparency Initiatives”(RTIs) to shift attitudes, norms, and behaviors in research communities. RTIs are usually a collaboration between one or more journals and one or more funders involving a funding call for a special section/issue of journals to promote new behaviors....
Preprint
Preprints (early, complete versions of manuscripts made available online before journal-organized peer review) are shifting the scholarly publishing model by accelerating open access and, potentially, open review. Making these tools interoperable with preprint infrastructure will increase the confidence of the scientific community in preprints. In...
Preprint
In theory, Registered Reports eliminate publication bias against negative results because publication decisions are made without knowledge of the results; increase clarity between planned (hypothesis testing; confirmatory) and unplanned (hypothesis generating; exploratory) analyses, thereby increasing the diagnosticity of statistical inferences; an...
Preprint
Full-text available
The Community of Open Scholarship Grassroots Networks (COSGN), includes 120 grassroots networks, representing virtually every region of the world and every research discipline. These networks communicate and coordinate on topics of common interest. We propose, using an NSF 19-501 Full-Scale implementation grant, to formalize governance and coordina...
Preprint
Preregistration is the act of submitting a study plan, ideally also with analytical plan, to a registry prior to conducting the work. Preregistration increases the discoverability of research even if it does not get published further. Adding specific analysis plans can clarify the distinction between planned, confirmatory tests and unplanned, explo...
Article
Full-text available
We report a randomized trial of a research ethics training intervention designed to enhance ethics communication in university science and engineering laboratories, focusing specifically on authorship and data management. The intervention is a project-based research ethics curriculum that was designed to enhance the ability of science and engineeri...
Preprint
Interpreting a failure to replicate is complicated by the fact that the failure could be due to the original finding being a false positive, unrecognized moderating influences between the original and replication procedures, or faulty implementation of the procedures in the replication. One strategy to maximize replication quality is involving the...
Article
Full-text available
Serious concerns about the way research is organized collectively are increasingly being raised. They include the escalating costs of research and lower research productivity, low public trust in researchers to report the truth, lack of diversity, poor community engagement, ethical concerns over research practices, and irreproducibility. Open scien...
Preprint
There is shared support by Riley et al., (RRL1) and Wolfe & Kanwisher (WK2) for the principles to “increase transparency, rigor, and reproducibility of science” and “[fulfill] an inherent commitment to study participants and the public”1. These principles are motivating the expansion of NIH Guidelines requiring study registration and outcome report...
Preprint
Credibility of scientific claims is established with evidence for their replicability using new data. According to common understanding, replication is repeating a study’s procedure and observing whether the prior finding recurs. This definition is intuitive, easy to apply, and incorrect. We propose that replication is a study for which any outcome...
Preprint
Preregistration clarifies the distinction between planned and unplanned research by reducing unnoticed flexibility. This improves credibility of findings and calibration of uncertainty. However, making decisions before conducting analyses requires practice. During report writing, respecting both what was planned and what actually happened requires...
Article
Full-text available
Using a novel technique known as network meta-analysis, we synthesized evidence from 492 studies (87,418 participants) to investigate the effectiveness of procedures in changing implicit measures, which we define as response biases on implicit tasks. We also evaluated these procedures’ effects on explicit and behavioral measures. We found that impl...
Article
Preregistration clarifies the distinction between planned and unplanned research by reducing unnoticed flexibility. This improves credibility of findings and calibration of uncertainty. However, making decisions before conducting analyses requires practice. During report writing, respecting both what was planned and what actually happened requires...
Article
Full-text available
Most scientific research is conducted by small teams of investigators who together formulate hypotheses, collect data, conduct analyses, and report novel findings. These teams operate independently as vertically integrated silos. Here we argue that scientific research that is horizontally distributed can provide substantial complementary value, aim...
Preprint
The Center for Open Science (COS) will create an ECR Data Resource Hub to facilitate rigorous and reproducible research practices such as data sharing and study registration. The Hub will integrate training materials, infrastructure, community engagement, and innovation in research to advance rigorous research skills and behavior across the STEM ed...
Preprint
The Center for Open Science (COS) proposes to undertake a Mid-scale RI-1 Implementation Project entitled, Expanding open infrastructure for robust, open, reproducible research, primarily in association with the SBE Directorate. Reproducibility is the ability to obtain independent evidence supporting scientific findings. Recent investigations sugges...
Article
Full-text available
Serious concerns about the way research is organized collectively are increasingly being raised. They include the escalating costs of research and lower research productivity, low public trust in researchers to report the truth, lack of diversity, poor community engagement, ethical concerns over research practices, and irreproducibility. Open scien...
Article
Beliefs shape how people interpret information and may impair how people engage in logical reasoning. In three studies, we show how ideological beliefs impair people’s ability to (1) recognize logical validity in arguments that oppose their political beliefs and (2) recognize the lack of logical validity in arguments that support their political be...
Preprint
Full-text available
In this manuscript, we (a) briefly describe proposed open-science practices to increase transparency of research in special education and related disciplines, and (b) provide recommendations for research funders, professional societies, journal editors and publishers, and individual researchers to support awareness, exploration, and adoption of ope...
Preprint
Full-text available
In this manuscript, we (a) briefly describe proposed open-science practices to increase transparency of research in special education and related disciplines, and (b) provide recommendations for research funders, professional societies, journal editors and publishers, and individual researchers to support awareness, exploration, and adoption of ope...
Preprint
Beliefs shape how people interpret information and may impair how people engage in logical reasoning. In 3 studies, we show how ideological beliefs impair people's ability to: (1) recognize logical validity in arguments that oppose their political beliefs, and, (2) recognize the lack of logical validity in arguments that support their political bel...
Article
Social judgment is shaped by multiple biases operating simultaneously, but most bias-reduction interventions target only a single social category. In seven preregistered studies (total N > 7,000), we investigated whether asking participants to avoid one social bias affected that and other social biases. Participants selected honor society applicant...
Preprint
Understanding and improving reproducibility is crucial for scientific progress. Prediction markets and related methods of eliciting peer beliefs are promising tools to predict replication outcomes. We invited researchers in the field of psychology to judge the replicability of 24 studies replicated in the large scale Many Labs 2 project. We elicite...
Preprint
This letter is a reply to Ledgerwood's "The Preregistration Revolution Needs to Distinguish between Predictions and Analyses", which commented on our earlier paper, "The Preregistration Revolution" (https://osf.io/2dxu5/).
Article
Full-text available
Understanding and improving reproducibility is crucial for scientific progress. Prediction markets and related methods of eliciting peer beliefs are promising tools to predict replication outcomes. We invited researchers in the field of psychology to judge the replicability of 24 studies replicated in the large scale Many Labs 2 project. We elicite...
Article
Full-text available
Being able to replicate scientific findings is crucial for scientific progress. We replicate 21 systematically selected experimental studies in the social sciences published in Nature and Science between 2010 and 2015. The replications follow analysis plans reviewed by the original authors and pre-registered prior to the replications. The replicati...
Preprint
Being able to replicate scientific findings is crucial for scientific progress. We replicate 21 systematically selected experimental studies in the social sciences published in Nature and Science between 2010 and 2015. The replications follow analysis plans reviewed by the original authors and pre-registered prior to the replications. The replicati...
Article
Full-text available
Twenty-nine teams involving 61 analysts used the same data set to address the same research question: whether soccer referees are more likely to give red cards to dark-skin-toned players than to light-skin-toned players. Analytic approaches varied widely across the teams, and the estimated effect sizes ranged from 0.89 to 2.93 (Mdn = 1.31) in odds-...
Article
Scientific evidence should guide the selection of practice for individuals with disabilities. Scientific evidence, however, must be trustworthy to move special education toward greater empirical certainty and more effective policies and practices. Transparency, openness, and reproducibility increase the trustworthiness of evidence. We propose that...
Preprint
Most scientific research is conducted by small teams of researchers, who together formulate hypotheses, collect data, conduct analyses, and report novel findings. These teams are rather closed and operate as vertically integrated silos. Here we argue that scientific research that is horizontally distributed provides substantial complementary value...
Preprint
Full-text available
Scientific evidence should guide the selection of practice for individuals with disabilities. Scientific evidence, however, must be trustworthy to move special education toward greater empirical certainty and more effective policies and practices. Transparency, openness, and reproducibility increase the trustworthiness of evidence. We propose that...
Preprint
We examined how felt age and desired age differed from chronological age across the age span. With each passing Earth year, felt and desired age do grow older, it just takes longer for the year to go by. Past age 25 or so, subjective aging appears to occur on Mars, where one Earth decade equals only 5.3 Martian years. In some sense, our minds age m...
Preprint
Individual differences in general speed lead to a positive correlation between the mean and standard deviation of mean latency. This “coarse” scaling effect causes the mean latency difference (MLD) to be spuriously correlated with general speed. Within individuals, the correlation between the mean and standard deviation of trial latencies leads con...
Article
Progress in science relies in part on generating hypotheses with existing observations and testing hypotheses with new observations. This distinction between postdiction and prediction is appreciated conceptually but is not respected in practice. Mistaking generation of postdictions with testing of predictions reduces the credibility of research fi...
Article
Many areas of social psychological research investigate how social information may bias judgment. However, most measures of social judgment biases are (1) low in reliability because they use a single response, (2) not indicative of individual differences in bias because they use between-subjects designs, (3) inflexible because they are designed for...
Preprint
In order to increase the replicability of scientific work, the scientific community has called for practices designed to increase the transparency of research (McNutt, 2014; Nosek et al., 2015). The validity of a scientific claim depends not on the reputation of those making the claim, the venue in which the claim is made, or the novelty of the res...
Preprint
Social judgment is shaped by multiple biases operating simultaneously, but most bias-reduction interventions target only a single social category. In four pre-registered studies (Total N > 4,800), we investigated whether raising awareness of one social bias impacted that and other social biases. Participants selected honor society applicants based...
Preprint
Registered Reports (RRs) is a publishing model in which initial peer review is conducted prior to knowing the outcomes of the research. In-principle acceptance of papers at this review stage combats publication bias, and provides a clear distinction between confirmatory and exploratory research. Some editors raise a practical concern about adopting...
Article
Full-text available
We propose to change the default P-value threshold for statistical significance for claims of new discoveries from 0.05 to 0.005.
Preprint
Implicit prejudice are social preferences that exist outside of conscious awareness or conscious control. We summarize evidence for three mechanisms that influence the expression of implicit prejudice: associative change, contextual change, and change in control over implicit prejudice. We then review the evidence (or lack thereof) for five open is...
Preprint
Full-text available
"We propose to change the default P-value threshold forstatistical significance for claims of new discoveries from 0.05 to 0.005."
Preprint
Here, we advance the ideological migration hypothesis — individuals choose to live in communities with ideologies similar to their own to satisfy their need to belong. In Study 1, incongruity between personal and community ideology predicted greater residential mobility and attraction to more ideologically-congruent communities. In Study 2, partici...
Preprint
This proposal is a response to NIH's call for creation of a Data Commons (RM-17-026). The Commons must support use cases of many stakeholders who need access to scholarly process, content, and outcomes in pursuit of knowledge. Moreover, the Commons must be flexible enough to respect researchers’ idiosyncratic workflows, yet specific enough to solve...
Preprint
Progress in science relies on generating hypotheses with existing observations and testing hypotheses with new observations. This distinction between postdiction and prediction is appreciated conceptually, but is not respected in practice. Mistaking generation of postdictions with testing of predictions reduces the credibility of research findings....
Preprint
In their review of validity of Implicit Association Test and affective priming, De Houwer, Teige-Mocigemba, Spruyt, and Moors (2009) identified validity with establishment of “basic theoretical understanding” of the measures. We agree that theoretical understanding has an important role in making measures more valid and useful. Nevertheless, we con...
Article
Co-occurrence of an object and affective stimuli does not always mean that the object and the stimuli are the same valence (e.g., false accusations that Richard is a crook). Contemporary theory posits that information about the (in)validity of co-occurrence has stronger influence on deliberate than automatic evaluation. However, available evidence...