Fig 2 - available via license: Creative Commons Attribution 4.0 International
Content may be subject to copyright.
Tacit inferences across cultures. Higher means in Condition 1 than Condition 2 reflect false memories consistent with linking traditional work and sex morality. As seen in the figure, participants from all samples made such tacit inferences. Error bars represent standard errors.
Source publication
How can we maximize what is learned from a replication study? In the creative destruction approach to replication, the original hypothesis is compared not only to the null hypothesis, but also to predictions derived from multiple alternative theoretical accounts of the phenomenon. To this end, new populations and measures are included in the design...
Similar publications
Previous research has demonstrated an inverse relation between subjective social class (SSC) and performance on emotion recognition tasks. Study 1 (N = 418) involved a preregistered replication of this effect using the Reading the Mind in the Eyes Task and the Cambridge Mindreading Face-Voice Battery. The inverse relation replicated; however, explo...
Citations
... Underlying these definitional differences are shared values in the conduct and dissemination of science, and the need to move toward the principles and behaviors of open science has been widely recognized across the sciences. Research communities across many disciplines have begun to develop stronger norms inspired by open science, including psychology [2,[14][15][16], genetics [17], biomedicine [18], animal behavior [4,19], economics [20][21][22][23][24], education [21,[25][26][27][28][29], political science [30], public health [31,32], science and technology studies [33], scientometrics [34], and sociology [35,36], among others (see ( [37]). Despite some progress, all stakeholders in the system need to do better at adopting and implementing open science practices, and our focus is on how to help editors accomplish this. ...
Journal editors have a large amount of power to advance open science in their respective fields by incentivising and mandating open policies and practices at their journals. The Data PASS Journal Editors Discussion Interface (JEDI, an online community for social science journal editors: www.dpjedi.org ) has collated several resources on embedding open science in journal editing ( www.dpjedi.org/resources ). However, it can be overwhelming as an editor new to open science practices to know where to start. For this reason, we created a guide for journal editors on how to get started with open science. The guide outlines steps that editors can take to implement open policies and practices within their journal, and goes through the what, why, how, and worries of each policy and practice. This manuscript introduces and summarizes the guide (full guide: https://doi.org/10.31219/osf.io/hstcx ).
... The "creative destruction" approach to replication serves as a case in point. This method contrasts the original predictions to pre-registered predictions from alternative theoretical accounts, often leading to the identification of additional moderating or mediating factors, such as new measures or testing conditions [51]. Hence, the consideration of counterfactuals and alternate theories is not a mere intellectual exercise. ...
We examine the opportunities and challenges of expert judgment in social sciences, scrutinizing the way social scientists make predictions. While social scientists show above chance accuracy in predicting lab-based phenomena, they often struggle to predict real-world societal changes. We argue that most causal models used in social sciences are oversimplified, confuse levels of analysis to which a model applies, misalign nature of the models with the nature of the phenomena, and fail to consider factors beyond the scientist’s pet theory. Taking cues from physical sciences and meteorology, we advocate for an approach that integrates broad foundational models with context-specific time series data. We call for a shift in the social sciences towards more precise, daring predictions, and greater intellectual humility.
... Prediction markets of research credibility. In recent years, researchers have employed prediction markets to assess the credibility of research findings [94][95][96][97][98][99] . Here, researchers invite experts or non-experts to estimate the replicability of different studies or claims. ...
The emergence of large-scale replication projects yielding successful rates substantially lower than expected caused the behavioural, cognitive, and social sciences to experience a so-called ‘replication crisis’. In this Perspective, we reframe this ‘crisis’ through the lens of a credibility revolution, focusing on positive structural, procedural and community-driven changes. Second, we outline a path to expand ongoing advances and improvements. The credibility revolution has been an impetus to several substantive changes which will have a positive, long-term impact on our research environment.
... In recent years, researchers have employed prediction markets to assess the credibility of research findings [137][138][139][140][141][142] . Here, researchers invite experts or non-experts to estimate the replicability of different studies or claims. ...
The emergence of large-scale replication projects yielding successful rates substantially lower than expected caused the behavioural, cognitive, and social sciences to experience a so-called ‘replication crisis’. In this Perspective, we reframe this ‘crisis’ through the lens of a credibility revolution, focusing on positive structural, procedural and community-driven changes. Second, we outline a path to expand ongoing advances and improvements. The credibility revolution has been an impetus to several substantive changes which will have a positive, long-term impact on our research environment.
... Forecasting Survey. Following previous efforts (91,93), we asked independent scientists (n = 238) recruited via social media advertisements to attempt to predict the outcomes of the generalizability tests while blinded to the results. Each forecaster was provided with the original article's title; abstract; full text, including the original sample size and all associated analyses; the key statistical test from the paper; and a narrative summary of the focal finding and attempted to predict both its direct reproducibility and generalizability to different time periods. ...
This initiative examined systematically the extent to which a large set of archival research findings generalizes across contexts. We repeated the key analyses for 29 original strategic management effects in the same context (direct reproduction) as well as in 52 novel time periods and geographies; 45% of the reproductions returned results matching the original reports together with 55% of tests in different spans of years and 40% of tests in novel geographies. Some original findings were associated with multiple new tests. Reproducibility was the best predictor of generalizability—for the findings that proved directly reproducible, 84% emerged in other available time periods and 57% emerged in other geographies. Overall, only limited empirical evidence emerged for context sensitivity. In a forecasting survey, independent scientists were able to anticipate which effects would find support in tests in new samples.
... Forecasting Survey. Following previous efforts (91,93), we asked independent scientists (n = 238) recruited via social media advertisements to attempt to predict the outcomes of the generalizability tests while blinded to the results. Each forecaster was provided with the original article's title; abstract; full text, including the original sample size and all associated analyses; the key statistical test from the paper; and a narrative summary of the focal finding and attempted to predict both its direct reproducibility and generalizability to different time periods. ...
This initiative examined systematically the extent to which a large set of archival research findings generalizes across contexts. We repeated the key analyses for 29 original strategic management effects in the same context (direct reproduction) as well as in 52 novel time periods and geographies; 45% of the reproductions returned results matching the original reports together with 55% of tests in different spans of years and 40% of tests in novel geographies. Some original findings were associated with multiple new tests. Reproducibility was the best predictor of generalizability—for the findings that proved directly reproducible, 84% emerged in other available time periods and 57% emerged in other geographies. Overall, only limited empirical evidence emerged for context sensitivity. In a forecasting survey, independent scientists were able to anticipate which effects would find support in tests in new samples.
... io/ zwpbr/). Hypothesis 1 is a directional hypothesis based on previous literature suggesting that forecasted predictions are positively associated with the observed results [19][20][21][22][23][24]34 . Hypotheses 2-6 are non-directional and reflect that we assumed that there was variation across groups and outcomes, but we remained neutral towards the direction of the forecasters' predictions: ...
Scientists and policymakers seek to choose effective interventions that promote preventative health measures. We evaluated whether academics, behavioral science practitioners, and laypeople (N = 1034) were able to forecast the effectiveness of seven different messages compared to a baseline message for Republicans and Democrats separately. These messages were designed to nudge mask-wearing attitudes, intentions, and behaviors. When examining predictions across political parties, forecasters predicted larger effects than those observed for Democrats compared to Republicans and made more accurate predictions for Republicans compared to Democrats. These results are partly driven by a lack of nudge effects on Democrats, as reported in Gelfand et al. (J Exp Soc Psychol, 2021). Academics and practitioners made more accurate predictions compared to laypeople. Although forecasters' predictions were correlated with the nudge interventions, all groups overestimated the observed results. We discuss potential reasons for why the forecasts did not perform better and how more accurate forecasts of behavioral intervention outcomes could potentially provide insight that can help save resources and increase the efficacy of interventions.
... recognition without identification paradigm; Catling et al., 2021) or using a creative destruction approach (i.e. pre-specifying alternative results by competing hypotheses on a complex set of experimental findings; Tierney et al., 2020;Tierney et al., 2021). ...
According to the lexical quality hypothesis (Perfetti, 2007), differences in the orthographic, semantic, and phonological representations of words will affect individual reading performance. Whilst several studies have focused on orthographic precision and semantic coherence, few have considered phonological precision. The present study used a suite of individual difference measures to assess which components of lexical quality contributed to competition resolution in a masked priming experiment. The experiment measured form priming for word and pseudoword targets with dense and sparse neighbourhoods in 84 university students. Individual difference measures of language and cognitive skills were also collected and a principal component analysis was used to group these data into components. The data showed that phonological precision and NHD interacted with form priming. In participants with high phonological precision, the direction of priming for word targets with sparse neighbourhoods was facilitatory, while the direction for those with dense neighbourhoods was inhibitory. In contrast, people with low phonological precision showed the opposite pattern, but the interaction was non-significant. These results suggest that the component of phonological precision is linked to lexical competition for word recognition and that access to the mental lexicon during reading is affected by differing levels of phonological processing.
... In the shift towards a more 'open' way of doing research, tools have been proposed to combat questionable research practices by improving its openness, rigour, and transparency. This includes, for example, preregistration of a study's hypotheses and analysis plan prior to data collection and/or analyses (e.g., Lindsay et al., 2016;Nosek et al., 2015), open data sharing (Houtkoop et al., 2018), considering metaanalytical perspective (Torpor et al., under review) and a focus on replication studies to evaluate the robustness of key findings and scientific theories (Open Science Collaboration, 2015; Tierney et al., 2020Tierney et al., , 2021. Although this movement has been primarily informed by a quantitative perspective, qualitative researchers are also considering how they can adopt more open practices (e.g., Haven & van Grootel, 2019;Haven et al., 2020). ...
Recently, there has been a growing emphasis on embedding open and reproducible approaches into research. One essential step in accomplishing this larger goal is to embed such practices into undergraduate and postgraduate research training. However, this often requires substantial time and resources to implement. Also, while many pedagogical resources are regularly developed for this purpose, they are not often openly and actively shared with the wider community. The creation and public sharing of open educational resources is useful for educators who wish to embed open scholarship and reproducibility into their teaching and learning. In this article, we describe and openly share a bank of teaching resources and lesson plans on the broad topics of open scholarship, open science, replication, and reproducibility that can be integrated into taught courses, to support educators and instructors. These resources were created as part of the Society for the Improvement of Psychological Science (SIPS) hackathon at the 2021 Annual Conference, and we detail this collaborative process in the article. By sharing these open pedagogical resources, we aim to reduce the labour required to develop and implement open scholarship content to further the open scholarship and open educational materials movement.
... In order to address these shortcomings and in order to get concurrent data and reduce the retrospective nature of reporting by adults, future studies with a longitudinal design, using a creative destruction approach (i.e. pre-specifying alternative results by competing hypotheses on a complex set of experimental findings; Tierney et al., 2020aTierney et al., , 2020b are needed. ...
This study assessed the prevalence of childhood stuttering in adults with dyslexia (AWD) and the prevalence of dyslexia in adults who stutter (AWS). In addition, the linguistic profiles of 50 AWD, 30 AWS and 84 neurotypical adults were measured. We found that 17 out of 50 AWD (34%) reported stuttering during childhood compared to 1% of the neurotypical population. This was moderated by the severity of dyslexia: People with mild dyslexia showed a lower prevalence rate (15%) of childhood stuttering than those with severe dyslexia (47%). In addition, we observed that 50% of the AWS (n = 30) fulfilled the diagnostic criteria of dyslexia, even though they had never been diagnosed as dyslexic. Compared to neurotypical adults, phonological working memory, awareness, and retrieval were similarly reduced in AWS and AWD. The findings supports the view that stuttering and dyslexia may share a phonological deficit.
