ArticlePDF Available

Publishing Research With Undergraduate Students via Replication Work: The Collaborative Replications and Education Project

Frontiers
Frontiers in Psychology
Authors:

Figures

This content is subject to copyright.
OPINION
published: 13 February 2019
doi: 10.3389/fpsyg.2019.00247
Frontiers in Psychology | www.frontiersin.org 1February 2019 | Volume 10 | Article 247
Edited by:
Traci Giuliano,
Southwestern University,
United States
Reviewed by:
Kimberley A. Phillips,
Trinity University, United States
Ranjana Dutta,
Saginaw Valley State University,
United States
*Correspondence:
Jordan R. Wagge
jordan.wagge@avila.edu
Specialty section:
This article was submitted to
Educational Psychology,
a section of the journal
Frontiers in Psychology
Received: 27 November 2018
Accepted: 25 January 2019
Published: 13 February 2019
Citation:
Wagge JR, Brandt MJ, Lazarevic LB,
Legate N, Christopherson C,
Wiggins B and Grahe JE (2019)
Publishing Research With
Undergraduate Students via
Replication Work: The Collaborative
Replications and Education Project.
Front. Psychol. 10:247.
doi: 10.3389/fpsyg.2019.00247
Publishing Research With
Undergraduate Students via
Replication Work: The Collaborative
Replications and Education Project
Jordan R. Wagge 1
*, Mark J. Brandt 2, Ljiljana B. Lazarevic 3, Nicole Legate 4,
Cody Christopherson 5, Brady Wiggins 6and Jon E. Grahe 7
1School of Psychology, Avila University, Kansas City, MO, United States, 2Department of Social Psychology, Tilburg
University, Tilburg, Netherlands, 3Institute of Psychology, University of Belgrade, Belgrade, Serbia, 4Department of
Psychology, Illinois Institute of Technology, Chicago, IL, United States, 5Department of Psychology, Southern Oregon
University, Ashland, OR, United States, 6Department of Psychology, Brigham Young University-Idaho, Rexburg, ID,
United States, 7Department of Psychology, Pacific Lutheran University, Tacoma, WA, United States
Keywords: replication, pedagogy, psychology, publishing, undergraduates, teaching, projects, open science
The Collaborative Replications and Education Project (CREP; http://osf.io/wfc6u) is a framework
for undergraduate students to participate in the production of high-quality direct replications.
Staffed by volunteers (including the seven authors1of this paper) and incorporated into
coursework, CREP helps produce high-quality data using existing resources and provides structure
for research projects from conceptualization to dissemination. Most notably, student research
generated through CREP make an impact: data from these projects are available for meta-analyses,
some of which are published with student authors.
The call for direct replications of published psychological research has been pronounced and
sustained in recent years (e.g., Lindsay, 2015), yet accomplishing this in light of the current
incentive structure for faculty is challenging (Nosek et al., 2012). There is pressure for faculty to
publish original research in high-impact journals and report significant effects (Franco et al., 2014),
and so replication work often does not get the attention that it requires or deserves (Martin and
Clarke, 2017). CREP harnesses the potential of student research to answer this call.
CREP BACKGROUND
CREP’s primary purpose is educational: to teach students good scientific practices by performing
direct replications of highly cited works in the field using open science methods. The focus on
students is what sets CREP apart from other large-scale collaborations with similar methodological
priorities, such as the ongoing Psych Science Accelerator (Moshontz et al., 2018), and one-off
projects such as the Reproducibility Project: Psychology (Open Science Collaboration., 2015) and
the Many Labs projects (Ebersole et al., 2016; Klein et al., 2018). The CREP approach also differs
from typical undergraduate research projects because CREP results are aimed to have an impact on
psychological science as a field.
To select the studies for crowdsourced direct replications, the CREP team samples the most
highly cited papers from the top-cited journals in each of nine sub-disciplines published 3 years
before the present year (e.g., 2010 in 2013, 2015 in 2018). From this sample, our administrative
1Jon Grahe is the Executive Director of CREP, Jordan Wagge is the Associate Director, and the other five authors are Executive
Reviewers. The CREP was founded in 2013 by Jon Grahe, Mark Brandt, and Hans IJzerman.
Wagge et al. The Collaborative Replications and Education Project
advisors (CREP student alumni) rate papers for how feasible2
they would be for a student to replicate in a semester, as
well as how interesting students would find the topic. If there
is more than one study in a paper, the CREP team selects
just one for replication (typically the one judged as most
feasible). The top-rated studies are then reviewed by one or
more Executive Reviewers before making a final selection as a
group. The CREP team then notifies the original authors of the
study selections and requests materials and replication guidance
with the goal of creating the most high-fidelity replication
possible. Documentation of the study selection process can be
found at osf.io/9kzje/.
For a student, the CREP process ideally looks like this: they
are introduced to CREP by a faculty instructor at their home
institution—typically in a research methods course, capstone
course, or individual laboratory. Figure 1 shows the CREP
process from that point on from the students’ perspective.
Student groups usually conduct direct replications, but can also
include additional measures or conditions that the students add
to test their own, original hypotheses. This Direct+replication
option can be performed out of student interest (e.g., theory-
driven and based on previous findings) or out of a course
or departmental requirement that students develop and test
original hypotheses.
Figure 1 highlights that students are, along the way,
participating in some of the critical requirements of open science
and transparent methodology: open methods, open data, and
preregistration of hypotheses (Kidwell et al., 2016). Students are
also engaged in standard scholarly peer-review processes that
many students do not get exposed to in their curricula. One
notable piece of this process is that the CREP team participates
in a revise-and-resubmit procedure of their project page until it
meets the high standards the review team has set for replication
fidelity and quality both before and after data collection. Being
told about peer-review is one thing, but being a participant
in the revise-and-resubmit process lends a greater appreciation
for published scholarly work and how the peer review process
works. For students who will enter academia, this training is
essential for their careers. For students not pursuing academic
careers, they gain skills in critically evaluating scientific claims
by asking whether reported research has engaged these practices.
For students who complete CREP projects and contribute
to manuscripts, it prepares them for the revise-and-resubmit
process that happens during the publication process.
DISSEMINATION OF STUDENT WORK
CREP may be a more likely vehicle for student publication and
citation compared to other teaching models that rely on student-
generated hypotheses and single-site data collection. Student
projects are rarely powered well enough for publication on their
own. In a recent survey of instructors, who supervise research
projects, we found that less than a third of instructors agreed with
the statement that “Enough data is collected to make appropriate
statistical conclusions” (only 4.9% strongly agreed) and less than
2Feasibility considerations include sample size, sample characteristics, access to
technology and equipment, and duration of study.
a third of students complete a power analysis prior to data
collection ((Wagge et al., manuscript in preparation). While close
to 2/3of instructors reported that the research questions for the
projects were student-generated, only just over half agreed that
student-generated hypotheses are interesting and <20% agreed
that student research questions are typically grounded in theory.
Unsurprisingly, these typical student projects completed as part
of courses are not likely to lead to publication. Indeed, while
instructors said that 79.5% of students presented their projects
in class, just 30.4% reported presentations outside of class, and
only 4.6% published in a journal. We believe these estimates may
also be high given the nature of our specific sample (recruited
from Twitter and Facebook methods groups, with large networks
of open science advocates). For CREP replications, we anticipate
that all completed student projects that meet our specifications
will be included in meta-analyses. Indeed, this has been the
case for our meta-analyses that have been published or are
under review. The data are practically guaranteed life beyond the
institution walls.
We strongly discourage contributors from writing their single
studies for publication because any single CREP replication is not
sufficiently powered to draw a strong inference. Instead, we wait
until at least five samples are completed to begin a meta-analysis.
Ultimately, the goal of the CREP is for completed projects to
be reported in peer-reviewed manuscripts. There are currently
several CREP meta-analyses in various stages of publication: two
have been published (Leighton et al., 2018; Wagge et al., 2019),
one has been submitted for publication (Ghelfi et al., 2018), one
is in preparation (Lazarevic et al., manuscript in preparation),
and an additional Phase 1 Registered Replication Report is in the
review process (Hall et al., 2018) for a pilot partnership with the
Psychological Science Accelerator (Moshontz et al., 2018).
Generally speaking, CREP can help students get first-hand
experience with scientific dissemination in three ways. The
first and most obvious way is that students can present their
replication results at a conference (e.g., Peck et al., 2017).
Second, students who complete replications that are used
in CREP manuscripts have their OSF pages cited in those
manuscripts. Students can therefore meaningfully contribute
to science without needing the time and skill to write a
professional paper themselves. OSF pages are also permanently
and publicly available for other researchers to use. Our meta-
analyses include only CREP direct replications, but other external
meta-analyses may consist of conceptual replications and other,
non-CREP direct replications. For example, a meta-analysis
by Lehmann et al. (2018) of the red-rank-romance effect
(e.g., Elliot et al., 2010) cites many of the individual projects
completed by CREP groups. Therefore, by doing nothing beyond
making their datasets publicly available (a requirement for CREP
projects), students who completed replications for this project
automatically gain cited authorship of their project’s OSF page
in a scholarly publication.
Third, and most importantly, students are invited to
contribute to the authorship process when enough data has
been collected for a meta-analysis. CREP has not been tracking
student conference presentations systematically, but 27 CREP
projects have been cited in three manuscripts currently published
or under review, and 17 co-authors on these manuscripts were
Frontiers in Psychology | www.frontiersin.org 2February 2019 | Volume 10 | Article 247
Wagge et al. The Collaborative Replications and Education Project
FIGURE 1 | The CREP process for students.
student CREP contributors. When possible, the CREP Executive
team avoids taking lead authorship roles on meta-analysis
manuscripts, offering these roles first to motivated students who
have collected data and junior faculty who have supervised teams.
Replication work may be more likely to help students get
published than other research models—while direct replications
and null effect findings might not typically be considered
“interesting” for journals, both null and confirmatory effects are
interesting and important when they are replications of highly
cited published works. For example, Royal Society Open Science
recently committed to publishing close replications of work that
was originally published in their journal (“Replication Studies”3
Further, the Psi Chi Journal has taken a step toward encouraging
replications by offering authors a “Replication” badge in addition
to the standard badges developed by the Center for Open
Science (Kidwell et al., 2016). Recently, as a result, the first
official CREP publication received the first “Replication” badge
offered by any journal (Leighton et al., 2018). This publication
3Replication studies (n.d.). Retrieved from http://rsos.royalsocietypublishing.org/
page/replication-studies).
included a student co-author and cited seven completed projects
by students.
While we face many of the same challenges as other
approaches to publishing with undergraduates (e.g., difficulty
contacting former students to request their involvement), we
believe that this approach is generally more productive than
single-site projects as this has been the experience of several
of us who have served as supervisors as well as manuscript
authors. First, individual projects don’t require collection from
more participants than would be feasible for student teams in a
typical semester. Second, students don’t need a deep background
in theory and the literature to run a CREP study and contribute
to the manuscript. Third, publication doesn’t require multiple
studies or pretests, and we are unlikely to get feedback that more
data needs to be collected to publish results.
BENEFITS OF CREP
Data from direct replications help establish credibility for
the discipline. CREP also has the benefits for students and
instructors. Students get training in cutting-edge research
Frontiers in Psychology | www.frontiersin.org 3February 2019 | Volume 10 | Article 247
Wagge et al. The Collaborative Replications and Education Project
practices including pre-registration, open data, open materials,
and large-scale collaboration. The selection of a replication study
may lower barriers for beginning researchers, as students are not
required to have extensive knowledge of a literature or research
design before making a contribution with impact.
Instructors benefit from using CREP in four ways. First,
CREP offers a supportive entry-point for faculty who are new
to open science and large-scale collaborations. Second, because
the data collected are meant to be included in a high-quality
meta-analysis, CREP helps with fidelity and quality checks.
Third, CREP eliminates the need for instructors to vet every
hypothesis and design for student research projects. Instructors
need not be experts in a topic to determine whether the
hypothesis and design are relevant to the field and because
we also try to provide stimuli and code for replications
they do not need to learn new programs. Fourth, CREP is
a rare opportunity for instructors to have a documentable
experience blending teaching, scholarship, and close mentoring.
These experiences are useful for tenure and promotion
reviews. Faculty who choose who work as reviewers at CREP
have an additional opportunity for meaningful international
service experience.
In 5 years, more than 120 student groups have initiated CREP
projects, and we hope to broaden the project’s impact in future
years. These projects offer the power, the rigor, and the fidelity
needed for good replication work, all while providing the student
the chance to learn, to publish, and to apprentice by following in
the footsteps of scholars in the field. Given the CREP’s benefits
and initial success, we also believe this model can be successfully
applied in other scientific disciplines.
AUTHOR CONTRIBUTIONS
JW wrote the original manuscript draft, revised the draft,
and coordinated collaboration. MB, LL, NL, CC, BW, and JG
provided feedback on drafts. All authors have made significant
intellectual and time contributions to the CREP project.
ACKNOWLEDGMENTS
We would like to thank the Center for Open Science and
Psi Chi for providing early funding for CREP projects, and
Hans IJzerman for early leadership and continued contributions
to projects.
REFERENCES
Ebersole, C. R., Atherton, O. E., Belanger, A. L., Skulborstad, H. M., Allen, J. M.,
Banks, J. B., et al. (2016). Many labs 3: evaluating participant pool quality
across the academic semester via replication. J. Exp. Soc. Psychol. 67, 68–82.
doi: 10.1016/j.jesp.2015.10.012
Elliot, A. J., Kayser, N. D., Greitemeyer, T., Lichtenfeld, S., Gramzow, R. H., Maier,
M. A., et al. (2010). Red, rank, and romance in women viewing men. J. Exp.
Psychol. Gen. 139, 399–417. doi: 10.1037/a0019689
Franco, A., Malhotra, N., and Simonovits, G. (2014). Publication bias in
the social sciences: unlocking the file drawer. Science 345, 1502–1505.
doi: 10.1126/science.1255484
Ghelfi, E., Christopherson, C. D., Urry, H. L., Lenne, R. L., Legate, N., Fischer, M.
A., et al. (2018). Reexamining the effect of gustatory disgust on moral judgment:
a multi-lab direct replication of Eskine, Kacinik, and Prinz (2011). PsyArXiv.
doi: 10.31234/osf.io/349pk
Hall, B. F., Wagge, J. R., Chartier, C. R., Pfuhl, G., Stieger, S., Vergauwe, E., et al.
(2018). Accelerated CREP - RRR: Turri, Buckwalter, & Blouw (2015). PsyArXiv.
doi: 10.31234/osf.io/zeux9
Kidwell, M. C., Lazarevi´
c, L. B., Baranski, E., Hardwicke, T. E., Piechowski, S.,
Falkenberg, L. S., et al. (2016). Badges to acknowledge open practices: a simple,
low-cost, effective method for increasing transparency. PLoS Biol. 14:e1002456.
doi: 10.1371/journal.pbio.1002456
Klein, R. A., Vianello, M., Hasselman, F., Adams, B. G., Adams, R. B. Jr.,
Alper, S., et al. (2018). Many labs 2: investigating variation in replicability
across samples and settings. Adv. Methods Pract. Psychol. Sci. 1, 443–490.
doi: 10.1177/2515245918810225
Lehmann, G. K., Elliot, A. J., and Calin-Jageman, R. (2018). Meta-analysis
of the effect of red on perceived attractiveness. Evol. Psychol. 16, 1–27.
doi: 10.1177/1474704918802412
Leighton, D. C., Legate, N., LePine, S., Anderson, S. F., and Grahe, J. (2018).
Self-esteem, self-disclosure, self-expression, and connection on facebook: a
collaborative replication meta-analysis. Psi Chi J. Psychol. Res. 23, 98–109.
doi: 10.24839/2325-7342.JN23.2.98
Lindsay, S. D. (2015). Replication in psychological science. Psychol. Sci. 26,
1827–1832. doi: 10.1177/0956797615616374
Martin, G. N., and Clarke, R. M. (2017). Are psychology journals anti-
replication? A snapshot of editorial practices. Front. Psychol. 8:523.
doi: 10.3389/fpsyg.2017.00523
Moshontz, H., Campbell, L., Ebersole, C. R., IJzerman, H., Urry, H. L., Forscher,
P. S., et al. (2018). The psychological science accelerator: advancing psychology
through a distributed collaborative network. Adv. Methods Pract. Psychol. Sci.
1, 501–515. doi: 10.1177/2515245918797607
Nosek, B. A., Spies, J. R., and Motyl, M. (2012). Scientific Utopia II: restructuring
incentives and practices to promote truth over publishability. Perspect. Psychol.
Sci. 7, 615–631. doi: 10.1177/1745691612459058
Open Science Collaboration. (2015). Estimating the reproducibility of
psychological science. Science 349:aac4716. doi: 10.1126/science.aac4716
Peck, T., McDonald, T., Murphy, M., Paulk, A., Perkinson, P., Sheelar, K., et al.
(2017). “Moral judgment and personality: neuroticism as a moderator of the
moral judgment and disgust relationship, in Poster Presented at the Annual
Meeting of the Western Psychological Association, Sacramento, CA.
Wagge, J. R., Baciu, C., Banas, K., Nadler, J. T., Schwarz, S., Weisberg, Y.,
et al. (2019). A demonstration of the collaborative replication and education
project: replication attempts of the red-romance effect. Collab. Psychol. 5:5.
doi: 10.1525/collabra.177
Conflict of Interest Statement: The authors declare that the research was
conducted in the absence of any commercial or financial relationships that could
be construed as a potential conflict of interest.
Copyright © 2019 Wagge, Brandt, Lazarevic, Legate, Christopherson, Wiggins and
Grahe. This is an open-access article distributed under the terms of the Creative
Commons Attribution License (CC BY). The use, distribution or reproduction in
other forums is permitted, provided the original author(s) and the copyright owner(s)
are credited and that the original publication in this journal is cited, in accordance
with accepted academic practice. No use, distribution or reproduction is permitted
which does not comply with these terms.
Frontiers in Psychology | www.frontiersin.org 4February 2019 | Volume 10 | Article 247
... In addition to these more tangible results, joint faculty-student research output can also be viewed as an example of teaching excellence, which is gaining importance in discussions about diversifying academic careers and valuing scholarship and teaching. For instance, reporting on an example from the field of psychology, Wagge et al. (2019) write that faculty-student research partnerships offer "a rare opportunity for instructors to have a documentable experience blending teaching, scholarship, and close mentoring" (p. 4, see also Jalbert, 2008, p. 114). ...
... At the same time, this may not always be realistic. Several authors therefore suggest considering other forms of output, such as blogs, datasets, or policy papers (Griffiths, 2015;Wagge et al., 2019). ...
Chapter
This chapter engages with the theme of doing research with students. As many of us are confronted with conflicting demands to publish excellent research, do fabulous teaching, and engage in related (and sometimes not so related) administrative tasks, it is worth considering ways in which to combine teaching and research. While one may ideally want to teach subjects that exactly fit our research expertise, we are not always able to do so, for instance, due to constraints that follow curriculum design. Another possible way of combining teaching and research is through research partnerships between faculty and students. Anecdotal evidence suggests that such partnerships are relatively rare in political science. However, faculty–student research partnerships are quite common in disciplines such as engineering, psychology, and the sciences. Consequently, there is also abundant literature in those fields that engages with the design of such partnerships and the accompanying benefits and challenges. What can political science learn from this literature? This chapter outlines the main benefits and challenges of faculty–student partnerships through a discussion of literature blended with practical experience. It concludes by offering advice that might inspire you to engage in faculty–student partnerships yourself.
... The Reproducibility Project: Psychology and the Many Labs projects provide excellent examples but they do not offer templates that can be readily adopted for future projects because they were developed early and are completed. However, the Collaborative Replications and Education Project (CREP) (Wagge et al., 2019) has many guidelines and templates that were developed over a decade to help guide students to replicate studies (see the "Getting Started" page, http:// osf.io/ stdgm/ ). The centralized leadership structure of the CREP supports a large group of volunteers who help in study selection and assist contributors in navigating the replication process. ...
Chapter
This chapter presents practical examples and makes recommendations about studying emerging adulthood using crowd-sourced research approaches. Examples are drawn from three studies (one ongoing) to assess the Emerging Adulthood Measured at Multiple Institutions project. Spanning 20 years, it is one of the earliest crowd-sourcing research projects in psychology; these studies offer lessons based on findings and the trials and errors in managing them. In addition to reviewing each study and its implications, this chapter discusses the management of crowd-sourcing projects, including administration and project management. Ideally, readers will gain an understanding of the most realistic way to study emerging adulthood with a crowd.
... Open Science modules and activities were introduced in first-year Biology courses [10], a project-based seminar on qualitative research [11], and Psychology courses focused on robust and reproducible research methods [12], [13]. Furthermore, The Collaborative Replications and Education Project [14] and The Hagen Cumulative Science Project [15] implemented projects around replicating existing research in undergraduate education. The increasing effort and awareness around Open Science and reproducible research in undergraduate education inspires a new approach to undergraduate courses that introduce fundamentals of research. ...
... In-class replications have previously been proposed for college-level education (Meng, 2020) and for psychology education Hawkins et al., 2018), to understand correlates of replicability (Boyce et al., 2023;Frank & Saxe, 2012). Furthermore, data analysis replication efforts have previously been used for comprehensive meta-analyses (Wagge, Baciu, et al., 2019;Wagge, Brandt, et al., 2019), based on multiple studies rather than on a single replication attempt (Boyce et al., 2023;Perry & See, 2022;Shrout & Rodgers, 2018). ...
... Meta-research projects are likely becoming more popular (e.g., Clarke et al., 2023) and can provide students with the opportunity to engage deeply with methodological issues in the literature. Replication studies using existing data are also feasible (e.g., Coyle et al., 2020) and the advantages of replication for learning and guidance on choosing what to replicate for teaching purposes are discussed in-depth elsewhere (see Janz, 2016;Wagge et al., 2019). Using existing data reduces research waste by minimising the unnecessary collection of new data, reducing the overall burden on potential participants, and reducing the burden on research ethics committees. ...
Article
Full-text available
The undergraduate research dissertation in psychology is the capstone demonstration of research skills including project planning and design, considering and resolving ethical issues, and the analysis and dissemination of findings. The dissertation represents an opportunity for learning as well as an opportunity to contribute to the research literature in the student’s chosen area; however, few articles have considered both dimensions in detail. This article provides a roadmap for undergraduate thesis supervision, for early-career supervisors and supervisors aiming to better align their supervision and research activities and/or engage their students in open research practices via the dissertation. Specifically, we review prior literature on undergraduate psychology research supervision and identify several dimensions that vary in existing approaches. Drawing on our own supervision experiences, we describe four key recommendations for undergraduate supervision in psychology and discuss how these can support student learning as well as benefit research.
... The batch of replications presented here were pedagogical replications, done as part of a class. Trainees must learn experimental methods, and conducting replications as part of methods classes serves several purposes: it enables students to learn to do experiments in a supportive context, it often leads to more useful results than if students designed their own experiments from scratch, and it creates a resource for studying the literature [34][35][36][37]. We believe that this kind of pedagogy has an important role to play in improving methodological practices in psychology more broadly. ...
Article
Full-text available
Cumulative scientific progress requires empirical results that are robust enough to support theory construction and extension. Yet in psychology, some prominent findings have failed to replicate, and large-scale studies suggest replicability issues are widespread. The identification of predictors of replication success is limited by the difficulty of conducting large samples of independent replication experiments, however: most investigations reanalyse the same set of 170 replications. We introduce a new dataset of 176 replications from students in a graduate-level methods course. Replication results were judged to be successful in 49% of replications; of the 136 where effect sizes could be numerically compared, 46% had point estimates within the prediction interval of the original outcome (versus the expected 95%). Larger original effect sizes and within-participants designs were especially related to replication success. Our results indicate that, consistent with prior reports, the robustness of the psychology literature is low enough to limit cumulative progress by student investigators.
Chapter
In a time where new research methods are constantly being developed and science is evolving, researchers must continually educate themselves on cutting-edge methods and best practices related to their field. The second of three volumes, this Handbook provides comprehensive and up-to-date coverage of a variety of issues important in developing, designing, and collecting data to produce high-quality research efforts. First, leading scholars from around the world provide an in depth explanation of various advanced methodological techniques. In section two, chapters cover general important methodological considerations across all types of data collection. In the third section, the chapters cover self-report and behavioral measures and their considerations for use. In the fourth section, various psychological measures are covered. The final section of the handbook covers issues that directly concern qualitative data collection approaches. Throughout the book, examples and real-world research efforts from dozens of different disciplines are discussed.
Article
Full-text available
In their commentary on Hall et al., Buckwalter and Friedman (2024) claimed that the replication should have been interpreted as successful, argued that the researchers’ conclusions were incorrect, and implied that the replication effort was misguided. As a subset of contributors to the Hall et al. replication, we appreciate the opportunity to respond to their comment. Although we recognize the potential for disagreement in the interpretation of research results, Buckwalter and Friedman’s critique ignored several key features of the research, and many of their arguments and proposed interpretations were already addressed by Hall et al. (2024). In response to their comment, we (a) explain why Hall et al. did not replicate all of the original findings, (b) emphasize how Hall et al. were accurate and nuanced in the description and interpretation of their results, and (c) caution against focusing on the mechanisms underlying a psychological phenomenon before it is clearly established
Preprint
Full-text available
In their replication of Turri et al. (2015; Experiment 1), Hall et al. (2024) identified a key condition difference that was not observed in the original experiment. In their commentary, Buckwalter and Friedman (2024) posited that the evidence presented by Hall et al. should be interpreted as fully replicating Turri et al. While Hall et al. may have observed a similar effect size as the one they calculated from Turri et al.’s data, they did not replicate Turri et al.’s statistical results, and their findings were not consistent with Turri et al.’s claims from Experiment 1. Further, Hall et al. presented a nuanced discussion of their findings that acknowledged many of Buckwalter and Friedman critiques. We opine that, through selectively referencing portions of Hall et al.’s results, Buckwalter and Friedman mischaracterize the key findings and conclusions of Hall et al.
Article
Research shows that questionable research practices (QRPs) are present in undergraduate final-year dissertation projects. One entry-level Open Science practice proposed to mitigate QRPs is “study preregistration,” through which researchers outline their research questions, design, method, and analysis plans before data collection and/or analysis. In this study, we aimed to empirically test the effectiveness of preregistration as a pedagogic tool in undergraduate dissertations using a quasi-experimental design. A total of 89 UK psychology students were recruited, including students who preregistered their empirical quantitative dissertation ( n = 52; experimental group) and students who did not ( n = 37; control group). Attitudes toward statistics, acceptance of QRPs, and perceived understanding of Open Science were measured both before and after dissertation completion. Exploratory measures included capability, opportunity, and motivation to engage with preregistration, measured at Time 1 only. This study was conducted as a Registered Report; Stage 1 protocol: https://osf.io/9hjbw (date of in-principle acceptance: September 21, 2021). Study preregistration did not significantly affect attitudes toward statistics or acceptance of QRPs. However, students who preregistered reported greater perceived understanding of Open Science concepts from Time 1 to Time 2 compared with students who did not preregister. Exploratory analyses indicated that students who preregistered reported significantly greater capability, opportunity, and motivation to preregister. Qualitative responses revealed that preregistration was perceived to improve clarity and organization of the dissertation, prevent QRPs, and promote rigor. Disadvantages and barriers included time, perceived rigidity, and need for training. These results contribute to discussions surrounding embedding Open Science principles into research training.
Article
Full-text available
The present article reports the results of a meta-analysis of nine student replication projects of Elliot et al.’s (2010) findings from Experiment 3, that women were more attracted to photographs of men with red borders (total 'n' = 640). The eight student projects were part of the Collaborative Replication and Education Project (CREP; https://osf.io/wfc6u/), a research crowdsourcing project for undergraduate students. All replications were reviewed by experts to ensure high quality data, and were pre-registered prior to data collection. Results of this meta-analysis showed no effect of red on attractiveness ratings for either perceived attractiveness (mean ratings difference = –0.07, 95% CI [–0.31, 0.16]) or sexual attractiveness (mean ratings difference = –0.06, 95% CI [–0.36, 0.24]); this null result held with and without Elliot et al.’s (2010) data included in analyses. Exploratory analyses examining whether being in a relationship moderated the effect of color on attractiveness ratings also produced null results.
Article
Full-text available
We conducted preregistered replications of 28 classic and contemporary published findings, with protocols that were peer reviewed in advance, to examine variation in effect magnitudes across samples and settings. Each protocol was administered to approximately half of 125 samples that comprised 15,305 participants from 36 countries and territories. Using the conventional criterion of statistical significance (p< .05), we found that 15 (54%) of the replications provided evidence of a statistically significant effect in the same direction as the original finding. With a strict significance criterion (p< .0001), 14 (50%) of the replications still provided such evidence, a reflection of the extremely high-powered design. Seven (25%) of the replications yielded effect sizes larger than the original ones, and 21 (75%) yielded effect sizes smaller than the original ones. The median comparable Cohen’s ds were 0.60 for the original findings and 0.15 for the replications. The effect sizes were small (< 0.20) in 16 of the replications (57%), and 9 effects (32%) were in the direction opposite the direction of the original effect. Across settings, the Q statistic indicated significant heterogeneity in 11 (39%) of the replication effects, and most of those were among the findings with the largest overall effect sizes; only 1 effect that was near zero in the aggregate showed significant heterogeneity according to this measure. Only 1 effect had a tau value greater than .20, an indication of moderate heterogeneity. Eight others had tau values near or slightly above .10, an indication of slight heterogeneity. Moderation tests indicated that very little heterogeneity was attributable to the order in which the tasks were performed or whether the tasks were administered in lab versus online. Exploratory comparisons revealed little heterogeneity between Western, educated, industrialized, rich, and democratic (WEIRD) cultures and less WEIRD cultures (i.e., cultures with relatively high and low WEIRDness scores, respectively). Cumulatively, variability in the observed effect sizes was attributable more to the effect being studied than to the sample or setting in which it was studied
Article
Full-text available
We conducted preregistered replications of 28 classic and contemporary published findings with protocols that were peer reviewed in advance to examine variation in effect magnitudes across sample and setting. Each protocol was administered to approximately half of 125 samples and 15,305 total participants from 36 countries and territories. Using conventional statistical significance (p < .05), fifteen (54%) of the replications provided evidence in the same direction and statistically significant as the original finding. With a strict significance criterion (p < .0001), fourteen (50%) provide such evidence reflecting the extremely high powered design. Seven (25%) of the replications had effect sizes larger than the original finding and 21 (75%) had effect sizes smaller than the original finding. The median comparable Cohen’s d effect sizes for original findings was 0.60 and for replications was 0.15. Sixteen replications (57%) had small effect sizes (< .20) and 9 (32%) were in the opposite direction from the original finding. Across settings, 11 (39%) showed significant heterogeneity using the Q statistic and most of those were among the findings eliciting the largest overall effect sizes; only one effect that was near zero in the aggregate showed significant heterogeneity. Only one effect showed a Tau > 0.20 indicating moderate heterogeneity. Nine others had a Tau near or slightly above 0.10 indicating slight heterogeneity. In moderation tests, very little heterogeneity was attributable to task order, administration in lab versus online, and exploratory WEIRD versus less WEIRD culture comparisons. Cumulatively, variability in observed effect sizes was more attributable to the effect being studied than the sample or setting in which it was studied.
Preprint
Full-text available
According to the Justified True Belief (JTB) account of knowledge, a person’s ability to know something is defined by having a belief that is both justified and true (i.e., knowledge is justified true belief). However, this account fails to consider the role of luck. In 1963, Gettier argued that JTB is insufficient because it does not account for certain situations, called Gettier cases, wherein a person is justified for believing something true but only because of luck. It is unclear whether lay people’s intuitions about knowledge lead them to agree with Gettier, such that lay people believe that individuals in these cases lack knowledge (referred to as Gettier intuitions). We attempt to provide a robust estimate of the Gettier intuition effect size by replicating Turri and colleagues’ (2015) Experiment 1. The Collaborative Replications and Education Project (CREP) selected this study for replication based on its undergraduate appeal, feasibility, and pedagogical value. However, in light of some inconsistent results, suboptimal designs, and inconsistent evidence for cultural variation (e.g., Machery et al., 2015; Nagel, et al., 2013; Seyedsayamdost et al., 2015; Starman & Friedman, 2012; Weinberg et al., 2001), the improved methodology of Turri et al. (2015) make it an important study to replicate cross-culturally. Therefore, we propose a multisite collaborative preregistered replication of Turri and colleague's (2015) Experiment 1 (35 labs from 14 countries across 4 continents signed up at time of submission; expected minimum N = 1,500). Results of this study are expected to provide a clearer picture of the Gettier intuition effect size, lay people’s theory and practice of knowledge, and potentially cross-cultural similarities and differences.
Article
Full-text available
We conducted meta-analyses of studies that test the red-romance hypothesis, which is that the color red enhances heterosexual attraction in romantic contexts. For men rating women, we found a small, statistically significant effect (d = 0.26 [0.12, 0.40], p = .0004, N = 2,961), with substantial heterogeneity, Q(44) = 172.5, pQ < .0001, I² = 89% [82, 94], and equivocal results regarding the possibility of upward bias in the estimate. For women rating men, we found a very small effect (d = 0.13 [0.01, 0.25], p = .03, N = 2,739), with substantial heterogeneity, Q(35) = 73.0, pQ = .0002, I² = 53% [33, 80], and evidence of upward bias in the estimate. Moderator analyses suggest effect sizes may have declined over time (both genders), may be largest when an original shade of red is used (men only), and may be smaller in preregistered studies (women only). We present contrasting interpretations and suggestions for future research.
Article
Full-text available
Concerns about the veracity of psychological research have been growing. Many findings in psychological science are based on studies with insufficient statistical power and nonrepresentative samples, or may otherwise be limited to specific, ungeneralizable settings or populations. Crowdsourced research, a type of large-scale collaboration in which one or more research projects are conducted across multiple lab sites, offers a pragmatic solution to these and other current methodological challenges. The Psychological Science Accelerator (PSA) is a distributed network of laboratories designed to enable and support crowdsourced research projects. These projects can focus on novel research questions or replicate prior research in large, diverse samples. The PSA’s mission is to accelerate the accumulation of reliable and generalizable evidence in psychological science. Here, we describe the background, structure, principles, procedures, benefits, and challenges of the PSA. In contrast to other crowdsourced research networks, the PSA is ongoing (as opposed to time limited), efficient (in that structures and principles are reused for different projects), decentralized, diverse (in both subjects and researchers), and inclusive (of proposals, contributions, and other relevant input from anyone inside or outside the network). The PSA and other approaches to crowdsourced psychological science will advance understanding of mental processes and behaviors by enabling rigorous research and systematic examination of its generalizability.
Article
Full-text available
Recent research in psychology has highlighted a number of replication problems in the discipline, with publication bias – the preference for publishing original and positive results, and a resistance to publishing negative results and replications- identified as one reason for replication failure. However, little empirical research exists to demonstrate that journals explicitly refuse to publish replications. We reviewed the instructions to authors and the published aims of 1151 psychology journals and examined whether they indicated that replications were permitted and accepted. We also examined whether journal practices differed across branches of the discipline, and whether editorial practices differed between low and high impact journals. Thirty three journals (3%) stated in their aims or instructions to authors that they accepted replications. There was no difference between high and low impact journals. The implications of these findings for psychology are discussed.
Preprint
Eskine, Kacinik, and Prinz’s (2011) influential experiment demonstrated that gustatory disgust triggers feelings of moral disgust. This is a report of a large-scale multi-site direct replication of this study, conducted by participants in the Collaborative Replications and Education Project (CREP). Participants in each sample were randomly assigned to one of three beverage conditions: bitter/disgusting, control, or sweet. After consuming the assigned beverage, participants made a series of judgments indicating the moral wrongness of the behavior depicted in each of six vignettes. In the original study, drinking the bitter beverage led to higher ratings of moral wrongness than drinking the control and sweet beverages. The original authors found that a beverage contrast (bitter versus both control and sweet) was significant among conservative participants and not among liberal participants. In this report, random effects meta-analyses across all participants (N = 1,137 in k = 11 studies), conservative participants (N = 162, k = 3), and liberal participants (N = 648, k = 7) revealed standardized effect sizes that were smaller than reported in the original study. Most were in the opposite of the predicted direction and had 95% confidence intervals containing zero; all were smaller than the effect size the original authors had 33% power to detect. In sum, the overall pattern does not provide strong support for the theory that physical disgust via taste perception contributes to moral disgust. We also discuss limitations including low reliability of the moral judgment measure and low numbers of conservative participants across samples.
Article
This replication meta-analysis explored the robustness of a highly cited study showing that those with low self-esteem perceived benefits for self-disclosure through Facebook compared to face-to-face interactions (i.e., Forest & Wood, 2012, Study 1). Seven preregistered direct replication attempts of this study were conducted by research teams as part of the Collaborative Replication and Education Project (CREP), and results were meta-analyzed to better understand the strength and consistency of the effects reported in the original study. Half of the original results were clearly supported: Self-esteem negatively predicted perceived safety of self-disclosure on Facebook as compared to face-to-face interactions (meta-analytic effect size = -.28, original effect size = -.31), and self-esteem did not relate to perceived opportunities for self-expression; across the 7 replications, all 95% confidence intervals (CIs) for effect sizes included 0. However, 2 other findings received less support: Self-esteem only weakly and inconsistently predicted perceived advantages of self-disclosure on Facebook (meta analytic effect size = -.16, original effect size = -.30), and contrary to the original study, there was no evidence for self-esteem predicting perceived opportunities for connection with others on Facebook (6 of the 7 replication effect size CIs contained 0). The results provided further evidence regarding the original study's generalizability and robustness. The implications of the research and its relevance to social compensation theory is presented, and considerations for future multisite replications are proposed. Full text open access at https://www.psichi.org/resource/resmgr/journal_2018/23-2_specialissue_leighton.pdf
Article
The university participant pool is a key resource for behavioral research, and data quality is believed to vary over the course of the academic semester. This crowdsourced project examined time of semester variation in 10 known effects, 10 individual differences, and 3 data quality indicators over the course of the academic semester in 20 participant pools (N = 2696) and with an online sample (N = 737). Weak time of semester effects were observed on data quality indicators, participant sex, and a few individual differences—conscientiousness, mood, and stress. However, there was little evidence for time of semester qualifying experimental or correlational effects. The generality of this evidence is unknown because only a subset of the tested effects demonstrated evidence for the original result in the whole sample. Mean characteristics of pool samples change slightly during the semester, but these data suggest that those changes are mostly irrelevant for detecting effects.