Article

Alogna et al. (2014). Registered replication report: Schooler & Englster-Schooler (1990)

Authors:
To read the full-text of this research, you can request a copy directly from the authors.

Abstract

Trying to remember something now typically improves your ability to remember it later. However, after watching a video of a simulated bank robbery, participants who verbally described the robber were 25% worse at identifying the robber in a lineup than were participants who instead listed U.S. states and capitals—this has been termed the " verbal overshadowing " effect (Schooler & Engstler-Schooler, 1990). More recent studies suggested that this effect might be substantially smaller than first reported. Given uncertainty about the effect size, the influence of this finding in the memory literature, and its practical importance for police procedures, we conducted two collections of preregistered direct replications (RRR1 and RRR2) that differed only in the order of the description task and a filler task. In RRR1, when the description task immediately followed the robbery, participants who provided a description were 4% less likely to select the robber than were those in the control condition. In RRR2, when the description was delayed by 20 min, they were 16% less likely to select the robber. These findings reveal a robust verbal overshadowing effect that is strongly influenced by the relative timing of the tasks. The discussion considers further implications of these replications for our understanding of verbal overshadowing. Multilab direct replication of: Study 4 (modified) and Study 1 from Schooler, J. W., & Engstler-Schooler, T. Y. (1990). Verbal overshadowing of visual memories: Some things are better left unsaid. Cognitive Psychology, 22, 36–71.

No full-text available

Request Full-text Paper PDF

To read the full-text of this research,
you can request a copy directly from the authors.

... Yet another competing theory proposes that implicit orientations towards work and sexuality are consistent across cultures, perhaps due to common evolutionary roots. In addition to directly replicating the original study designs (Simons, 2014), this initiative strategically included new measures and samples-permitting not only a comparison of the original theoretical predictions (Poehlman, 2007;Uhlmann et al., 2008Uhlmann et al., , 2009Uhlmann et al., , 2011 with the null hypothesis of no condition or group differences, but also tests of further ideas. We were then able to examine which theory best accounts for the results across multiple key outcomes and contexts. ...
... The scientific community's shaken faith in original effects that do not emerge in a single direct replication (same method, new observations; Simons, 2014) has been documented in the context of a prediction market . More generally, debate and discussion regarding replications centers largely on the existence or nonexistence of a given finding, as opposed to testing competing predictions of positive effects against one another. ...
... Notably, we held methods and materials constant across these populations to allow for direct replication (Simons, 2014). One can also make iterative modifications to the materials across research sites, assessing mediating states each time, in an effort to achieve psychological rather than methodological equivalence (Fabrigar, Wegener, & Petty, in press;Schwarz & Strack, 2014;Stroebe & Strack, 2014). ...
Article
Drawing on the concept of a gale of creative destruction in a capitalistic economy, we argue that initiatives to assess the robustness of findings in the organizational literature should aim to simultaneously test competing ideas operating in the same theoretical space. In other words, replication efforts should seek not just to support or question the original findings, but also to replace them with revised, stronger theories with greater explanatory power. Achieving this will typically require adding new measures, conditions, and subject populations to research designs, in order to carry out conceptual tests of multiple theories in addition to directly replicating the original findings. To illustrate the value of the creative destruction approach for theory pruning in organizational scholarship, we describe recent replication initiatives re-examining culture and work morality, working parents’ reasoning about day care options, and gender discrimination in hiring decisions.
... The semantic mismatch effect is thus a newly discovered phenomenon with high theoretical leverage for theories on auditory distraction and working memory as it allows one to draw conclusions about the fate of to-be-ignored auditory information. However, before theories on attention and working memory are adapted to account for a novel phenomenon, it is essential to establish that this phenomenon is indeed reproducible (Simons, 2014). While it seems promising that the semantic mismatch effect has already been successfully replicated in several experiments, an important caveat at this point is that these replications have all been reported in the same language from a single laboratory. ...
... In sum, the present results confirm the semantic mismatch effect in a preregistered multiplelanguage, multiple-laboratory replication. Reproducibility across laboratories is seen as an important step in establishing trust in a newly discovered phenomenon (e.g., Hüffmeier et al., 2016;Simons, 2014). The fact that the semantic mismatch effect can be obtained in different languages and independently of the specific circumstances prevailing in individual laboratories shows that the effect does not depend on highly specific boundary conditions and thus suggests that the effect may reflect a general property of the processing of to-be-ignored auditory information. ...
Article
Visual-verbal serial recall is disrupted when task-irrelevant background speech has to be ignored. Contrary to previous suggestion, it has recently been shown that the magnitude of disruption may be accentuated by the semantic properties of the irrelevant speech. Sentences ending with unexpected words that did not match the preceding semantic context were more disruptive than sentences ending with expected words. This particular instantiation of a deviation effect has been termed the semantic mismatch effect. To establish a new phenomenon, it is necessary to show that the effect can be independently replicated and does not depend on specific boundary conditions such as the language of the stimulus material. Here we report a preregistered replication of the semantic mismatch effect in which we examined the effect of unexpected words in 4 different languages (English, French, German, and Swedish) across 4 different laboratories. Participants performed a serial recall task while ignoring sentences with expected or unexpected words that were recorded using text-to-speech software. Independent of language, sentences ending with unexpected words were more disruptive than sentences ending with expected words. In line with previous results, there was no evidence of habituation of the semantic mismatch effect in the form of a decrease in disruption with repeated exposure to the occurrence of unexpected words. The successful replication and extension of the effect to different languages indicates the expression of a general and robust mechanism that reacts to violations of expectancies based on the semantic content of the irrelevant speech. (PsycInfo Database Record (c) 2021 APA, all rights reserved).
... A distinction is often drawn between 'exact' (or 'direct') and 'conceptual' replication. On that basis, it is suggested by some methodologists that scientists should strive for exact replications (for example, Cesario, 2014;Hüffmeier et al., 2016;Pashler & Harris, 2012;Simons, 2014), whereas others vouch for the priority of conceptual replications (Crandall & Sherman, 2016;Lynch et al., 2015;Stroebe & Strack, 2014). But Feest (2019) has recently argued that the notion of replication itself, whether exact or conceptual, is flawed (Feest, 2019), whereas Machery (2020) has argued that, although the notion of replication is not flawed, we should dispense nevertheless with the distinction between exact and conceptual replication. ...
... Other methodologists, by comparison, are less strict on this point. Simons (2014) responds directly to Cesario by requiring replications to at least be performed in different labs, thus ensuring that a generated effect is not due to some idiosyncratic feature of the original laboratory experiment, one unrelated to the reality of the effect (77). In this respect, Hüffmeier et al. (2016) distinguish between 'exact' and 'close' replications, the latter being a case where the same experimental process is performed, to whatever degree possible, by independent researchers (84). ...
Article
Full-text available
What does it mean to replicate an experiment? A distinction is often drawn between ‘exact’ (or ‘direct’) and ‘conceptual’ replication. However, in recent work, Uljana Feest argues that the notion of replication in itself, whether exact or conceptual, is flawed due to the problem of systematic error, and Edouard Machery argues that, although the notion of replication is not flawed, we should nevertheless dispense with the distinction between exact and conceptual replication. My plan in this paper is to defend the value of replication, along with the distinction between exact and conceptual replication, from the critiques of Feest and Machery. To that end, I provide an explication of conceptual replication, and distinguish it from what I call ‘experimental’ replication. On the basis, then, of a tripartite distinction between exact, experimental and conceptual replication, I argue in response to Feest that replication is still informative despite the prospect of systematic error. I also rebut Machery’s claim that conceptual replication is fundamentally confused and wrongly conflates replication and extension, and in turn raise some objections to his own Resampling Account of replication.
... Despite their appeal from the standpoint of construct validity, conceptual replications have also faced criticisms. In particular, a number of commentators have expressed reservations regarding the utility of conceptual replications in evaluating the reliability of previously documented effects (e.g., LeBel & Peters, 2011;Nosek, Spies, & Motyl, 2012;Pashler & Harris, 2012;Simons, 2014). Central to these reservations has been the concern that any failure to reproduce an earlier finding in a conceptual replication is ultimately open to a variety of interpretations. ...
Article
Full-text available
Many recent discussions have focused on the role of replication in psychological science. In this article, we examine three key issues in evaluating the conclusions that follow from results of studies at least partly aimed at replicating previous results: the evaluation and status of exact versus conceptual replications, the statistical evaluation of replications, and the robustness of research findings to potential existing or future “non-replications.” In the first section of the article, we discuss the sources of ambiguity in evaluating failures to replicate in exact as well as conceptual replications. In addressing these ambiguities, we emphasize the key role of psychometric invariance of the independent and dependent variables in evaluations of replications. In the second section of the article, we use a meta-analytic framework to discuss the statistical status of replication attempts. We emphasize meta-analytic tools that have been used too sparingly, especially in evaluation of sets of studies within a single article or focused program of research. In the final section of the article, we extend many of these meta-analytic tools to the evaluation of the robustness of a body of research to potential existing or future failures to replicate previous statistically significant results.
... While the original report conceptually replicated the effect across studies, inconsistencies in methods make them studies difficult to compare (Pashler & Harris, 2012). In contrast, direct replications provide an opportunity to understand changes in effect sizes which provide information about the reliability of an effect (Simons, 2014). ...
Article
Full-text available
The implicit association test (IAT) is widely used to measure evaluative associations towards groups or the self but is influenced by other traits. Siegel, Dougherty, and Huber (2012, Journal of Experimental Social Psychology) found that manipulating cognitive control via false feedback (Study 3) changed the degree to which the IAT was related to cognitive control versus evaluative associations. We conducted two replications of this study and a mini meta-analysis. Null-hypothesis tests, meta-analysis, and a small telescope approach demonstrated weak to no support for the original hypotheses. We conclude that the original findings are unreliable and that both the original study and our replications do not provide evidence that manipulating cognitive control affects IAT scores.
... Our failure to replicate in Study 1 could mean that the results from Wood et al. (2009) study were due to a Type I error or sampling variability (Maxwell et al., 2015). Regarding Study 2, we modified Wood et al. (2009) procedures and measures to make the methodology stronger but these changes also add the possibility that the failure to replicate was due to those modifications (Cesario, 2014;Maxwell et al., 2015;Simons, 2014;Stroebe & Strack, 2014). Similarly, the procedures and measures used in Study 2 varied from other values writing studies (e.g., comparison group was active; more targeted writing prompt), which, again, could explain the lack of difference found in Study 2 between the conditions on outcome variables (e.g., Cohen, Garcia, Purdie-Vaughns, Apfel, & Brzustoski, 2009;Creswell et al., 2005;Sherman et al., 2013). ...
Article
Objectives Previous research has established a differential effect of positive affirmations, whereby they may be helpful for individuals with high trait self-esteem and possibly iatrogenic for individuals with low self-esteem (Wood, Perunovic, & Lee, 2009). The current studies are replications of Wood et al. (2009) study. The aim of Study 1 was to replicate this study by examining the efficacy of a positive affirmation intervention on mood and state self-esteem and to see if trait self-esteem made the intervention more or less efficacious. The aim of Study 2 was to attempt to conceptually replicate and extend the original study by examining the efficacy of positive affirmation and values writing interventions on mood, self-esteem, and goal completion. We also examined whether trait self-esteem and psychological inflexibility moderated the relationship between condition and outcome variables. Design For Study 1, participants were randomly assigned to either the positive affirmation or no statement condition. Outcome measures were completed immediately following the intervention. For Study 2, participants randomly assigned to the positive affirmations or values writing intervention. Outcome measures were completed immediately following the intervention and at a three-day follow-up. Setting Participants completed this analogue study in a university setting. Participants Approximately 225 individuals participated in Study 1 and 237 in Study 2. Main outcome measures The main outcome measures of Study 1 were state self-esteem and mood measures. The outcome measures for Study 2 were state self-esteem, mood, goal competition, and challenge of self-reported goal. Results Results from Study 1 revealed a failure to replicate as there were no difference between conditions on all outcome variables and trait-self esteem did not interact with condition to predict any of the outcome variables. Findings from Study 2, again, yielded a failure to replicate, as there were no differences between conditions on mood, state self-esteem, self-reported level of goal challenge, and goal completion. Conclusions State self-esteem did not moderate the effect of a positive affirmation intervention, failing to replicate the findings of Wood et al. (2009). Implications regarding the failure to replicate the original study are discussed.
... Direct replications involve repeating the precise methodology of a previously conducted study, and conceptual replications involve testing the same hypothesis using different methods (Schmidt, 2009). The purpose of a direct replication is to determine the reliability of an effect, whereas a conceptual replication provides a new test of a theory (Simons, 2014). ...
Article
Full-text available
This paper presents a side-by-side consideration of multiplicity control procedures and replication as solutions to the problem of multiplicity. Several independent theoretical arguments are presented which demonstrate that replication serves several important functions, and that multiplicity control procedures have a number of serious flaws. Subsequently, the results of a simulation study are provided, showing that under typical conditions, replication provides similar familywise error control and power as multiplicity control procedures. Taken together, these theoretical and statistical arguments lead to the conclusion that researchers who are concerned about the problem of multiplicity should shift their attention away from multiplicity control procedures and towards increased use of replication.
... Необхідною умовою оприлюднення інформації про отримане нове знання є такий його опис у публікаціях, який дозволяє іншим дослідникам відтворити або спростувати результати дослідів. На жаль, така мовби тривіальна для природознавців вимога не виконується в більшості досліджень у царині соціальних та гуманітарних наук, наприклад в психології, де, як стверджується, результати до 70% психологічних експериментів [Randall, Welser, 2018;Simons, 2014; Replication crisis, 2021] не відтворюються. Головна епістемологична проблема полягає у тім, що на ґрунті таких «експериментів» будуються різні «теорії», які слугують як дороговказ для певних практичних дій. ...
... Psychologists disagree about the merits of such 'direct' replications (e.g., Simons, 2014;Stroebe & Strack, 2014;Zwaan et al., 2018) and it is not our intention to summarize the various viewpoints here. 2 Instead, we would like to highlight that replication is not only important from a statistical conclusion validity perspective, but also for reasons of internal validity, construct validity and external validity. ...
... The effect of working memory capacity on visual search efficiency has important theoretical implications. However, before addressing the implications for executive attention theory of working memory, it is essential to establish that this phenomenon is indeed reproducible (Simons, 2014). On this point, the presence of a large missing-letter effect with an RSVP procedure and prose passages extends previous findings (see., e.g., Newman et al., 2013;Saint-Aubin & Klein, 2010;Saint-Aubin et al., 2003, 2010 and further show that the missing-letter effect is not a by-product of eye-movements as assumed by some models (Corcoran, 1966;Hadley & Healy, 1991). ...
Preprint
Full-text available
Individual differences in working memory capacity (WMC) are related to variations in a wide range of cognitive tasks. Surprisingly, effects of individual differences in working memory capacity are somewhat limited in visual search tasks. Here we tested the hypothesis that such an effect would be robust when search was one component of a dual task. Participants were presented strings of letters using rapid serial visual presentation and were required to detect all instances of a particular target letter. In Experiment 1, participants performed the letter search task in three contexts, while: a) reading a prose passage, b) processing a stream of random words, or c) processing a random stream of non-words. In the absence of the dual task of reading prose, and in line with much of the literature on individual differences in WMC and visual search, search performance was unaffected by WMC. As hypothesized, however, higher working memory capacity participants detected more target letters than lower capacity participants in the “true” dual task (searching while reading prose). The hypothesized results from the prose passage were replicated in Experiment 2. These results show that visual search efficiency is dramatically affected by WMC when searching is combined with another cognitive task but not when it is performed in isolation. Our findings are consistent with recent suggestions that visual search efficiency will be affected by WMC so long as searching is embedded in a context that entails managing resource allocation between concurrent tasks.
... The underlying assumption of this intellectual project is that the preceding layers of knowledge are stable because they have met the criterion of scientific review (Pulverer 2015). One of the most basic criteria for research validity is reproducibility (Caplan and Redman 2018;Simons 2014;Van Bavel et al. 2016). Several types of replication tests exist: Close replication, Constructive replication, Conceptual replication in the laboratory, Conceptual replication in the field and repeatability (Hüffmeier et al. 2016). ...
Article
Full-text available
The literature discusses causes of low reproducibility of scientific publications. Our article adds another main cause—uncritical adherence to accepted research procedures. This is evident in: (1) anachronistically requiring researchers to base themselves on theoretical background even if the studies cited were not tested for reproducibility; (2) conducting studies suffering from a novelty effect bias; (3) forcing researchers who use data mining methods and field-based theory, with no preliminary theoretical rationale, to present a theoretical background that allegedly guided their work—as a precondition for publication of their findings. It is possible to increase research validity in relation to the above problems by the following means: (1) Conducting a longitudinal study on the same participants and only on them; (2) Trying to shorten the time period between laboratory experiments and those on humans, based on cost–benefit considerations, anchored in ethical norms; (3) Reporting the theoretical background in a causal modular format; (4) Giving incentives to those who meet the above criteria while moderating the pressure for fast output.
... Replicability asks the question, can a study's results can be repeated with new data? Direct replication studies evaluate the ability of a particular method to produce the same results upon repetition in the same setting (Simons, 2014). In IS, direct replication would include assessing an implementation strategy or bundle again in the same setting, with the same type of target population, and with the same EBP (Machery, 2020). ...
Article
Full-text available
Adapting the classic line from the 1967 film Cool Hand Luke, the title is meant to convey that implementation science (IS), like other fields, has not been embracing replication studies, which is a key component to the open science movement. The purpose of this article is to review what is known about replication of implementation trials and identify the gaps and next steps to continue increasing the transparency, openness, and replicability of implementation research. After presenting an overview of study replication and how it is a key component of open science, the article will examine how replication of implementation studies has (or more accurately has not) been approached in IS. As will be discussed, replication in IS shares some challenges with studies that attempt to replicate interventions, but also presents unique challenges. This article discusses different types of replications (e.g., direct vs. conceptual) and how they can benefit the field of IS. The article then presents a specific example of an implementation strategy called Getting To Outcomes© to describe how to design a replication study and interpret the results. The article ends with multiple options implementation scientists could consider to improve the likelihood and quality of replication studies. The discussion also envisions how implementation science can enable researchers and practitioners to work together in real-world contexts to encourage wide replication of implementation studies and advance the goal of improving public health.
... However, despite the potential significant impact that survey research can have, researchers face a high potential for mistakes and errors in collecting data for such research, which can impact replicability (Collins & Tabak, 2014;The Economist, 2013;Li, Hu, Xie, & He, 2015;Marsden & Pingry, 2018). Researchers can impact replicability of survey research in several ways: for instance, researchers may generalize results despite contextual factors without sufficiently documenting the contextual factors (Lynch, Bradlow, Huber, & Lehmann, 2015;Stroebe & Strack, 2014), they may use poorly designed data-collection procedures and practices (Simons, 2014), or they may fraudulently falsify data (Fang et al., 2012). ...
... To her, this would mean that they are objective (Hawkins & Nosek, 2012;Ziman, 1996;Stegenga, 2011). 2 Objectivity can be attributed, among others, to scientific measurements, tools for development/improvement of scientific theories, and/or to true-to-nature explanations. It ensures that study outcomes are not biased (e.g., over estimation of drug efficacy, under estimation of risk; Goldacre, 2014), positive research results are not false-positives (to a larger proportion than is allowed by the statistical method; Simmons et al., 2011), and are independently reproducible by other scientists (Simons, 2014;Lindsay, 2015;Altmejd et al., 2019;van Bavel et al., 2016). Dr. Summers considers objectivity to be essential to science 3 and its absence to be a cause of the crisis that threatens the foundations of her research field. ...
Article
Full-text available
In the last decade, many problematic cases of scientific conduct have been diagnosed; some of which involve outright fraud (e.g., Stapel, 2012) others are more subtle (e.g., supposed evidence of extrasensory perception; Bem, 2011). These and similar problems can be interpreted as caused by lack of scientific objectivity. The current philosophical theories of objectivity do not provide scientists with conceptualizations that can be effectively put into practice in remedying these issues. We propose a novel way of thinking about objectivity for individual scientists; a negative and dynamic approach.We provide a philosophical conceptualization of objectivity that is informed by empirical research. In particular, it is our intention to take the first steps in providing an empirically and methodologically informed inventory of factors that impair the scientific practice. The inventory will be compiled into a negative conceptualization (i.e., what is not objective), which could in principle be used by individual scientists to assess (deviations from) objectivity of scientific practice. We propose a preliminary outline of a usable and testable instrument for indicating the objectivity of scientific practice.
... Clinicians and clients alike can leverage the quality and nature of social relationships in concert with clients' personality profiles to inform treatment targets, such as maintaining meaningful romantic relationships and gainful employment. Beyond the importance of attempting to replicate previous findings (e.g., Open Science Collaboration, 2015; Simons, 2014), and given the nature of personality dimensions and their relationship with BPD-relevant outcomes in the context of close social relationships, this paper supports considerations of the intersections between underlying personality traits and situational context (i.e., CSC) in understanding affective processes in BPD. ...
Preprint
Full-text available
Dimensional models of personality, such as the Five Factor Model (FFM), have demonstrated strong coherence with the presentation of personality disorders, including Borderline Personality Disorder (BPD). Given that select personality trait elevations have been linked to impairments in multiple life domains across diagnostic groups, we sought to replicate findings from a previous investigation of the utility of the FFM in predicting BPD-relevant outcomes (i.e., negative affect [NA] intensity and instability, impulsivity, and interpersonal disagreements) in the daily lives of those with BPD (Hepp et al., 2016) and community participants. As interpersonal context is instrumental in determining the strength of effects observed in studies examining individuals with BPD, we utilized ecological momentary assessment across 3 weeks (6 times daily; ntotal=15,889) to test whether close social contact (CSC) would moderate the effects of personality on momentary outcomes. Overall, results suggest that CSC is an important moderator between the effects of personality and daily life outcomes for individuals with BPD (N=56), but not for community individuals (N=60). For individuals with BPD, CSC may function as both a protective buffer and a risk factor, depending on outcome. For example, CSC attenuates experience of NA intensity for individuals with elevated neuroticism, but CSC may predict more frequent disagreements for individuals who report lower agreeableness. We replicated approximately half of the original study’s findings and results support that FFM personality is predictive of BPD-relevant outcomes broadly. However, interpersonal context is key to understanding these relationships for individuals with BPD.
... But the problems outlined in the opening on replication failures and the WEIRD people problem remain. For example, behavioural priming is often unreliable (Simons, 2014), behaviour in economic games such as the dictator game varies from 47% offers in the US to 26% offers among the Hadza (Henrich et al., 2010b). ...
... Strategies for success lead as well to investigators' exploiting RDFs by making minor alterations to their research design and, as one psychologist described it, by "capitalizing on chance" (Bakker et al., 2012, pp. 544-545;John et al., 2012;Lilienfield, 2017;Pashler & De Ruiter, 2017;Simons, 2014). Diagnosing behavioral faults as caused by incentive structures draws upon economic language of incentives, investment of resources, opportunities, costs, self-interest, payoff, and maximizing. ...
Article
Psychology’s current crisis attends most visibly to perceived problems with statistical models, methods, publication practices, and career incentives. Rarely is close attention given to the objects of inquiry—to ontological matters—yet the crisis-related literature does features statements about the nature of psychology’s objects. Close analysis of the ontological claims reveals discrepant understandings: some researchers assume objects to be stable and singular while others posit them to be dynamic and complex. Nevertheless, both views presume the objects under scrutiny to be real. The analysis also finds each of these ontological claims to be associated not only with particular method prescriptions but also with distinct notions of the scientific self. Though both take the scientific self to be objective, one figures the scientist as not always a rational actor and, therefore, requiring some behavior regulation, while the other sees the scientist as largely capable of self-governing sustained through painstakingly acquired expertise and self-control. The fate of these prevalent assemblages of object, method, and scientific self remains to be determined, yet as conditions of possibility they portend quite different futures. Following description of the assemblages, the article ventures a futuristic portrayal of the scientific practices they each might engender.
... As Fiske and Campbell (1992) noted several decades later, citations do not solve problems. Many psychologists would argue that the most relevant problem in psychological science today is the replication crisis (Simmons et al., 2011;Simons, 2014). ...
Thesis
Full-text available
The purpose of this thesis was to assess the relationship between group rapport and nonverbal expressivity using three data sources: self-report, observer ratings, and test data. Assessing these constructs using multiple data sources enabled the construction of multitrait-multimethod matrices. These matrices allowed for a critical evaluation of the convergent and discriminant validity of the group rapport and nonverbal expressivity constructs. Participants (N = 162) were randomly assigned to small groups of 5-7 (24 groups total) and tasked with completing a puzzle activity in collaboration with their group members. Rapport has been colloquially defined as the “clicking, chemistry, and harmony” shared between interactants. After the activity, participants rated their rapport experience. Groups were filmed while completing the activity and objective raters assessed the groups on domains derived from the rapport (Tickle-Degnen & Rosenthal, 1987) and entitativity (Campbell, 1958) literatures. Group rapport has been theorized to be relevant for successful group collaboration in many applied contexts (e.g., business, health care, and engineering), therefore the primary outcome (test) measure of group rapport was whether groups successfully completed the puzzle activity before the other groups assigned to complete the same puzzle. It was expected that nonverbal expressivity (defined as the extent to which an individual uses their face, gestures, body, and voice to transmit emotion) would be associated with group rapport because expressive individuals are easier to accurately read and respond to compared to their unexpressive counterparts. Nonverbal expressivity had a weak relationship with group rapport, indicating that nonverbal expressivity may not be as important for effective group collaboration as it is for dyadic exchanges. In addition to the self-reports, observer ratings of group rapport and entitativity based on only ten-second segments (thin slices) of group behavior were associated with whether groups won the puzzle competition. Based on these findings, a development to group rapport theory is proposed that includes entitativity as a primary component of rapport in small groups. It is recommended that future investigations empirically test this supposition in addition to evaluating the utility of short segments of behavior (thin slices) to predict applied group outcomes.
... It is a complex question that the same concept of replicability of studies can partially answer; no single study can provide absolute truths. Replicability is a vital part of strengthening the scientific credibility of research itself (Francis, 2012;Ioannidis, 2012;Jasny, Chin, Chong, & Vignieri, 2011;Simons, 2014). ...
Article
Full-text available
The Covid pandemic has opened new challenges for education, especially for the social and emotional wellbeing of children and adolescents who had to face unprecedented and upsetting changes in their daily lives. The paper explores the possibilities offered by the social-emotional intelligence framework in helping children and youths develop the good emotional literacy needed for facing such a challenging time and growing as wholesome adults. This is done through an in-depth analysis of the concept of replication and generalization and by proposing a perspective working model for embedding social and emotional learning in daily teaching and learning activities. Promuovere l’intelligenza emotiva nel post-Covid. Approcci flessibili per insegnare le competenze sociali e emotive. La pandemia di Covid ha introdotto nuove sfide nel mondo dell’educazione, in modo particolare per quanto riguarda il benessere sociale e emotivo di bambini e adolescenti che hanno dovuto affrontare cambiamenti sconvolgenti senza precedenti nel loro vivere quotidiano. L’articolo esplora le possibilità offerte dal costrutto di educazione socio-emotiva a supporto dello sviluppo in bambini e ragazzi di un’alfabetizzazione emotiva solida, necessaria per affrontare un periodo così sfidante e per la loro crescita futura. Tutto questo è fatto attraverso un’analisi approfondita dei concetti di replicabilità e generalizzazione e attraverso la proposta di un nuovo modello di lavoro per integrare l’educazione socio-emotiva all’interno delle azioni didattiche quotidiane
... Along with initiating a discussion about the relationship between embeddedness and commitment in an HRD context, another contribution of this paper was to present the importance of replication and generalizability of theorized relationships in behavioural management research. The present research adhered to recent calls for replicating findings (Simons, 2014;Vazire, 2018) by using two cross-sectional samples surveyed at two different time points. This research is an example of Köhler and Cortina's (2021) dependent replication, which is achieved when the same authors replicate the study of a phenomenon more than once. ...
Article
Purpose This paper aims to apply the theoretical perspective of job embeddedness to delineate how organizations could bundle and implement specific HRD practices that cater to fit, connections and the psychological costs of leaving to influence employees’ organizational commitment. Design/methodology/approach Using a dual-study approach, the current research uses survey responses collected from two samples of working adults to test the theorized framework using structural equation modelling. Findings Replicated results reveal that on-the-job embeddedness predicts affective commitment. There was no association between embeddedness at the community level and organizational commitment in either study. Originality/value This research offers a fresh perspective to explore the direct influence that embeddedness has on organizational commitment in the context of HRD practices.
... A direct replication attempts to mimic the experimental methodology used in the original study. Some prefer direct over conceptual replications, which instead aim to extend a finding by using different procedures (e.g., Pashler & Harris, 2012;Simons, 2014). The authors of the ManyLabs project worked with us to ensure that their materials were nearly identical to ours. ...
Preprint
This manuscript contains our responses to several commentaries about the Many Labs Project (Klein et al., 2014).
... A direct replication attempts to mimic the experimental methodology used in the original study. Some prefer direct over conceptual replications, which instead aim to extend a finding by using different procedures (e.g., Pashler & Harris, 2012;Simons, 2014). The authors of the ManyLabs project worked with us to ensure that their materials were nearly identical to ours. ...
Preprint
This dataset is from the Many Labs Replication Project [1] in which 13 effects were replicated across 36 samples and over 6,000 participants. Data from the replications are included, along with demographic variables about the participants and contextual information about the environment in which the replication was conducted. Data were collected in-lab and online through a standardized procedure administered via an online link. The dataset is stored on the Open Science Framework website. These data could be used to further investigate the results of the included 13 effects or to study replication and generalizability more broadly.
... Here, rather than replicating one published protocol, we propose a paradigmatic replication (inspired by, but not identical to the replication format as used by Vohs et al., 2021). A direct replication aims to repeat a specific published protocol to produce the effect of interest (Simons, 2014). Two design choices in our paradigmatic replication deserve special attention. ...
Preprint
Full-text available
Post-traumatic stress disorder is characterised by recurring memories of a traumatic experience as well as the deliberate avoidance of those memories in order to forget. However, can intentional suppression really lead to forgetting? The Think/No-Think (TNT) task has been used widely in the laboratory to study suppression-induced forgetting. The idea is that actively suppressing the retrieval of a memory when faced with a reminder reduces its strength, and hence, the memory can become inaccessible across multiple suppression attempts. During the TNT task, participants first learn a series of cue-target word pairs (e.g., WAFFLE-MAPLE). Subsequently, they are presented with a subset of the cue-words and are instructed to either think (respond items) or not think about the corresponding target (suppression items; baseline items are not shown). Successful suppression-induced forgetting is thought to reduce recall of the suppressed items compared to baseline items in subsequent memory tests. Although recent meta-analyses have reported small-to-moderate effect sizes in this paradigm, the current replication represents a collaborative effort to evaluate this paradigm following pre-registration. In particular, we propose an online experimenter-present version inclusive of both the same- (e.g., WAFFLE) and independent-probe (e.g., TREE-M) tests in English-speaking healthy individuals using a direct suppression instruction.
Article
Davis's (1971) article “That's interesting! Towards a phenomenology of sociology and a sociology of phenomenology” is regarded by many management researchers as a classic work and a basis for guiding management studies; in the wake of its publication, an interesting research advocacy gradually emerged. However, from the perspective of scientific research, Davis's core argument that great theories have to be interesting is seriously flawed. Interestingness is not regarded as a virtue of a good scientific theory and thus has little value in science. Moreover, obsession with interestingness can lead to at least five detrimental outcomes, namely promoting an improper way of doing science, encouraging post hoc hypothesis development, discouraging replication studies, ignoring the proper duties of a researcher, and undermining doctoral education.
Thesis
Full-text available
With its context-independent rules valid in any setting, mathematics is considered to be the champion of abstraction, and for a long time human mathematical reasoning was thought to follow nothing but the laws of logic. However, the idea that mathematics is grounded in nature has gained traction over the past decades, and the context-independency of mathematical reasoning has come to be questioned. The thesis we defend concerns the role played by general, non-mathematical knowledge on individuals' understanding of numerical situations. We propose that what we count has a crucial impact on how we count, in the sense that human's representation of numerical information is dependent on the semantic context in which it is embedded. More specifically, we argue that general, non-mathematical knowledge about the entities described in a mathematical word problem can shape its interpretation and foster one of two representations: either a cardinal encoding, or an ordinal encoding. After introducing a new framework of arithmetic word problem solving accounting for the interactions between mathematical knowledge and world knowledge in the encoding, recoding and solving of arithmetic word problems, we present a series of 16 experiments assessing how world knowledge about specific quantities can promote one of two problem representations. Using isomorphic arithmetic word problems involving either cardinal quantities (weights, prices, collections) or ordinal quantities (durations, heights, number of floors), we investigate the pervasiveness of the cardinal-ordinal distinction in a wide range of activities, including problem categorization, problem comparison, algorithm selection, problem solvability assessment, problem recall, sentence recognition, drawing production and transfer of strategies. We gather data using behavioral measures (success rates, algorithm use, response times) as well as eye tracking (fixation times, saccades, pupil dilation), to show that the difference between problems meant to foster either a cardinal or an ordinal encoding has a far-reaching influence on participants from diverse populations (N = 2180), ranging from 2nd graders and 5th graders to lay adults, expert mathematicians and math teachers. We discuss the general educational implications of these effects of semantic (in)congruence, and we propose new directions for future research on this crucial issue. We conclude that these findings illustrate the extent to which human reasoning is constrained by the content on which it operates, even in domains where abstraction is praised and trained.
Preprint
The content of this dissertation spans four years of work, which was carried out in the Netherlands (Tilburg University and University of Amsterdam) and Italy (University of Turin). It is part of the ERC project “Making Scientific Inference More Objective” led by professor Jan Sprenger, for which philosophy of science and empirical research were combined. The dissertation can be summarized as a small set of modest attempts to contribute to improving scientific practice. Each of these attempts was geared towards either increasing understanding of a particular problem or making a contribution to how science can be practiced. The general focus was on philosophical nuance while remaining methodologically practicable. The five papers contained in this dissertation are both methodologically and philosophically diverse. The first three (Chapters 2 through 4) are more empirical in nature and are focused on understanding and evaluating how science is practiced: a meta-analysis of semantic intuitions research in experimental philosophy; a systematic review on essay literature on the null hypothesis significance test; and an experiment on how teams of statisticians analyze the same data. The last two (Chapters 5 and 6) are focused on the improvement of scientific practice by providing tools for the improvement of empirical research with a strong philosophical foundation: a practicable and testable definition of scientific objectivity and a Bayesian operationalization of Popper’s concept of a severe test.
Article
Machery (2020) has recently proposed a "resampling" account of experimental replication to dissolve a debate in psychology about the relative merits of direct and conceptual replication. We argue that (i) on matters of replication’s function and typology, the resampling account is not substantially different from the functional account of replication extant in the literature; (ii) on what generalizations can be drawn from replications, the resampling account is too restrictive and relies on a misunderstanding of the relation between random sampling and generalizability; and (iii) Machery’s reading of the debate on the relative importance of direct and conceptual replication elides a deeper debate about values and the distribution of research resources in science.
Article
Full-text available
Verbal overshadowing describes the phenomenon in which verbalisation negatively affects performance on a task related to the verbalised material. Within the verbal overshadowing literature, three accounts exist which attempt to explain this phenomenon: content, processing, and criterion accounts. The content account refers to the notion that the specific contents of verbalisation interfere with later performance, processing refers to a proposed shift in processing caused by verbalisation, and criterion deals with the possibility that verbalisation leads to a reliance on more conservative choosing. The current manuscript reviews evidence for the existing accounts, while describing advantages and disadvantages of each account and attempting to reconcile these various accounts. The authors provide a framework for understanding verbal overshadowing as caused by one unified mechanism, or several. Finally, an outline for future research is suggested that should aid in reconciling the existing accounts for verbal overshadowing.
Article
Full-text available
Recent studies have demonstrated that requesting individuals to produce a verbal description of a previously seen face can hinder subsequent attempts at identification. This phenomenon, termed ‘verbal overshadowing’, has been studied rather extensively in the face-identification paradigm; however, studies have not always replicated the general effect. Based upon both practical and theoretical interests in the phenomenon, a meta-analysis of 29 effect size comparisons (N = 2018) was conducted. Across the sample of studies there was a small, yet significant, negative effect (Fisher's Zr = −0.12), indicating some degree of verbal impairment or overshadowing. A fixed-effects analysis of several moderating variables demonstrated a significant effect of post-description delay and type of description instruction. The pattern of means indicated that overshadowing effects were more likely to occur when the identification task immediately followed the description task, and when participants were given an elaborative, as opposed to a standard (free recall), instruction during the description task. Inconsistencies in the literature are discussed, as well as various theoretical and applied issues regarding the verbal overshadowing effect. Copyright © 2001 John Wiley & Sons, Ltd.
Article
Full-text available
Empirical replication has long been considered the final arbiter of phenomena in science, but replication is undermined when there is evidence for publication bias. Evidence for publication bias in a set of experiments can be found when the observed number of rejections of the null hypothesis exceeds the expected number of rejections. Application of this test reveals evidence of publication bias in two prominent investigations from experimental psychology that have purported to reveal evidence of extrasensory perception and to indicate severe limitations of the scientific method. The presence of publication bias suggests that those investigations cannot be taken as proper scientific studies of such phenomena, because critical data are not available to the field. Publication bias could partly be avoided if experimental psychologists started using Bayesian data analysis techniques.
Article
Full-text available
Some effects diminish when tests are repeated. Jonathan Schooler says being open about findings that don't make the scientific record could reveal why.
Article
Full-text available
Three experiments explored the role of perceptual expertise in mediating the finding (termed verbal overshadowing) that describing a face can impair later recognition. In Experiment 1, verbalization impaired White participants' recognition of White faces (expert domain) but not African American faces (novice domain). In Experiment 2, judges attempted to identify targets on the basis of the verbal descriptions generated in Experiment 1. Experiment 2 revealed a significant relationship between verbalization participants' recognition performance and yoked judges' identification performance for other-race but not own-race faces, suggesting that other-race recognition may involve a unique reliance on "verbalizable" information. In Experiment 3, the interaction between verbalization and race of face was replicated with upright faces but was attenuated with inverted recognition arrays (a manipulation that reduces the influence of configural information). Collectively, these findings suggest that verbalization may disrupt the nonreportable configural processes associated with recognizing stimuli with which one is an expert.
Article
Full-text available
This article investigated the role of the recognition criterion in the verbal overshadowing effect (VOE). In 3 experiments, people witnessed an event, verbally described a perpetrator, and then attempted identification. The authors found in Experiment 1, which included a "not present" response option and both perpetrator-present (PP) and perpetrator-absent (PA) lineups, an increased reluctance to identify a person from both lineup types after verbalization. Experiment 2 incorporated a forced-choice procedure, and the authors found no effect of verbalization on identification performance. Experiment 3 replicated the essential aspects of these results. Consequently, the VOE may reflect a change in recognition criterion rather than a changed processing style or alteration of the underlying memory trace. This conclusion was confirmed by computational modeling of the data.
Article
It is widely believed that verbal processing generally improves memory performance. However, in a series of six experiments, verbalizing the appearance of previously seen visual stimuli impaired subsequent recognition performance. In Experiment 1, subjects viewed a videotape including a salient individual. Later, some subjects described the individual's face. Subjects who verbalized the face performed less well on a subsequent recognition test than control subjects who did not engage in memory verbalization. The results of Experiment 2 replicated those of Experiment 1 and further clarified the effect of memory verbalization by demonstrating that visualization does not impair face recognition. In Experiments 3 and 4 we explored the hypothesis that memory verbalization impairs memory for stimuli that are difficult to put into words. In Experiment 3 memory impairment followed the verbalization of a different visual stimulus: color. In Experiment 4 marginal memory improvement followed the verbalization of a verbal stimulus: a brief spoken statement. In Experiments 5 and 6 the source of verbally induced memory impairment was explored. The results of Experiment 5 suggested that the impairment does not reflect a temporary verbal set, but rather indicates relatively long-lasting memory interference. Finally, Experiment 6 demonstrated that limiting subjects' time to make recognition decisions alleviates the impairment, suggesting that memory verbalization overshadows but does not eradicate the original visual memory. This collection of results is consistent with a recording interference hypothesis: verbalizing a visual memory may produce a verbally biased memory representation that can interfere with the application of the original visual memory.
Article
Three experiments tested the effect of verbal description on face identification accuracy. Based on verbal overshadowing research, it was predicted that enhancing verbal description of a face would reduce subsequent face identification accuracy. Experiment 1 tested and confirmed this hypothesis using the cognitive interview to enhance verbal description; face identification accuracy was reduced significantly following the cognitive interview, compared with a standard police interview. Experiments 2 and 3 tested and confirmed the hypothesis that verbal overshadowing would be reduced when a delay is inserted between verbal description and face identification, hence resulting in "release from verbal overshadowing." These results suggest that in the verbal overshadowing task, the verbal description does not overwrite the visually based representation of the face in memory but rather makes it less accessible at the time of face identification. The cognitive interview reduces face identification accuracy only when the identification follows description immediately--a rare situation in real criminal cases.
The truth wears off: Is there something wrong with the scientific method? The New Yorker
  • J Lehrer
Lehrer, J. (2010, December 13). The truth wears off: Is there something wrong with the scientific method? The New Yorker, 52-57.