Content uploaded by Ali H. Al-Hoorie
Author content
All content in this area was uploaded by Ali H. Al-Hoorie on Mar 24, 2022
Content may be subject to copyright.
FROM REPLICATION TO SUBSTANTIATION:
A COMPLEXITY THEORY PERSPECTIVE
Ali H. Al-Hoorie, Royal Commission for Jubail and Yanbu, Saudi Arabia
Phil Hiver, Florida State University, USA
Diane Larsen-Freeman, University of Michigan, USA
Wander Lowie, University of Groningen, Netherlands
Presented at AAAL, Pittsburgh, Pennsylvania
20-Mar-2022
OUTLINE
•Why replication?
•Classifications of replication research
•Limitations of these classifications
•Substantiation framework
•Complexity perspective
WHY REPLICATION?
WHY REPLICATION?
•Why should journals accept (let alone encourage) replication?
•Isn’t science about finding new things, not repeating what was previously done?
•The problem with this is
•Creating an incentive structure that favors innovation over verification
•Unreplicable findings will accumulate
•Replication crisis, confidence crisis, devaluing science
•The replication dilemma
•If you find the same results, the journal & reviewers will say: we already know that
•If you don’t find the same results, the journal, reviewers & initial authors may blame you
•Your expertise
•Design deviation
CLASSIFICATIONS OF REPLICATION RESEARCH
CLASSIFICATIONS OF REPLICATION
CLASSIFICATIONS OF REPLICATION
(Clemens, 2017)
CLASSIFICATIONS OF REPLICATION
CLASSIFICATIONS OF REPLICATION
CLASSIFICATIONS OF REPLICATION
We do not want to add to the confusion here with
more definitions! Rather, we intend to present you
with three existing definitions and define a
practical use for each within a systematic,
cumulative approach to replication work. The
intention is to set out firm replication research
series which are interdependent, and in which you
can participate –first through close, followed by
approximate, and perhaps then supplemented by
conceptual replications. (p. 72)
Close: Change in one variable
Approximate: Change in two variables
Conceptual: More deviation
•A lot of confusion over terminology
•Different fields have different conventions
•In the field of language learning, some have similarly proposed certain terms
•Thus, replication terms not set in stone
CLASSIFICATIONS OF REPLICATION
LIMITATIONS OF THESE CLASSIFICATIONS
LIMITATIONS
•How do we interpret failed replications?
•If failed replications do not question the initial results in some way, what’s the point?
•Does all this apply to open systems?
•Unlike lab-controlled settings, many phenomena are too complex to control
•Even if they follow deterministic structures (Lorenz, 1963)
•Does it make sense to attempt replication, if we already know it will fail (sensitivity to initial conditions)?
•Do failed replications mean question initial results, or our own lack of knowledge?
•Should we call all these “replications”?
•Or are some “extensions”? (e.g., partial replication)
•Is it a conceptual replication, or a completely different study?
•“Conceptual replications… are in a weaker position for ascribing different findings to the
adaptations made to the initial study” (Marsden et al., 2018, p. 366)
•“[Conceptual replications] can be a more high-risk undertaking than close or approximate
replications in the sense that failure to replicate will leave us with little or anything to say about
the original” (Porte & McNanus, 2019, p. 94)
•“if our study did not come to the same conclusion, can we say we have “conceptually not
replicated” our original? Obviously not” (Porte & McNanus, 2019, pp. 93–94)
LIMITATIONS
•“when a conceptual replication fails to support a theory, rather than reduce our belief in the
theory, we are tempted to explain the failure in terms of methodological problems with the
operationalization of the key variables. As such, conceptual replication has been described by
critics as solely a mechanism for confirmation bias” (Crandall & Sherman, 2016, p. 97)
•“the likelihood of being able to publish a conceptual replication failure in a journal is very low.
But here, the failure will likely generate no gossip—there is nothing interesting enough to talk
about here” (Pashler & Harris, 2012, p. 533)
LIMITATIONS
SUBSTANTIATION FRAMEWORK
SUBSTANTIATION FRAMEWORK
•Three conditions need to exist:
•Result interpretability
•Theoretical maturity
•Terminological precision
SUBSTANTIATION FRAMEWORK
•1) Result interpretability
•Replication should not be defined in relation to its operational characteristics (how similar to
the initial study: direct, partial, conceptual)
•Instead, “a study for which any outcome would be considered diagnostic evidence about a
claim from prior research” (Nosek & Errington, 2020, p. 1)
•Therefore,
•positive results must support initial study findings
•Negative results must question initial study findings
SUBSTANTIATION FRAMEWORK
•2) Theoretical maturity
•Theory explains necessary and sufficient conditions to obtain a finding
•Therefore, even with some design deviations a replication can still be direct
•Constraints-on-Generality statements “The current publishing model incentivizes authors to make the
strongest possible claims of generality; broadly generalizable findings are more likely to be published
and more likely to be influential” (Simons et al., 2017, p. 1124)
•Failed conceptual replications can still hold evidentiary value
•Verification of postulated processes
•Testing the underlying theory
SUBSTANTIATION FRAMEWORK
•3) Terminological precision
•Are all types really “replication”?
•E.g., partial and conceptual replications are actually:
•an extension,
•a follow-up, or
•a generalizability test
•A conceptual replication is ‘a practical oxymoron’ (Freese & Peterson, 2018, p. 302)
•Repeat what?
•Is it appropriate to use “replication” as the umbrella term?
SUBSTANTIATION FRAMEWORK
SUBSTANTIATION FRAMEWORK
Al-Hoorie et al. (in press)
•For open complex system, also:
•Interactions: the way that components of any treatment/process influence each other
•Iterativity: what happens in the next step in the process depends on the preceding step
•Interdependence: various nested processes that unfold over many time interdependent timescales
•Self-organization: attractors emerge spontaneously when components and their interactions become
self-sustaining
•When CDST “replicates” an effect, it does not refer to a simple relationship, but to something
that takes place in the form of complex, iterative, time-scaled, situated processes (Hiver & Al-
Hoorie, 2020; Hiver et al., in press)
•Can this be a direct replication? Or only conceptual extension?
COMPLEXITY PERSPECTIVE
•“This paper provides evidence that contextual factors are associated with reproducibility,
even after adjusting for other methodological variables reported or hypothesized to impact
replication success. Attempting a replication in a different time or place or with a different
sample can alter the results of what are otherwise considered ‘direct replications.’ The results
suggest that many variables in psychology and other social sciences cannot be fully
understood apart from the cultural and historical contexts that define their meanings”
•Van Bavel et al. (2016, p. 6457)
COMPLEXITY PERSPECTIVE
VIABLE DIRECTIONS
•Theoretical maturity
•Constraints of generality
•Mini-theory
•Encourage exploratory (descriptive) research, not just confirmatory
•“Confirmatory research bias”?
•Open science and transparency
•Expect effect sizes from individual studies to be overestimated
•Adopt a meta-analytic mindset to disconfirm results
•Not only “replication” but broader “substantiation” efforts
•“Many solutions suggested for the [reproducibility] concerns… are decades old
(Meehl 1967). The reproducibility crisis presents a sober occasion to revisit them,
given our accumulating research experience.”
•“Inevitably, psychology will still face a reproducibility problem 20 years from now,
even when recommendations such as preregistration, open materials, data, and code
are standard (cf. Meehl 1990a).”
•Alexander and Moors (2018, pp. 13–14)
IN CONCLUSION
•Replication is HARD
•Especially with open systems
•Just like we realized that preregistration is HARD (Nosek et al., 2019)
•It would be naïve to think of replication as “just doing the same study again”
•If there is one thing that we have learned from the history of science:
•“Anything that is complex, upon closer examination becomes more complex” (Hansen, 2011, p. 119).
WITH THANKS TO
Phil Hiver Diane Larsen-Freeman Wander Lowie
REFERENCES
•Alexander, D. M., & Moors, P. (2018). If we accept that poor replication rates are mainstream. Behavioral and Brain Sciences, 41, e121.
https://doi.org/10.1017/S0140525X18000572
•Al-Hoorie, A. H., Hiver, P., Larsen-Freeman, D., & Lowie, W. (in press). From replication to substantiation: A complexity theory perspective. Language Teaching.
•Clemens, M. A. (2017). The meaning of failed replications: A review and proposal. Journal of Economic Surveys, 31(1), 326–342. https://doi.org/10.1111/joes.12139
•Crandall, C. S., & Sherman, J. W. (2016). On the scientific superiority of conceptual replications for scientific progress. Journal of Experimental Social Psychology,
66, 93–99. https://doi.org/10.1016/j.jesp.2015.10.002
•Freese, J., & Peterson, D. (2018). The emergence of statistical objectivity: Changing ideas of epistemic vice and virtue in science. Sociological Theory, 36(3), 289–
313. https://doi.org/10.1177/0735275118794987
•Hansen, W. B. (2011). Was Herodotus correct? Prevention Science, 12(2), 118–120. https://doi.org/10.1007/s11121-011-0218-5
•Hiver, P., & Al-Hoorie, A. H. (2020). Research methods for complexity theory in applied linguistics. Multilingual Matters.
•Hiver, P., Al-Hoorie, A. H., & Reid, E. (in press). Complex dynamic systems theory in language learning: A scoping review of 25 years of research. Studies in
Second Language Acquisition. https://doi.org/10.1017/S0272263121000553
•Lorenz, E. N. (1963). Deterministic nonperiodic flow. Journal of Atmospheric Sciences, 20(2), 130–141. https://doi.org/10.1175/1520-
0469(1963)020<0130:DNF>2.0.CO;2
•Marsden, E., Morgan-Short, K., Thompson, S., & Abugaber, D. (2018). Replication in second language research: Narrative and systematic reviews and
recommendations for the field. Language Learning, 68(2), 321–391. https://doi.org/10.1111/lang.12286
•Nosek, B. A., & Errington, T. M. (2020). What is replication? PLOS Biology, 18(3), e3000691. https://doi.org/10.1371/journal.pbio.3000691
REFERENCES
•Nosek, B. A., Beck, E. D., Campbell, L., Flake, J. K., Hardwicke, T. E., Mellor, D. T., van ’t Veer, A. E., & Vazire, S. (2019). Preregistration is hard, and worthwhile.
Trends in cognitive sciences, 23(10), 815–818. https://doi.org/10.1016/j.tics.2019.07.009
•Pashler, H., & Harris, C. R. (2012). Is the replicability crisis overblown? Three arguments examined. Perspectives on Psychological Science, 7(6), 531–536.
https://doi.org/10.1177/1745691612463401
•Plesser, H. E. (2018). Reproducibility vs. replicability: A brief history of a confused terminology. Frontiers in Neuroinformatics, 11, Article 76.
https://doi.org/10.3389/fninf.2017.00076
•Porte, G. K., & McManus, K. (2019). Doing replication research in applied linguistics. Routledge.
•Simons, D. J., Shoda, Y., & Lindsay, D. S. (2017). Constraints on Generality (COG): A proposed addition to all empirical papers. Perspectives on Psychological
Science, 12(6), 1123–1128. https://doi.org/10.1177/1745691617708630
•Van Bavel, J. J., Mende-Siedlecki, P., Brady, W. J., & Reinero, D. A. (2016). Contextual sensitivity in scientific reproducibility. Proceedings of the National
Academy of Sciences, 113(23), 6454-6459. https://doi.org/10.1073/pnas.1521897113
Thank You
@Ali_AlHoorie
hoorie_ali@hotmail.com
www.ali-alhoorie.com