Methodological Innovations in Studying Complex Systems in Applied Linguistics

Abstract

A quarter of a century ago, scholars first proposed that applied linguistics issues could benefit by being viewed explicitly from the lens of complex dynamic systems theory (CDST). Since then, CDST has gained considerable currency in the field, and it has yielded significant insights in domains as diverse as educational linguistics, second language development, sociolinguistics, and multilingualism. There has also been a recent increase in practical guidance around research methods that can be applied to studying complex systems in applied linguistics. In this talk, we first highlight the innovative assumptions underlying CDST research methods and the consequences of adopting these—namely, that when researching human and social phenomena everything counts and everything is connected (i.e., the relational principle), and everything changes (i.e., the adaptive principle). We then report a scoping review of the heterogenous body of research adopting this framework by looking back at the methodological characteristics of two decades of empirical CDST studies in the field to note trends and tendencies in designs and analytical choices. We discuss the many strands of applied linguistics research that have been informed by CDST and its innovative perspectives. We also highlight the substantive contributions this body of research has made to the field and the ways that CDST research has allowed the field to adopt a transdisciplinary stance that is more problem-oriented and transcends disciplinary boundaries. Finally, we showcase an innovative analytical method—mixed-effects location scale models—that enables the assessment of systematic change in within-person variability and is useful for predicting between-person volatility, stability, and variation in development. Combining it with generalized additive models, we can also examine nonlinearity in the change in variability. This talk both examines ways CDST expands the toolbox of research methods available and describes practical templates and methods suited to studying dynamic change in context and interconnectedness.

Full text

Methodological Innovations in Studying Complex
Systems in Applied Linguistics
Methodological Innovation in Applied Linguistics Research: Perspectives,
Strategies, and Trends (organized by Shaofeng Li & Matthew Prior)
Ali Al-Hoorie
Saudi TESOL Association
hoorie_a@rcjy.edu.sa
Phil Hiver
Florida State University
phiver@fsu.edu
Akira Murakami
University of Birmingham
a.murakami@bham.ac.uk

“The starting point of research is, as emphasized by Einstein
and others, astonishment. As long as a problem remains
unsolved we imagine many solutions. … [But] the future is not
given and therefore we have only a probabilistic description
and there is no certainty.… Uncertainty and surprise are part
of human destiny.”
Ilya Prigogine (2005, p. 16)
Why innovate methodologically?

Figure from (Sahin et al., 2020)
Wicked Problems
“Wicked” problems (Hiver et al., 2022)
are diabolical because they:
•span multiple social levels,
•affect the lives of many individuals,
institutions, societies,
•are intricately interconnected with other
problems,
•have no easily defined end-points,
•defy easy solutions and resist most
attempts to resolve them,
•have no consensus understandings or
readily apparent resolution.

“There are many interacting factors at play which determine the trajectory of one’s
develop[ment]: the source language, the target language, the markedness of the L1, the
markedness of the L2, the amount and type of input, the amount and type of interaction, the
amount and type of feedback received, whether it is acquired in untutored or tutored
contexts…age, aptitude, sociopsychological factors such as motivation and attitude,
personality factors, cognitive style, hemisphericity, learning strategies, sex, birth order,
interests, etc. Perhaps no one of these by itself is a determining factor, the interaction
of them, however, has a very profound effect.
Larsen-Freeman (1997, pp. 151-152)
What’s the “problem”?
The relational principle
Everything is connected and everything counts;
Human and social phenomena only make sense
in relation to other phenomena.
The adaptive principle
Everything changes; Human and social
phenomena are non-stationary and history
matters.

Chasing “wicked” problems…
“SLA must be particularly responsive to the needs of people who learn to live—and in fact do live—with more
than one language at various points in their lives, with regard to their education, their multilingual and multiliterate
development, social integration, and performance…across private and public, material and digital social contexts in a
multilingual world…. to meet the challenge of responding to the pressing needs of additional language users,
their education, their multilingual and multiliterate development, social integration, and performance across diverse
globalized, technologized, and transnational contexts”
The Douglas Fir Group (2016, pp. 20-24)

•Which type of conversation dyad (L1er-L1er; L1er-L2er; or L2er-L2er) features more
negotiation?
Varonis, E., & Gass, S. (1985). Non-native/non-native conversations: A model for negotiation of meaning.
Applied Linguistics, 6(1), 71-90.
Loschky, L. (1994). Comprehensible input and second language acquisition: What is the relationship?
Studies in Second Language Acquisition, 16(3), 65-89.
•How does input, with and without interactional modification, lead to comprehension and
retention of L2 features?
Egi, T. (2007). Interpreting recasts as linguistic evidence: The roles of linguistic target, length, and degree of
change. Studies in Second Language Acquisition, 29(4), 511-537.
•How do learners interpret recasts (given their inherent ambiguity)?
•How do recast features (i.e., target, length, number of changes) affect those interpretations?
Methodological innovation in action…
cited bycited by

•How does the type of modified output learners produce (i.e., no modified output, partial
modified output, or full modified output) relate to their accurate noticing of the feedback
provided in FTF and in SCMC environments?
Gurzynski-Weiss, L., & Baralt, M. (2014). Does type of modified output correspond to learner noticing of
feedback? A closer look in face-to-face and computer-mediated task-based interaction. Applied
Psycholinguistics, 36, 1393-1420.
Fu, M., & Li, S. (2021). The associations between implicit and explicit language aptitude and the effects of
timing of corrective feedback. Studies in Second Language Acquisition, 43(3), 498-522.
•What is the nature of the interface between learners’ language aptitudes (implicit; explicit) with the
timing of feedback provided to them (i.e., immediate CF; delayed CF; and task only) in the context
of focused communicative tasks.
Innovation in action (continued)…
cited bycited by

Mechanisms of acquisition/development
The (in)stability of the developing language system
Individual differences
New Ways of Seeing…
Kliesch, M., & Pfenninger, S. (2021). Cognitive and socio-affective predictors of L2 microdevelopment in late
adulthood: A longitudinal intervention study. The Modern Language Journal, 105(1), 237-266.
• What do overall trajectories of L2 learning look like?
• When do periods of significant growth in learners’ oral and written L2 performance occur?
• How well does the overall pattern of L2 development represent individual trajectories?
•Which cognitive variables predict L2 development?
•Which cognitive factors predict between-participant variation in L2 development and which predict within-
participant variation?
•How do the relationships between cognitive variables and L2 performance vary over time?
• Do socio-affective and background variables (education, age, and multilingualism) moderate the relationship
between cognition and L2 development?
Definition of learning
The effects of instruction

•We update our priors:
With each study we know more about more. Each future study rests on and extends from those
assumptions.
•We run down specific threads of knowledge:
We know more about more focused topics/questions. Each future study probes for more specific
nuanced knowledge.
•Methodological sophistication advances:
We learn methodological lessons as the field’s methodological knowledge grows and as meta-science
encourages greater transparency and rigor.
•We deal with increasingly “wicked problems” and ask questions that are increasingly
complex:
Methodological innovation can provide solutions for this “pain point”.
Why innovate methodologically?

Taking stock of research innovation for complex systems
Since it was first introduced in language learning (SLRF paper given by Larsen-Freeman in 1994)
CDST has made important contributions to the study of.
• language development/acquisition (de Bot, 2008)
• language attrition (Schmid et al., 2013)
• language change (Kretzschmar, 2015)
• language ecology (Cowley, 2011)
• language evolution (Ke & Holland, 2006)
• language policy and planning (Larsen-Freeman, 2018)
• language pedagogy (Mercer, 2016)
• language learners (Gurzynski-Weiss, 2020).
!What methodological innovations have accompanied these advances?

Systematic Review: objectives
1. Examine methodological characteristics of empirical CDST studies; identify
trends and tendencies in designs and analytical choices.
2. Assess contributions CDST research has made to the field.
3. Evaluate rigor of CDST empirical work (i.e., limitations and potential areas
for enhancing) future directions.
RQ1: What are the methodological characteristics of CDST studies in the field (including participants,
contexts, timescales, and analytic strategy)?
RQ2: What are the substantive contributions of these CDST studies to the field (i.e., theoretical and
practical)?
RQ3: What, if any, areas for improving CDST study quality are apparent?

•Initial Search: a search for studies spanning the 25-year period of interest (1994–
2019); Database (i.e., ERIC, LLBA, MLA, ProQuest, and PsychINFO)
•Search scope: peer-reviewed articles, book chapters, conference papers and
proceedings, and doctoral dissertations.
•This search returned a total of 2,341 hits from the combined database.
Database Search

Inclusion Criteria
1. It must involve an empirical design (whether quantitative, qualitative, or mixed method).
2. It must explicitly identify itself as operating within, or informed by, CDST or its terminological
antecedents.
3. It must be related to language learning.
4. It must be in English.
5. It must be available by 2019.
Based on our inclusion criteria, the final report pool (k= 158) included 89 journal articles and 69 dissertations.

Participants
Nearly 80% of all studies featured a sample size of N≤ 50.
The largest sample size in the article pool was N= 924
(Mdn = 13.5). The largest sample size in the dissertation
pool was N= 1,723 (Mdn = 16).
Studies with younger participants were in the minority,
with 112 studies (70.8%) sampling either university
students or adults aged 18 or older.
The rarest were studies with participants aged seven
years and younger (4 studies) followed by those with
respondents aged 7–12 (10 studies).

Timescales
Study design: over a third (59 studies) were cross-
sectional, more than 53% of studies (84 studies) were
longitudinal in design.
Study length: data elicitation took place most often over
a span of months (54 studies), followed by studies with a
timespan of weeks (33 studies), years (32 studies), hours
(9 studies), and days (5 studies). Study length ranged
from 90 minutes to four years.

General Designs
Over 80% of studies (130 studies) were exploratory and
only 28 studies had a falsificatory aim.
The unit of analysis in 73 studies was the group, and in 70
studies it was the individual.
The choice of method was split across qualitative (74
studies), quantitative (46 studies), and mixed methods (36
studies).

Data collection technique most frequently adopted
was interviews and focus groups (68 studies; 43%).
Qualitative coding and analysis methods were employed
most often in the reviewed studies (64 studies; 40.5%).
Twenty-four other studies (15% of the total) adopted
dynamic statistical analysis.
Analytical Strategies

Contributions
• Advances have been made in
describing complex systems and
identifying various dynamic changes.
•The important role of context in
understanding development is
clearly apparent.
• Work has begun to model complex
mechanisms and dynamic patterns
in learners’ development.
•Work is still needed to understand
how to intervene or influence
systems’ behavior.

•As a research community, the field has developed new ways of operating that are accompanied by
and that “require a different framing” (Larsen-Freeman, 2020, p. 202).
•There is a growing recognition of the importance of innovating with new modes of data elicitation
and dynamic analytical strategies, whether case-based or variable-based (Hiver & Al-Hoorie, 2020).
•We turn now to describing/showcasing one of these methodological innovations.
What have CDST’s methodological innovations done for
our field so far?
•dynamic network analysis
• idiodynamic method
• process tracing
• agent-based modeling
• single-case designs
•panel designs
•growth curve modeling
• design-based intervention research
• state space grids/models
•experience sampling method
•retrodictive qualitative modeling
•experimental ethnography
•time series analysis
• location-scale models
•qualitative comparative analysis
•generalized additive mixed-effects models
•longitudinal cluster analysis
• change point analysis

Analyzing variability
•CDST holds that variability, which has been traditionally considered as ‘noise’, is
informative as to one’s development (e.g., van Geert & van Dijk, 2002).
•However, there have been limited ways to analyze variability quantitatively (e.g.,
Spoelman & Verspoor, 2010; Verspoor & van Dijk, 2011).
•In order to increase the repertoire of the techniques to analyze variability, we will
introduce location-scale models (e.g., Hedeker et al., 2008; Williams et al., 2019;
see also Coupé, 2018).
•We will showcase their use in the context of the longitudinal development of
syntactic complexity in L2 English writing.

Location-scale models
•A location-scale model consists of two sub-models: A location model and a scale
model.
•A location model pertains to the relationship between the mean (or any other
location in the probability distribution) of the outcome variable and predictor
variables (e.g., as X increases, Y decreases).

Location-scale models
•A scale model, on the other hand, pertains to the relationship between the
variability of the outcome variable and predictors (e.g., as X increases, the
variance of Y decreases).
•As in location models, it can be of any complexity (e.g., random effects, nonlinear
relationships through splines).

Case study
•92 Saudi learners of English were asked to write seven essays (i.e., seven
waves) over the course of a semester.
•The task type was description in all the occasions, but the topics varied across
the seven assignments (e.g., Describe your best friend. Describe what you
usually do on weekends).
•Target measure: clause length
•Each essay was syntactically parsed automatically (Klein & Manning, 2003), and
the number of clauses was counted in each sentence based on the Tregex
pattern (Levy & Andrew, 2006) used in L2 Syntactic Complexity Analyzer (Lu,
2010).

Descriptive figure
•Large individual variation in the level
of mean clause length.
•The overall developmental pattern is
less clear.
•The variability differs somewhat
across waves, and the error bar
appears to be larger in Waves 1, 4,
6, and 7 than in the other waves.

Analysis
•It is generally inappropriate to model count(-based) variables based on a typical
linear regression model (e.g., Murakami, 2020; Winter & Bürkner, 2021).
•A negative binomial regression model was employed to model clause length.
•Observational unit: Sentence
•Bayesian model with brms (Bürkner, 2017), a front-end R package of Stan
(Carpenter et al., 2017)

Location model
•Generalized additive mixed model (Murakami, 2016; Wieling, 2018)
•Outcome variable: Number of words in each sentence (minus 1)
•Offset: Number of clauses in the sentence
•Predictor variable: nonlinear wave + learner-wave factor smooths
•R formula: token ~ s(wave, k = 4) + offset(log(clause)) + s(wave,
learner_id, bs = "fs", m = 1, k = 4)

Scale model
•The negative binomial distribution has the parameter called phi (), whose value
is larger when the variability is smaller.
•We modeled phi as a function of some combination of by-learner random
intercepts, (linear and nonlinear) wave, and the difference in mean clause length
between the essays at Time t and t + 1.
•The last variable examined whether there is a systematic pattern between the
magnitude of development and the variability of clause length.

Results of the location model
•Clause length does not change much in the first few waves.
•From Wave 3, it consistently increases.
•Large individual variability in the level of clause length but much less in the
developmental pattern.

Results of the scale model
•Larger waves were associated with slightly smaller phi.
•Longitudinal development was associated with slightly larger variability.
•The variability does not necessarily vary across the learners.
•No systematic pattern was identified between the magnitude of development and
the variability of clause length.

Bürkner, P.-C. (2018). Advanced Bayesian multilevel modeling with the R package brms. The R Journal, 10, 395–411.
Carpenter, B., Gelman, A., Hoffman, M., Lee, D., Goodrich, B., Betancourt, M., … & Riddell, A. (2017). Stan: A probabilistic programming language. Journal of
Statistical Software, 76(1).
Hiver, P., & Al-Hoorie, A.H. (2020). Research methods for complexity theory in applied linguistics. Multilingual Matters.
Hiver, P., Al-Hoorie, A.H., & Larsen-Freeman, D. (2022). Toward a transdisciplinary integration of research purposes and methods for complex dynamic systems
theory: Beyond the quantitative–qualitative divide. International Review of Applied Linguistics, 60(1), 7-22.
Klein, D., & Manning, C. (2003). Accurate unlexicalized parsing. Proceedings of the 41st Annual Meeting on Association for Computational Linguistics (Volume 1).
Association for Computational Linguistics. 423–430.
Larsen–Freeman, D. (1997). Chaos/complexity science and second language acquisition. Applied Linguistics, 18, 141–165.
Larsen-Freeman, D. (2020). Complexity theory. Relational systems in interaction and in interlocutor differences in second language development. In L. Gurzynski-
Weiss (Ed.), Cross-theoretical explorations of interlocutors and their individual differences (pp. 189–208). John Benjamins
Levy, R., & Andrew, G. (2006). Tregex and Tsurgeon: Tools for querying and manipulating tree data structures. Proceedings of the 5th International Conference on
Language Resources and Evaluation. ELRA. 2231-2234.
Murakami, A. (2016). Modeling systematicity and individuality in nonlinear second language development: The case of English grammatical morphemes.
Language Learning, 66, 834–871.
Murakami, A. (2020). On the sample size required to identify the longitudinal L2 development of complexity and accuracy indices. In W. Lowie, M. Michel, A.
Rousse-Malpat, M. Keijzer, R. Steinkrauss (Eds.), Usage-based dynamics in second language development (pp. 20–49). Multilingual Matters.
Sahin, O., Salim, H., Suprun, E., Richards, R., MacAskill, S., Heilgeist, S., … & Beal, C. D. (2020). Developing a preliminary causal loop
diagram for understanding the complexity of the COVID-19 pandemic. Systems, 8(2), 20.
The Douglas Fir Group. (2016). A transdisciplinary framework for SLA in a multilingual world. The Modern Language Journal, 100(S1), 19–47.
Verspoor, M., & de Bot, K. (2022). Measures of variability in transitional phases in second language development. International Review of Applied Linguistics,60(1),
85-101.
Wieling, M. (2018). Analyzing dynamic phonetic data using generalized additive mixed modeling: A tutorial focusing on articulatory differences between L1 and L2
speakers of English. Journal of Phonetics, 70, 86–116.
Winter, B., & Bürkner, P-C. (2021). Poisson regression for linguists: A tutorial introduction to modelling count data with brms. Language and Linguistics Compass,
15(11), e12439.
Wood, S. N. (2003) Thin plate regression splines. Journal of the Royal Statistical Society Series B, 65, 95–114.
Wood, S. N. (2017). mgcv: Mixed gam computation vehicle with automatic smoothness estimation. https://cran.r-project.org/web/packages/mgcv.
Selected References