Research Syntheses in L2
A Scoping Review
Ali H. Al-Hoorie, Saudi TESOL Association
Joseph P. Vitta, Kyushu University
Christopher Nicklin, Rikkyo University
JALT Vocabulary SIG's 2022 Symposium on Vocabulary & Learning
Tokyo International University - Kawagoe, Japan
October 29, 2022
–Vocabulary Research has gone from a niche to a mainstay of SLA research
–L2 Research Syntheses has likewise grown in popularity from Norris & Ortega
(2000) and Ross (1998)
–The time is now ripe to look back at what synthesizers of L2 vocabulary
research has been interested in and what they have found.
–Driven by RQs
–Designed to cast a wider net than systematic reviews
–Uncovers patterns and trends for future research
Flow of this talk
–Report Pool Creation Process
–Results by RQ
Report Pool Creation (31 WoS research synthesis reports)
Research Question 1
What types of vocabulary-based research syntheses were conducted between
January 1990 through May 2022?
A. Meta-analyses vs. Systematic reviews
B. For meta-analyses: Correlational vs. Experimental
C. For systematic reviews: What type (e.g., methodological syntheses)?
Research Question 2
A. What are the descriptive characteristics of these reports (e.g., size of report
pool, aggregate sample size)?
B. To what extent did meta-analyses in the report pool employ inferential
testing? Which tests were employed?
It was also observed via post-hoc analysis that time and the frequency of reports were positively associated (ρ = .74).
“Of the 25 meta-analyses, 17 featured Q-testing as the inferential assessment
testing the significance of its reported aggregated effect size(s) and/or moderator
analyses. Of the eight meta-analyses not using Q-testing completely, five featured
parametric testing where procedures were stated to treat each effect size as a
‘case.’ Three reports had no inferential assessment with only Nicklin and Vitta
(2021) offering a justification as to why. In the case of Nicklin and Vitta, Q-testing
was avoided to capture the full heterogeneity of studies’ effect sizes using a simple
approach to ascertain effect sizes for a series of power analyses.”
Research Question 3
What areas have L2 vocabulary research syntheses investigated?
(categories only created if 4+ reports were coded as being in a given category)
Technology and Vocabulary
MALL, CALL, digital games
Vocabulary as a Predictor of
1 of the 4 (Zhang & Zhang,
2020) also focused on
Trends are present among
these reports however (e.g.,
Webb and colleagues’ work)
Research Question 4
A. When reviewing meta-analyses in the pool, which areas emerge as being
conclusively addressed? and which areas emerge as requiring further
B. What are apparent gaps in the existing body of L2 vocabulary systematic
●Technology enhances the acquisition of vocabulary
●Glossing enhances the acquisition of vocabulary
●Vocabulary predicts reading proficiency (and listening proficiency [see Zhang &
**There is now a need to ask second and third generation questions (Zanna &
Potential application of 2nd/3rd generation RQs
“What such research projects would look like are modeled using examples from the report pool.
Consider glossing as an example with Abraham’s (2008) finding that glossing had strong yet
nonsignificant effects for more proficient learners. A second-generation research question-
governed project could address the conditions under which this finding holds true. To illustrate
the potential of a third-generation research question consider Zhang and Zhang (2020), who
presented a moderator analysis detailing that vocabulary knowledge when measured via
meaning recall had a significantly stronger association with reading than when such vocabulary
knowledge was measured via form recall or meaning recognition. To explain the ‘how’ of this
finding, researchers could design mixed-methods studies where qualitative data helps to explain
what meaning recall is testing to give it its significantly stronger association.”
–Difficult to answer with the findings we observed.
**More methods work is required.
**The curious finding of Choi and Zhang (2021) warrants more discussion and
research: They reviewed multivariate models where vocabulary and grammar
predicted reading and concluded that it was inclusive as to which construct
provided the stronger association. (17 studies / no inferential testing)
Concluding Thoughts (open questions)
Why do synthesizers of vocabulary research favor experimental RQs and
hypotheses to correlation ones?
*Does this pattern exist among primary vocabulary research?
Are vocabulary researchers relatively less engaged with the ongoing L2 methods
reform movement? (only 4 of 31 SRs were methods focus)
Will 2nd and 3rd generation RQs be addressed by future L2 vocabulary
●Abraham, L. B. (2008). Computer-mediated glosses in second language reading comprehension and vocabulary learning: A meta-
analysis. Computer Assisted Language Learning, 21(3), 199–226. https://doi.org/10.1080/09588220802090246
●Choi, Y., & Zhang, D. B. (2021). The relative role of vocabulary and grammatical knowledge in L2 reading comprehension: a systematic
review of literature. International Review of Applied Linguistics in Language Teaching, 59(1), 1–30. https://doi.org/10.1515/iral-2017-
●Elgort, I. (2018). Technology-mediated second language vocabulary development: A review of trends in research methodology. CALICO
Journal, 35(1), 1–29. https://doi.org/10.1558/cj.34554
●Lin, J. J., & Lin, H. (2019). Mobile-assisted ESL/EFL vocabulary learning: A systematic review and meta-analysis. Computer Assisted
Language Learning, 32(8), 878–919. https://doi.org/10.1080/09588221.2018.1541359
●Nicklin, C., & Vitta, J. P. (2021). Effect-driven sample sizes in second language instructed vocabulary acquisition research. The Modern
Language Journal, 105(1), 218–236. https://doi.org/10.1111/modl.12692
●O’Connor, D., Green, S., & Higgins, J. P. (2008). Defining the review question and developing criteria for including studies. In J. P. Higgins
& S. Green (Eds.), Cochrane handbook for systematic reviews of interventions (pp. 81–94). Wiley-Blackwell.
●Yang, X., Kuo, L. J., Eslami, Z. R., & Moody, S. M. (2021). Theoretical trends of research on technology and L2 vocabulary learning: A
systematic review. Journal of Computers in Education, 8(4), 465–483. https://doi.org/10.1007/s40692-021-00187-8
●Zanna, M. P., Fazio, R. H. (1982). The attitude-behavior relation: Moving toward a third generation of research. In M. P. Zanna, E. T.
Higgins, & C. P. Herman (Eds.), Consistency in social behavior: The Ontario symposium: Volume 2 (pp. 283–301). Lawrence Erlbaum.
●Zhang, S., Zhang, X. (2020). The relationship between vocabulary knowledge and L2 reading/listening comprehension: A meta-analysis.
Language Teaching Research, 26(4), 696–725. https://doi.org/10.1177/1362168820913998