Content uploaded by Ali H. Al-Hoorie
Author content
All content in this area was uploaded by Ali H. Al-Hoorie on Jun 10, 2020
Content may be subject to copyright.
Methodological and technological
considerations in flipped language
learning interventions:
A systematic review
Joseph P. Vitta, Rikkyo University, Japan (vittajp@rikkyo.ac.uk)
Ali H. Al-Hoorie (hoorie_a@jic.edu.sa)
JALTCALL 2020
June 6, 2020
Agenda
•Defining flipped learning
•Flipped learning effectiveness
•Systematic review
•Methodological features
•Power, reliability & use of pre-test
•Review of technology use
•Videos & interactive technology use
•Gaps for feature Research
•21st century skills
Flipped Learning as a
Debated Construct
•Minimum definition: new content
precedes class time in the form of
homework/outside of class study
activity.
•Higher order thinking and agency
as the defining features of flipped
vs.
•Technology as the defining feature
of flipped
Flipped as communicative language teaching?
•Webb and Doman (2016): Flipped Learning ≠ CLT
•Hung (2015) and Ishikawa et al. (2015) both compared flipped
treatment groups with CLT comparison groups.
•Chen Hsieh et al. (2017) had students draft “the final dialog
collaboratively” (p. 4) under the conventional learning condition.
Ideal Conditions for Flipped (Mehring, 2018)
•Procedural and conceptual outcomes
•Motivated and trustworthy students
•Technologically literacy on both the student and teacher ends
Flipped Learning’s
Effectiveness
•Usually tested via experimental designs
where flipped is a treatment compared
with non-flipped learning conditions
•Past meta-analyses have found flipped
groups perform better by a magnitude of
0.3 to 0.5 standard deviations (Cohen’s
d) across different educational domains
•Humanities, subsuming second/foreign
language learning, have slightly higher
effects observed (e.g., Cheng et al., 2019,
gor corrected d= 0.63)
From our meta-analysis of L2 flipped interventions
- Magnitude of L2 d (Plonsky & Oswald, 2014): 0.4 (small), 0.7 (medium), 1.0 (large)
- Overall effect g (corrected d) = 0.99, but it corrects to 0.58 when considering
possible publication bias
L2 outcome
Studies
gLower
95% CI Upper
95% CI
Writing
13 1.50 1.00 1.99
Listening
4 1.42 0.62 2.21
Speaking
8 1.14 0.81 1.48
Multi
-skill 14 1.03 0.65 1.41
Grammar
5 1.01 0.38 1.63
Vocabulary
9 0.25 0.03 0.47
Standardized test performance
4 0.33 –0.07 0.72
Reading
3 1.25 –0.09 2.59
•From our meta-analysis,
concerning observations:
•Lack of details on the ‘flip’
•Much higher effects in non-
SSCI journals (points to
publication bias and
methods issues) –Scopus g
= 1.39
•Executed a systematic
review in response
Overview of Study
•Systematic review is a research synthesis that identifies and quantifies
trends in a report pool:
•RQ1: What are the observed methodological issues in L2 flipped experimental
reports?
•Power, reliability, use of pre-test
•RQ2: How do L2 flipped interventions employ technology in flipping the content?
•Videos, web 1.0 vs. web 2.0
•RQ3: What additional ‘gaps’ in relation to age, L2, and learning outcomes emerge
from a review of L2 flipped experimental reports?
•Judgements were validated via inter-rater reliability checking reported in the larger
meta-analysis report – 85% to 88% agreement (.7 ≤ κ ≤ .86)
The report pool
•Many L2 meta-analyses and
systematic report pools limit
their report search to a few
journals but this goes against
the spirit of research
synthesis endeavors where
comprehensiveness is king.
•Robust search ending in 56
experimental reports –more
than double the L2 reports in
either flipped review studies
RQ1. Power, i.e. minimum sample size
•Finding: No report engaged in an a priori power analysis
•Why is this an issue? Power calculations tie research to the existing body
of literature. Underpowered and no-power samples may not be
untrustworthy in both directions: 1) they can miss true but small effects
in the population (type 2 error) and 2) detected large effects could be
flukes (Brysbaert, 2019)
•How to fix it? Let’s assume d= 0.58, calculate power G*Power software:
•2 groups of 48 –basic design that tests flipped vs. non-flipped RQ
•3 groups of 40 –treatment, comparison, control design + ANOVA testing
•Values from other meta-analyses can be substituted
RQ1. Reliability
•53% of reports (30 of them) reported the reliability of the measurement
of the outcome/dependent variable
•Why is it important? Without reliability, validity is impossible to satisfy,
and so the study could be flawed. Validity of instruments, especially in L2,
can be assumed across studies, BUT reliability must be checked in every
instance (Al-Hoorie & Vitta, 2019).
•How to fix it? Inter-rater agreement when outcome is a proficiency
judgment. Cronbach’s alpha (KR-21) on test items. When using
standardized tests that have been externally marked, clearly identify this in
report.
•Chen Hsieh et al. (2017): positive example
RQ1. Pretest Usage
•Finding = 39 reports (~70%) employed pre-tests to empirically
demonstrate pre-treatment equivalence before the treatment.
•In all, this was a positive observation and speaks well of our field.
•While experimental designs in the strictest sense only require post-
test comparisons, pre-treatment equivalence is important in L2
research as we’re pushing the boundaries of experimental research as
it is, i.e., we’re not in a lab.
Curious example: 2017 study (reading outcome) with g= 2.89
RQ2 –100% of reports used technology
Video Usage
•75% (42 reports) used videos
•Video as preferred medium of
flipped applications
•Easy for teachers to made/control
•Accessibility of YouTube etc.
Interactive Technology
•41% (23 reported) used
interactive technologies
•WhatsApp was a popular choice
•Learning management systems
such as Edmodo
•Web 2.0 as the natural
complement to flipped learning?
RQ3. Gaps Emerging within Report Pool
•91% of reports (51 studies) saw English as the L2
•75% reports (42 studies) had samples of university language learners
•L2 outcomes involving competencies underpinning proficiency skills
appears to be under-researched:
•Vocabulary –~11% (6 studies)
•Grammar –~9% (5 studies)
•Pronunciation –0 studies
Implications for Researchers Going Forward
•There is room for improvement in flipped experimental designs,
especially in relation to power analysis and psychometric checking.
•There is a need for future research to investigate the effects of flipped
learning on non-university students and learners of other languages
besides English.
•The positive and strong effects of flipped learning on procedural
proficiency skills appears to be established, but flipped learning
effects on outcomes such as grammar and vocabulary are still unclear.
Implications for Teachers Going Forward
•Overall, the research supports the view
that flipped learning is effective in our
classrooms
•Try to flip using interactive technology
•Process/skill outcomes seem suitable for
flipped learning
•You can join the flipped learning
academic discussion via frontline
qualitative/action research into how
flipped is working
•21st century skills provides a justification
for using flipped learning in your
classrooms
References
•Al-Hoorie, A. H., & Vitta, J. P. (2019). The seven sins of L2 research: A review of 30 journals’ statistical quality
and their CiteScore, SJR, SNIP, JCR Impact Factors. Language Teaching Research, 23(6), 727-744.
doi:10.1177/1362168818767191
•Brysbaert, M. (2019). How many participants do we have to include in properly powered experiments? A
tutorial of power analysis with reference tables. Journal of Cognition, 2(1), 1-38.
doi:10.5334/joc.72
•Chen Hsieh, J. S., Wu, W.-C. V., & Marek, M. W. (2017). Using the flipped classroom to enhance EFL learning.
Computer Assisted Language Learning, 30(1–2), 1–21. doi:10.1080/09588221.2015.1111910
•Cheng, L., Ritzhaupt, A. D., & Antonenko, P. (2019). Effects of the flipped classroom instructional
strategy on students’ learning outcomes: A meta-analysis. Educational Technology Research
and Development, 67(4), 793–824. doi:10.1007/s11423-018-9633-7
•Hung, H.-T. (2015). Flipping the classroom for English language learners to foster active learning.
Computer Assisted Language Learning, 28(1), 81–96. doi:10.1080/09588221.2014.967701
•Ishikawa, Y., Akahane-Yamada, R., Smith, C., Kondo, M., Tsubota, Y., & Dantsuji, M. (2015). An EFL
flipped learning course design: Utilizing students’ mobile online devices. In F. Helm, L. Bradley, M.
Guarda, & S. Thouësny (Eds.), Critical CALL –Proceedings of the 2015 EUROCALL Conference, Padova,
Italy (pp. 261–267). Dublin, Ireland: Researchpublishing.net.
•Mehring, J. (2018). The flipped classroom. In J. Mehring & A. Leis (Eds.), Innovations in flipping the
language classroom: Theories and practices (pp. 1–10). New York, NY: Springer Berlin Heidelberg
•Plonsky, L., & Oswald, F. L. (2014). How big Is “big”? Interpreting effect sizes in L2 research. Language
Learning, 64(4), 878-912. doi:10.1111/lang.12079
•Webb, M., & Doman, E. (2016). Does the flipped classroom lead to increased gains on learning outcomes in
ESL/EFL contexts? CATESOL Journal, 28(1), 39–67.
Thank you!