Flipped Classroom and Gamification Approach: Its Impact on Performance and Academic Commitment on Sustainable Learning in Education

Abstract

The onset of the COVID-19 global pandemic has negatively impacted sustainable learning in education (SLE). During city lockdowns, higher education institutes (HEIs) have transitioned from adopting solely traditional didactic classroom teaching to including innovative, flexible learning approaches such as flipped classrooms. Gamification is a new techno-pedagogy that has been integrated into flipped classrooms to promote learner achievement and engagement. Grounded in self-determination theory, the objectives of this exploratory study were to analyse the influence of the flipped classroom and gamification on SLE concerning learner achievement and engagement. Participants were recruited from postgraduate business education programmes in China, and three instructional interventions were applied for a semester of 10 weeks. The three instructional interventions applied were: gamified flipped classroom (n = 25), non-gamified flipped classroom (n = 24) and gamified traditional classroom (n = 19). A mixed-methods approach was used, and both quantitative and qualitative data were analysed. The results indicated gamified traditional classrooms promote learner achievement, and the gamified flipped classrooms promote learner engagement. Furthermore, learning culture, such as teacher-dependency, also influence learner achievement and engagement. The class observation reports and learner interviews suggested that both gamified flipped classrooms and gamified traditional classrooms support SLE in the time of academic uncertainty during the COVID-19 pandemic.

Full text

Citation: Ng, L.-K.; Lo, C.-K. Flipped
Classroom and Gamification
Approach: Its Impact on Performance
and Academic Commitment on
Sustainable Learning in Education.
Sustainability 2022,14, 5428. https://
doi.org/10.3390/su14095428
Academic Editors: Michael D.
Kickmeier-Rust, Danial Hooshyar
and Nour El Mawas
Received: 31 March 2022
Accepted: 28 April 2022
Published: 30 April 2022
Publisher’s Note: MDPI stays neutral
with regard to jurisdictional claims in
published maps and institutional affil-
iations.
Copyright: © 2022 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article
distributed under the terms and
conditions of the Creative Commons
Attribution (CC BY) license (https://
creativecommons.org/licenses/by/
4.0/).
sustainability
Article
Flipped Classroom and Gamification Approach: Its Impact on
Performance and Academic Commitment on Sustainable
Learning in Education
Lui-Kwan Ng * and Chung-Kwan Lo
Department of Mathematics and Information Technology, The Education University of Hong Kong,
Hong Kong 999077, China; chungkwanlo@eduhk.hk
*Correspondence: s1122637@s.eduhk.hk
Abstract:
The onset of the COVID-19 global pandemic has negatively impacted sustainable learning
in education (SLE). During city lockdowns, higher education institutes (HEIs) have transitioned from
adopting solely traditional didactic classroom teaching to including innovative, flexible learning
approaches such as flipped classrooms. Gamification is a new techno-pedagogy that has been in-
tegrated into flipped classrooms to promote learner achievement and engagement. Grounded in
self-determination theory, the objectives of this exploratory study were to analyse the influence of
the flipped classroom and gamification on SLE concerning learner achievement and engagement.
Participants were recruited from postgraduate business education programmes in China, and three
instructional interventions were applied for a semester of 10 weeks. The three instructional interven-
tions applied were: gamified flipped classroom (n= 25), non-gamified flipped classroom (n= 24) and
gamified traditional classroom (n= 19). A mixed-methods approach was used, and both quantitative
and qualitative data were analysed. The results indicated gamified traditional classrooms promote
learner achievement, and the gamified flipped classrooms promote learner engagement. Furthermore,
learning culture, such as teacher-dependency, also influence learner achievement and engagement.
The class observation reports and learner interviews suggested that both gamified flipped classrooms
and gamified traditional classrooms support SLE in the time of academic uncertainty during the
COVID-19 pandemic.
Keywords: COVID-19; flipped classroom; gamification; self-determination theory; business education;
adult learning
1. Introduction
The onset of the COVID-19 global pandemic has imposed a negative impact on the
Sustainable Development Goals (SDGs) of Education 2030 (SDG 4) declared by the United
Nations in 2015 [
1
,
2
]. The goal of SDG 4 is to equip adult learners with the knowledge and
skills necessary for the benefit of global sustainable development by providing flexible edu-
cation pathways in higher education [
3
]. Higher education institutes (HEIs) have to equip
adult learners with the knowledge and skills of sustainable learning in education (SLE) to
cope with challenging and complicated circumstances [
4
,
5
]. Thus, HEIs are responsible for
assisting adult learners in overcoming barriers and interruptions by providing sustainable
quality education, which is essential for the country’s SDGs [
6
,
7
]. HEIs were compelled
to rapidly redesign teaching approaches and classroom arrangements to offer flexible and
sustainable learning pathways for SLE in the times of COVID-19 lockdown [8,9].
Instructors in China innovatively planned their teaching by providing learners with
pre-class self-study materials during in-campus classroom lessons due to the uncertainty
of intermittent lockdowns [
10
]. Advancements in information communication technol-
ogy have made such learning arrangements that combine pre-class and in-classroom
Sustainability 2022,14, 5428. https://doi.org/10.3390/su14095428 https://www.mdpi.com/journal/sustainability

Sustainability 2022,14, 5428 2 of 23
learning activities feasible. These arrangements are critical for the sustainability of educa-
tion programmes under enforced home confinements and campus lockdowns during the
pandemic [11,12].
Most importantly, these new pedagogical approaches can alleviate the
threats of the pandemic on the sustainability of the country’s education system [
13
] and
promote SLE amid COVID-19 [4].
Learners with flipped classroom approaches were provided with instructional ma-
terials for self-study at home before participating in practical learning activities in the
classroom, reducing the reliance on face-to-face teaching instead of a traditional classroom
arrangement that is totally reliant on it [
14
]. Learner achievement and engagement are two
important indicators of the success of flipped classrooms in HEIs [
15
,
16
]. Research also
indicates that the success of flipped classrooms depends on sustained learner engagement
in learning activities [
17
,
18
]. According to self-determination theory (SDT), gamification
has the potential to promote learner achievement and motivate learner engagement in the
learning activities of flipped classrooms [
19
]. SDT is an empirically derived theory in social
contexts that differentiates human motivation in terms of autonomous and controlled, in
which autonomously motivated learners thrive in educational settings [
20
,
21
]. Gamifica-
tion is the use of game-design elements in non-gaming contexts and has the potential to
motivate learning [
22
,
23
]. Based on SDT, gamification is hypothesised to foster intrinsic
motivation and satisfy the psychological needs for autonomy, competence, and social
relatedness in flipped classrooms [19].
Flipped classroom approaches have also been adopted in HEI programmes during
the COVID-19 pandemic for sustainable learning in higher education [
24
]. Instructors
provide prerecorded instructional videos to enable learners to study on their own dur-
ing home confinement periods. In addition to the flexibility, the flipped classroom ap-
proach also allows more in-class time for promoting SLE. The key components of SLE are
(1) active learning; (2) independent learning; (3) collaborative learning; (4) renewing and
relearning; (5) knowledge and skills transferability [
4
]. The key learning activities in flipped
classrooms are pre-class self-study (i.e., independent learning), knowledge application
(i.e., transferability), problem-solving (i.e., renewing and relearning), and peer-assisted
learning (i.e., collaborative and active learning) [
8
,
19
,
25
,
26
]; these are especially advanta-
geous in promoting SLE. HEIs in China flexibly adopted flipped classroom approaches in
response to the government’s policy of “suspension of classes without interrupting learn-
ing” during the COVID-19 pandemic [
27
,
28
]. However, additional research is required to
investigate the impact of the abrupt pedagogical change caused by flipped classrooms in the
long history of Chinese education, which is dominated by traditional didactic teaching [
29
].
This study investigated the impact of the flipped classroom and gamification ap-
proaches on learner achievement and engagement in postgraduate business programmes
for SLE in China during the COVID-19 pandemic. Three classes with different instructional
interventions were applied: gamified flipped classroom (GFC), non-gamified flipped class-
room (NFC) and gamified traditional classroom (GTC). Furthermore, the study was guided
by the following two research questions:
•
RQ1. How does the adoption of the flipped classroom approach influence (a) learner
achievement and (b) learner engagement compared to non-flipped approaches?
•
RQ2. How does the adoption of gamification influence (a) learner achievement and
(b) learner engagement compared to non-gamified approaches?
2. Literature Review
We have the following three sections of the literature review. First, we discuss the SLE
and flipped classroom approach during the COVID-19 pandemic. Second, we review how
the SDT framework supports and motivates SLE and flipped classrooms. Third, we explore
the integration of gamification with flipped and traditional classrooms for SLE.

Sustainability 2022,14, 5428 3 of 23
2.1. SLE and Flipped Classroom during COVID-19 Pandemic
SLE is a philosophy of learning and teaching, which is not limited to education for or
about sustainability [
30
]. SLE supports SDGs in education for adult learners, whose past
knowledge and skills are viewed as flexible and receptive to modification [
31
]. SLE is an
emerging and timely concept designed to enable learners to keep pace with technological
and social changes [
8
]. HEIs with SLE initiatives provide learners with individual and
group learning [
32
]. SLE promotes learners’ willingness to participate and contribute to the
learning process, reflected in learners’ behavioural, emotional, and cognitive engagement
in learning activities [33,34]. Ben-Eliyahu [4] summarised the key components of SLE as:
1. Active learning: refers to seeking information actively and intentionally;
2.
Independent learning: refers to self-sustained learning, which involves learning
by oneself, being autodidactic, identifying and finding sources for what needs to
be learned;
3.
Collaborative learning is conducted in groups with peers of different levels of knowl-
edge and is a process of acquiring and creating knowledge;
4.
Renewing and relearning is the self-reflection and awareness of our knowledge that
might have been forgotten or need to be relearned because of advancements in a field;
5.
Knowledge and skills transferability: refers to using knowledge and skills learned in
one setting (e.g., in the classroom) to another setting (e.g., at work).
SLE could be integrated into education settings with information communication
technology (ICT), online courses (e.g., MOOC), and pre-class and in-class learning [
8
]. Such
flexible pedagogical approaches are especially beneficial to ensure learning sustainability
when classroom teaching is suspended during uncertain and complex situations [35].
The COVID-19 pandemic impedes learning sustainability by imposing academic
uncertainty on HEIs. Learners reported negative emotions and decreased satisfaction due
to interruptions to their sustainable learning [
36
,
37
]. HEIs in China needed to rapidly
adjust their instructional practices to anticipate the changes in the country’s confinement
policies [
38
]. In sync with this development, educators in China quickly adopted the flipped
classroom approach with the aid of information communication technologies [28,29]. The
flipped classroom approach is a technology-enhanced pedagogy that free ups class time by
providing pre-class instructional videos [
39
]. Flipped classrooms emphasise learner-centric
and problem-solving activities inside the classroom [
40
], which enable independent and
collaborative learning [
41
]. Peer-to-peer and peer-to-teacher interactions promote active
learning, and problem-solving activities encourage knowledge renewing and relearning
of SLE [
4
,
29
]. In addition, problem-solving activities for knowledge transferability also
enhance perceived learning and learner achievement [42,43].
Chinese learners are very concerned about the delays in their academic progress [
44
].
Therefore, they favoured flipped classroom approaches as they could benefit from the
self-paced independent learning enabled by pre-class instructional materials, such as pre-
recorded videos, even during confinement and closures of campus. They could then look
forward to increased engagement in collaborative learning once classroom teaching was
allowed [
17
]. However, the flipped classroom approach was not common in HEIs world-
wide or in China before the onset of the COVID-19 pandemic [
45
]. HEIs in China have long
adopted teacher-centric didactic pedagogy in which programmes are primarily conducted
face-to-face with scheduled timetables [
46
]. The COVID-19 pandemic precipitated an
abrupt transition in traditional classroom teaching approaches. Thus, additional research
with a solid theoretical foundation is required to investigate the impact of this abrupt
pedagogical change on learning, mainly learner achievement and engagement [
24
,
47
]. This
is especially important during the interruptions and frequent transitions between pre-class
and in-class modes of learning caused by the COVID-19 pandemic [48–50].
2.2. Self-Determination Theory
Motivation is the main catalyst for sustainable learning behaviour [
51
]. SDT posits
that humans inherently possess the propensity to be curious and interested in learning

Sustainability 2022,14, 5428 4 of 23
and developing [
52
,
53
]. The desire for sustainable development of business executives and
entrepreneurs helps to attain the SDGs of the country [
54
]. SDT states that when the three
basic psychological needs of learners (i.e., autonomy, competence, and relatedness) are
fulfilled, they are motivated and are more likely to engage in education contexts [
55
,
56
].
Learner engagement is characterised by learners’ investment in learning, desire to exceed
standard requirements, and preference for challenges of varying difficulty levels [
57
].
Learner engagement includes behavioural, emotional, and cognitive dimensions [
58
].
Studies by Abeysekera and Dawson [
40
] and Sergis et al. [
59
] indicate that SDT could
be drawn upon to support learning in flipped classroom environments. As less class time
is consumed by lectures, learners have more opportunities for self-regulated, independent,
peer-to-peer collaborative learning and hands-on problem-solving activities for knowledge
transfer [
59
]. Hence, flipped classrooms facilitate autonomy, and supportive feedback from
teachers and peers promotes competence and relatedness [60].
A recent study of HEIs during the COVID-19 pandemic found that learning persistence
and sustainability were also directly impacted by learners’ needs for autonomy, related-
ness, and competence [
61
]. Therefore, higher engagement resulting from motivation, as
explained by SDT, is a prerequisite for the sustainability of education programmes during
the COVID-19 pandemic [
62
]. Supportive feedback from teachers is especially crucial for
engaging learners and can be conveyed via game-design elements such as points, badges,
and leaderboards. Learner engagement is positively related to perceived learning and
sustainable learning [
5
,
63
,
64
]. Thus, flipped and gamified classrooms based on the SDT
framework may promote learning through knowledge transferability in problem-solving
activities and collaborative and active learning [
19
,
65
]. Furthermore, the adoption of flipped
classrooms and game-design elements based on SDT has the potential to promote SLE in
HEIs during the COVID-19 pandemic [4,66].
2.3. Gamification
Gamification is the use of game-design elements in a non-game environment [
67
],
which can be a sustainable method to achieve the United Nations’ Sustainable Development
Goals for quality education (SDG 4) [
68
]. In the context of education, both gamification
and SDT aim to promote learner achievement and engagement [
69
]. Gamification is typi-
cally used in flipped classrooms to provide feedback to encourage learners to take on task
challenges with progressively increasing difficulty levels and to motivate independent and
collaborative learning [
70
]. Feedback and challenges are important to promote learning
achievement and engagement [
71
]. A progressive increase in the difficulty level intrinsi-
cally motivates learners to complete more challenging tasks, especially those tasks that
are relevant to their personal goals [54,72]. One major personal goal for learners in higher
education business programmes is to translate knowledge to practice (i.e., knowledge
transferability), as it enhances their employability [
73
]. Gamification promotes active learn-
ing and engagement in problem-solving activities that bridge the knowledge-to-practice
gap [
74
,
75
]. Game-design elements engage learners by helping them to build new knowl-
edge (i.e., renewing and relearning) and support learning achievement [
70
]. Therefore,
pedagogies with gamification promote SLE (i.e., renewing and relearning, collaborative
and active learning, and knowledge transferability) [4,76].
The most common game-design elements used to motivate learning in higher edu-
cation are points, badges, and leaderboards [
22
]. These game-design elements serve the
following specific purposes [77] (Table 1):
1.
Points capture granular feedback directly related to learners’ specific actions, e.g., par-
ticipation in in-class learning exercises and pursuing optional and challenging tasks.
2.
Badges capture sustained feedback to recognise learners’ progress and contributions
to the activity group tasks.
3.
Leaderboards capture cumulative feedback on a series of actions performed by the
learner and their contributions to completing tasks by displaying the number of
badges earned in activity groups during the entire course.

Sustainability 2022,14, 5428 5 of 23
Table 1. Game-design elements and feedback.
Game-Design Element Picture Feedback Description
Points
Sustainability 2022, 14, x FOR PEER REVIEW 5 of 23
2. Badges capture sustained feedback to recognise learners’ progress and contributions
to the activity group tasks.
3. Leaderboards capture cumulative feedback on a series of actions performed by the
learner and their contributions to completing tasks by displaying the number of
badges earned in activity groups during the entire course.
Table 1. Game-design elements and feedback.
Game-Design Ele-
ment Picture Feedback Description
Points
Granular feedback
- Rewards for specific actions and participa-
tion in learning exercises.
- Encouragement for pursuing optional and
increasing difficulty levels of tasks.
Badges
Sustained feedback
- Rewards and recognition for progress and
contributions to problem-solving exercises in the
activity groups (e.g., case studies).
Leaderboards
Cumulative feedback - Ranking of learners in activity groups ac-
cording to total badges earned.
Gamification can be adopted in flipped classrooms to motivate learners to increase
their efforts and participation during the COVID-19 pandemic [78]. Recent studies found
that gamification [79] and flipped classrooms [70] support positive learning achievement
and engagement. Further, SDT provides the theoretical foundation on which gamification
and flipped classroom pedagogies can be grounded [59,80]. Nevertheless, there is a pau-
city of research on gamification applied in flipped classrooms for business education
among adults [81].
3. Research Methods
We adopted an explanatory sequential design with a mixed-methods approach in
which both quantitative and qualitative research methods were used. When the quantita-
tive phase is followed by the qualitative phase, it helps to explain the mechanism behind
the quantitative results [82]. This approach can also be adopted to provide an insightful
comparison between flipped classrooms with gamification to traditional classrooms in the
existing study [60]. In this section, we first introduce participants, followed by the research
design, data collection, and analysis.
3.1. Participants
Participants were adult business executives (aged 25–42 years) studying postgradu-
ate business programmes of HEI in eastern China. Their participation was voluntary, and
participants could withdraw from the study without negative consequences at any time.
They were assigned to one of the three different instructional approaches (GFC, n = 25;
NFC, n = 24; GTC, n = 19) for the second module in weeks 6–10. For flipped classrooms
(i.e., GFC and NFC), the learners were provided with pre-class instructional videos rec-
orded by the teacher via the institution’s learning management system (Moodle) four
weeks before the scheduled in-class lessons. The videos were intended to impart basic
knowledge before the in-class lessons. After watching the videos, the learners could com-
plete self-evaluation multiple-choice quizzes during their free time. Owing to the high
degree of trust and coordination between the government and Chinese higher education
institutes [14], universities reopened soon after local COVID-19 cases were contained. Rig-
orous hygiene measures enabled the resumption of regular face-to-face lessons in
Granular feedback
- Rewards for specific actions and
participation in learning exercises.
- Encouragement for pursuing optional and
increasing difficulty levels of tasks.
Badges
Sustainability 2022, 14, x FOR PEER REVIEW 5 of 23
2. Badges capture sustained feedback to recognise learners’ progress and contributions
to the activity group tasks.
3. Leaderboards capture cumulative feedback on a series of actions performed by the
learner and their contributions to completing tasks by displaying the number of
badges earned in activity groups during the entire course.
Table 1. Game-design elements and feedback.
Game-Design Ele-
ment Picture Feedback Description
Points
Granular feedback
- Rewards for specific actions and participa-
tion in learning exercises.
- Encouragement for pursuing optional and
increasing difficulty levels of tasks.
Badges
Sustained feedback
- Rewards and recognition for progress and
contributions to problem-solving exercises in the
activity groups (e.g., case studies).
Leaderboards
Cumulative feedback - Ranking of learners in activity groups ac-
cording to total badges earned.
Gamification can be adopted in flipped classrooms to motivate learners to increase
their efforts and participation during the COVID-19 pandemic [78]. Recent studies found
that gamification [79] and flipped classrooms [70] support positive learning achievement
and engagement. Further, SDT provides the theoretical foundation on which gamification
and flipped classroom pedagogies can be grounded [59,80]. Nevertheless, there is a pau-
city of research on gamification applied in flipped classrooms for business education
among adults [81].
3. Research Methods
We adopted an explanatory sequential design with a mixed-methods approach in
which both quantitative and qualitative research methods were used. When the quantita-
tive phase is followed by the qualitative phase, it helps to explain the mechanism behind
the quantitative results [82]. This approach can also be adopted to provide an insightful
comparison between flipped classrooms with gamification to traditional classrooms in the
existing study [60]. In this section, we first introduce participants, followed by the research
design, data collection, and analysis.
3.1. Participants
Participants were adult business executives (aged 25–42 years) studying postgradu-
ate business programmes of HEI in eastern China. Their participation was voluntary, and
participants could withdraw from the study without negative consequences at any time.
They were assigned to one of the three different instructional approaches (GFC, n = 25;
NFC, n = 24; GTC, n = 19) for the second module in weeks 6–10. For flipped classrooms
(i.e., GFC and NFC), the learners were provided with pre-class instructional videos rec-
orded by the teacher via the institution’s learning management system (Moodle) four
weeks before the scheduled in-class lessons. The videos were intended to impart basic
knowledge before the in-class lessons. After watching the videos, the learners could com-
plete self-evaluation multiple-choice quizzes during their free time. Owing to the high
degree of trust and coordination between the government and Chinese higher education
institutes [14], universities reopened soon after local COVID-19 cases were contained. Rig-
orous hygiene measures enabled the resumption of regular face-to-face lessons in
Sustained feedback
- Rewards and recognition for progress and
contributions to problem-solving exercises
in the activity groups (e.g., case studies).
Leaderboards
Sustainability 2022, 14, x FOR PEER REVIEW 5 of 23
2. Badges capture sustained feedback to recognise learners’ progress and contributions
to the activity group tasks.
3. Leaderboards capture cumulative feedback on a series of actions performed by the
learner and their contributions to completing tasks by displaying the number of
badges earned in activity groups during the entire course.
Table 1. Game-design elements and feedback.
Game-Design Ele-
ment Picture Feedback Description
Points
Granular feedback
- Rewards for specific actions and participa-
tion in learning exercises.
- Encouragement for pursuing optional and
increasing difficulty levels of tasks.
Badges
Sustained feedback
- Rewards and recognition for progress and
contributions to problem-solving exercises in the
activity groups (e.g., case studies).
Leaderboards
Cumulative feedback - Ranking of learners in activity groups ac-
cording to total badges earned.
Gamification can be adopted in flipped classrooms to motivate learners to increase
their efforts and participation during the COVID-19 pandemic [78]. Recent studies found
that gamification [79] and flipped classrooms [70] support positive learning achievement
and engagement. Further, SDT provides the theoretical foundation on which gamification
and flipped classroom pedagogies can be grounded [59,80]. Nevertheless, there is a pau-
city of research on gamification applied in flipped classrooms for business education
among adults [81].
3. Research Methods
We adopted an explanatory sequential design with a mixed-methods approach in
which both quantitative and qualitative research methods were used. When the quantita-
tive phase is followed by the qualitative phase, it helps to explain the mechanism behind
the quantitative results [82]. This approach can also be adopted to provide an insightful
comparison between flipped classrooms with gamification to traditional classrooms in the
existing study [60]. In this section, we first introduce participants, followed by the research
design, data collection, and analysis.
3.1. Participants
Participants were adult business executives (aged 25–42 years) studying postgradu-
ate business programmes of HEI in eastern China. Their participation was voluntary, and
participants could withdraw from the study without negative consequences at any time.
They were assigned to one of the three different instructional approaches (GFC, n = 25;
NFC, n = 24; GTC, n = 19) for the second module in weeks 6–10. For flipped classrooms
(i.e., GFC and NFC), the learners were provided with pre-class instructional videos rec-
orded by the teacher via the institution’s learning management system (Moodle) four
weeks before the scheduled in-class lessons. The videos were intended to impart basic
knowledge before the in-class lessons. After watching the videos, the learners could com-
plete self-evaluation multiple-choice quizzes during their free time. Owing to the high
degree of trust and coordination between the government and Chinese higher education
institutes [14], universities reopened soon after local COVID-19 cases were contained. Rig-
orous hygiene measures enabled the resumption of regular face-to-face lessons in
Cumulative feedback - Ranking of learners in activity groups
according to total badges earned.
Gamification can be adopted in flipped classrooms to motivate learners to increase
their efforts and participation during the COVID-19 pandemic [
78
]. Recent studies found
that gamification [
79
] and flipped classrooms [
70
] support positive learning achievement
and engagement. Further, SDT provides the theoretical foundation on which gamification
and flipped classroom pedagogies can be grounded [
59
,
80
]. Nevertheless, there is a paucity
of research on gamification applied in flipped classrooms for business education among
adults [81].
3. Research Methods
We adopted an explanatory sequential design with a mixed-methods approach in
which both quantitative and qualitative research methods were used. When the quantitative
phase is followed by the qualitative phase, it helps to explain the mechanism behind the
quantitative results [
82
]. This approach can also be adopted to provide an insightful
comparison between flipped classrooms with gamification to traditional classrooms in the
existing study [
60
]. In this section, we first introduce participants, followed by the research
design, data collection, and analysis.
3.1. Participants
Participants were adult business executives (aged 25–42 years) studying postgraduate
business programmes of HEI in eastern China. Their participation was voluntary, and
participants could withdraw from the study without negative consequences at any time.
They were assigned to one of the three different instructional approaches (GFC, n= 25; NFC,
n= 24; GTC, n= 19) for the second module in weeks 6–10. For flipped classrooms (i.e., GFC
and NFC), the learners were provided with pre-class instructional videos recorded by
the teacher via the institution’s learning management system (Moodle) four weeks before
the scheduled in-class lessons. The videos were intended to impart basic knowledge
before the in-class lessons. After watching the videos, the learners could complete self-
evaluation multiple-choice quizzes during their free time. Owing to the high degree of trust
and coordination between the government and Chinese higher education institutes [
14
],
universities reopened soon after local COVID-19 cases were contained. Rigorous hygiene
measures enabled the resumption of regular face-to-face lessons in classrooms. Teachers
focused more on advanced topics during the in-class lessons once the institute was allowed
to reopen. In this study, the learners in the flipped classrooms spent more in-class time on
learning activities such as knowledge application for solving real and simulated business
problems and assignment discussions.
The learners in the gamified classrooms (i.e., GFC and GTC) were given an account
and access code to Qitoupiao. Guidelines and descriptions of expectations throughout the

Sustainability 2022,14, 5428 6 of 23
pre-class (GFC only) and in-class learning stages (GFC and GTC) were provided. These
learners were not graded or provided marks for the points or badges they gained in the
learning activities to prevent them from aiming at high scores instead of being motivated
by the game-design elements. A summary of the guidelines and necessary information was
made available on Moodle, as shown in Figure 1.
Sustainability 2022, 14, x FOR PEER REVIEW 6 of 23
classrooms. Teachers focused more on advanced topics during the in-class lessons once
the institute was allowed to reopen. In this study, the learners in the flipped classrooms
spent more in-class time on learning activities such as knowledge application for solving
real and simulated business problems and assignment discussions.
The learners in the gamified classrooms (i.e., GFC and GTC) were given an account
and access code to Qitoupiao. Guidelines and descriptions of expectations throughout the
pre-class (GFC only) and in-class learning stages (GFC and GTC) were provided. These
learners were not graded or provided marks for the points or badges they gained in the
learning activities to prevent them from aiming at high scores instead of being motivated
by the game-design elements. A summary of the guidelines and necessary information
was made available on Moodle, as shown in Figure 1.
(a) (b)
Figure 1. Summary of guidelines. (a) Guidelines for pre-class learning activities; (b) Guidelines for
in-class learning activities.
Participants in the three classrooms had to submit a post-class assignment in an essay
of around 2000 words after completing classroom lessons at the end of each module. As-
signment questions mainly focused on knowledge application for solving real business
problems the learners are facing at work. Furthermore, all assignment questions were
evaluated and approved by the programme academic team and external examiners.
3.2. Research Design
The study was conducted in the context of adult postgraduate business education
programmes in China during the COVID-19 pandemic from November 2021 to January
2022. The programmes consisted of two modules. Each module lasted for five weeks and
consisted of 16 h of in-class lessons (eight hours each day for Day-1 and Day-2). Three
kinds of intervention, namely the gamified flipped classroom (GFC, experimental group-
1), the non-gamified flipped classroom (NFC, experimental group-2), and the gamified
traditional classroom (GTC, experimental group-3), were introduced in the second mod-
ule, which started at the sixth week. To evaluate the influence of flipped classrooms on
learner achievement and engagement (i.e., RQ 1), we compared the GFC and GTC exper-
imental groups, and to evaluate the influence of gamification on learner achievement and
Pre-class learning activities (for GFC & NFC)
- Watch the lecture videos in Moodle
- Review and study the instructional contents
- Do the self-check multiple-choice quizzes
In-class learning activities
- Attend case study lessons and attempt to solve the
problems in the cases (GFC, NFC & GTC)
- Attend knowledge application and assignment discussion
lessons and attempt to solve the problems and challenges
in assignment questions (GFC & NFC only)
- Results can be presented in the following three ways (GFC,
NFC & GTC):
a) a summary report, or
b) a summary report with comments and real examples, or
c) a plan to solve the problems
- Obtain badges by innovative ideas and solutions, number
of badges accumulated is displayed on Qitoupiao
leaderboard (GFC & GTC)
Figure 1.
Summary of guidelines. (
a
) Guidelines for pre-class learning activities; (
b
) Guidelines for
in-class learning activities.
Participants in the three classrooms had to submit a post-class assignment in an
essay of around 2000 words after completing classroom lessons at the end of each module.
Assignment questions mainly focused on knowledge application for solving real business
problems the learners are facing at work. Furthermore, all assignment questions were
evaluated and approved by the programme academic team and external examiners.
3.2. Research Design
The study was conducted in the context of adult postgraduate business education
programmes in China during the COVID-19 pandemic from November 2021 to January
2022. The programmes consisted of two modules. Each module lasted for five weeks and
consisted of 16 h of in-class lessons (eight hours each day for Day-1 and Day-2). Three kinds
of intervention, namely the gamified flipped classroom (GFC, experimental
group-1
), the
non-gamified flipped classroom (NFC, experimental group-2), and the gamified traditional
classroom (GTC, experimental group-3), were introduced in the second module, which
started at the sixth week. To evaluate the influence of flipped classrooms on learner
achievement and engagement (i.e., RQ 1), we compared the GFC and GTC experimental
groups, and to evaluate the influence of gamification on learner achievement and learner
engagement (i.e., RQ 2), we compared the GFC and NFC experimental groups. The
experimental design is summarised in Table 2.

Sustainability 2022,14, 5428 7 of 23
Table 2. Experimental design.
Experimental
Groups and
Approaches
Experimental Group 1:
Gamified Flipped
Classroom (GFC)
Experimental Group 2:
Non-Gamified Flipped
Classroom (NFC)
Experimental Group 3:
Gamified Traditional
Classroom (GTC)
Research Question
and Group
Comparisons
Flipped classroom Yes Yes No RQ1: GFC and GTC
Gamified classroom Yes No Yes RQ2: GFC and NFC
3.2.1. Class Rundown
All in-class lessons were conducted face-to-face after city lockdowns were lifted. In
the first module (weeks 1–5), the students in all three experimental groups learned under
the traditional classroom approach (i.e., teacher-centric didactic approach with no pre-class
videos and gamification). The assignment marks from the first module were recorded as
pre-intervention references. The three instructional interventions were adopted for the
respective experimental groups in the second module during weeks 6–10 (Figure 2). For
the flipped classrooms (GFC and NFC), eight sessions of pre-recorded instructional videos
with a duration of 30 min each were recorded by the module teacher and provided via
the institution’s learning management system (Moodle). Self-evaluation multiple-choice
quizzes with only ten questions were offered to avoid heavy cognitive load in the pre-class
stage of learning [40,83].
Sustainability 2022, 14, x FOR PEER REVIEW 7 of 23
learner engagement (i.e., RQ 2), we compared the GFC and NFC experimental groups.
The experimental design is summarised in Table 2.
Table 2. Experimental design.
Experimental
Groups and Ap-
proaches
Experimental Group 1:
Gamified Flipped Class-
room (GFC)
Experimental Group 2:
Non-Gamified Flipped
Classroom (NFC)
Experimental Group 3:
Gamified Traditional
Classroom (GTC)
Research Question
and Group Com-
parisons
Flipped class-
room Yes Yes No RQ1: GFC and GTC
Gamified class-
room Yes No Yes RQ2: GFC and NFC
3.2.1. Class Rundown
All in-class lessons were conducted face-to-face after city lockdowns were lifted. In the
first module (weeks 1–5), the students in all three experimental groups learned under the
traditional classroom approach (i.e., teacher-centric didactic approach with no pre-class vid-
eos and gamification). The assignment marks from the first module were recorded as pre-
intervention references. The three instructional interventions were adopted for the respec-
tive experimental groups in the second module during weeks 6–10 (Figure 2). For the
flipped classrooms (GFC and NFC), eight sessions of pre-recorded instructional videos with
a duration of 30 min each were recorded by the module teacher and provided via the insti-
tution’s learning management system (Moodle). Self-evaluation multiple-choice quizzes
with only ten questions were offered to avoid heavy cognitive load in the pre-class stage of
learning [40,83].
Figure 2. Class rundown.
3.2.2. Lesson Schedule
The face-to-face lesson schedules were the same across the three experimental groups
for the first four morning hours of lecture lessons. This was followed by two hours of case
study lessons in the early afternoon for all three classrooms (GFC, NFC, and GTC). The
case study lessons served to expand the scope of learning and knowledge through peer
interactive and collaborative learning. For the flipped classrooms (GFC and NFC), there
were two hours of knowledge application and assignment discussion lessons in the late
afternoon of Day-1 and Day-2, respectively. In contrast, for the non-flipped classroom
Figure 2. Class rundown.
3.2.2. Lesson Schedule
The face-to-face lesson schedules were the same across the three experimental groups
for the first four morning hours of lecture lessons. This was followed by two hours of
case study lessons in the early afternoon for all three classrooms (GFC, NFC, and GTC).
The case study lessons served to expand the scope of learning and knowledge through
peer interactive and collaborative learning. For the flipped classrooms (GFC and NFC),
there were two hours of knowledge application and assignment discussion lessons in the
late afternoon of Day-1 and Day-2, respectively. In contrast, for the non-flipped classroom
(GTC), there were lecture lessons in the late afternoon of both Day-1 and Day-2. Figure 3
provides details of the lesson schedule.

Sustainability 2022,14, 5428 8 of 23
Sustainability 2022, 14, x FOR PEER REVIEW 8 of 23
(GTC), there were lecture lessons in the late afternoon of both Day-1 and Day-2. Figure 3
provides details of the lesson schedule.
Figure 3. Lesson schedule.
The case study lessons format was the same for all three classrooms, which started
with a business case relevant to the morning lectures. Learners were grouped into activity
groups of five to seven peer members to discuss the case according to the guiding ques-
tions. For example, the learners discussed the case of emotional pricing and its implica-
tions for marketing strategy in Figure 4. Each activity group could then choose one of
three levels of difficulty to present their results: a summary report of the discussion (Easy
level), real application with examples (Medium level), or real application with a business
plan (Hard level). The learners’ choices were recorded in class observation reports by
teaching assistants.
Figure 4. Example of an in-class case study learning activity with three difficulty levels for presenting
the results.
For the two flipped classrooms (GFC and NFC), there were two hours of knowledge
application (Figure 5) and two hours of assignment discussion (Figure 6) during the late
afternoon of Day-1 and Day-2, respectively. In the knowledge application lessons, the
teachers introduced a topic or questions that focused on applying the knowledge learned.
Figure 3. Lesson schedule.
The case study lessons format was the same for all three classrooms, which started
with a business case relevant to the morning lectures. Learners were grouped into activity
groups of five to seven peer members to discuss the case according to the guiding questions.
For example, the learners discussed the case of emotional pricing and its implications
for marketing strategy in Figure 4. Each activity group could then choose one of three
levels of difficulty to present their results: a summary report of the discussion (Easy
level), real application with examples (Medium level), or real application with a business
plan (Hard level). The learners’ choices were recorded in class observation reports by
teaching assistants.
Sustainability 2022, 14, x FOR PEER REVIEW 8 of 23
(GTC), there were lecture lessons in the late afternoon of both Day-1 and Day-2. Figure 3
provides details of the lesson schedule.
Figure 3. Lesson schedule.
The case study lessons format was the same for all three classrooms, which started
with a business case relevant to the morning lectures. Learners were grouped into activity
groups of five to seven peer members to discuss the case according to the guiding ques-
tions. For example, the learners discussed the case of emotional pricing and its implica-
tions for marketing strategy in Figure 4. Each activity group could then choose one of
three levels of difficulty to present their results: a summary report of the discussion (Easy
level), real application with examples (Medium level), or real application with a business
plan (Hard level). The learners’ choices were recorded in class observation reports by
teaching assistants.
Figure 4. Example of an in-class case study learning activity with three difficulty levels for presenting
the results.
For the two flipped classrooms (GFC and NFC), there were two hours of knowledge
application (Figure 5) and two hours of assignment discussion (Figure 6) during the late
afternoon of Day-1 and Day-2, respectively. In the knowledge application lessons, the
teachers introduced a topic or questions that focused on applying the knowledge learned.
Figure 4.
Example of an in-class case study learning activity with three difficulty levels for presenting
the results.
For the two flipped classrooms (GFC and NFC), there were two hours of knowledge
application (Figure 5) and two hours of assignment discussion (Figure 6) during the late
afternoon of Day-1 and Day-2, respectively. In the knowledge application lessons, the
teachers introduced a topic or questions that focused on applying the knowledge learned.
For example, as shown in Figure 3, the learners learned different pricing strategies in
the morning lessons and then attempted to apply them in real or simulated business
scenarios. These lessons were intended to strengthen their ability to apply their knowledge

Sustainability 2022,14, 5428 9 of 23
for solving real or simulated business problems, which was knowledge transferability and
most relevant to their jobs and employment. Lastly, the assignment discussion lessons on
Day-2 allowed the learners to discuss the individual assignment questions and learn with
their peers collaboratively. Peer-to-peer interactive and collaborative learning could inspire
further thoughts, ideas, and solutions to the business problems listed in the assignment
questions. Learners could renew and relearn in the process. Because the learners in
the flipped classrooms (GFC and NFC) were aided by the knowledge application and
assignment discussion lessons, they subsequently required less time to complete the same
assignment writing required for all three experimental groups after the classroom lessons.
By contrast, those in GTC had to work on their assignment writing from the beginning and
required more time to complete it by themselves after classroom lessons.
Sustainability 2022, 14, x FOR PEER REVIEW 9 of 23
For example, as shown in Figure 3, the learners learned different pricing strategies in the
morning lessons and then attempted to apply them in real or simulated business scenar-
ios. These lessons were intended to strengthen their ability to apply their knowledge for
solving real or simulated business problems, which was knowledge transferability and
most relevant to their jobs and employment. Lastly, the assignment discussion lessons on
Day-2 allowed the learners to discuss the individual assignment questions and learn with
their peers collaboratively. Peer-to-peer interactive and collaborative learning could in-
spire further thoughts, ideas, and solutions to the business problems listed in the assign-
ment questions. Learners could renew and relearn in the process. Because the learners in
the flipped classrooms (GFC and NFC) were aided by the knowledge application and as-
signment discussion lessons, they subsequently required less time to complete the same
assignment writing required for all three experimental groups after the classroom lessons.
By contrast, those in GTC had to work on their assignment writing from the beginning
and required more time to complete it by themselves after classroom lessons.
Figure 5. Example of a knowledge application lesson.
Figure 6. Example of an assignment discussion lesson.
Learner surveys, interviews, class observation reports, and written feedback were
collected from all three classes at the end of Day-2 after the completion of all the in-class
lessons of the second module (i.e., in week 10).
3.2.3. Application of Game-Design Elements
During the case study lessons, the learners were divided into activity groups of 5–7
peers per group. Game-design elements were applied for the gamified classrooms (GFC
and GTC). To ensure that gamification was effective, points, badges, and leaderboards
were displayed through the Chinese classroom application Qitoupiao (‘Voting Together’).
These game-design elements and their use in the two gamified classrooms are described
below and in Table 3.
Figure 5. Example of a knowledge application lesson.
Sustainability 2022, 14, x FOR PEER REVIEW 9 of 23
For example, as shown in Figure 3, the learners learned different pricing strategies in the
morning lessons and then attempted to apply them in real or simulated business scenar-
ios. These lessons were intended to strengthen their ability to apply their knowledge for
solving real or simulated business problems, which was knowledge transferability and
most relevant to their jobs and employment. Lastly, the assignment discussion lessons on
Day-2 allowed the learners to discuss the individual assignment questions and learn with
their peers collaboratively. Peer-to-peer interactive and collaborative learning could in-
spire further thoughts, ideas, and solutions to the business problems listed in the assign-
ment questions. Learners could renew and relearn in the process. Because the learners in
the flipped classrooms (GFC and NFC) were aided by the knowledge application and as-
signment discussion lessons, they subsequently required less time to complete the same
assignment writing required for all three experimental groups after the classroom lessons.
By contrast, those in GTC had to work on their assignment writing from the beginning
and required more time to complete it by themselves after classroom lessons.
Figure 5. Example of a knowledge application lesson.
Figure 6. Example of an assignment discussion lesson.
Learner surveys, interviews, class observation reports, and written feedback were
collected from all three classes at the end of Day-2 after the completion of all the in-class
lessons of the second module (i.e., in week 10).
3.2.3. Application of Game-Design Elements
During the case study lessons, the learners were divided into activity groups of 5–7
peers per group. Game-design elements were applied for the gamified classrooms (GFC
and GTC). To ensure that gamification was effective, points, badges, and leaderboards
were displayed through the Chinese classroom application Qitoupiao (‘Voting Together’).
These game-design elements and their use in the two gamified classrooms are described
below and in Table 3.
Figure 6. Example of an assignment discussion lesson.
Learner surveys, interviews, class observation reports, and written feedback were
collected from all three classes at the end of Day-2 after the completion of all the in-class
lessons of the second module (i.e., in week 10).
3.2.3. Application of Game-Design Elements
During the case study lessons, the learners were divided into activity groups of
5–7 peers
per group. Game-design elements were applied for the gamified classrooms
(GFC and GTC). To ensure that gamification was effective, points, badges, and leaderboards
were displayed through the Chinese classroom application Qitoupiao (‘Voting Together’).
These game-design elements and their use in the two gamified classrooms are described
below and in Table 3.

Sustainability 2022,14, 5428 10 of 23
Table 3. Application of game-design elements for gamified classrooms (GFC and GTC).
Game-Design Element Purpose How They Were Awarded
Points
Sustainability 2022, 14, x FOR PEER REVIEW 10 of 23
1. Points: Learners received points on completing tasks in the in-class learning activi-
ties. Points serve as feedback to encourage learners to complete subsequent learning
activities and achieve more advanced challenges together in a group [84,85]. One
point was allotted to activity groups that complete their discussion and presented the
results as a summary report, which was the lowest level of difficulty (Easy) to com-
plete their task. Two and three points were allotted to activity groups that chose the
intermediate (Medium) or the most challenging (Hard) levels of difficulty to present
their discussion results in a real application with examples and a business plan, re-
spectively.
2. Badges: Learners received badges when they provided innovative ideas and solu-
tions to problems and questions during in-class learning activities. Badges serve as
recognitions of a learner’s contributions and encourage participation during learning
activities [60]. Badges also promote social validation as they provide opportunities
for learners to show their conformity and progress towards the expected learning
behaviour with their peers [86]. Learners in an activity group obtained one badge
when any member of the group presented a good idea, new knowledge, or an inno-
vative solution to a problem or question. A ‘good idea’ refers to a new way to apply
the knowledge learned in the class, ‘new knowledge’ refers to a point that was not
mentioned or taught in the class, and an ‘innovative solution’ is a solution to a prob-
lem-solving task that was not mentioned or taught in the class.
3. Leaderboards: Teamwork was encouraged within each activity group and the learn-
ers were also encouraged to compare their performance with those of other activity
groups in the same classroom [87]. Thus, the numbers of accumulated badges of each
activity group were displayed on the class leaderboard. The leaderboard was in-
tended to inspire intragroup peer collaborative learning and a healthy intergroup
competition amongst the groups to achieve a prominent position on the leaderboard
by contributing to learning activities for more badges [88,89].
Table 3. Application of game-design elements for gamified classrooms (GFC and GTC).
Game-Design Ele-
ment Purpose How They Were Awarded
Points
Feedback to encourage task
completion and progress to-
wards more advanced and
challenging learning activities.
Point(s) for in-class learning and presentation activities based on
level of difficulty:
 Easy—one point for a summary report
 Medium—two points for real application with examples
 Hard—three points for real application with a business plan
Badges
Recognition of learning efforts,
participation, and contribu-
tions; promoting social recogni-
tion.
Badges were given to learners who presented good ideas, new
knowledge or innovative solutions to a problem.
Leaderboards
Inspire intragroup peer learn-
ing and intergroup competition
for learning advancement.
The number of accumulated badges obtained in each activity
group.
3.3. Data Collection
Both quantitative and qualitative data were collected during the in-class and post-
class stages (Table 4). Class observation reports were collected by teaching assistants dur-
ing the in-class stages. Data were from the post-class stage and were collected based on
three sources: learner survey, learner interview, and assignment marks. The survey used
a 5-point Likert scale that ranged from ‘Strongly Agree’ (5) through ‘Neutral’ (3) to
Feedback to encourage task
completion and progress
towards more advanced and
challenging learning activities.
Point(s) for in-class learning and presentation activities based on
level of difficulty:
•Easy—one point for a summary report
•Medium—two points for real application with examples
•Hard—three points for real application with a business plan
Badges
Sustainability 2022, 14, x FOR PEER REVIEW 10 of 23
1. Points: Learners received points on completing tasks in the in-class learning activi-
ties. Points serve as feedback to encourage learners to complete subsequent learning
activities and achieve more advanced challenges together in a group [84,85]. One
point was allotted to activity groups that complete their discussion and presented the
results as a summary report, which was the lowest level of difficulty (Easy) to com-
plete their task. Two and three points were allotted to activity groups that chose the
intermediate (Medium) or the most challenging (Hard) levels of difficulty to present
their discussion results in a real application with examples and a business plan, re-
spectively.
2. Badges: Learners received badges when they provided innovative ideas and solu-
tions to problems and questions during in-class learning activities. Badges serve as
recognitions of a learner’s contributions and encourage participation during learning
activities [60]. Badges also promote social validation as they provide opportunities
for learners to show their conformity and progress towards the expected learning
behaviour with their peers [86]. Learners in an activity group obtained one badge
when any member of the group presented a good idea, new knowledge, or an inno-
vative solution to a problem or question. A ‘good idea’ refers to a new way to apply
the knowledge learned in the class, ‘new knowledge’ refers to a point that was not
mentioned or taught in the class, and an ‘innovative solution’ is a solution to a prob-
lem-solving task that was not mentioned or taught in the class.
3. Leaderboards: Teamwork was encouraged within each activity group and the learn-
ers were also encouraged to compare their performance with those of other activity
groups in the same classroom [87]. Thus, the numbers of accumulated badges of each
activity group were displayed on the class leaderboard. The leaderboard was in-
tended to inspire intragroup peer collaborative learning and a healthy intergroup
competition amongst the groups to achieve a prominent position on the leaderboard
by contributing to learning activities for more badges [88,89].
Table 3. Application of game-design elements for gamified classrooms (GFC and GTC).
Game-Design Ele-
ment Purpose How They Were Awarded
Points
Feedback to encourage task
completion and progress to-
wards more advanced and
challenging learning activities.
Point(s) for in-class learning and presentation activities based on
level of difficulty:
 Easy—one point for a summary report
 Medium—two points for real application with examples
 Hard—three points for real application with a business plan
Badges
Recognition of learning efforts,
participation, and contribu-
tions; promoting social recogni-
tion.
Badges were given to learners who presented good ideas, new
knowledge or innovative solutions to a problem.
Leaderboards
Inspire intragroup peer learn-
ing and intergroup competition
for learning advancement.
The number of accumulated badges obtained in each activity
group.
3.3. Data Collection
Both quantitative and qualitative data were collected during the in-class and post-
class stages (Table 4). Class observation reports were collected by teaching assistants dur-
ing the in-class stages. Data were from the post-class stage and were collected based on
three sources: learner survey, learner interview, and assignment marks. The survey used
a 5-point Likert scale that ranged from ‘Strongly Agree’ (5) through ‘Neutral’ (3) to
Recognition of learning efforts,
participation, and contributions;
promoting social recognition.
Badges were given to learners who presented good ideas, new
knowledge or innovative solutions to a problem.
Leaderboards
Sustainability 2022, 14, x FOR PEER REVIEW 10 of 23
1. Points: Learners received points on completing tasks in the in-class learning activi-
ties. Points serve as feedback to encourage learners to complete subsequent learning
activities and achieve more advanced challenges together in a group [84,85]. One
point was allotted to activity groups that complete their discussion and presented the
results as a summary report, which was the lowest level of difficulty (Easy) to com-
plete their task. Two and three points were allotted to activity groups that chose the
intermediate (Medium) or the most challenging (Hard) levels of difficulty to present
their discussion results in a real application with examples and a business plan, re-
spectively.
2. Badges: Learners received badges when they provided innovative ideas and solu-
tions to problems and questions during in-class learning activities. Badges serve as
recognitions of a learner’s contributions and encourage participation during learning
activities [60]. Badges also promote social validation as they provide opportunities
for learners to show their conformity and progress towards the expected learning
behaviour with their peers [86]. Learners in an activity group obtained one badge
when any member of the group presented a good idea, new knowledge, or an inno-
vative solution to a problem or question. A ‘good idea’ refers to a new way to apply
the knowledge learned in the class, ‘new knowledge’ refers to a point that was not
mentioned or taught in the class, and an ‘innovative solution’ is a solution to a prob-
lem-solving task that was not mentioned or taught in the class.
3. Leaderboards: Teamwork was encouraged within each activity group and the learn-
ers were also encouraged to compare their performance with those of other activity
groups in the same classroom [87]. Thus, the numbers of accumulated badges of each
activity group were displayed on the class leaderboard. The leaderboard was in-
tended to inspire intragroup peer collaborative learning and a healthy intergroup
competition amongst the groups to achieve a prominent position on the leaderboard
by contributing to learning activities for more badges [88,89].
Table 3. Application of game-design elements for gamified classrooms (GFC and GTC).
Game-Design Ele-
ment Purpose How They Were Awarded
Points
Feedback to encourage task
completion and progress to-
wards more advanced and
challenging learning activities.
Point(s) for in-class learning and presentation activities based on
level of difficulty:
 Easy—one point for a summary report
 Medium—two points for real application with examples
 Hard—three points for real application with a business plan
Badges
Recognition of learning efforts,
participation, and contribu-
tions; promoting social recogni-
tion.
Badges were given to learners who presented good ideas, new
knowledge or innovative solutions to a problem.
Leaderboards
Inspire intragroup peer learn-
ing and intergroup competition
for learning advancement.
The number of accumulated badges obtained in each activity
group.
3.3. Data Collection
Both quantitative and qualitative data were collected during the in-class and post-
class stages (Table 4). Class observation reports were collected by teaching assistants dur-
ing the in-class stages. Data were from the post-class stage and were collected based on
three sources: learner survey, learner interview, and assignment marks. The survey used
a 5-point Likert scale that ranged from ‘Strongly Agree’ (5) through ‘Neutral’ (3) to
Inspire intragroup peer learning
and intergroup competition for
learning advancement.
The number of accumulated badges obtained in each activity group.
1.
Points: Learners received points on completing tasks in the in-class learning activities.
Points serve as feedback to encourage learners to complete subsequent learning
activities and achieve more advanced challenges together in a group [
84
,
85
]. One
point was allotted to activity groups that complete their discussion and presented
the results as a summary report, which was the lowest level of difficulty (Easy) to
complete their task. Two and three points were allotted to activity groups that chose
the intermediate (Medium) or the most challenging (Hard) levels of difficulty to
present their discussion results in a real application with examples and a business
plan, respectively.
2.
Badges: Learners received badges when they provided innovative ideas and solu-
tions to problems and questions during in-class learning activities. Badges serve as
recognitions of a learner’s contributions and encourage participation during learning
activities [
60
]. Badges also promote social validation as they provide opportunities
for learners to show their conformity and progress towards the expected learning
behaviour with their peers [
86
]. Learners in an activity group obtained one badge
when any member of the group presented a good idea, new knowledge, or an in-
novative solution to a problem or question. A ‘good idea’ refers to a new way to
apply the knowledge learned in the class, ‘new knowledge’ refers to a point that was
not mentioned or taught in the class, and an ‘innovative solution’ is a solution to a
problem-solving task that was not mentioned or taught in the class.
3.
Leaderboards: Teamwork was encouraged within each activity group and the learners
were also encouraged to compare their performance with those of other activity
groups in the same classroom [
87
]. Thus, the numbers of accumulated badges of
each activity group were displayed on the class leaderboard. The leaderboard was
intended to inspire intragroup peer collaborative learning and a healthy intergroup
competition amongst the groups to achieve a prominent position on the leaderboard
by contributing to learning activities for more badges [88,89].
3.3. Data Collection
Both quantitative and qualitative data were collected during the in-class and post-class
stages (Table 4). Class observation reports were collected by teaching assistants during
the in-class stages. Data were from the post-class stage and were collected based on three
sources: learner survey, learner interview, and assignment marks. The survey used a
5-point Likert scale that ranged from ‘Strongly Agree’ (5) through ‘Neutral’ (3) to ‘Strongly
Disagree’ (1). The learner survey and interview focused on the learners’ perceptions and
their suggestions for pedagogical improvement.

Sustainability 2022,14, 5428 11 of 23
Table 4. Data sources and collection.
Stage Data Purpose RQs Addressed
In-class
Class observation
report for levels of
difficulty and
participation
To evaluate learner engagement
under the three instructional
approaches (GFC, NFC,
and GTC).
Learner engagement
(RQ1 and RQ2)
Post-class
Learner survey To evaluate learners’ perceived
achievement and engagement
under the three
instructional approaches.
Learner perceived
achievement and
engagement
(RQ1 and RQ2)
Learner interview
Assignment marks
To evaluate the learner’s overall
achievement under the three
instructional approaches
Learner achievement
(RQ1 and RQ2)
The effects of classroom approaches (i.e., GFC, NFC, and GTC) on learner achievement
were evaluated based on the assignment marks. The teachers graded the assignment strictly
according to the marking scheme and rubrics provided by the institute. All in-class learning
activities, topics and assignment questions were approved by the school appointed external
examiner and the academic programme committee. Thirty per cent of the assignment
marks were countermarked and sent to the external examiner for final review to ensure
accurate assessment. Discrepancies in marks were discussed and were resolved in the
board of examiners’ meetings.
The learners were invited to complete a survey at the end of classroom lessons in
the second module. The survey consisted of questions on four themes: (i) perceived
learning (Items 1–3), (ii) behavioural engagement (Items 4–8), (iii) emotional engagement
(Items 9–13),
and (iv) cognitive engagement (Items 14–17). It also contained one open-
ended question (Item 18). The learners were free to respond, and their responses were
tagged using anonymised labels, e.g., GFC-Learner 1, NFC-Learner 2, and GTC-Learner 3.
Sample items for each theme from the survey are presented in Table 5.
Table 5. Sample items of the learner survey.
Survey Items Supporting Citation
Perceived learning (Items 1–3)
I learned more because of the classroom format. (Item 2) Lo and Hew [60]
Behavioural engagement (Items 4–8)
I participated in in-class activities and discussions. (Item 6) Skinner et al. [90]
Emotional engagement (Items 9–13)
The class was fun. (Item 11) Skinner et al. [90]
Cognitive Engagement (Items 14–17)
I was so involved that I forgot everything around me. (Item 17)
Rotgans et al. [91]
Qualitative data sources included Item 18 of the learner survey, class observation
reports, and learner interviews. Class observation was adopted for evaluating learner en-
gagement in in-class activities in accordance with the recommendation of a prior study [
92
].
Studies have found learners’ behavioural and cognitive engagement to be reflected in their
participation in learning activities [
60
,
93
]. Another study found that the willingness and
effort to exceed minimum requirements and adopt more challenging presentation methods
indicated a higher learning engagement [
94
]. Thus, teaching assistants recorded class
observation reports which focused on two aspects of the learners’ engagement: (1) the level
of difficulty that they chose for their case study results presentation and (2) their learning
participation. Learning participation ranges from passive receiving, active manipulating,
and constructive generating to the most engaging interactive dialoguing [
92
]. Learner

Sustainability 2022,14, 5428 12 of 23
engagement and their participation also reflected the different components of SLE [
4
].
Teaching assistants recorded the learners’ participation in in-class activities by checking one
box each for the level of difficulty and level of participation, respectively, that most closely
matched their observations of classes during the in-class case study lessons (Figure 7).
Sustainability 2022, 14, x FOR PEER REVIEW 12 of 23
(2) their learning participation. Learning participation ranges from passive receiving, ac-
tive manipulating, and constructive generating to the most engaging interactive dialogu-
ing [92]. Learner engagement and their participation also reflected the different compo-
nents of SLE [4]. Teaching assistants recorded the learners’ participation in in-class activ-
ities by checking one box each for the level of difficulty and level of participation, respec-
tively, that most closely matched their observations of classes during the in-class case
study lessons (Figure 7).
Figure 7. Class observation report form with SLE components.
Interviews help to understand learners’ behaviour, feelings, and interpretation of the
learning environment [95]. Therefore, we conducted learner interviews after classroom
lessons in the second module (Week 10). The interview adopted a semi-structured ap-
proach with a protocol (Table 6). The topics covered in the interview were based on the
engagement framework of Fredricks et al. [96] and McCallum et al. [97]. Topics for explor-
ing learner engagement included (1) behavioural engagement (e.g., task participation and
completion), (2) emotional engagement (e.g., enjoyment or boredom), and (3) cognitive
engagement (e.g., investing effort in learning). All interviews were conducted and rec-
orded in Chinese by the researcher. Some transcripts were translated into English for re-
porting purposes. The interviewees were invited to check all original transcriptions, and
discrepancies were corrected to ensure accuracy. Learners’ participation in the survey and
interview was voluntary, and no incentive was offered. After the completion of the second
module lessons, we received responses for 49 surveys (GFC n = 20, NFC n = 17, GTC n =
12) and 26 learner interviews (GFC n = 7, NFC n = 11, GTC n = 8).
Table 6. Sample questions of the semi-structured learner interview protocol.
Dependent
Variable Question Citation Refer-
ence
Behavioural
engagement
How did the classroom approach change the way
you prepared for studying this module differently
from other or previous classes you have attended?
Fredricks et al.
[96]
Emotional
engagement
What did you find most/least interesting in your
studying of this module? MaCullum et al.
[97]
Cognitive
engagement
Did you do anything extra that helped your learn-
ing when studying this module
Figure 7. Class observation report form with SLE components.
Interviews help to understand learners’ behaviour, feelings, and interpretation of the
learning environment [
95
]. Therefore, we conducted learner interviews after classroom
lessons in the second module (Week 10). The interview adopted a semi-structured approach
with a protocol (Table 6). The topics covered in the interview were based on the engagement
framework of Fredricks et al. [
96
] and McCallum et al. [
97
]. Topics for exploring learner
engagement included (1) behavioural engagement (e.g., task participation and completion),
(2) emotional engagement (e.g., enjoyment or boredom), and (3) cognitive engagement
(e.g., investing effort in learning). All interviews were conducted and recorded in Chinese
by the researcher. Some transcripts were translated into English for reporting purposes.
The interviewees were invited to check all original transcriptions, and discrepancies were
corrected to ensure accuracy. Learners’ participation in the survey and interview was
voluntary, and no incentive was offered. After the completion of the second module lessons,
we received responses for 49 surveys (GFC n= 20, NFC n= 17, GTC n= 12) and 26 learner
interviews (GFC n= 7, NFC n= 11, GTC n= 8).
Table 6. Sample questions of the semi-structured learner interview protocol.
Dependent
Variable Question Citation Reference
Behavioural
engagement
How did the classroom approach change the way
you prepared for studying this module differently
from other or previous classes you have attended?
Fredricks et al. [96]
Emotional
engagement
What did you find most/least interesting in your
studying of this module? MaCullum et al. [97]
Cognitive
engagement
Did you do anything extra that helped your
learning when studying this module

Sustainability 2022,14, 5428 13 of 23
3.4. Data Analysis
3.4.1. Quantitative Data Analysis
To determine the most appropriate statistical test for quantitative data analysis, quan-
titative data were first tested for normality [
98
]. A Kolmogorov–Smirnov test showed a
significant deviation from normality for learner achievement (according to assignment
marks) across the three classes (first module p< 0.001; second module p= 0.017). Therefore,
non-parametric tests were used to analyse the quantitative data on learner achievement [
98
].
The Kruskal–Wallis non-parametric test for multiple group comparisons was per-
formed on the assignment marks from the first module, which was taught in traditional
classrooms, to check the initial equivalence of the three experimental groups (i.e., GFC,
NFC, and GTC). The results showed that the three experimental groups were statistically
equivalent (H= 0.256, p= 0.880) for the first module. The assignment marks from the second
module of the three experimental groups were then tested with the Kruskal–Wallis test
at a significance level of 0.05. Multiple Mann–Whitney tests were conducted for post hoc
pairwise comparisons when significant differences were found [
98
]. Bonferroni correction
was applied to avoid Type I errors [
99
]. Therefore, the post hoc analyses reported effects
at a significance level of 0.05/3 = 0.0167. The effect size (r) was then calculated using the
following formula [100]:
r=
z
√N0
where zis the z-score, and N
0
is the number of participants in the two experimental groups
in each pairwise comparison.
3.4.2. Qualitative Data Analysis
The qualitative data were transcribed in Chinese and analysed thematically by organ-
ising the data into categories. Direct quotations from participant interviews were used to
ensure data validity [
101
]. The qualitative analysis procedures followed the suggestion
of Creswell et al. [
82
]. Coding started with the shortest interview transcripts using some
exact wordings of the participants and concepts from the literature. All assigned codes
were reviewed and grouped with redundant codes to produce a preliminary list of codes,
which were then used to analyse the rest of the qualitative data. Exemplary quotes that
clearly illustrated new emerging themes were identified and added to the list of codes, and
similar codes were organised into subthemes.
Double-coding was adopted when data were descriptively and inferentially meaning-
ful but not in neat or isolated units [
102
,
103
]. Thirty percent of the qualitative data were
double-coded by an independent coder, and the intercoder agreement rate was checked.
Disagreements between the coders were resolved through discussion. The data were val-
idated by member checking to avoid any misunderstanding or misinterpretation [
104
].
Qualitative data were translated into English for reporting in this study.
Qualitative data from the written comments, feedback from learner surveys (Item 18),
teaching assistant interviews, and class observation reports and teacher interviews were
analysed with triangulation for comprehensive understanding [
105
] and to better explain
the quantitative results [82].
4. Results
4.1. Learner Achievement
The Kruskal–Wallis test indicated a significant difference between the three classrooms
in terms of the assignment marks obtained in the second module (H= 7.550, p= 0.023).
Figure 8shows a boxplot of the results.

Sustainability 2022,14, 5428 14 of 23
Sustainability 2022, 14, x FOR PEER REVIEW 14 of 23
4. Results
4.1. Learner Achievement
The Kruskal–Wallis test indicated a significant difference between the three class-
rooms in terms of the assignment marks obtained in the second module (H = 7.550, p =
0.023). Figure 8 shows a boxplot of the results.
Figure 8. Boxplots of assignment marks from the second module by classroom.
Mann–Whitney tests for pairwise comparisons showed no statistically significant dif-
ferences between GFC and GTC (p = 0.199) or between GFC and NFC (p = 0.117). However,
the learners in GTC scored significantly higher assignment marks than those in NFC (U =
112.00, z = −2.875, p = 0.004) with an effect size r = 0.410. Table 7 shows the results for the
pairwise comparison of assignment marks between NFC and GTC.
Table 7. Pairwise comparison of assignment marks between NFC and GTC for the second module.
Assignment Marks n Mean SD Mean Rank Pairwise Comparison
NFC 24 75.44 3.73 17.17 GTC > NFC *
GTC 19 78.79 3.05 28.11
* p < 0.0167 (Bonferroni correction).
Although the qualitative data from the interviews revealed several benefits of gami-
fication for learning achievement, the learners trusted knowledge delivered by teachers
more than knowledge received from their peers, regardless of whether their peers had
more firsthand experience in specific business scenarios. Interviews with both learners
and teaching assistants indicated that the knowledge learned from teachers was reflected
more frequently in the learners’ submissions of their assignments, which required them
to apply the knowledge learned in the module to provide a solution to a real business
problem in around 2000 words. Table 8 shows the major benefits of GTC identified from
the learner interviews. These benefits included the generation of excitement and curiosity
and the promotion of interactions and discussions in the classroom. In contrast, the learn-
ers in NFC did not experience a sense of ‘belonging’ (NFC-Learner 8) to the class. Keywords
that occurred multiple times in the interviews of learners from NFC were ‘boring/dry’
(NFC-Learners 1 and 7) and ‘no interaction makes me sleepy/passive’ (NFC-Learners 8 and
10).
Figure 8. Boxplots of assignment marks from the second module by classroom.
Mann–Whitney tests for pairwise comparisons showed no statistically significant
differences between GFC and GTC (p= 0.199) or between GFC and NFC (p= 0.117).
However, the learners in GTC scored significantly higher assignment marks than those
in NFC (U= 112.00, z=
−
2.875, p= 0.004) with an effect size r= 0.410. Table 7shows the
results for the pairwise comparison of assignment marks between NFC and GTC.
Table 7. Pairwise comparison of assignment marks between NFC and GTC for the second module.
Assignment Marks nMean SD Mean Rank Pairwise Comparison
NFC 24 75.44 3.73 17.17 GTC > NFC *
GTC 19 78.79 3.05 28.11
*p< 0.0167 (Bonferroni correction).
Although the qualitative data from the interviews revealed several benefits of gami-
fication for learning achievement, the learners trusted knowledge delivered by teachers
more than knowledge received from their peers, regardless of whether their peers had
more firsthand experience in specific business scenarios. Interviews with both learners
and teaching assistants indicated that the knowledge learned from teachers was reflected
more frequently in the learners’ submissions of their assignments, which required them
to apply the knowledge learned in the module to provide a solution to a real business
problem in around 2000 words. Table 8shows the major benefits of GTC identified from the
learner interviews. These benefits included the generation of excitement and curiosity and
the promotion of interactions and discussions in the classroom. In contrast, the learners
in NFC did not experience a sense of ‘belonging’ (NFC-Learner 8) to the class. Keywords
that occurred multiple times in the interviews of learners from NFC were ‘boring/dry’
(NFC-Learners 1 and 7)
and ‘no interaction makes me sleepy/passive’ (NFC-Learners 8 and 10).

Sustainability 2022,14, 5428 15 of 23
Table 8. Benefits of GTC.
Theme Example Quote
Excitement/curiosity ‘Very excited, and engaged’ (GTC-Learner 3)
Peer learning
‘Classmates encouraged each other to choose the difficult learning
option’ (GTC-Learner 8)
Exchanges/interactions ‘More feedback from the teacher, enabled by the rhythm of the
teaching process, made me more attentive’ (GTC-Learner 5)
4.2. Learner Engagement
The Kolmogorov–Smirnov test showed that the data from the learner surveys sig-
nificantly deviated from normality (p< 0.001); therefore, nonparametric Kruskal–Wallis
tests were performed. The difference between GFC and NFC was significant (p= 0.014)
for perceived learning. Post hoc pairwise Mann–Whitney tests showed that the learners in
GFC scored significantly higher than those in NFC (U = 102, z=
−
2.448, p= 0.0140), with
effect size r=−0.402. These results are shown in Table 9.
Table 9. Comparison between GFC and NFC for learner survey Item 2.
Item Survey Question Class nMean SD Mean
Rank
Pairwise
Comparison
2I learned more because of
the classroom format.
GFC 20 4.80 0.41 22.40 GFC > NFC *
NFC 17 4.35 0.61 15.00
*p< 0.0167 (Bonferroni correction).
Class observation reports reflected that the learners in GFC engaged at the interactive
dialoguing level during learning activities [
93
]. This shows that GFC was the most engaging
of the three classrooms and performed all five SLE components. Those in NFC engaged
half their time at the active manipulation level and the other half at the constructive
generating level. This reveals that NFC was the least engaging among the three classrooms.
Gamification elevated the participation level of learners in GTC to constructive generating,
which showed three out of the five SLE components, indicating that GTC was more
engaging than NFC (Table 10).
Table 10.
Level of difficulty and participation in in-class learning activities from class observation
reports with SLE components. The order of the classrooms in terms of learner engagement and the
number of SLE components that were performed was GFC > GTC > NFC.
Classroom Level of Difficulty Level of
Participation SLE Components
GFC Medium Interactive
dialoguing (100% *)
Active learning
Independent learning
Collaborative learning
Renewing and relearning
Transferability
NFC Medium Active manipulating (50% *) Active learning
Independent learning
Constructive
generating (50% *)
Active learning
Independent learning
Renewing and relearning
GTC Medium Constructive
generating (100% *)
Active learning
Independent learning
Renewing and relearning
* % of the time learners engaged at the level of participation.

Sustainability 2022,14, 5428 16 of 23
Keywords that occurred multiple times in learner interviews from the gamified class-
rooms were ‘active participation’ (GFC-Learners 2 and 8), ‘focused and engaged’
(GFC-Learner 7,
GTC-Learner 3), and ‘the class was interesting, inspired my thoughts and stimulated deep learning’
(GFC-Learner 5). One point of contrast in the feedback between GFC and GTC related to
peer-dependent vs. teacher-dependent learning:
GFC-Learner 2: ‘The presence of such exceptional classmates made me more actively engaged
(in learning)’.
GTC-Learner 5: ‘In the classroom, I followed the rhythm and pace of the teacher, who enabled
me to focus during the interactive learning lesson’.
The feedback from the interviews with those in NFC, in contrast, reflected that the
learners faced certain obstacles to active participation:
NFC-Learner 1: ‘I wish the teacher could control classmates that engaged too much in
irrelevant discussions.’
NFC-Learner 5: ‘I am not very familiar (close) with the classmates, and the (learning)
interactions and bonding between us were not strong’.
In summary, there was a significant difference in learner achievement between GTC
and NFC (p= 0.004). The learners in GTC scored significantly higher than those in NFC for
learner achievement (U= 112.00, z=
−
2.875, p= 0.004), with effect size r= 0.410. Regarding
learner engagement, the results of class observation reports and learner surveys indicated
that the learners in GFC had the highest level of participation and perceived learning, and
performed all five components of SLE.
5. Discussion
In this study, we compared the influence of flipped classrooms and gamification on
learner achievement and engagement for SLE in the context of postgraduate business edu-
cation during the COVID-19 pandemic. The findings are discussed in the following three
subsections. First is the role of flipped classrooms and gamification during the COVID-19
pandemic. Second, teacher-dependency and learning interactions. Lastly, lessons learned
for the further improvement of flipped classrooms and gamification pedagogical design for
SLE in HEIs.
5.1. Role of Flipped Classroom and Gamification during COVID-19 Pandemic
Flipped classrooms were welcomed by learners [
106
] and helped to maintain the sus-
tainability of learning programmes in higher education during COVID-19 lockdowns [
24
].
However, learner motivation declines in the absence of learning interactions [
48
,
107
]. Gam-
ification promotes learning interactions and social networking [
108
]. The qualitative results
of class observation reports from the teaching assistants on learner engagement concur
with those of Lo and Hew [
60
] that learners in gamified classrooms (GFC and GTC) exhibit
a stronger sense of engagement, as evidenced by their level of participation during in-class
learning activities. In the context of adult postgraduate business education, the quantitative
results of the post-class assignment in our study showed that GTC offers advantages for
learner achievement and GFC for learner engagement, respectively. GFC improved learn-
ers’ perceived learning and engagement, but there was no significant impact on learner
achievement. Several other studies have found similar results, suggesting the need for
further investigation [
109
–
111
]. Our results confirm the observation of Bredow et al. [
81
]
that the benefit of a simple flipped classroom (i.e., NFC) may not be apparent. According
to McLean et al. [
110
], one reason for this might be that flipped classrooms challenge the
learners’ perception of the teacher’s role as a knowledge provider to playing the combined
role of knowledge application and synthesis facilitators.
5.2. Teacher-Dependency and Learning Interactions
A prior study found that, despite the autonomy that the flipped classroom pedagogy
affords to learners, in-class interactive learning was still preferred over pre-class self-
regulated study [
39
]. Peer-to-teacher interaction and learner preference for teachers to play

Sustainability 2022,14, 5428 17 of 23
the role of authority for knowledge delivery are dominant themes in the Asian context [
112
].
In flipped classrooms, teacher roles change from being like those of sages to those of
facilitators, thereby conflicting with existing expectations [
113
]. Like previously published
findings [
114
], our results found better learning achievement in a traditional classroom
approach supplemented by gamification (i.e., GTC), as teacher recognition and feedback
in the form of game-design elements were valued more than responses from peers [
115
].
Such teacher dependency was also observed in previous studies [
116
–
118
]. Teachers should
introduce the importance and value of lesson content at the beginning of each lesson [
119
],
offer a brief review of pre-class materials [
120
], and facilitate peer-to-teacher and peer-to-
peer interactions [25].
In-class learning interactions played an important role in flipped classrooms [
121
], and
there is a need to explore this in further detail [
60
,
122
]. Learning interactions can be either
peer-to-peer or peer-to-teacher interactions. Our interviews with the teaching assistants
indicated that the learners were used to a teacher-dependent style of learning, which may
impede peer-to-peer learning. Learners needed the teacher to motivate the discussion when
the learners’ participation waned. The following is a transcript from an interview with our
experienced teaching assistant (JG):
‘Although the flipped classroom and gamified classroom may be the trends in future
education, our learners are relatively older adults, and the education model that they
experienced during their school ages was traditional. Many people are still inclined
towards the traditional methods of education. We have to use different teaching modes
according to the learners and their preferences and rely on teachers to facilitate changes’.
Therefore, to facilitate SLE in flipped classrooms, the cultural background and learning
styles of the learners, especially the type of schooling that the learners experienced, must be
considered. This is imperative if more instructional content is going to be imparted using
the flipped classroom and gamification approaches during the COVID-19 pandemic [24].
5.3. Lessons Learned
The flipped classroom and gamification designs adopted in this study were theoret-
ically grounded in SDT [
56
]. The three pedagogical interventions (GFC, NFC, and GTC)
provided valuable insight into the practical application of the theories in the context of
adult business education during the COVID-19 pandemic. Based on our findings, we
provide three observations and recommendations for the design of flipped classrooms and
gamification for SLE.
First, gamification plays an important role in motivating adult learners for learning
participation and engagement, which in turn promotes SLE [
123
]. From the perspective
of SDT, timely and evaluative feedback from teachers is vital to promoting learners’ sense
of engagement [
55
]. Second, teachers should facilitate learning interactions at the earliest
stage of flipped classrooms, such as at the beginning of the pre-class learning stage. Similar
inferences have also been made in other studies [
112
,
124
]. Knowledge learned from pre-
class materials must be recollected to set as the foundation of more advanced in-class
learning [
125
]. This is critical, especially in a highly teacher-dependent learning culture.
Teachers’ early participation and facilitation in both pre-class and in-class lessons would
trigger learners’ curiosity and interest and stimulate initial discussions, leading to more
learning participation [
83
]. Last, although we used the same game-design elements in both
GFC and GTC, learner achievement in the flipped GFC was lower than that of the traditional
GTC. Our results resonate with the findings of Jensen et al. [
124
] that the adoption of
flipped classrooms to create more time for in-class learning may not be effective even
with gamification. Teacher-dependency and learning culture must be considered. Adult
learners in our study benefitted more from teacher-led lectures (i.e., from the traditional
GTC) than the two flipped classroom approaches (GFC and NFC). This mirrors the study
of Magana et al. [126].
Nonetheless, the flipped classroom is an effective pedagogical approach to ensuring
the sustainability of education programmes in HEIs during COVID-19 lockdowns [
24
].

Sustainability 2022,14, 5428 18 of 23
Pre-class self-study videos and materials ameliorate the anxiety of learners caused by
academic uncertainty and interruptions in learning [
127
]. For example, one learner from
GTC remarked, ‘if the school can provide pre-class videos, I can prepare in advance, integrate
my previous knowledge and develop clarity on the areas in which I need to learn more (in the
coming in-class lessons)’ (GTC-Learner 3). Therefore, a flipped classroom with gamification
is a valuable pedagogical approach for the sustainability of higher business education,
especially under the dynamic COVID-Zero strategy in China [
128
]. However, teachers
cannot rely solely on flipped classrooms or gamification pedagogies. Indeed, the early
presence of teachers along the entire learning journey, including in the pre-class stage, to
facilitate peer-to-teacher and peer-to-peer facilitations is deemed necessary.
6. Conclusions and Recommendations for Future Study
This study compared learner achievement and engagement across three pedagogical
approaches (i.e., GFC, NFC, and GTC). The learners in GTC performed the best in terms
of learner achievement, whereas those in GFC reported the highest perceived learning
and engagement for SLE. Qualitative findings from surveys and class observation reports
reflected that gamification made the lessons more interesting and exciting, and promoted
participation in in-class learning activities. Learners in GTC and GFC showed higher levels
of participation and performed more key components of SLE, which were the reasons for
higher learner achievement and engagement in comparison to NFC.
We provided three recommendations based on the study. First, flipped classrooms
can be leveraged to maintain the continuity and sustainability of education programmes,
especially under the uncertainty of lockdowns during the COVID-19 pandemic. Second,
gamification plays a key role in improving learner achievement, and should therefore be
integrated into flipped classrooms with the consideration of the learning culture and styles
of the learners. Third, teachers must facilitate and be involved in all stages of flipped
classrooms, especially in highly teacher-dependent learning cultures.
Notwithstanding the aforementioned contributions, there are certain limitations in
the study. First, the sample sizes of the three classrooms were small, which reduces the
generalisability of the results. Second, the study was conducted in a postgraduate adult
business education setting and may therefore not be equally applicable to other educational
contexts. Third, the duration of the study was limited to 10 weeks owing to restrictions
caused by the COVID-19 pandemic. Therefore, the findings must not be interpreted as
establishing that any one of the pedagogical approaches is better than the others. Further
studies are necessary to explore the differences between flipped classrooms with and
without gamification in larger samples and across a longer duration.
Author Contributions:
Formal analysis, investigation, writing—original draft preparation, writing
—review and editing, project administration, and visualization, L.-K.N.; conceptualization, method-
ology, validation, data curation, and supervision, C.-K.L. All authors have read and agreed to the
published version of the manuscript.
Funding: This research received no external funding.
Institutional Review Board Statement:
The study was conducted according to the guidelines of the
Declaration of Helsinki and approved by the Human Research Ethics Committee of the Education
University of Hong Kong (protocol code: 2020-2021-0436; date of approval: 28 October 2021).
Informed Consent Statement:
Informed consent was obtained from all subjects involved in the study.
Data Availability Statement:
The data samples and detailed coding procedures can be accessed by
contacting the author.
Acknowledgments:
Thank you for the valuable and constructive feedback from the editor and
professional reviewers. This study is in partial fulfilment of the Research Project/Thesis Requirements
of the Education University of Hong Kong.
Conflicts of Interest: The authors declare no conflict of interest.

Sustainability 2022,14, 5428 19 of 23
References
1.
Chen, F.H.; Tsai, C.C.; Chung, P.Y.; Lo, W.S. Sustainability Learning in Education for Sustainable Development for 2030:
An Observational Study Regarding Environmental Psychology and Responsible Behavior through Rural Community Travel.
Sustainability 2022,14, 2779. [CrossRef]
2.
Leicht, A.; Heiss, J.; Byun, W.J. Issues and Trends in Education for Sustainable Development; UNESCO Publishing: Paris, France, 2018;
Volume 5.
3.
Martin, M.; Godonoga, A. SDG 4-Policies for Flexible Learning Pathways in Higher Education: Taking Stock of Good Practices
Internationally; IIEP-UNESCO: Paris, France, 2020.
4. Ben-Eliyahu, A. Sustainable learning in education. Sustainability 2021,13, 4250. [CrossRef]
5. Pilotti, M.A.; Al Ghazo, R. Sustainable education starts in the classroom. Sustainability 2020,12, 9573. [CrossRef]
6.
Rovio-Johansson, A. Students’ Knowledge Progression: Sustainable Learning in Higher Education. Int. J. Learn. High.
2016
,28,
427–439.
7.
Yassin, A.A.; Razak, N.A.; Qasem, Y.A.; Mohammed, M.A.S. Intercultural learning challenges affecting international students’
sustainable learning in Malaysian higher education institutions. Sustainability 2020,12, 7490. [CrossRef]
8.
Hays, J.; Reinders, H. Sustainable learning and education: A curriculum for the future. Int. Rev. Educ.
2020
,66, 29–52. [CrossRef]
9.
Pfeiffer, A.; Bezzina, S.; Denk, N.; Kriglstein, S.; Wimmer, S.; Thielen, C. Life-World Oriented Education in Times of the COVID-19
Lockdown. Twitch.tv as Pedagogical Tool. ICERI2020 Proc. 2020,9, 10.
10.
Yu, K.; Wu, L.; Zhou, L. Research on the Mixed Education Mode for the Safety Engineering Major during the Coronavirus
(COVID-19) Epidemic. Int. J. Environ. Res. Public Health 2022,19, 1967. [CrossRef]
11.
Pérez-Jorge, D.; Rodríguez-Jiménez, M.C.; Ariño-Mateo, E.; Barragán-Medero, F. The effect of COVID-19 in university tutoring
models. Sustainability 2020,12, 8631. [CrossRef]
12.
Murphy, M.P.A. COVID-19 and emergency eLearning: Consequences of the securitization of higher education for post-pandemic
pedagogy. Contemp. Secur. Policy 2020,41, 492–505. [CrossRef]
13.
Le, K. Pre-Recorded Lectures, Live Online Lectures, and Student Academic Achievement. Sustainability
2022
,14, 2910. [CrossRef]
14. Yang, R. China’s higher education during the COVID-19 pandemic: Some preliminary observations. High. Educ. Res. Dev. 2020,
39, 1317–1321. [CrossRef]
15.
Ardhaoui, K.; Lemos, M.S.; Silva, S. Effects of new teaching approaches on motivation and achievement in higher education
applied chemistry courses: A case study in Tunisia. Educ. Chem. Eng. 2021,36, 160–170. [CrossRef]
16.
Bowden, J.L.H.; Tickle, L.; Naumann, K. The four pillars of tertiary student engagement and success: A holistic measurement
approach. Stud. High. Educ. 2021,46, 1207–1224. [CrossRef]
17.
Jang, H.Y.; Kim, H.J. A meta-analysis of the cognitive, affective, and interpersonal outcomes of flipped classrooms in higher
education. Educ. Sci. 2020,10, 115. [CrossRef]
18.
Huang, B.; Hew, K.F.; Lo, C.K. Investigating the effects of gamification-enhanced flipped learning on undergraduate students’
behavioral and cognitive engagement. Interact. Learn. Environ. 2019,27, 1106–1126. [CrossRef]
19.
Sailer, M.; Sailer, M. Gamification of in-class activities in flipped classroom lectures. Br. J. Educ. Technol.
2021
,52, 75–90. [CrossRef]
20.
Deci, E.L.; Ryan, R.M. Self-determination theory. In Handbook of Theories of Social Psychology; Van Lange, P.A.M., Kruglanski, A.W.,
Higgins, E.T., Eds.; Sage Publications Ltd.: New York, NY, USA, 2012; pp. 416–436. [CrossRef]
21.
Reeve, J. Self-determination theory applied to educational settings. In Handbook of Self-Determination Research: Theoretical and
Applied Issues; Deci, E.L., Ryan, R.M., Eds.; University of Rochester Press: Rochester, NY, USA, 2002; Volume 2, pp. 183–204.
22.
Deterding, S.; Dixon, D.; Khaled, R.; Nacke, L. From game design elements to gamefulness: Defining “gamification”. In
Proceedings of the 15th International Academic MindTrek Conference: Envisioning Future Media Environments, Tampere,
Finland, 28–30 September 2011.
23.
Dicheva, D.; Dichev, C.; Agre, G.; Angelova, G. Gamification in education: A systematic mapping study. J. Educ. Technol. Soc.
2015,18, 75–88.
24.
Collado-Valero, J.; Rodríguez-Infante, G.; Romero-González, M.; Gamboa-Ternero, S.; Navarro-Soria, I.; Lavigne-Cerván, R.
Flipped classroom: Active methodology for sustainable learning in higher education during social distancing due to COVID-19.
Sustainability 2021,13, 5336. [CrossRef]
25.
Akram, A.; Sattar, K.; Ahmad, T.; Abdulghani, H.M.; John, J.; Yusoff, M.S.B. Flipping the “Learning Skills” Course during
COVID-19: A Mixed-Modality Study. Educ. Med. J. 2021,13, 31–41. [CrossRef]
26.
French, H.; Arias-Shah, A.; Gisondo, C.; Gray, M.M. Perspectives: The flipped classroom in graduate medical education.
NeoReviews 2020,21, e150–e156. [CrossRef] [PubMed]
27.
Peters, M.A.; Wang, H.; Ogunniran, M.O.; Huang, Y.; Green, B.; Chunga, J.O.; Quianoo, E.A.; Ren, Z.; Hollings, S.; Hayes, S.; et al.
China’s internationalized higher education during COVID-19: Collective student autoethnography. Postdigit. Sci. Educ.
2020
,2,
968–988. [CrossRef]
28.
Wu, Z. How a Top Chinese University Is Responding to Coronavirus. 2020. Available online: https://www.weforum.org/
agenda/2020/03/coronavirus-china-the-challenges-of-online-learning-for-universities/ (accessed on 28 March 2022).
29.
Li, S.; Liao, X.; Burdick, W.; Tong, K. The Effectiveness of Flipped Classroom in Health Professions Education in China:
A Systematic Review. J. Med. Educ. Curric. 2020,7, 2382120520962838. [CrossRef] [PubMed]

Sustainability 2022,14, 5428 20 of 23
30.
Filho, W.L.; Raath, S.; Lazzarini, B.; Vargas, V.R.; de Souza, L.; Anholon, R.; Quelhas, O.L.G.; Haddad, R.; Klavins, M.;
Orlovic, V.L.
The role of transformation in learning and education for sustainability. J. Clean. Prod 2018,199, 286–295. [CrossRef]
31.
Willats, J.; Erlandsson, L.; Molthan-Hill, P.; Dharmasasmita, A.; Simmons, E. A university wide approach to embedding the
sustainable development goals in the curriculum—A case study from the Nottingham Trent University’s Green Academy. In
Implementing Sustainability in the Curriculum of Universities; Springer: Cham, Switzerland, 2018; pp. 63–78.
32.
Hays, J. Chaos to Capability: Educating Professionals for the 21st Century; Unitec ePress Monograph Series (1); Unitec Institute of
Technology: Auckland, New Zealand, 2015. Available online: https://hdl.handle.net/10652/3053 (accessed on 28 March 2022).
33.
Azevedo, R. Defining and measuring engagement and learning in science: Conceptual, theoretical, methodological, and analytical
issues. Educ. Psychol. 2015,50, 84–94. [CrossRef]
34.
Ben-Eliyahu, A.; Linnenbrink-Garcia, L. Integrating the regulation of affect, behavior, and cognition into self-regulated learning
paradigms among secondary and post-secondary students. Metacogn. Learn. 2015,10, 15–42. [CrossRef]
35.
Hettiarachchi, S.; Damayanthi, B.W.R.; Heenkenda, S.; Dissanayake, D.M.S.L.B.; Ranagalage, M.; Ananda, L. Student Satisfaction
with Online Learning during the COVID-19 Pandemic: A Study at State Universities in Sri Lanka. Sustainability
2021
,13, 11749.
[CrossRef]
36.
Li, S.; Wang, Y.; Xue, J.; Zhao, N.; Zhu, T. The impact of COVID-10 epidemic declaration on psychological consequences: A study
on active weibo users. Int. J. Environ. Res. Public Health 2020,17, 2032. [CrossRef]
37.
Šakan, D.; Žuljevi´c, D.; Rokvi´c, N. The role of basic psychological needs in well-being during the COVID-19 outbreak: A self-
determination theory perspective. Front. Public Health 2020,8, 713. [CrossRef]
38.
Carrillo, C.; Flores, M.A. COVID-19 and teacher education: A literature review of online teaching and learning practices. Eur. J.
Teach. Educ. 2020,43, 466–487. [CrossRef]
39.
Bishop, J.; Verleger, M.A. The flipped classroom: A survey of the research. In Proceedings of the 2013 ASEE Annual Conference &
Exposition, Atlanta, GA, USA, 23–26 June 2013; pp. 23–1200.
40.
Abeysekera, L.; Dawson, P. Motivation and cognitive load in the flipped classroom: Definition, rationale and a call for research.
High. Educ. Res. Dev. 2015,34, 1–14. [CrossRef]
41.
Mahmud, S.N.D.; Husnin, H.; Tuan Soh, T.M. Teaching presence in online gamified education for sustainability learning.
Sustainability 2020,12, 3801. [CrossRef]
42. Galindo-Dominguez, H. Flipped Classroom in the Educational System. J. Educ. Technol. Soc. 2021,24, 44–60.
43.
Hsia, L.H.; Lin, Y.N.; Hwang, G.J. A creative problem solving-based flipped learning strategy for promoting students’ performing
creativity, skills and tendencies of creative thinking and collaboration. Br. J. Educ. Technol. 2021,52, 1771–1787. [CrossRef]
44.
Cao, W.; Fang, Z.; Hou, G.; Han, M.; Xu, X.; Dong, J.; Zheng, J. The psychological impact of the COVID-19 epidemic on college
students in China. Psychiatr. Res. 2020,287, 112934. [CrossRef] [PubMed]
45.
Hernández, D.J.; Ortiz, J.J.G.; Abellán, M.T. Metodologías Activas en la Universidad y su relación con los Enfoques de Enseñanza.
Profr. Rev. De Currículum Y Form. Del Profr. 2020,24, 76–94. [CrossRef]
46.
Xiong, J.; Yan, J.; Fu, K.; Wang, K.; He, Y. Innovation in an authoritarian society: China during the pandemic crisis. J. Bus. Strategy
2021,43, 79–86. [CrossRef]
47.
Zainuddin, Z. Students’ learning performance and perceived motivation in gamified flipped-class instruction. Comput. Educ.
2018,126, 75–88. [CrossRef]
48.
Ahshan, R. A framework of implementing strategies for active student engagement in remote/online teaching and learning
during the COVID-19 pandemic. Educ. Sci. 2021,11, 483. [CrossRef]
49.
Heo, H.; Bonk, C.J.; Doo, M.Y. Enhancing learning engagement during COVID-19 pandemic: Self-efficacy in time management,
technology use, and online learning environments. J. Comput. Assist. Learn. 2021,37, 1640–1652. [CrossRef]
50.
Kuhfeld, M.; Soland, J.; Tarasawa, B.; Johnson, A.; Ruzek, E.; Liu, J. Projecting the potential impact of COVID-19 school closures
on academic achievement. Educ. Res. 2020,49, 549–565. [CrossRef]
51.
Luria, E.; Shalom, M.; Levy, D.A. Cognitive Neuroscience Perspectives on Motivation and Learning: Revisiting Self-Determination
Theory. Mind Brain Educ. 2021,15, 5–17. [CrossRef]
52. Ryan, R.M. Psychological needs and the facilitation of integrative processes. J. Pers. 1995,63, 397–427. [CrossRef] [PubMed]
53. Silvia, P.J. Interest—The curious emotion. Curr. Dir. Psychol. Sci. 2008,17, 57–60. [CrossRef]
54.
Ashari, H.; Abbas, I.; Abdul-Talib, A.N.; Zamani, S.N.M. Entrepreneurship and Sustainable Development Goals: A Multigroup
Analysis of the Moderating Effects of Entrepreneurship Education on Entrepreneurial Intention. Sustainability
2021
,14, 431.
[CrossRef]
55.
Niemiec, C.P.; Ryan, R.M. Autonomy, competence, and relatedness in the classroom: Applying self-determination theory to
educational practice. Theory Res. Educ. 2009,7, 133–144. [CrossRef]
56.
Ryan, R.M.; Deci, E.L. Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being.
Am. Psychol. 2000,55, 68–78. [CrossRef]
57.
Connell, J.P.; Wellborn, J.G. Competence, autonomy, and relatedness. In A Motivational Analysis of Self-System Processes;
Gunnar, M.R., Sroufe, L.A., Eds.; Lawrence Erlbaum Associates, Inc.: Hillsdale, MI, USA, 1991; pp. 43–77.
58.
Newmann, F.M. Student Engagement and Achievement in American Secondary Schools; Teachers College Press: New York, NY,
USA, 1992.

Sustainability 2022,14, 5428 21 of 23
59.
Sergis, S.; Sampson, D.G.; Pelliccione, L. Investigating the impact of Flipped Classroom on students’ learning experiences:
A Self-Determination Theory approach. Comput. Hum. Behav. 2018,78, 368–378. [CrossRef]
60.
Lo, C.K.; Hew, K.F. A comparison of flipped learning with gamification, traditional learning, and online independent study: The
effects on students’ mathematics achievement and cognitive engagement. Interact. Learn. Environ. 2020,28, 464–481. [CrossRef]
61.
Pelikan, E.R.; Korlat, S.; Reiter, J.; Holzer, J.; Mayerhofer, M.; Schober, B.; Spiel, C.; Hamzallari, O.; Uka, A.; Chen, J.; et al. Distance
learning in higher education during COVID-19: The role of basic psychological needs and intrinsic motivation for persistence
and procrastination—A multi-country study. PLoS ONE 2021,16, e0257346. [CrossRef]
62.
Chiu, T.K. Applying the self-determination theory (SDT) to explain student engagement in online learning during the COVID-19
pandemic. J. Res. Technol. Educ. 2021,54 (Suppl. 1), S14–S30. [CrossRef]
63.
Bayoumy, H.M.M.; Alsayed, S. Investigating relationship of perceived learning engagement, motivation, and academic perfor-
mance among nursing students: A multisite study. Adv. Med. Educ. Pract. 2021,12, 351. [CrossRef] [PubMed]
64.
Panigrahi, R.; Srivastava, P.R.; Panigrahi, P.K. Effectiveness of e-learning: The mediating role of student engagement on perceived
learning effectiveness. Inf. Technol. People 2020,34, 1840–1862. [CrossRef]
65.
Lozano-Jiménez, J.E.; Huéscar, E.; Moreno-Murcia, J.A. Effects of an autonomy support intervention on the involvement of higher
education students. Sustainability 2021,13, 5006. [CrossRef]
66.
Orazbayeva, B.; van der Sijde, P.; Baaken, T. Autonomy, competence and relatedness–the facilitators of academic engagement in
education-driven university-business cooperation. Stud. High. Educ. 2021,46, 1406–1420. [CrossRef]
67.
Deterding, S.; Sicart, M.; Nacke, L.; O’Hara, K.; Dixon, D. Gamification. Using game-design elements in non-gaming contexts. In
CHI’11 Extended Abstracts on Human Factors in Computing Systems; Association for Computing Machinery: New York, NY, USA,
2011; pp. 2425–2428.
68.
Park, S.; Kim, S. Is sustainable online learning possible with gamification?—The effect of gamified online learning on student
learning. Sustainability 2021,13, 4267. [CrossRef]
69. Ekici, M. A systematic review of the use of gamification in flipped learning. Educ. Inf. Technol. 2021,26, 3327–3346. [CrossRef]
70.
Hammill, J.; Nguyen, T.; Henderson, F. Encouraging the flip with a gamified process. Int. J. Educ. Res.
2021
,2, 100085. [CrossRef]
71.
Langendahl, P.A.; Cook, M.; Mark-Herbert, C. Gamification in Higher Education: Toward a Pedagogy to Engage and Motivate;
Department of Economics, Swedish University of Agricultural Sciences: Uppsala, Sweden, 2016.
72.
Deci, E.L.; Ryan, R.M. Optimizing students’ motivation in the era of testing and pressure: A self-determination theory perspective.
In Building Autonomous Learners; Springer: Singapore, 2016; pp. 9–29.
73.
Rosenbaum, M.S.; Russell-Bennett, R.; Contreras-Ramírez, G. Business education in profound disruption. J. Serv. Mark.
2021
,35,
553–558. [CrossRef]
74.
Chang, D.; Hwang, G.J.; Chang, S.C.; Wang, S.Y. Promoting students’ cross-disciplinary performance and higher order thinking:
A peer assessment-facilitated STEM approach in a mathematics course. Educ. Technol. Res. Dev. 2021,69, 3281–3306. [CrossRef]
75.
Kressler, B.; Kressler, J. Diverse student perceptions of active learning in a large enrollment STEM course. J. Scholarsh. Teach. Learn.
2020,20, 40–64. [CrossRef]
76.
Jusas, V.; Barisas, D.; Janˇciukas, M. Game Elements towards More Sustainable Learning in Object-Oriented Programming Course.
Sustainability 2022,14, 2325. [CrossRef]
77.
Sailer, M.; Hense, J.U.; Mayr, S.K.; Mandl, H. How gamification motivates: An experimental study of the effects of specific game
design elements on psychological need satisfaction. Comput. Hum. Behav. 2017,69, 371–380. [CrossRef]
78. Rincon-Flores, E.G.; Santos-Guevara, B.N. Gamification during COVID-19: Promoting active learning and motivation in higher
education. Australas. J. Educ. Technol. 2021,37, 43–60. [CrossRef]
79.
Santos-Villalba, M.J.; Olivencia, J.J.L.; Navas-Parejo, M.R.; Benítez-Márquez, M.D. Higher Education Students’ Assessments
towards Gamification and Sustainability: A Case Study. Sustainability 2020,12, 8513. [CrossRef]
80.
Ahmed, H.D.; Asiksoy, G. The Effects of Gamified Flipped Learning Method on Student’s Innovation Skills, Self-Efficacy towards
Virtual Physics Lab Course and Perceptions. Sustainability 2021,13, 10163. [CrossRef]
81.
Bredow, C.A.; Roehling, P.V.; Knorp, A.J.; Sweet, A.M. To flip or not to flip? A meta-analysis of the efficacy of flipped learning in
higher education. Rev. Educ. Res. 2021,91, 878–918. [CrossRef]
82.
Creswell, J.W.; Klassen, A.C.; Clark, V.L.P.; Smith, K.C. Best practices for mixed methods research in the health sciences. Bethesda
Natl. Inst. Health 2013,12, 541–545.
83.
Ho, C.M.; Yeh, C.C.; Wang, J.Y.; Hu, R.H.; Lee, P.H. Pre-class online video learning and class style expectation: Patterns, association,
and precision medical education. Ann. Med. 2021,53, 1390–1401. [CrossRef]
84.
Furdu, I.; Tomozei, C.; Kose, U. Pros and Cons Gamification and Gaming in Classroom.: Discovery Service for University of
South Carolina. BRAIN Broad Res. Artif. Intell. Neurosci. 2017,8, 56–62.
85.
Mazarakis, A. Using Gamification for Technology Enhanced Learning: The Case of Feedback Mechanisms. Bull. Tech. Comm.
Learn. Technol. 2015,17, 6–9.
86.
Hamari, J. Do badges increase user activity? A field experiment on the effects of gamification. Comput. Hum. Behav.
2017
,71,
469–478. [CrossRef]
87.
Burguillo, J.C. Using game theory and competition-based learning to stimulate student motivation and performance. Comput.
Educ. 2010,55, 566–575. [CrossRef]
88. Okura, M.; Carfì, D. Coopetition and game theory. J. Appl. Econ. Sci. 2014,9, 29.

Sustainability 2022,14, 5428 22 of 23
89. Seaborn, K.; Fels, D.I. Gamification in theory and action: A survey. Int. J. Hum. Comput. Stud. 2015,74, 14–31. [CrossRef]
90.
Skinner, E.; Furrer, C.; Marchand, G.; Kindermann, T. Engagement and disaffection in the classroom: Part of a larger motivational
dynamic? J. Educ. Psychol. 2008,100, 765. [CrossRef]
91.
Rotgans, J.I.; Schmidt, H.G. Cognitive engagement in the problem-based learning classroom. Adv. Health Sci. Educ. Theory Pract.
2011,16, 465–479. [CrossRef]
92.
Al-Zahrani, A.M. From passive to active: The impact of the flipped classroom through social learning platforms on higher
education students’ creative thinking. Br. J. Educ. Technol. 2015,46, 1133–1148. [CrossRef]
93.
Chi, M.T.; Wylie, R. The ICAP framework: Linking cognitive engagement to active learning outcomes. Educ. Psychol.
2014
,49,
219–243. [CrossRef]
94.
Hew, K.F.; Huang, B.; Chu, K.W.S.; Chiu, D.K. Engaging Asian students through game mechanics: Findings from two experiment
studies. Comput. Educ. 2016,92, 221–236. [CrossRef]
95.
Merriam, S.B.; Grenier, R.S. (Eds.) Qualitative Research in Practice: Examples for Discussion and Analysis; John Wiley & Sons: San
Francisco, CA, USA, 2019.
96.
Fredricks, J.A.; Blumenfeld, P.C.; Paris, A.H. School engagement: Potential of the concept, state of the evidence. Rev. Educ. Res.
2004,74, 59–109. [CrossRef]
97.
McCallum, S.; Schultz, J.; Sellke, K.; Spartz, J. An examination of the flipped classroom approach on college student academic
involvement. Int. J. Learn. High. 2015,27, 42–55.
98. Field, A. Discovering Statistics Using SPSS, 3rd ed.; Sage Publications Ltd.: London, UK, 2009.
99.
Weisstein, E.W. Bonferroni Correction. 2004. Available online: https://mathworld.wolfram.com/BonferroniCorrection.html
(accessed on 28 March 2022).
100.
Tomczak, M.; Tomczak, E. The need to report effect size estimates revisited. An overview of some recommended measures of
effect size. Trends Sport Sci. 2014,1, 19–25.
101. Johnson, R.B. Examining the validity structure of qualitative research. Education 1997,118, 282–292.
102. Glesne, C. Becoming Qualitative Researchers: An Introduction; Pearson: Upper Saddle River, NJ, USA, 2016.
103. Saldana, J. Fundamentals of Qualitative Research; Oxford University Press: Oxford, UK, 2011.
104. Maxwell, J.A. Qualitative Research Design: An Interactive Approach; Sage Publications: Newbury Park, CA, USA, 2012.
105. Flick, U. Designing Qualitative Research; Sage Publications: Newbury Park, CA, USA, 2018.
106.
Li, B.Z.; Cao, N.W.; Ren, C.X.; Chu, X.J.; Zhou, H.Y.; Guo, B. Flipped classroom improves nursing students’ theoretical learning in
China: A meta-analysis. PLoS ONE 2020,15, e0237926. [CrossRef] [PubMed]
107.
Kormos, J.; Csizer, K. The interaction of motivation, self-regulatory strategies, and autonomous learning behavior in different
learner groups. Tesol Q. 2014,48, 275–299. [CrossRef]
108.
Hajarian, M.; Diaz, P. Effective Gamification: A Guideline for Gamification Workshop of WEEF-GEDC 2021 Madrid Conference.
In Proceedings of the 2021 World Engineering Education Forum/Global Engineering Deans Council (WEEF/GEDC), Madrid,
Spain, 15–18 November 2021; pp. 506–510.
109.
Adams, A.E.; Garcia, J.; Traustadóttir, T. A quasi experiment to determine the effectiveness of a “partially flipped” versus “fully
flipped” undergraduate class in genetics and evolution. CBE Life Sci. Educ. 2016,15, 11. [CrossRef]
110.
McLean, S.; Attardi, S.M.; Faden, L.; Goldszmidt, M. Flipped classrooms and student learning: Not just surface gains. Adv. Physiol.
Educ. 2016,40, 47–55. [CrossRef]
111.
Smallhorn, M. The flipped classroom: A learning model to increase student engagement not academic achievement. Stud. Success
2017,8, 43–53. [CrossRef]
112.
Loh, C.Y.R.; Teo, T.C. Understanding Asian students learning styles, cultural influence and learning strategies. J. Educ. Soc. Pol.
2017,7, 194–210.
113.
Van Sickle, J.R. Discrepancies between student perception and achievement of learning outcomes in a flipped classroom.
J. Scholarsh. Teach. Learn. 2016,16, 29–38. [CrossRef]
114.
Zheng, L.; Bhagat, K.K.; Zhen, Y.; Zhang, X. The effectiveness of the flipped classroom on students’ learning achievement and
learning motivation. J. Educ. Technol. Soc. 2020,23, 1–15.
115.
Zhao, H. Investigating learners’ use and understanding of peer and teacher feedback on writing: A comparative study in a
Chinese English writing classroom. Assess. Writ. 2010,15, 3–17. [CrossRef]
116.
Hew, K.F. Student perceptions of peer versus instructor facilitation of asynchronous online discussions: Further findings from
three cases. Instr. Sci. 2015,43, 1938. [CrossRef]
117. Roorda, D.L.; Zee, M.; Koomen, H.M. Don’t forget student-teacher dependency! A Meta-analysis on associations with students’
school adjustment and the moderating role of student and teacher characteristics. Attach Hum. Dev.
2021
,23, 490–503. [CrossRef]
118.
Vaziri, S.; Vaziri, B.; Novoa, L.J.; Torabi, E. Academic Motivation in Introductory Business Analytics Courses: A Bayesian
Approach. INFORMS Trans. Educ. 2022,22, 121–129. [CrossRef]
119.
Long, T.; Logan, J.; Waugh, M. Students’ perceptions of the value of using videos as a pre-class learning experience in the flipped
classroom. TechTrends 2016,60, 245–252. [CrossRef]
120.
Lo, C.K.; Hew, K.F. The impact of flipped classrooms on student achievement in engineering education: A meta-analysis of
10 years of research. J. Eng. Educ. 2019,108, 523–546. [CrossRef]

Sustainability 2022,14, 5428 23 of 23
121.
Lestari, I.W. Flipped classroom in Indonesian higher education: A mixed-method study on students’ attitudes and experiences.
Stud. Engl. Lang. Educ. 2021,8, 243–257. [CrossRef]
122. Maciejewski, W. Flipping the calculus classroom: An evaluative study. Teach. Math. Appl. 2016,35, 187–201. [CrossRef]
123.
Karra, S.; Karampa, V.; Paraskeva, F. Gamification design framework based on self determination theory for adult motivation. In
Proceedings of International Workshop on Learning Technology for Education in Cloud; Springer: Cham, Switzerland, 2019; pp. 67–78.
124.
Jensen, J.L.; Holt, E.A.; Sowards, J.B.; Ogden, T.H.; West, R.E. Investigating strategies for pre-class content learning in a flipped
classroom. J. Sci. Educ. Technol. 2018,27, 523–535. [CrossRef]
125.
Yorganci, S. Implementing flipped learning approach based on ‘first principles of instruction’ in mathematics courses. J. Comput.
Assist. Learn. 2020,36, 763–779. [CrossRef]
126.
Magana, A.J.; Vieira, C.; Boutin, M. Characterizing engineering learners’ preferences for active and passive learning methods.
IEEE Trans. Educ. 2017,61, 46–54. [CrossRef]
127.
Arribathi, A.H.; Suwarto; Rosyad, A.M.; Budiarto, M.; Supriyanti, D.; Mulyati. An analysis of student learning anxiety during the
COVID-19 Pandemic: A Study in Higher Education. J. Contin. High. Educ. 2021,69, 192–205. [CrossRef]
128. Liu, J.; Liu, M.; Liang, W. The Dynamic COVID-Zero Strategy in China. China CDC Wkly. 2022,4, 74–75. [CrossRef]