ArticlePDF Available

Abstract and Figures

Gamification is increasingly employed in learning environments as a way to increase student motivation and consequent learning outcomes. However, while the research on the effectiveness of gamification in the context of education has been growing, there are blind spots regarding which types of gamification may be suitable for different educational contexts. This study investigates the effects of the challenge-based gamification on learning in the area of statistics education. We developed a gamification approach, called Horses for Courses, which is composed of main game design patterns related to the challenge-based gamification; points, levels, challenges and a leaderboard. Having conducted a 2 (read: yes vs. no) x 2 (gamification: yes vs. no) between-subject experiment, we present a quantitative analysis of the performance of 365 students from two different academic majors: Electrical and Computer Engineering (n=279), and Business Administration (n=86). The results of our experiments show that the challenge-based gamification had a positive impact on student learning compared to traditional teaching methods (compared to having no treatment and treatment involving reading exercises). The effect was larger for females and for students at the School of Electrical and Computer Engineering.
Content may be subject to copyright.
Contents lists available at ScienceDirect
International Journal of Human-Computer Studies
journal homepage: www.elsevier.com/locate/ijhcs
The eect of challenge-based gamication on learning: An experiment in the
context of statistics education
Nikoletta-Zampeta Legaki
,a,b
, Nannan Xi
b
, Juho Hamari
b
, Kostas Karpouzis
a
,
Vassilios Assimakopoulos
a
a
School of Electrical and Computer Engineering, National Technical University of Athens, Zografou 15780, Greece
b
Gamication Group, Faculty of Information Technology and Communication Sciences, Tampere University, Tampere 33100, Finland
ARTICLE INFO
Keywords:
Gamication
Applications in education
Statistics education
Teaching forecasting
Human-Computer interface
ABSTRACT
Gamication is increasingly employed in learning environments as a way to increase student motivation and
consequent learning outcomes. However, while the research on the eectiveness of gamication in the context of
education has been growing, there are blind spots regarding which types of gamication may be suitable for
dierent educational contexts. This study investigates the eects of the challenge-based gamication on learning
in the area of statistics education. We developed a gamication approach, called Horses for Courses, which is
composed of main game design patterns related to the challenge-based gamication; points, levels, challenges
and a leaderboard. Having conducted a 2 (read: yes vs. no) x 2 (gamication: yes vs. no) between-subject
experiment, we present a quantitative analysis of the performance of 365 students from two dierent academic
majors: Electrical and Computer Engineering (n=279), and Business Administration (n=86). The results of our
experiments show that the challenge-based gamication had a positive impact on student learning compared to
traditional teaching methods (compared to having no treatment and treatment involving reading exercises). The
eect was larger for females or for students at the School of Electrical and Computer Engineering.
1. Introduction
Gamication approaches are being applied with increasing fre-
quency in an attempt to positively aect behavior and cognitive pro-
cesses by enhancing the system or service with motivational aor-
dances and eventually by bringing similar experiences as games
do (Huotari and Hamari, 2017). Motivational aordances have been
widely used in many elds such as business (Alcivar and Abad, 2016; Xi
and Hamari, 2020), crowdsourcing (Morschheuser et al., 2017),
healthcare (Johnson et al., 2016) and education (Dichev and Dicheva,
2017; Hanus and Fox, 2015; Koivisto and Hamari, 2019; Majuri et al.,
2018; Osatuyi et al., 2018; Seaborn and Fels, 2015). Additionally, ga-
mication has been employed in many education related contexts,
across dierent educational levels (Caponetto et al., 2014; Dicheva
et al., 2015; Simões et al., 2013; de Sousa Borges et al., 2014) and in
various subjects (Dichev and Dicheva, 2017; Dicheva et al., 2015;
Kasurinen and Knutas, 2018; Seaborn and Fels, 2015), showing its
potential to improve learning outcomes (Koivisto and Hamari, 2019;
Seaborn and Fels, 2015).
According to reviews of gamication literature, gamication has
been employed mostly in the eld of education (Koivisto and Hamari,
2019; Majuri et al., 2018; Seaborn and Fels, 2015). Gamied educa-
tional applications have been applied in non-academic areas as well:
language teaching (Duolingo counts 300 million active users
1
) or soft-
ware using (Ribbon Hero by Microsoft). Other popular gamied ap-
plications are: Kahoot and Quizizz, which can be easily congured and
used in a variety of subjects, bringing game elements into classrooms
without any special eort. Although gamication has an important
position in education both inside and outside universities, there is still
little eective guidance on how to combine dierent gamication fea-
tures to enhance learning performance in dierent educational
contexts (Hanus and Fox, 2015; Koivisto and Hamari, 2019; Seaborn
and Fels, 2015).
Beyond research problems pertaining to the general interest in ga-
mication and its eect on education, statistics education is an in-
creasingly fundamental skill to understand the world around us. The
lack of data literacy has been deemed one of the main causes behind our
inability to act against climate change, to properly ratify means towards
e.g. COVID-19 or generally as a hindrance for public understanding of
science. Therefore, there is a need to make teaching methods in
https://doi.org/10.1016/j.ijhcs.2020.102496
Received 12 November 2019; Received in revised form 9 June 2020; Accepted 12 June 2020
Corresponding author.
E-mail addresses: zampeta.legaki@tuni.,zabbeta@fsu.gr,nzabbeta@gmail.com (N.-Z. Legaki).
1
https://ai.duolingo.com/
International Journal of Human-Computer Studies 144 (2020) 102496
Available online 14 June 2020
1071-5819/ © 2020 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license
(http://creativecommons.org/licenses/BY-NC-ND/4.0/).
T
statistics and forecasting more engaging (Love and Hildebrand, 2002).
The acceleration of the daily data production, and ever-greater capacity
to store and process this information, has boosted the necessity for
students with a strong background in statistics and predictive analytics
skills, in business environments or even in everyday life. Consequently,
both statistics and forecasting techniques are of vital importance in the
economics curriculum (Loomis and Cox Jr, 2003) and in other elds
such as business (Makridakis et al., 2008) or social problems, where
data may help to make better decisions. However, often forecasting
courses are not even oered as an independent course in business
schools (Hanke, 1989) and when they are, students are discouraged to
participate in the courses because they nd the topic too
complicated (Albritton and McMullen, 2006; Gardner, 2008; Snider and
Eliasson, 2013; Torres et al., 2018) and demanding (Craighead, 2004).
Therefore, regarding the education and especially the education in the
eld of statistics and forecasting, student motivation is crucial for their
participation and understanding in order to reach their learning po-
tential, meet business needs and get insights of the data to support the
decision-making process.
Despite the long tradition in educational business games, thus far
there have only been a few studies on gamication or simple gamied
exercises, combined with traditional teaching methods in the area of
statistical forecasting. These studies have mostly used: score
(Craighead, 2004), spreadsheets (Gardner, 2008),
competition (Snider and Eliasson, 2013) and real-world forecasting
problems (Buckley and Doyle, 2016a; Gavirneni, 2008) in order to
encourage students' participation, without examining gamication ef-
fects. Other more quantitative studies have used forecasting in the
context of a prediction market as a tool to motivate students rather than
to teach forecasting aspects (Buckley and Doyle, 2016a; 2016b; 2017;
Buckley et al., 2011). While there are several types of games and ga-
mication designs, the challenge-based gamication (e.g points, levels,
leaderboard, clear goals/ tasks), as opposed to the immersion- and the
social-based gamication, has been suggested (Dicheva et al., 2015;
de Sousa Borges et al., 2014; Zichermann and Cunningham, 2011) and
applied to a high degree in practice as gamication design in
education (Koivisto and Hamari, 2019; Seaborn and Fels, 2015;
de Sousa Borges et al., 2014). One or more of these gamication ele-
ments have been used with promising results even in educational topics
relative to forecasting (Craighead, 2004; Gavirneni, 2008; Gel et al.,
2014; Snider and Eliasson, 2013). However, there is still a lack of ef-
fective design guides and empirical data on the combination or in-
tegration of these features in the context of educational information
systems (Koivisto and Hamari, 2019). Challenge-based gamication
introduces a design approach of integrating achievement gamication
features, positively related with intrinsic need satisfaction (Xi and
Hamari, 2019), in an educational service or application, in order to
explore its potential, motivate users and eventually improve learning.
The present study examines the impact of three treatments on stu-
dentsperformance i) reading, ii) use of a challenge-based gamied
application, iii) the combination of the two. In order to do that, we
consider a variety of student characteristics such as gender, level of
studies, academic major, expertise in the English language, and use of
personal computers and games. We designed and implemented a web-
based gamied application, called Horses for Courses and we conducted
a series of experiments over the last 4 years. The total sample is com-
posed of 365 students, with 279 undergraduate and MBA students at
the School of Electrical and Computer Engineering of the National
Technical University of Athens, Greece (hereafter ECE, NTUA) and
86 undergraduate students at the Business Administration Department
in the School of Business and Economics of the University of Thessaly,
Greece (hereafter Business Administration). Our ndings show that
challenge-based gamication improves studentslearning outcomes on
a statistics course, contributing to the knowledge of challenge-based
gamicationseect on statistics/stem education and eventually on
gamied pedagogy.
2. Background
2.1. Gamication in education
Gamication refers to a method of designing systems, services, or-
ganizations and activities in order to create similar experiences and
motivations to those experienced when playing games, with the added
educational goal of aecting user behavior (Huotari and
Hamari, 2017). Games are known to motivate and engage
players (Dichev and Dicheva, 2017) because of the enjoyment and the
excitement that this activity oers (Koivisto and Hamari, 2019). In this
regard, gamication aspires to create this experience in dierent con-
texts. This is usually attempted by using game mechanics or other
game-like designs in the target environment (Deterding et al., 2011).
Over the last decade, gamication research has aected a variety of
domains that deal with education (Koivisto and Hamari, 2019). The
educational domain is continuously evolving, incorporating the latest
developments in information technology even in elementary
schools (Karpouzis et al., 2007). Nonetheless still demands students'
commitment and persistence in order for them to gain in-depth
knowledge. Consequently, gamication has been of great interest to
educators who have been exploring its potential in improving student
learning (Dichev and Dicheva, 2017; Dicheva et al., 2015; Hamari,
2013; Koivisto and Hamari, 2019; Majuri et al., 2018; Seaborn and Fels,
2015). This potential has led to a growing literature on the eectiveness
of gamication, mainly in universities but also in other academic
contexts (Caponetto et al., 2014; Koivisto and Hamari, 2019; Seaborn
and Fels, 2015; de Sousa Borges et al., 2014) and in a variety of
subjects (Dichev and Dicheva, 2017; Kasurinen and Knutas, 2018). To
name a few: information technology (Osatuyi et al., 2018), math/
science (Attali and Arieli-Attali, 2015; Christy and Fox, 2014) and
taxation (Buckley and Doyle, 2016a; 2017).
Gamication types and types of game design have commonly been
horizontally categorized into main three overarching categories of
achievement/challenge-, immersion-, and social-based (Hamari and
Tuunanen, 2014; Koivisto and Hamari, 2019; Snodgrass et al., 2013; Xi
and Hamari, 2019; Yee, 2006; Yee et al., 2012) beyond the common
vertical categorization of e.g. the MDA model that separated game
design into mechanics, dynamics and aesthetics (Hunicke et al., 2004).
The immersion-based game design attempts primarily to engulf the
player or user into a story, roleplay and audiovisual richness. The so-
cial-based game design is commonly focused on dierent forms of
competition and collaboration. Finally, the achievement/challenge-
based game design is focused on overcoming challenges, progressing
and earning rewards and feeling competent. Within the achievement/
challenge-based gamication, the most commonly embodied mechanics
have been points, challenges, leaderboards, levels and badges (Koivisto
and Hamari, 2019; Majuri et al., 2018; Pedreira et al., 2015). According
to the self-determination theory, the use of these elements, which are
considered as achievement related features and immediate performance
indicators, is associated with intrinsic motivation for students (Xi and
Hamari, 2019). In this regard, these elements form challenge-based
gamication underpinnings in order to motivate students to maximize
their knowledge acquisition.
Review studies about the eectiveness of gamication are generally
optimistic, mainly listing either positive or mixed results of applied
gamied strategies (Buckley and Doyle, 2017; Caponetto et al., 2014;
Dicheva et al., 2015; Koivisto and Hamari, 2019; Lambruschini and
Pizarro, 2015; Majuri et al., 2018; Nah et al., 2014; Osatuyi et al., 2018;
Reiners et al., 2012; Seaborn and Fels, 2015; de Sousa Borges et al.,
2014). Nevertheless, they mention the need for more controlled ex-
perimental research on the impact of gamication, independently of the
application domain or used gamied strategy (Buckley and Doyle,
2017; Caponetto et al., 2014; Dichev and Dicheva, 2017; Dicheva et al.,
2015; Hanus and Fox, 2015; Koivisto and Hamari, 2019; Lambruschini
and Pizarro, 2015; Landers et al., 2018; Majuri et al., 2018; Nah et al.,
N.-Z. Legaki, et al. International Journal of Human-Computer Studies 144 (2020) 102496
2
2014; Osatuyi et al., 2018; Reiners et al., 2012; Seaborn and Fels, 2015;
de Sousa Borges et al., 2014).
The eects of gamication are bound together with the target au-
dience and the context (Buckley and Doyle, 2017; Dichev and Dicheva,
2017; Hanus and Fox, 2015; Koivisto and Hamari, 2019; Seaborn and
Fels, 2015). Hence, the results of gamication vary regarding the sub-
ject and the eld of application (Hanus and Fox, 2015; Sánchez-Martín
et al., 2017). Therefore, researchers generally agree on the need for
stronger empirical results (Buckley and Doyle, 2017; Hanus and Fox,
2015; Koivisto and Hamari, 2019; Landers et al., 2018; Maican et al.,
2016; Morschheuser et al., 2017). This study contributes to this body of
research with empirical data drawn from a series of experiments on the
impact of gamication with control and treatment groups in the context
of a forecasting course.
2.2. Teaching forecasting in higher education
As described in Gareld and Ben-Zvi (2007), statistics and statistical
literacy are of paramount importance, especially in the rapidly chan-
ging business environments. As interest in the available technology and
statistics is growing, forecasting skills are becoming more
sophisticated (Kros and Rowe, 2016) and the process of teaching fore-
casting is becoming more dicult and demanding. Statistics courses
focus on data analysis (Cobb, 1992), as competitive business environ-
ments require graduate students to interpret data and be able to use
statistical and judgmental forecasting methods and
applications (Giullian et al., 2000; Kros and Rowe, 2016). The im-
portance of forecasting skills is not a new discovery (Albritton and
McMullen, 2006; Craighead, 2004; Giullian et al., 2000; Kros and
Rowe, 2016; Loomis and Cox Jr, 2003; Makridakis et al., 2008; Snider
and Eliasson, 2013). However, recently these skills have have become
even more important since business decision-making must be supported
by data-based evidence and projections (Giullian et al., 2000). Another
aspect of forecasting that highlights its importance is its multi-
disciplinary nature, since the forecasting techniques are an essential
component in a number of elds such as business
statistics (Tabatabai and Gamble, 1997), supply chain
management (Gavirneni, 2008) and management
science (Makridakis et al., 2008).
However, the eagerness of the business sector to equip students with
a strong background in forecasting techniques is only partially reected
in the education that universities and business schools provide. Thirty-
ve years ago, 58% of the surveyed universities oered an independent
forecasting course (Hanke, 1984). The percentage is reduced to 34.48%
of the surveyed business schools based on a more recent study
by Kros and Rowe (2016) and it is almost the same (50%) regarding the
top 50 US Business Programs, which requires a forecasting time-series
course. Moreover, there is a variety of e-learning-in-statistics in-
itiatives, but even these modules do not focus on time-series and
forecasting methodologies (Gel et al., 2014).
Despite the growing popularity of and need for forecasting skills,
business schools slowly address this demand, and they generally dis-
regard the need for increasing student motivation (Debnath et al.,
2007). Business forecasting or statistical forecasting methods are
usually considered complicated (Albritton and McMullen, 2006;
Craighead, 2004; Gardner, 2008; Snider and Eliasson, 2013; Torres
et al., 2018), making it dicult for students to remain
motivated (Craighead, 2004). Taking this into account, Chu (2007);
Donihue (1995);Loomis and Cox (2000);Loomis and Cox Jr (2003);
McEwen (1994) suggest alternative teaching guidelines such as the use
of a software or new technology in combination with real data and
forecasting problems. Active learning has also been proposed (Love and
Hildebrand, 2002) in order to address the need to update the fore-
casting educational process. So the digitalization, which we experience
has boosted the statistical skillsimportance. However, universities and
business schools have not responded immediately to this challenge and
they have been criticized for not placing enough focus on the specic
skills that will improve the studentsfuture job
performance (McEwen, 1994) and career success (Pfeer and
Fong, 2002).
2.3. Gamication and teaching statistical forecasting
This study puts emphasis on gamication only in terms of simple
educational activities or systems, usually including game
mechanics (Bunchball, 2010; Deterding et al., 2011). In this direction,
we reviewed journal articles that discuss simple active learning events,
gamied exercises or games in the context of a forecasting course. Our
results indicate that the use of score (Craighead, 2004),
spreadsheets (Gardner, 2008) and competition (Snider and
Eliasson, 2013) during lectures has positive eects regarding students
attitude, but strong empirical data is not presented. The use of a cus-
tomized software (Spithourakis et al., 2015) and studentsparticipation
in prediction of a basketball score appeared benecial in the context of
an undergraduate forecasting techniques course.
Other active learning exercises have used competition based on
studentsforecasting accuracy in order to increase studentsparticipa-
tion and improve learning outcomes in a management
course (Buckley et al., 2011) and in a taxation course (Buckley and
Doyle, 2016a). Another simple game, named: FREDCAST has been de-
signed and recently used in order to teach forecasting in a macro-
economic course (Mendez-Carbajo, 2018). All of these examples, show
that forecasting due to its nature can be considered as a kind of an
artistic eld (Gavirneni, 2008), where gamication could be eciently
integrated in order to make it attractive to its audience.
Our preliminary review mainly positions gamication as a bene-
cial tool in education of forecasting and related elds such as man-
agement (Buckley et al., 2011; Makridakis et al., 2008), decision-
making (Makridakis et al., 2008), taxation (Buckley and Doyle, 2016b),
supply chain management (Gavirneni, 2008) but highlights the need for
more empirical results. Additionally, an overview of teaching fore-
casting shows the importance of forecasting courses in economics
syllabus (Loomis and Cox Jr, 2003) or business school
curriculum (Buckley et al., 2011; Gavirneni, 2008). The need for fore-
casting skills is increasing as well, due to technological changes, as
management seeks data-based approaches in dealing with decision-
making on market opportunities, environmental factors and technolo-
gical resources. However, forecasting courses are not adequately sup-
ported by students' participation or university and business school
programming (Albritton and McMullen, 2006; Snider and Eliasson,
2013). A possible approach to address this issue could be gamication,
which under proper design guidelines has produced promising results
regarding student motivation and learning outcomes in management
courses (Buckley and Doyle, 2016a; Craighead, 2004; Gardner, 2008;
Snider and Eliasson, 2013) and in a forecasting
module (Gavirneni, 2008). Nevertheless, thus far, there have not been a
lot of studies on the eects of gamication on learning outcomes, in the
specic area of statistical forecasting. Therefore, this study experi-
mentally examines the potential of the challenge-based gamication, by
designing from scratch and using a gamied application in order to
improve student learning in a forecasting course.
3. Material and methods
3.1. Participants
A series of experiments were conducted at the ECE, NTUA and at the
Business Administration. More precisely, we performed the experiments
in dierent classes and academic majors, as follows:
49 undergraduate students (class of 2015) at the ECE, NTUA.
37 MBA students (class of 2015) at the ECE, NTUA.
N.-Z. Legaki, et al. International Journal of Human-Computer Studies 144 (2020) 102496
3
60 undergraduate students (class of 2016) at the ECE, NTUA.
52 undergraduate students (class of 2018) at the ECE, NTUA.
21 MBA students (class of 2018) at the ECE, NTUA.
86 undergraduate students (class of 2018) at the Business
Administration.
60 undergraduate students (class of 2019) at the ECE, NTUA.
The total sample is composed of 365 students; 270 students are
males and 95 are females. The experiments were performed in the
context of a forecasting course, with fourth-year undergraduate stu-
dents and second-year MBA students, respectively, at the ECE, NTUA.
At the Business Administration, the experiment was conducted in the
context of an information technology course, with rst-year students.
However, the Business Administrations curriculum contains an opera-
tional research course, which includes forecasting techniques. The ex-
perimental design of our experiments was followed strictly, in-
dependently of the academic level of the students, as described in the
next section.
3.2. Experimental design
We conducted a 2 (read: yes vs. no) x 2 (gamication: yes vs no)
factorial experiment. The dependent variable was student performance
in the learning task. Participants were randomly assigned to one of the
conditions of the experiment: i) Group Control: no treatment, ii) Group
Read: treatment of reading a research paper, named thenceforth as task
Read (see 3.3.3), iii) Group Play: treatment of using challenge-based
gamication, named thenceforth as task Play (see 3.3.4) and iv) Group
Read&Play: both tasks: Readand Play. Time was controlled and
was equal to 15 minutes for each task. Table 1, depicts the design of the
evaluation of our experiment and all the treatments are explained in the
following section.
3.3. Materials
3.3.1. Lecture
The learning objectives of this lecture were for students to under-
stand and be able to apply the Method Selection Protocolsfor reg-
ular/fast-moving and intermittent demand time-series based on specic
academic work of Petropoulos et al. (2014). During the lecture, the aim
of the Petropoulos et al. (2014) research was mentioned, along with the
data, and the research methodology used. Special attention was paid on
the results, the practical implications and the conclusions of this study
about the Method Selection Protocols. More precisely, we further
explained the relation between the time-series features and one stra-
tegic decision and the forecasting accuracy of the proposed methods for
both regular/fast-moving and intermittent demand time-series. The
visual material of the lecture was composed of 17 slides and lasted 15
minutes. The lecture content was focused on specialized knowledge of
the research of Petropoulos et al. (2014) that both undergraduate and
MBA students in any of the stages of their studies would not have
otherwise been taught.
3.3.2. Final evaluation form
An evaluation form at the end was used to measure students'
learning performance via 30 close-ended questions (i.e. questions
where the participants would select the right answer among possible
answers) of equivalent grade about the ndings
of Petropoulos et al. (2014). This was the last task for all participants,
independently of the group to which they were assigned. The answers
to all the questions were covered in the lecture material. All the ques-
tions were about topics that have been discussed in the lecture de-
scribed at subsection 3.3.1, and therefore, in principle it would be
possible to attain the highest score by only participating in the lecture.
3.3.3. Task read
The material of the reading task was: Petropoulos, F., Makridakis,
S., Assimakopoulos, V., Nikolopoulos, K., 2014. horses for coursesin
demand forecasting. European Journal of Operational Research 237,
152 -163.. The paper is 12 pages and is the foundational material of
the lecture content. The students read the article using a computer in
the computer lab.
3.3.4. Task play: Challenge-based gamication
Since there is a lack of free, computationally non-complex gamied
applications, specically created to teach statistical forecasting, we
developed a gamication approach called Horses for Courses.Itisa
simple gamied application, which is composed of main design patterns
related to challenge-based gamication, as described in Section 3.3.4.2.
Horses for Courses aims to motivate students' participation, to improve
their learning outcomes regarding the choice of simple but accurate
statistical forecasting methods, and consequently enhance their fore-
casting skills. It is structured based on the above-mentioned founda-
tional material: Petropoulos, F., Makridakis, S., Assimakopoulos, V.,
Nikolopoulos, K., 2014. horses for coursesin demand forecasting.
European Journal of Operational Research 237, 152 -163..
3.3.4.1 Horses for Courses Architecture
In order to implement Horses for Courses, we considered the methods
and design principles of both gamication (Morschheuser et al., 2018;
Zichermann and Cunningham, 2011) and software
development (Barnett et al., 2005; Gallaugher and Ramanathan, 1996;
Lewandowski, 1998). As far as the architecture of the application is
concerned, a focus on exibility, accessibility, high-level programming
and the ability to be integrated in dierent platforms led us to build a
web application on the Microsoft.NET framework (Barnett et al., 2005)
and to use an MS-SQL database. Horses for Courses is a web-based ga-
mied application, structured as a three-tier system with simple ap-
plication logic layer, which is fully accessible to registered users via a
browser. Users register with an email and a password in order to save
their progress in a database scheme, which serves as a data tier. A Data
Moduleis used to retrieve data from the database and to build it into
functional objects. A class named Actions, enables the interaction
between users and the Data Modulein order to save updated data and
the users progress back into the database, composing the logic tier of
the application. The presentation tier consists of a graphical user in-
terface, including time-series, data visualization and system functions.
A compact graphical representation of the Horses for Courses application
is found in Fig. 1.
3.3.4.2 Horses for Courses Design
Guidelines for the design of the challenge-based gamication and
consequently of Horses for Courses application were divided into two
main directions: (1) the eective use of motivational aordances in
Table 1
Design of the evaluation of the experiments.
Task Description Group Control Group Read Group Play Group Read&Play
Attend Lecture (see 3.3.1)
Read the Paper (see 3.3.3)
Play (see 3.3.4)
Evaluation Form (see 3.3.2)
N.-Z. Legaki, et al. International Journal of Human-Computer Studies 144 (2020) 102496
4
learning environments (Deterding et al., 2011; Dicheva et al., 2015;
DomíNguez et al., 2013; González and Area, 2013; Hanus and Fox,
2015; Maican et al., 2016; Nah et al., 2014; Pedreira et al., 2015;
da Rocha Seixas et al., 2016; Yildirim, 2017) and (2) the design and
development of gamied applications (Kapp, 2013; Morschheuser et al.,
2018; Zichermann and Cunningham, 2011). The most frequently used
and assessed motivational aordances in education and in general, so
far are: points, levels, badges/achievements and
leaderboards (Alhammad and Moreno, 2018; Koivisto and Hamari,
2019; Majuri et al., 2018; Pedreira et al., 2015). Thus, we incorporated
points, a level setting, challenges and a leaderboard into our application
in order to maximize the external validity of the experiment - i.e. to
mimic a possible real world implementation of this gamication style.
More precisely, Table 2 describes the motivational aordances in Horses
for Courses, along with their denitions from the literature and the
purpose they serve.
Apart from these motivational aordances, our design decisions
regarding Horses for Courses were also determined by a desire to create
a user-friendly and agile interface and work ow, with clear player
guidance and instructions (Kapp, 2013). Fig. 2 describes a full round of
the game. Initially, students had to register or sign in. Later, for each
level they had to select the most suitable forecasting method based on
the provided data and information, as it is depicted in Fig. 3. Then
participants win points according to their choices. Instructions are ea-
sily accessible as well as theMethod Selection Protocols, through
colorful buttons as Fig. 3 illustrates, as well. New challenges arise at
each level, for example to identify time-series components for the real
data time-series as it is depicted in Fig. 4, encouraging the students to
assess their knowledge and win more points. The aim is to achieve a
high ranking on the nal leaderboard, based on the collected points.
3.4. Procedure
The experiments took place as a replacement of a normal lecture of
the respective course. Therefore, participants (students) arrived at the
lecture as normal at the designated computer lab at the standard time.
After arriving at the class, the participants were informed about the
experiment and their informed consent was obtained. At ECE, NTUA,
participation was voluntary, however, the incentive for participation
was a bonus of 0.5/10 in the coursesnal grade, instead of an
equivalent exercise in the nal examination. In such manner, every
student, participating or not in the experiment, could receive the
highest grade in the nal examination. The participation of under-
graduate students at Business Administration was mandatory as part of
the course and there were no additional incentive for them to take part
in the experiment.
Participants were instructed that they should pick a computer sta-
tion at the classroom (a computer lab), attend a lecture and then
complete an evaluation form, which was based on the content of the
lecture. They were informed that their performance in the evaluation
form would not aect their course grade, however, they should try to
correctly answer the questions based on their understanding of the
topic described in the lecture. The experimenter mentioned that the
participants would be randomly divided into four groups, without
further information, but the importance of the evaluation form along
with the time constraints for all groups was highlighted.
First part of the experiment procedure was a 15-minute lecture
about the ndings of Petropoulos et al. (2014) research (see 3.3.1).
After the lecture, the participants were randomly assigned to dierent
conditions of the experiment: Group Control, Group Read, Group Play
and Group Read&Play. All students, independently of their group re-
ceived the same incentive in order to eliminate the recruitment bias.
Then, the experimenter informed the participants about their next task
and the available time. Instructions were given for each group respec-
tively. Each group had 15 minutes to complete the respective task. As
described above, Group Control did not have an extra task, Group Read
had 15 minutes to read the paper (task Read), Group Play had 15
minutes to fulll a full round in the gamied application (task Play).
Group Read&Play had 30 (15 + 15) minutes to fulll the task Read and
then the task Play. Finally, all groups had to complete the evaluation
Fig. 1. Horses for Courses architecture.
Table 2
Integrated motivational aordances in Horses for Courses application.
Aordance Denition Purpose in Horses for Courses
Points Numeric measure of playersperformances. Reward for the correct application of method selection protocol.
Levels Diculty moderated based on playersexpertise. Indicator of progression and diculty.
Challenges Predened quests and increasingly more dicult objectives. Positive impetus to keep players engaged to maximize their points.
Leaderboard Direct comparison of playersperformance. Increase of competition among students.
Source: (Buckley and Doyle, 2017; Bunchball, 2010; Kapp, 2013; Maican et al., 2016; Nah et al., 2014; Seaborn and Fels, 2015; Zichermann and Cunningham, 2011).
N.-Z. Legaki, et al. International Journal of Human-Computer Studies 144 (2020) 102496
5
form within 15 minutes, which measured their performance (see Ma-
terials 3.3.2). Participants completed all of the tasksstages in-
dependently and were not allowed to communicate with other parti-
cipants. The procedure was the same at both participating campuses,
including the materials and the experimenter.
4. Results
The objective of this study is to identify the impact of the challenge-
based gamication on studentsperformances in the evaluation form
and consequently on their comprehension. Thus, we examine the re-
lative performance of a control group in comparison with that of the
treatment groups. Studentsperformances on the evaluation form were
calculated as the sum of the right answers on the questionnaire, nor-
malized to a maximum of 100. Summary statistics of results are pre-
sented in Table 3 for each treatment along with the studentsgender,
their academic major: ECE, NTUA or Business Administration and their
educational level: Undergraduate or MBA. The distribution of the per-
formances is illustrated in percentiles with box-plot diagrams in Fig. 5,
which are further separated into dierent treatments, including gender,
major and educational level. Finally, for a subset of the sample
(N=146), three extra variables have been examined along with the
treatment groups, namely studentsexpertise in English, and use of
personal computers and games. Studentsresponses were ranging from
1=beginner to 5=procient. Horses for Courses is a web-based gami-
ed application which demands the use of a personal computer and the
language of its interface is English. Therefore, we examined the re-
lationship between these variables and studentsperformances by
testing if these variables would be statistically signicant in students
performances in conjunction with the treatment received.
The analysis of the results was conducted in three steps. First, we
investigated the mean values of the studentsperformances and their
statistically signicant dierences with respect to specic treatments.
Table 3 presents the number of students per treatment, the mean value
of the studentsperformances, and the standard deviation in each
treatment group, regarding their gender, academic major and educa-
tional level. Overall, the groups that experienced the challenge-based
gamication achieved greater mean values of performances than the
other groups. More precisely, Group Read&Play, which read the re-
spective paper and used the gamied application, reached the highest
mean performance of 58.05 out of 100, and had the second lower level
of standard deviation in results (SD=17.00). Group Play, which only
experienced the gamied application, had the second highest mean
performance of 52.55 out of 100 and the highest level of standard de-
viation in results (SD=19.74). Group Read had a lower mean perfor-
mance of 46.13 out of 100 (SD=18.68). Finally, Group Control had the
Fig. 2. Horses for Coursesowchart of a full game round.
N.-Z. Legaki, et al. International Journal of Human-Computer Studies 144 (2020) 102496
6
lowest mean performance, 36.13 out of 100, but also the lowest stan-
dard deviation (SD=11.57).
Based on the Shapiro-Wilk test on the ANOVA residuals, the as-
sumption of normality was violated. In order to study the signicant
dierences in the average values of all the groupsperformances, we ran
the non-parametric Kruskal-Wallis rank sum test (Kruskal and
Wallis, 1952). The null hypothesis of equal dierences is rejected (chi-
squared=70.842, df=3, p<0.001) and we can therefore establish
signicant dierences between the groups. Furthermore, we conducted
pairwise multiple comparisons without making assumptions about
normality, using the Dunn procedure (Dinno, 2015; Dunn, 1961; Zar
et al., 1999), with a condence interval equal to 95%. Kruskal-Wallis
with Dunns post-test was chosen to test the signicant dierences be-
cause data was not normally distributed in all cases. Table 4 presents
the outcomes concluding that all treatment groups resulted in sig-
nicantly higher performances compared to Group Control (p.
adj<0.001, Kruskal-Wallis with Dunns post test). Additionally, Table 4
displays the respective eect size of these treatments compared to
Group Control, based on non-parametric Clis Delta estimator (Cli,
2014; Macbeth et al., 2011; Wilcox, 2006). The only pairwise com-
parison without statistically signicant dierences in studentsperfor-
mances is Group Play versus Group Read&Play. Finally, Group
Read&Play outperformed all the other groups. This group noted the
highest improvement regarding the mean values of performances of
Group Control equal to 60.67%. Group Play and Group Read follow
with improvement equal to 45.45% and 27.68% respectively.
The performances of all groups were compared directly, focusing on
the assessment of questions in the evaluation form, despite that treat-
ments did not have the same duration. In order to deal with this
limitation of our study, we used independent binary variables for the
tasks Read and Play respectively. The value of the variable Read is
equal to 1 if the respective group completed the task, and 0 otherwise.
The same applies for the variable Play. Then, the Scheirer-Ray-Hare test
was performed (Scheirer et al., 1976), using studentsperformances and
these variables. Results show that each of the tasks: Read the research
(H=16.014, p<0.001) or Play with Horses for Courses application
(H=52.81, p<0.001) had a signicant impact on studentsperfor-
mances, but their interaction was not signicant (H=2.019, p=0.156).
Along with the impact of dierent treatments on studentsperfor-
mances, the independent variables gender, academic major and edu-
cational level were examined using the Scheirer-Ray-Hare test. While
the studentsgender and their academic major appeared to have a great
impact on their performances, based on the results presented in Table 5,
the interactions between them and the respective treatments they un-
derwent did not have signicant impact. The studentseducational level
was not an important variable, nor was its interaction with the treat-
ments.
Regarding the impact of additional variables on a subset of our
sample, only the impact of the studentsexpertise in English resulted in
statistically signicant dierences in studentsperformances. The last
rows of Table 5 show the results of the Scheirer-Ray-Hare test for the
respective samples. Results about the mean values and the standard
deviations of these extra variables are presented in Table 6. However,
these variables will not be further analyzed because students reported
their answers only in the recent experiments.
Table 7 demonstrates the impact of dierent treatments on students
performances, regarding the statistically signicant variables of gender
and academic major. We calculated the improvement of more specied
Fig. 3. View of 1
st
level challenges of Horses for Courses application.
N.-Z. Legaki, et al. International Journal of Human-Computer Studies 144 (2020) 102496
7
groups regarding these variables compared to the mean value of the
studentsperformances of Group Control (equal to 36.13) as a bench-
mark value. Students at the ECE, NTUA have noted the highest im-
provement regarding the mean values of their performances in the
evaluation form. Furthermore, female students, independently of their
major, who had used the gamied application beneted more from
gamication; their mean performances are higher from those of the
respective groups composed of male participants. These ndings do not
apply in non-gamied groups.
To conclude, we divided all the data of studentsperformances into
two larger groups, instead of four: the non-gamied group, composed of
165 students (M=41.04, SD=16.22), who did not use the Horses for
Courses application (Group Control and Group Read) and 200 students
(M=55.30, SD=18.58) who used it (Group Play and Group
Read&Play), the gamied group. We adopted this approach in order to
examine the overall impact of the challenge-based gamication on
studentslearning outcomes. Fig. 6 illustrates the studentsperfor-
mances for each group in percentiles with box-plot diagrams. Normality
is not conrmed, thus Wilcoxon-Mann-Whitney rank sum test was
performed, with a condence interval equal to 95%. The null hypoth-
esis of equal dierences in means is rejected (W=23821, p<0.001),
while the use of Horses for Courses presents a moderate level of impact
Fig. 4. View of 4
th
level of Horses for Courses application.
Table 3
The challenge-based gamication results per treatment and variable.
Variable Group Control Group Read Group Play Group Read&Play
n M SD n M SD n M SD n M SD
Gender
Female 15 27.08 10.07 19 35.14 19.19 28 49.08 20.46 33 57.81 18.75
Male 69 38.10 10.98 62 49.49 17.32 72 53.90 19.43 67 58.16 16.21
Academic major
ECE, NTUA 61 39.56 10.58 65 50.56 16.55 74 59.78 15.67 79 61.53 15.28
Business Administration 23 27.04 8.97 16 28.13 16.18 26 31.97 15.19 21 44.94 17.07
Educational Level
UG 71 36.28 11.87 68 47.59 19.10 85 52.45 21.03 83 56.00 16.90
MBA 13 35.34 10.16 13 38.46 14.62 15 53.13 10.02 17 68.01 14.00
Total 84 36.13 11.57 81 46.13 18.68 100 52.55 19.74 100 58.05 17.00
N.-Z. Legaki, et al. International Journal of Human-Computer Studies 144 (2020) 102496
8
based on non-parametric Clis Delta estimator (delta estimate=0.44
(medium)) and an improvement regarding mean values of perfor-
mances, equal to 34.75%.
Studentsperformances in the nal evaluation form (questionnaire)
should not be confused with their game performances. Weak positive
correlation was found between studentsperformances at the nal
evaluation form and their game performances for students who ex-
perienced the challenge-based gamication (r(146)=0.339, p <0.001).
However, the value of the Pearsons Correlation Coecient is calcu-
lated only for a subset of students (N=148), who used the gamied
application, since there was no specic instruction to students to use
the same personal details in the gamied application and in the eva-
luation form.
Fig. 5. Studentsperformances per treatment and variable.
Table 4
Pairwise multiple comparisons among the groups based on Kruskal-Wallis with
Dunns post test and CliDelta eect size.
Groups Z P.adj Delta
estimate
Improvement (%)
Control vs. Read -3.70 0.001 0.35
(medium)
27.68%
Control vs. Play -6.16 <0.001 0.51 (large) 45.45%
Control vs. Read&Play -8.04 <0.001 0.69 (large) 60.67%
Play vs. Read 2.25 0.049 - -
Play vs. Read&Play -1.96 0.05 - -
Read&Play vs. Read 4.10 <0.001 - -
N.-Z. Legaki, et al. International Journal of Human-Computer Studies 144 (2020) 102496
9
5. Discussion of results
Overall, the results suggest that the challenge-based gamication
improves learning outcomes in a forecasting course. This type of ga-
mication presents the greatest improvement in studentsperformances
when it is combined with traditional teaching methods. However, our
results show that even this gamied application alone, integrated in a
lecture, may have a positive impact on learning outcomes.
In general, groups who experienced the challenge-based gamica-
tion have greater performances than the groups that only participated
in traditional teaching methods such as only attending the lecture
(Group Control) or reading the paper (Group Read). More precisely, the
group whose participants read the respective paper and used the ga-
mied application had the highest performance regardless of their
gender, academic major and level of studies. However, the mean value
of the performances of this group is not statistically signicant dierent
from the group who only used the gamied application. Despite the fact
that Group Read&Play had extra 15 minutes to read the respective re-
search and then use the gamied application, the interaction of these
two tasks did not seem to have a great impact on studentsperfor-
mances; while each of the tasks, Read or Play did. According
to Fisher et al. (1981), the amount of time that students are focused or
engaged in an activity is generally positively associated with their
learning outcomes. Given that, we might speculate that the students
who had to complete two tasks may not have been fully engaged
throughout the duration of the tasks. Nevertheless, the aim of our study
is to investigate the impact of gamication on studentsperformances.
So, based on our analysis the use of this gamied application presents
an improvement regarding the mean values of performances, equal to
34.75%. In addition, the challenge-based gamication may improve
studentsperformance by up to 89.45% compared to only being present
at a lecture. Under certain conditions, the use of gamication within
less time, may have almost the same impact as reading and using the
gamied application, as far as learning outcomes in forecasting are
concerned.
Moreover, we can state that a gamied application combined in a
lecture may improve learning outcomes at both schools. However, its
impact is even more important in the case of engineering students,
where both female and male participants had signicantly better per-
formances. This nding is in agreement with the fact that gamication
has already been incorporated into software engineering and math/
science education to a greater degree (Alhammad and Moreno, 2018;
Dicheva et al., 2015; Pedreira et al., 2015). Dicheva et al. (2015) argue
that one possible explanation could be the lack of fully customized
gamied applications in a variety of educational elds or the fact that
instructors in these schools are more qualied to develop such appli-
cations. Another explanation might be that gamication helps to in-
crease student interest in dicult concepts in
engineering (Markopoulos et al., 2015). Thus, engineering students
may benet more by experiencing these subjects as more manageable.
Although the research in this eld is at a preliminary stage (Alhammad
and Moreno, 2018; Pedreira et al., 2015), Pedreira et al. (2015) support
that gamication has great potential in software engineering education
mainly because of the nature of tasks, which demand high motivation
Table 5
Impact of the treatments, variables and their interactions.
Variables Gender Academic major Educational Level
Groups (N=365) df H Sig. df H Sign. df H Sign.
Treatment (df=3) 1 7.74 0.005 1 73.41 <0.001 1 0.20 0.657
(H=70.84, p<0.001)
Interaction 3 5.10 0.164 3 7.02 0.071 3 7.63 0.054
Variables English Prociency PC Expertise Game Expertise
Groups (n=146) df H Sig. df H Sign. df H Sign.
Treatment (df=3) 4 15.26 0.004 4 7.83 0.098 4 0.51 0.972
(H=24.77, p<0.001)
Interaction 9 2.58 0.995 9 5.41 0.797 12 11.31 0.502
Table 6
Studentsperformances per treatment and extra variables.
Group Performance English Prociency PC Expertise Game Expertise
Control M=32.3 M=3.36 M=3.43 M=3.29
(n=28) SD=11.4 SD=1.37 SD=1.14 SD=1.49
Read M=44.4 M=3.59 M=4 M=3.37
(n=27) SD=18.4 SD=1.39 SD=0.83 SD=1.04
Play M=43.2 M=3.77 M=3.96 M=3.26
(n=47) SD= 20.6 sd=1.37 SD=1.02 SD=1.15
Read&Play M=56.7 M=4.23 M=4.07 M= 3.27
(n=44) SD= 20.0 SD=0.96 SD=0.90 SD=1.34
Table 7
Improvement of studentsperformances per treatment, gender and academic major.
Group Academic major Gender n M SD Improvement (%) Delta Est.
ECE, NTUA Female 4 32.81 5.41 -9.18 -0.173 (small)
Control (9.49%) Male 57 40.03 10.72 10.80 0.195 (small)
(0%) Business Administration Female 11 25.00 10.73 -30.81 -0.524 (large)
(-25.17%) Male 12 28.91 6.94 -20.00 -0.388 (medium)
ECE, NTUA Female 12 44.44 16.74 23.00 0.283 (small)
Read (39.93%) Male 53 51.94 16.35 43.76 0.594 (large)
(27.66%) Business Administration Female 7 19.20 11.04 -46.87 -0.731 (large)
(-22.16%) Male 9 35.07 16.59 -2.94 -0.193 (small)
ECE, NTUA Female 15 59.95 16.07 65.91 0.775 (large)
Play (65.45%) Male 59 59.74 15.70 65.34 0.779 (large)
(45.44%) Business Administration Female 13 36.54 17.97 1.13 -0.068 (negligible)
(-11.51%) Male 13 27.40 10.61 -24.15 -0.424 (medium)
ECE, NTUA Female 16 68.45 13.70 89.45 0.935 (large)
Read&Play (70.30%) Male 63 59.77 15.26 65.43 0.764 (large)
(60.65%) Business Administration Female 17 47.79 17.53 32.28 0.400 (medium)
(24.38%) Male 4 32.81 7.86 -9.18 -0.179 (small)
N.-Z. Legaki, et al. International Journal of Human-Computer Studies 144 (2020) 102496
10
by students (Pedreira et al., 2015). Our results strengthen this state-
ment, indicating that challenge-based gamication was even more ef-
fective in engineering studentsperformances, without neglecting its
potential in business schools, as well.
Another nding, based on the results presented in Table 7, is that
female users of this gamied application, independently of their edu-
cational background, had higher performances and a higher level of
improvement compared to their male counterparts. However, this
nding does not apply in non-gamied groups (Group Control and
Group Read). More specically, the female participants of gamied
groups at the ECE, NTUA have achieved the highest performances and
the highest improvement. Dierences in motivations (Carr, 2005) and
activities that engage players (Codish and Ravid, 2017; Koivisto and
Hamari, 2014), within genders have already been mentioned. Thus,
since female participants are generally more motivated by challenge
than by competition (McDaniel et al., 2012), females students would
probably be even more motivated by challenge-based gamication,
which would lead to better performance. Another possible explanation
could be that female students receive higher levels of playfulness in a
gamied educational content (Codish and Ravid, 2015), which in-
creases their motivation and improves their learning outcomes. It is
important to note that while the dierence in the sample size of the
genders may be the cause of the results, this assumption requires fur-
ther exploration, and for the time being we may conclude that both
female and male students appear to benet from
gamication (ODonovan et al., 2013).
Last but not least, it is surprising that the variables of students
expertise in use of personal computers and games were not statistically
signicant for the subset of the participants who reported this data
(engineering students=60, business school students=66 and MBA
students=20). However, based on the results of Table 6, these groups
have similar mean values regarding these variables, sharing probably
common characteristics. The studentsexpertise in English is another
parameter which had a statistically signicant impact on their perfor-
mances, since the slides, the research and the interface of the applica-
tion are all in English. The dierence between the mean values of this
variable, achieved by the engineering students (M=4.45, SD=0.86)
and business school students (M=3, SD=1.29) could justify the lower
performances of the business school students in the evaluation form for
all groups.
5.1. Limitations
While challenge-based gamication has a positive impact on
learning outcomes for both engineering and business school students,
some limitations should be acknowledged. We gathered and compared
performances of all groups although the treatments did not have the
same duration as the tasks did. However, this study focuses on the as-
sessment of the evaluation form (questionnaire), which was the same
for all groups and experiments, in order to evaluate the challenge-based
gamication impact. Although conducting both pre- and post-test
would provide further methodological rigor, in the scope of the present
study, the extant knowledge of the participants could be assumed to be
homogeneous because of the specialized content of the lecture. The
content of the lecture, and consequently the topic of the gamied ap-
plication and the evaluation form are focusing on the specic topic of
Method Selection Protocols(Petropoulos et al., 2014). Thus, students
in any of the stages of their studies would have not otherwise learned
this topic. Therefore, the present studys design was economized by
conducting only the post-test of knowledge on the topic. Furthermore,
engineering students noted better performances than business school
students. This fact is probably due to the discrepancy between en-
gineering and business school students in their prociency in English
and in their years of studies.
Additionally, another possible limitation is that there is a dierence
between the two schools in terms of the incentives to participate in our
experiments. Finally, although non-parametric tests have been con-
ducted because of the fact that data was non-normal or heteroscedastic
or both, the dierences in the sample size of dierent groups should be
considered as a limitation (i.e. 307 undergraduate students versus
58 MBA students). Thus, further experiments could contribute to the
conclusions of this study.
6. Conclusion
Overall this study contributes to the core literature of how gami-
cation aects desired outcomes (i.e. skills, knowledge, motivations and
behavior). According to several state-of-the-art analyses of the
eld (Koivisto and Hamari, 2019; Majuri et al., 2018; Nacke and
Deterding, 2017; Rapp et al., 2019), there has been a relative gap of
randomized controlled experiments that could reliably show eects of
gamication. Therefore, this study contributes to the corpus via such an
Fig. 6. Studentsperformances of non-gamied and gamied groups.
N.-Z. Legaki, et al. International Journal of Human-Computer Studies 144 (2020) 102496
11
experiment by showing that challenge-based gamication (i.e. im-
plementation including points, levels, challenges and a leaderboard)
improves learning outcomes. Our ndings contribute to the literature of
serious games (Connolly et al., 2012) and game-based learning (Hamari
et al., 2016; Squire, 2003), as well. More specically, the study informs
the area of scientic and statistics education (Chu, 2007; Love and
Hildebrand, 2002), because the object of learning in the experiment
was in forecasting techniques. The contribution in this area gets more
importance by considering the need for updating and making more
engaging the traditional teaching methods (Love and Hildebrand, 2002;
Surendeleg et al., 2014), since the statistical skills are fundamentals to
get insight of the available data in order to increase awareness towards
social problems and understanding our world.
In this study, we conducted a factorial design experiment, using a
developed gamication approach named Horses for Courses, which
provides valuable empirical evidence on how challenge-based gami-
cation and reading dierently inuence learning performance. The
ndings of our empirical study, based on a quantitative analysis of our
results, are in line with the positive eects of gamication on
learning (Buckley and Doyle, 2016a; Hamari et al., 2016; Kuo and
Chuang, 2016; Maican et al., 2016; da Rocha Seixas et al., 2016; Simões
et al., 2013; Yildirim, 2017) as well as on software engineering edu-
cation (Alhammad and Moreno, 2018; Pedreira et al., 2015). The results
demonstrate that the challenge-based gamication improves students
performances by 34.75% regarding a statistical forecasting topic and
that the eect was larger for females or engineering students. The
greatest improvements take place when gamication is combined with
traditional methods such as reading, however even simply integrating a
gamied application into a lecture benets students.
This research sheds light upon the eect of challenge-based gami-
cation on statistics education by demonstrating improvement in
learning outcomes. Apart from the theoretical contribution, this study
also provides practical implications to gamication designers and
educators. Challenge-based gamication (i.e. points, levels, challenges
and leaderboard), can be eectively combined with traditional teaching
methods such as lectures and reading in order to improve the learning
outcomes in a variety of educational elds related to statistics and stem
education. Finally, gamication designers should take into account
studentsproles, since our results show that benets dier across
studentscharacteristics.
With our study, we position challenge-based gamication as a useful
educational tool in statistics education in dierent academic majors
under certain circumstances, but also we acknowledge its limitations.
Further investigation of the eects of individual game elements or
dierent gamied approaches in statistics or data-related courses with a
larger sample is necessary, in order to enhance the scope of the research
and further rene its ndings. An extension of our research could be to
investigate the impact of additional motivational aordances combined
or compared with challenge-based gamication, under proper and
cautious design. These additional aordances could be related to the
actual content of the course or the actual behaviors that the instructors
want to promote, e.g. social sharing or responding to forum questions.
CRediT authorship contribution statement
Nikoletta-Zampeta Legaki: Conceptualization, Methodology,
Software, Formal analysis, Investigation, Writing - original draft,
Writing - review & editing, Visualization, Validation, Funding acquisi-
tion. Nannan Xi: Conceptualization, Writing - original draft, Writing -
review & editing. Juho Hamari: Conceptualization, Methodology,
Writing - original draft, Writing - review & editing, Supervision,
Funding acquisition. Kostas Karpouzis: Methodology,
Conceptualization. Vassilios Assimakopoulos: Conceptualization,
Methodology, Supervision.
Declaration of Competing Interest
The authors declare that they have no known competing nancial
interests or personal relationships that could have appeared to inu-
ence the work reported in this paper.
Acknowledgments
Special thanks to the students of the forecasting courses at the
School of Electrical and Computer Engineering of the National
Technical University of Athens, Greece and to the students at the
Business Administration Department in the School of Business and
Economics of the University of Thessaly, Greece who participated in our
experiments and helped to investigate the eect of challenge-based
gamication on student learning.
An early version of this study was presented at the International
GamiFIN Conference 2018, at the HICSS-52 2019 52nd Hawaii
International Conference on System Sciences and at the International
GamiFIN Conference 2019. This work was supported by the European
Union's Horizon 2020 research and innovation program under the
Marie Sklodowska-Curie [grant agreement ID 840809]; Business of
Finland [Grant No. 5654/31/2018]; Business of Finland [Grant No.
4708/31/2019]; Liikesivistysrahasto [Grant No. 14-7798].
References
Albritton, M.D., McMullen, P.R., 2006. Classroom integration of statistics and manage-
ment science via forecasting. Decision Sci. J. Innovat. Educ. 4 (2), 331336.
Alcivar, I., Abad, A.G., 2016. Design and evaluation of a gamied system for erp training.
Comput. Human. Behav. 58, 109118.
Alhammad, M.M., Moreno, A.M., 2018. Gamication in software engineering education:
asystematic mapping. J. Syst. Softw. 141, 131150.
Attali, Y., Arieli-Attali, M., 2015. Gamication in assessment: do points aect test per-
formance? Comput. Educ. 83, 5763.
Barnett, B., Kirtland, M., Ganapathy, M., 2005. An architecture for distributed applica-
tions on the internet: overview of the microsoft?. net framework. J. STEM Educ. 4 (1).
Buckley, P., Doyle, E., 2016. Gamication and student motivation. Interact. Learn.
Environ. 24 (6), 11621175.
Buckley, P., Doyle, E., 2016. Using web-based collaborative forecasting to enhance in-
formation literacy and disciplinary knowledge. Interact. Learn. Environ. 24 (7),
15741589.
Buckley, P., Doyle, E., 2017. Individualising gamication: an investigation of the impact
of learning styles and personality traits on the ecacy of gamication using a pre-
diction market. Comput. Educ. 106, 4355.
Buckley, P., Garvey, J., McGrath, F., 2011. A case study on using prediction markets as a
rich environment for active learning. Comput. Educ. 56 (2), 418428.
Bunchball, I., 2010. Gamication 101: an introduction to the use of game dynamics to
inuence behavior. White paper 9.
Caponetto, I., Earp, J., Ott, M., 2014. Gamication and education: A literature review.
European Conference on Games Based Learning. 1. Academic Conferences
International Limited, pp. 50.
Carr, D., 2005. Contexts, gaming pleasures, and gendered preferences. Simulat. Gaming
36 (4), 464482.
Christy, K.R., Fox, J., 2014. Leaderboards in a virtual classroom: a test of stereotype
threat and social comparison explanations for womens math performance. Comput.
Educat. 78, 6677.
Chu, S., 2007. Some initiatives in a business forecasting course. J. Stat. Educ. 15 (2).
Cli, N., 2014. Ordinal Methods for Behavioral Data Analysis. Psychology Press.
Cobb, G., 1992. Teaching statistics. Heed. Call Change 22, 343.
Codish, D., Ravid, G., 2015. Detecting playfulness in educational gamication through
behavior patterns. IBM J. Res. Dev. 59 (6), 611.
Codish, D., Ravid, G., 2017. Gender Moderation in Gamication: Does One Size Fit All?.
Connolly, T.M., Boyle, E.A., MacArthur, E., Hainey, T., Boyle, J.M., 2012. A systematic
literature review of empirical evidence on computer games and serious games.
Comput. Educ. 59 (2), 661686.
Craighead, C.W., 2004. Right on target for time-series forecasting. Decis. Sci. J. Innovat.
Educ. 2 (2), 207212.
Debnath, S.C., Tandon, S., Pointer, L.V., 2007. Designing business school courses to
promote student motivation: an application of the job characteristics model. J.
Manag. Educ. 31 (6), 812831.
Deterding, S., Dixon, D., Khaled, R., Nacke, L., 2011. From game design elements to
gamefulness: Dening gamication. Proceedings of the 15th International
Academic MindTrek Conference: Envisioning Future Media Environments. ACM, New
York, NY, USA, pp. 915.
Dichev, C., Dicheva, D., 2017. Gamifying education: what is known, what is believed and
what remains uncertain: a critical review. Int. J. Educ. Technol. Higher Educ. 14
(1), 9.
N.-Z. Legaki, et al. International Journal of Human-Computer Studies 144 (2020) 102496
12
Dicheva, D., Dichev, C., Agre, G., Angelova, G., 2015. Gamication in education: a sys-
tematic mapping study. Educ. Technol. Soc. 18 (3), 7589.
Dinno, A., 2015. Nonparametric pairwise multiple comparisons in independent groups
using dunns test. Stata. J. 15 (1), 292300.
DomíNguez, A., Saenz-De-Navarrete, J., De-Marcos, L., FernáNdez-Sanz, L., PagéS, C.,
MartíNez-HerráIz, J.-J., 2013. Gamifying learning experiences: practical implications
and outcomes. Comput. Educ. 63, 380392.
Donihue, M.R., 1995. Teaching economic forecasting to undergraduates. J. Econ. Educ.
26 (2), 113121.
Dunn, O.J., 1961. Multiple comparisons among means. J. Am. Stat. Assoc. 56 (293),
5264.
Fisher, C.W., Berliner, D.C., Filby, N.N., Marliave, R., Cahen, L.S., Dishaw, M.M., 1981.
Teaching behaviors, academic learning time, and student achievement: an overview.
J. Classroom Interaction 17 (1), 215.
Gallaugher, J.M., Ramanathan, S.C., 1996. Choosing a client/server architecture: a
comparison of two-and three-tier systems. Inf. Syst. Manag. 13 (2), 713.
Gardner, L., 2008. Using a spreadsheet for active learning projects in operations man-
agement. INFORMS Trans. Educ. 8 (2), 7588.
Gareld, J., Ben-Zvi, D., 2007. How students learn statistics revisited: a current review of
research on teaching and learning statistics. Int. Stat. Rev. 75 (3), 372396.
Gavirneni, S., 2008. Teaching the subjective aspect of forecasting through the use of
basketball scores. Decision Sci. J. Innovat. Educ. 6 (1), 187195.
Gel, Y.R., OHara Hines, R.J., Chen, H., Noguchi, K., Schoner, V., 2014. Developing and
assessing e-learning techniques for teaching forecasting. J. Educ. Bus. 89 (5),
215221.
Giullian, M.A., Odom, M.D., Totaro, M.W., 2000. Developing essential skills for success in
the business world: a look at forecasting. J. Appl. Bus. Res. 16 (3), 5162.
González, C., Area, M., 2013. Breaking the rules: Gamication of learning and educa-
tional materials. Proceedings of the 2nd international workshop on interaction design
in educational environments. pp. 753.
Hamari, J., 2013. Transforming homo economicus into homo ludens: aeld experiment
on gamication in a utilitarian peer-to-peer trading service. Electron. Commer. Res.
Appl. 12 (4), 236245.
Hamari, J., Sherno, D.J., Rowe, E., Coller, B., Asbell-Clarke, J., Edwards, T., 2016.
Challenging games help students learn: an empirical study on engagement, ow and
immersion in game-based learning. Comput. Human. Behav. 54, 170179.
Hamari, J., Tuunanen, J., 2014. Player Types: A Meta-synthesis.
Hanke, J., 1984. The teachers/practitioners corner forecasting in business schools: a
survey. J. Forecast. 3 (2), 229234.
Hanke, J., 1989. Forecasting in business schools: a follow-up survey. Int. J. Forecast. 5
(2), 259262.
Hanus, M.D., Fox, J., 2015. Assessing the eects of gamication in the classroom: a
longitudinal study on intrinsic motivation, social comparison, satisfaction, eort, and
academic performance. Comput. Educ. 80, 152161.
Hunicke, R., LeBlanc, M., Zubek, R., 2004. Mda: a formal approach to game design and
game research. Proceedings of the AAAI Workshop on Challenges in Game AI. 4. pp.
1722.
Huotari, K., Hamari, J., 2017. A denition for gamication: anchoring gamication in the
service marketing literature. Electron. Markets 27 (1), 2131.
Johnson, D., Deterding, S., Kuhn, K.-A., Staneva, A., Stoyanov, S., Hides, L., 2016.
Gamication for health and wellbeing: a systematic review of the literature. Internet
Intervent. 6, 89106.
Kapp, K.M., 2013. The Gamication of Learning and Instruction Fieldbook: Ideas into
Practice. John Wiley & Sons.
Karpouzis, K., Caridakis, G., Fotinea, S.-E., Efthimiou, E., 2007. Educational resources and
implementation of a greek sign language synthesis architecture. Comput. Educ. 49
(1), 5474.
Kasurinen, J., Knutas, A., 2018. Publication trends in gamication: a systematic mapping
study. Comput. Sci. Rev. 27, 3344.
Koivisto, J., Hamari, J., 2014. Demographic dierences in perceived benets from ga-
mication. Comput. Human. Behav. 35, 179188.
Koivisto, J., Hamari, J., 2019. The rise of motivational information systems: areview of
gamication research. Int. J. Inf. Manage. 45, 191210.
Kros, J.F., Rowe, W.J., 2016. Business school forecasting for the real world. Advances in
Business and Management Forecasting. Emerald Group Publishing Limited, pp.
149161.
Kruskal, W.H., Wallis, W.A., 1952. Use of ranks in one-criterion variance analysis. J. Am.
Stat. Assoc. 47 (260), 583621.
Kuo, M.-S., Chuang, T.-Y., 2016. How gamication motivates visits and engagement for
online academic dissemination - an empirical study. Comput. Human. Behav. 55,
1627.
Lambruschini, B.B., Pizarro, W.G., 2015. Tech-gamication in university engineering
education: Captivating students, generating knowledge. 2015 10th International
Conference on Computer Science & Education (ICCSE). IEEE, pp. 295299.
Landers, R.N., Auer, E.M., Collmus, A.B., Armstrong, M.B., 2018. Gamication science, its
history and future: Denitions and a research agenda. Simulat. Gaming.
1046878118774385
Lewandowski, S.M., 1998. Frameworks for component-based client/server computing.
ACM Comput. Surv. (CSUR) 30 (1), 327.
Loomis, D.G., Cox, J.E., 2000. A course in economic forecasting: rationale and content. J.
Econ. Educ. 31 (4), 349357.
Loomis, D.G., Cox Jr, J.E., 2003. Principles for teaching economic forecasting. Int. Rev.
Econ. Educ. 2 (1), 6979.
Love, T.E., Hildebrand, D.K., 2002. Statistics education and the making statistics more
eective in schools of business conferences. Am. Stat. 56 (2), 107112.
Macbeth, G., Razumiejczyk, E., Ledesma, R.D., 2011. CliS delta calculator: a non-
parametric eect size program for two groups of observations. Universitas Psychol.
10 (2), 545555.
Maican, C., Lixandroiu, R., Constantin, C., 2016. Interactivia.ro - a study of a gamication
framework using zero-cost tools. Comput. Human. Behav. 61, 186197.
Majuri, J., Koivisto, J., Hamari, J., 2018. Gamication of education and learning: A re-
view of empirical literature. Proceedings of the 2nd International GamiFIN
Conference, GamiFIN 2018. CEUR-WS.
Makridakis, S., Wheelwright, S.C., Hyndman, R.J., 2008. Forecasting methods and ap-
plications. John wiley & sons.
Markopoulos, A.P., Fragkou, A., Kasidiaris, P.D., Davim, J.P., 2015. Gamication in en-
gineering education and professional training. Int. J. Mech. Eng. Educ. 43 (2),
118131.
McDaniel, R., Lindgren, R., Friskics, J., 2012. Using badges for shaping interactions in
online learning environments. 2012 IEEE International Professional Communication
Conference. IEEE, pp. 14.
McEwen, B.C., 1994. Teaching critical thinking skills in business education. J. Educ. Bus.
70 (2), 99103.
Mendez-Carbajo, D., 2018. Experiential learning in intermediate macroeconomics
through fredcast. Int. Rev. Econ. Educ.
Morschheuser, B., Hamari, J., Koivisto, J., Maedche, A., 2017. Gamied crowdsourcing:
conceptualization, literature review, and future agenda. Int. J. Hum. Comput. Stud.
106, 2643.
Morschheuser, B., Hassan, L., Werder, K., Hamari, J., 2018. How to design gamication? a
method for engineering gamied software. Inf. Softw. Technol. 95, 219237.
Nacke, L.E., Deterding, C.S., 2017. The maturing of gamication research. Comput. Hum.
Behav. 450454.
Nah, F.F.-H., Zeng, Q., Telaprolu, V.R., Ayyappa, A.P., Eschenbrenner, B., 2014.
Gamication of education: a review of literature. International Conference on hci in
Business. Springer, pp. 401409.
ODonovan, S., Gain, J., Marais, P., 2013. A case study in the gamication of a university-
level games development course. Proceedings of the South African Institute for
Computer Scientists and Information Technologists Conference. ACM, pp. 242251.
Osatuyi, B., Osatuyi, T., de la Rosa, R., 2018. Systematic review of gamication research
in is education: a multi-method approach. CAIS 42, 5.
Pedreira, O., García, F., Brisaboa, N., Piattini, M., 2015. Gamication in software en-
gineering - a systematic mapping. Inf. Softw. Technol. 57, 157168.
Petropoulos, F., Makridakis, S., Assimakopoulos, V., Nikolopoulos, K., 2014. Horses for
coursesin demand forecasting. Eur. J. Oper. Res. 237 (1), 152163.
Pfeer, J., Fong, C.T., 2002. The end of business schools? less success than meets the eye.
Acad. Manag. Learn. Educ. 1 (1), 7895.
Rapp, A., Hopfgartner, F., Hamari, J., Linehan, C., Cena, F., 2019. Strengthening gami-
cation studies: current trends and future opportunities of gamication research. Int.
J. Hum. Comput. Stud. 127, 16.
Reiners, T., Wood, L.C., Chang, V., Gütl, C., Herrington, J., Teräs, H., Gregory, S., 2012.
Operationalising Gamication in an Educational Authentic Environment.
da Rocha Seixas, L., Gomes, A.S., de Melo Filho, I.J., 2016. Eectiveness of gamication
in the engagement of students. Comput. Human. Behav. 58, 4863.
Sánchez-Martín, J., Dávila-Acedo, M.A., et al., 2017. Just a game? Gamifying a general
science class at university: collaborative and competitive work implications. Think.
Skill. Creat. 26, 5159.
Scheirer, C.J., Ray, W.S., Hare, N., 1976. The analysis of ranked data derived from
completely randomized factorial designs. Biometrics 32 (2), 429434.
Seaborn, K., Fels, D.I., 2015. Gamication in theory and action: a survey. Int. J. Hum.
Comput. Stud. 74, 1431.
Simões, J., Redondo, R.D., Vilas, A.F., 2013. A social gamication framework for a k-6
learning platform. Comput. Hum. Behav. 29 (2), 345353.
Snider, B.R., Eliasson, J.B., 2013. Beat the instructor: an introductory forecasting game.
Decis. Sci. J. Innovat. Educ. 11 (2), 147157.
Snodgrass, J.G., Dengah, H.F., Lacy, M.G., Fagan, J., 2013. A formal anthropological view
of motivation models of problematic mmo play: achievement, social, and immersion
factors in the context of culture. Transcult. Psychiatry 50 (2), 235262.
de Sousa Borges, S., Durelli, V.H., Reis, H.M., Isotani, S., 2014. A systematic mapping on
gamication applied to education. Proceedings of the 29th annual ACM symposium
on applied computing. ACM, pp. 216222.
Spithourakis, G.P., Petropoulos, F., Nikolopoulos, K., Assimakopoulos, V., 2015.
Amplifying the learning eects via a forecasting and foresight support system. Int. J.
Forecast. 31 (1), 2032.
Squire, K., 2003. Video games in education. Int. J. Intell. Games Simulat. 2 (1), 4962.
Surendeleg, G., Murwa, V., Yun, H.-K., Kim, Y.S., 2014. The role of gamication in
education-a literature review. Contempor. Eng. Sci. 7 (29), 16091616.
Tabatabai, M., Gamble, R., 1997. Business statistics education: content and software in
undergraduate business statistics courses. J. Educ. Bus. 73 (1), 4853.
Torres, C., Babo, L., Mendonça, J., 2018. Engaging students to learn forecasting methods.
10th annual International Technology, Conference on Education and New Learning
Technologies. INTED, pp. 1033710343.
Wilcox, R.R., 2006. Graphical methods for assessing eect size: some alternatives to co-
hens d. J. Exp. Educ. 74 (4), 351367.
Xi, N., Hamari, J., 2019. Does gamication satisfy needs? a study on the relationship
between gamication features and intrinsic need satisfaction. Int. J. Inf. Manage. 46,
210221.
Xi, N., Hamari, J., 2020. Does gamication aect brand engagement and equity? a study
in online brand communities. J. Bus. Res. 109, 449460.
Yee, N., 2006. Motivations for play in online games. CyberPsychol. Behav. 9 (6),
772775.
Yee, N., Ducheneaut, N., Nelson, L., 2012. Online gaming motivations scale: development
and validation. Proceedings of the SIGCHI conference on human factors in computing
N.-Z. Legaki, et al. International Journal of Human-Computer Studies 144 (2020) 102496
13
systems. ACM, pp. 28032806.
Yildirim, I., 2017. The eects of gamication-based teaching practices on student
achievement and studentsattitudes toward lessons. Internet Higher Educ. 33, 8692.
Zar, J.H., et al., 1999. Biostatistical analysis. Pearson Education India.
Zichermann, G., Cunningham, C., 2011. Gamication by design: Implementing game
mechanics in web and mobile apps. OReilly Media, Inc..
Nikoletta-Zampeta Legaki is a Marie - Curie Researcher at
Gamication Group, Tampere University. She pursuits her
PhD in School of Electrical and Computer Engineering of
National Technical University of Athens (NTUA), Greece.
She has been a researcher and teaching assistant in
Forecasting and Strategy Unit, School of Electrical and
Computer Engineering since 2012. During this period, she
has participated in various research projects about fore-
casting and data analytics and she has worked as a con-
sultant in Financial Services and Risk Management, in EY,
Greece. Her research interests lie on time series forecasting,
business forecasting information systems, gamication and
educational methods in teaching forecasting.
Dr. Nannan Xi is a Postdoctoral researcher in Gamication
Group. She got her PhD in marketing management from
Zhongnan University of Economic and Law (ZUEL), China
and holds M.Sc. in International Business from the
University of Lincoln, UK. Her dissertation was awarded as
the excellent Ph.D. dissertation in ZUEL. Xis current re-
search focuses on gamication in marketing, especially in
gamied interaction in brand management. In addition, her
research interests include customer management in gami-
cation and virtual reality/augmented reality/mixed rea-
lity in business and sharing economy.
Dr. Juho Hamari is a Professor of Gamication at the
Faculty of Information Technology and Communications,
Tampere University. He leads the Gamication Group. His
research covers several forms of information technologies
such as games, motivational information systems, new
media (social networking services, eSports), peer-to-peer
economies (sharing economy, crowdsourcing), and virtual
economies. Dr. Hamari has authored several seminal em-
pirical, theoretical and meta-analytical scholarly articles on
these topics from perspective of consumer behavior,
human-computer interaction, game studies and information
systems science. His research has been published in a
variety of prestigious venues.
Dr. Kostas Karpouzis is currently an Associate Researcher
at the Institute of Communication and Computer Systems
(ICCS) of the National Technical University of Athens
(NTUA) in Greece. His research interests lie in the areas of
human computer interaction, emotion understanding, af-
fective and natural interaction, serious games and games
based assessment and learning. He has participated in more
than twenty research projects at Greek and European level.
He is also a member of the Student Activities Chair for the
IEEE Greece Section and a member of the Editorial Board
for international journals.
Prof. Vassilios Assimakopoulos is a professor of
Forecasting Systems at the School of Electrical and
Computer Engineering of the National Technical University
of Athens (NTUA). He is the author of over than 60 original
publications and papers in international journals and con-
ferences. Moreover, he has conducted research on in-
novative tools for management support and decision sys-
tems design. He has participated and led numerous projects,
funded by National and European institutes. He specializes
in various elds of Strategic Management, Design and
Development of Information systems, Business Resource
Management, Statistical and Forecasting Techniques using
time series.
N.-Z. Legaki, et al. International Journal of Human-Computer Studies 144 (2020) 102496
14
... Resulta necesario que el recurso educativo posibilite una exploración del contenido, para ello debe plantear retos intelectuales, guiar al estudiante en la búsqueda de soluciones (Legaki et al., 2020), y generar actividades que propicien que la capacidad de aprender a aprender (Sun-Lin y Chiou, 2019). Se debe tener en cuenta la relevancia del material de estudio para desarrollar una instrucción que genere engagement, por lo que es aconsejable emplear un diseño didáctico que facilite el estudio y el aprendizaje independiente y permanente (Sombra et al., 2019), además de potenciar la contribución del contenido a la resolución de problemas de la profesión (Mavridis et al., 2017). ...
... Resultado que es coincidente con las investigaciones sobre este tema y nos hace evidenciar empíricamente la importancia de la retroalimentación para el engagement (Kyaruzi et al., 2019), poniendo en valor las posibilidades que ofrecen las TIC y cómo contribuyen al aprendizaje de los estudiantes (de-Pablos, 2018). Vale destacar que se logró una mayor exploración en la búsqueda de nuevos conocimientos por parte de los estudiantes generando en ellos la capacidad de aprender a aprender (Legaki et al., 2020;Sun-Lin y Chiou, 2019). Las actividades en el aula se caracterizaron por una mejor gestión del profesor, propiciando una mediación tecnológica certera que elevó los niveles de engagement hacia el aprendizaje (de-Pablos, 2018). ...
Article
Full-text available
Uno de los retos del profesorado es el diseño de materiales educativos para generar engagement en la enseñanza digital, aspecto relevante en materias universitarias en las que el rendimiento está influenciado por su complejidad y factores motivacionales. Este trabajo evalúa los efectos del diseño de un e-textbook en el engagement hacia el aprendizaje digital en una universidad cubana. La metodología responde a un diseño cuasiexperimental; una primera fase incluye el diseño del e-textbook y su validación por expertos; en la segunda fase se realizó el cuasiexperimento con la participación de 178 estudiantes (88 en el grupo experimental y 90 en el grupo de control). Los estudiantes que utilizaron el e-textbook consiguieron una mejora significativa del engagement y el rendimiento académico, estos resultados refuerzan la necesidad de que los profesores diseñen recursos interactivos para la enseñanza digital.
... Visually presenting learners' level of goal achievement so that they can monitor their progress Sanchez et al. (2020) Rewards Awarding appropriate rewards when learners perform sub-tasks successfully or achieve goals Domínguez et al. (2013) Leaderboards/Ranking Displaying the ranking of learners or learners with high scores Legaki et al. (2020) Time limit Constraining the length of time learners are allowed to complete missions or tasks Putz et al. (2020) Feedback Providing messages or signals based on learners' performance in the process of achieving goals Ding (2019) principle) and using visual and auditory cues to draw attention to important information (ie, the signalling principle) reduce students' unnecessary cognitive load and promote cognitive processing. These principles can be linked to interventions conducted in a gamified VBL. ...
... • Allow learners to check their relative performance in their own group. Providing leaderboards or rankings can promote learners' motivation by providing opportunities to recognize their performance compared with other students (Nah et al., 2014 evaluate their learning status objectively by comparing their performance with others themselves, which can provoke a sense of competition and thus promote learners' active participation in their learning (Legaki et al., 2020). The intervals of providing leaderboards should be considered (eg, Toda et al., 2019) to update the ranks so that the information can help learners track and monitor their learning status. ...
Article
Full-text available
Despite the rapid growth of video‐based learning in higher education, students face challenges in sustaining their full attention on video lectures. While gamification has been used to engage students, there have been few attempts to gamify in‐video learning. The study aimed to create a gamified instructional video and examine the effects on learning. We identified gamification principles that can be incorporated to gamify in‐video components. An experiment conducted for validation purposes involved 48 undergraduate and graduate students. The results revealed that the participants who learned from the gamified instruction video (ie, experimental group, n = 23) had higher levels of necessary cognitive load required for accomplishing the given tasks than the participants who learned from a conventional video (ie, control group, n = 25). Although there was no significant difference in situational interest, the experimental group also exhibited better achievement, higher levels of intrinsic motivation, and learning satisfaction than the control group. Interview results revealed that the experimental group had a positive learning experience and clarified how the gamification strategies contributed to the outcomes. We discuss the implications of this study for research and practice. Practitioner notes What is already known about this topic Video‐based learning (VBL) is recognized for its ability to accommodate students with diverse educational needs, and higher education has paid attention to its accessibility. During VBL, students face challenges in sustaining their full attention on video lectures unless they are highly motivated to learn. Moreover, students lack access to receive real‐time guidance and feedback. What this paper adds We created a gamified VBL environment by embedding game elements into an instructional video. The experimental results indicated that the gamified video did not significantly increase learners' situational interest but enhanced necessary cognitive load, intrinsic motivation, learning satisfaction, and academic achievement. Interview findings supported these results, revealing that the gamified video effectively reduced unnecessary cognitive load while increasing cognitive efforts toward achieving learning objectives, thereby fostering engaging learning experiences. Implications for practice and/or policy Our attempts to create gamified instructional videos demonstrated how to enhance video lectures through gamification strategies and offered guidance for the design of gamified instructional videos. We also contributed to the exploration of gamification strategies in education and the potential of relatively underexplored gamification elements.
... Although previous studies supported the advantages of using game-based learning in the learning activities, other studies revealed negative results [35], [43], and their impacts on students' outcomes remain questionable [13], [44]- [46]. For instance, students who practice game-based learning activities failed their tests [35], [43]. ...
Article
Full-text available
p>Students in academia are facing serious challenges such as engagement, which causes academicians to question the quality and effectiveness of the teaching pedagogy. As such, new adapted innovative technological tools such as gamifications have been introduced to learning activities experiences and skills acquisition. However, little is known about the implementation of such technologies, which have been relatively underexplored in education literature. The primary objective of this study is to identify the perceptions of students and evaluate their engagement considering the use of gamification in the learning process. There were 210 respondents enrolled in the private university in Jordan were surveyed using a quantitative questionnaire. The study used descriptive statistics and independent sample t-test analyses on the data. The results indicated that the students have positive perceptions of game-based technology in both learning and classroom activity engagement. In addition, student perceptions and engagement did not differ significantly based on age or gender, but significant differences based on user experience were identified. These findings provide insight into the dynamic interaction between students and gamified technology to enhance student perceptions and promote meaningful learning. This research also provides implications on gamification literacy education and the pedagogically rich design of educational games to enhance student engagement. The recommendations are enumerated based on the obtained findings.</p
... Survei terbaru dari UNESCO menunjukkan bahwa lebih dari 40% siswa merasa bosan dengan metode pembelajaran tradisional, terutama di negara berkembang. Ketidakmampuan metode tradisional untuk menyesuaikan diri dengan perubahan zaman berdampak pada rasa percaya diri siswa dalam menghadapi tugas akademik (Legaki et al., 2020;Subhash & Cudney, 2018). Teknologi menawarkan peluang besar untuk meningkatkan pengalaman belajar, salah satunya melalui integrasi elemen gamifikasi dalam pembelajaran. ...
Article
Full-text available
This study aims to evaluate the effect of gamification on students' learning motivation and self-confidence. The underlying problem is low learning motivation due to monotonous conventional learning methods. Using the PRISMA method, this study screened relevant literature and used a quantitative approach with an experimental design. The research subjects were 60 grade 5 students, who were divided into an experimental group (using Kupin Edu) and a control group (using traditional methods). Data were collected through pretest and posttest questionnaires, analyzed by statistical tests. The results showed that the experimental group experienced significant improvements in learning motivation, from a score of 65.3 to 82.7, and self-confidence, from 58.7 to 75.3. In contrast, the control group showed minimal improvement. Competitive elements, such as leaderboards, and collaborative elements, such as daily challenges, were shown to significantly increase student engagement. This study concludes that gamification is an effective approach to increase students' learning motivation and confidence, particularly at the primary education level. The practical implication is that gamification can be integrated in the curriculum to create a more engaging and relevant learning environment. This research contributes to expanding the application of educational technology to improve the quality of student learning.
... Game design is centered on achievement and challenge and focuses on overcoming obstacles, making progress, earning rewards, and fostering a sense of competence. In gamification that is based on success and challenges, the most frequently utilised mechanisms include points, challenges, rankings, levels, and badges (Legaki et al., 2020). Csikszentmihalyi (1991) proposed that game-based online activities should have several key characteristics: ...
Article
Full-text available
This systematic review explores the effectiveness of gamification in enhancing oral communication skills among English language learners. Employing the PRISMA framework, 50 studies published between 2019 and 2023 were analyzed to address the research question: "To What extent does gamification foster students’ oral communication?". Findings suggest that gamification positively impacts oral communication by increasing student motivation, fostering engagement, and reducing anxiety during speaking activities. Key factors contributing to these outcomes include the strategic use of game elements, collaborative learning tasks, and interactive learning environments. However, challenges such as the need for teacher training and the variability of gamification's effectiveness across different educational contexts were also identified.
Article
Full-text available
In an era marked by the omnipresence of Information and Communication Technologies, educators face the intimidating challenge of harnessing students' attention. Studies suggest that technology adopted early in life affects brain structure and cognition. Therefore, new generations require creative pedagogical approaches such as gamified learning. At the Information and Library Center of the Medical University of Łódź, we are exploring the optimal combination of gamification elements to improve learning outcomes in our specific academic environment. This article presents practical examples of more engaging materials that we have designed at the Information and Library Center to teach information literacy. Gamified materials were tested on various groups of library users (students, librarians, and academic staff). Although we have gained an interesting insight on how learners perceive gamified learning, a quantitative analysis is still needed.
Article
Purpose This study aims to examine the metaverse experiences of individuals traveling to Turkey, a nation with a complex history of migration, and to assess the influence of these experiences on their accommodation processes within host countries. Design/methodology/approach A qualitative approach, using semi-structured interviews, explored participants’ Metaverse experiences. Thematic analysis was applied to the interview data. The participant statements were coded, and concept maps were created. Findings The findings indicate that metaverse technologies can significantly facilitate planning for both short-term travelers and temporary migrants to migration hotspots, mitigating anxieties associated with the unfamiliar, and expediting the adaptation process by enabling virtual exploration of such destinations. Originality/value The research uniquely explores how metaverse and related technologies can enhance hospitality experiences in destination countries, particularly those facing challenges from irregular migration. Unlike previous studies, it specifically focuses on the metaverse experiences of people traveling to these destinations, aiming to answer the crucial question of how effective these technologies are in improving the hospitality experience in such contexts.
Article
Physical Education is one of the cornerstones of education, promoting the overall development of students on a physical, psychological and social level. It cultivates skills that contribute to physical health, motor ability and self-confidence, while enhancing collaboration and social interaction. While its importance is recognised, one of the main problems it faces is low student participation, mainly due to lack of interest or the inability to link activities to their needs and interests. In this context, approaches such as gamification offer solutions to enhance engagement and participation, creating a learning environment that is both fun and educational. Gamification is a pedagogical method that utilizes mechanisms from games, such as level progression, rewards, scores and challenges, in non-playful environments. Its main goal is to motivate students, creating positive motivation for participation and engagement. In a course like Physical Education, where engagement and active participation are critical, incorporating this method can be a catalyst for creating a more engaging and dynamic learning environment. This study focuses on the theoretical investigation of the connection between gamification and physical activity, analyzing how these mechanisms can enhance educational practice and provide solutions to chronic problems of participation. Through the analysis of the relevant literature and research findings, it is confirmed that gamification can act as a motivation and engagement tool. By creating challenges that meet students' potential, their sense of autonomy and achievement is enhanced, making them feel active members of the learning process. In addition, the ability to collect virtual rewards, such as medals or points, and participate in competitions help create a fun experience that motivates even the most reluctant students. This method helps maintain interest by fostering consistency and active participation over time. At the same time, gamification contributes to the development of social skills, encouraging cooperation through team activities and friendly competition. Students learn to work together to achieve common goals, cultivating communication and relationship management skills. Especially for more reluctant students or those with lower motor skills, the ability to participate in activities tailored to their abilities creates an inclusive environment, enhancing self-esteem and sense of belonging. Through this process, the Physical Education lesson is transformed from a formal activity into an experience that empowers all students. While gamification offers significant benefits, it is not devoid of challenges. One of the key problems is an overemphasis on extrinsic motivation, such as rewards and scores, that can undermine students' internal motivation. To address this issue, teachers need to focus on creating activities that enhance autonomy, a sense of achievement and social interaction. In addition, limitations such as lack of equipment, insufficient teacher training and different abilities of students can be obstacles to the effective implementation of the method. However, with proper training and the use of affordable technological tools, these obstacles can be overcome. Through this study, guidelines for the implementation of gamification in Physical Education are proposed. Specially designed digital tools can support the personalisation of teaching, providing feedback and progress data. At the same time, the connection of Physical Education with other subjects, such as technology, sciences and arts, can create an interdisciplinary environment that enriches the learning experience. The use of augmented reality and other innovative technologies offers further possibilities for student engagement. Overall, gamification is presented as an innovative method that can transform the teaching of Physical Education. This approach not only improves participation and performance, but also creates an environment that enhances students' autonomy, collaboration and personal development. Future research and practical applications can contribute to the further exploitation of this method, shaping a more participatory and effective educational future
Article
Full-text available
Gamification has become a popular technique in marketing. Many companies believe that gamification can potentially increase the engagement, awareness and loyalty of consumers with respect to the brand. However, there is current dearth of empirical evidence supporting these beliefs beyond the pervasive hype. In this study we investigated the relationships between gamification, brand engagement and brand equity among consumers (N = 824) of two online gamified brand communities. The results showed that achievement and social interaction -related gamification features were positively associated with all three forms of brand engagement (emotional, cognitive and social). Immersion -related gamification features were only positively associated with social brand engagement. Additionally, brand engagement was further positively associated with brand equity. The results imply that gamification can positively affect brand engagement and further increase brand equity, and that gamification appears to be an effective technique for brand management.
Article
Full-text available
Gamification is now a well-established technique in Human-Computer Interaction. However, research on gamification still faces a variety of empirical and theoretical challenges. Firstly, studies of gamified systems typically focus narrowly on understanding individuals. short-term interactions with the system, ignoring more difficult to measure outcomes. Secondly, academic research on gamification has been slow to improve the techniques through which gamified applications are designed. Third, current gamification research lacks a critical lens capable of exploring unintended consequences of designs. The 14 articles published in this special issue face these challenges with great methodological rigor. We summarize them by identifying three main themes: the determination to improve the quality and usefulness of theory in the field of gamification, the improvements in design practice, and the adoption of a critical gaze to uncover side-effects of gamification designs. We conclude by providing an overview of the questions that we feel must be addressed by future work in gamification. Gamification studies would benefit from a wider use of theories to account for the complexity of human behavior, a more thorough exploration of the many opportunities coming from the world of games, and an ethical reflection on the use of game design elements in serious domains.
Article
Full-text available
Background Compared to traditional persuasive technology and health games, gamification is posited to offer several advantages for motivating behaviour change for health and well-being, and increasingly used. Yet little is known about its effectiveness. Aims We aimed to assess the amount and quality of empirical support for the advantages and effectiveness of gamification applied to health and well-being. Methods We identified seven potential advantages of gamification from existing research and conducted a systematic literature review of empirical studies on gamification for health and well-being, assessing quality of evidence, effect type, and application domain. Results We identified 19 papers that report empirical evidence on the effect of gamification on health and well-being. 59% reported positive, 41% mixed effects, with mostly moderate or lower quality of evidence provided. Results were clear for health-related behaviours, but mixed for cognitive outcomes. Conclusions The current state of evidence supports that gamification can have a positive impact in health and wellbeing, particularly for health behaviours. However several studies report mixed or neutral effect. Findings need to be interpreted with caution due to the relatively small number of studies and methodological limitations of many studies (e.g., a lack of comparison of gamified interventions to non-gamified versions of the intervention).
Article
Full-text available
Today, our reality and lives are increasingly game-like, not only because games have become a pervasive part of our lives, but also because activities, systems and services are increasingly gamified. Gamification refers to designing information systems to afford similar experiences and motivations as games do, and consequently, attempting to affect user behavior. In recent years, popularity of gamification has skyrocketed and manifested in growing numbers of gamified applications, as well as a rapidly increasing amount of research. However, this vein of research has mainly advanced without an agenda, theoretical guidance or a clear picture of the field. To make the picture more coherent, we provide a comprehensive review of the gamification research (N = 819 studies) and analyze the research models and results in empirical studies on gamification. While the results in general lean towards positive findings about the effectiveness of gamification, the amount of mixed results is remarkable. Furthermore, education, health and crowdsourcing as well as points, badges and leaderboards persist as the most common contexts and ways of implementing gamification. Concurrently, gamification research still lacks coherence in research models, and a consistency in the variables and theoretical foundations. As a final contribution of the review, we provide a comprehensive discussion, consisting of 15 future research trajectories, on future agenda for the growing vein of literature on gamification and gameful systems within the information system science field.
Article
Full-text available
Background. Definitions of gamification tend to vary by person, both in industry and within academia. One particularly popular lay interpretation, introduced and popularized by Ian Bogost, and reiterated by Jan Klabbers, is that gamification is “bullshit” and “exploitationware.” They describe gamification as a marketing term or business practice invented to sell products rather than to represent a real and unique phenomenon relevant to a nascent game science. However, this view is an oversimplification, one which ignores a growing body of theory development and empirical research on gamification within a post-positivist epistemology. In fact, because gamification is so much more outcome-focused than general game design, current gamification research in many ways has a stronger footing in modern social science than much games research does. Aim. In this article, to address common misunderstandings like these, we describe the philosophical underpinnings of modern gamification research, define the relationship between games and gamification, define and situate gamification science as a subdiscipline of game science, and explicate a six-element framework of major concerns within gamification science: predictor constructs, criterion constructs, mediator constructs, moderator constructs, design processes, and research methods. This framework is also presented diagrammatically as a causal path model. Conclusion. Gamification science refers to the development of theories of gamification design and their empirical evaluation within a post-positivist epistemology. The goal of gamification scientist-practitioners should be to understand how to best meet organizational goals through the design of gamification interventions, drawing upon insights derived from both gamification science and games research more broadly.
Conference Paper
Presently, higher education institutions are faced with the challenge of developing student’s abilities and skills needed in their future workplace. In what concerns technological skills, the use of spreadsheets for calculations, analysis of data and forecasting is a common and important practice in companies. Particularly, the MS Excel software is widely used by professionals from all fields. In this sense, Economic, Business and Marketing graduates need competencies in forecasting methods, extremely useful for decision-making processes. It thus becomes imperative to implement pedagogical practices to encourage students to use these technological tools. The later will promote the development of competencies in forecasting methods to solve future real problems. In this paper, we attempt to address these issues by analysing the MS Excel software capabilities as a teaching tool in a forecasting methods course. It was proposed to the students to carry out a learning project involving statistical concepts, namely linear regression, performed in MS Excel. We examine the performance and engagement of two samples of students with different backgrounds and from distinct realities. One group is composed of ERASMUS’ students from several nationalities and fields of study and other group consists of Portuguese students of the Marketing Bachelor degree, both enrolled in an optional course of the bachelor degree in Marketing taught at Porto Accounting and Business School from Polytechnic of Porto. The effectiveness of this approach is shown through the analysis of results of students’ projects. We verify that students in both groups achieved the task proposed goals and applied appropriately the required concepts in an engaged way. Keywords: Higher education, Excel, forecasting, linear regression.
Article
This article discusses how to use FREDcast, an interactive economic forecasting game from the Federal Reserve Bank of St. Louis, to teach macroeconomics. It provides instructions on how to set the game up and organize related teaching materials such as graphs and data points through a FRED (Federal Reserve Economic Data) Dashboard. It also puts forward a series of classroom discussion questions and suggested assignments. We argue that this pedagogical strategy provides an engaging context for the discussion of core topics in macroeconomics while contributing to the development of data-related expected student proficiencies.