The Key Elements of Gamification in Corporate Training - The Delphi Method

Abstract

Gamification of training, the application of gamification to modify training content and methodology, has become a hot topic in corporate training and school education in recent years. However, the relevant key elements have not been identified. Therefore, we aimed to engender and prioritize a list of key elements of effective gamification in the course of corporate training.
Using the Delphi method, this study surveyed 14 corporate training gamification experts to evaluate the key elements of gamification – 12 were identified and categorized into six design principles: integration with training goals, rapid feedback, team competition, clear rules, goal-oriented challenges, and freedom to fail. Points and leaderboards were also named as the keys to effective game mechanics.
The integration of educational objectives with gamification design, and the balancing between game mechanics and design principles were the conception for gamification in corporate training. This study also provided qualitative data, presenting the rich and differentiated views of corporate training experts regarding the key elements of gamification, to provide some practical tips for teachers in the development of training gamification.

Full text

Entertainment Computing 40 (2022) 100463
Available online 12 October 2021
1875-9521/© 2021 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/).
The key elements of gamication in corporate training – The Delphi method
Yung-Fu Wang
a
,
*
, Ya-Fang Hsu
b
, Kwoting Fang
a
a
Department of Information Management, National Yunlin University of Science and Technology, Yunlin, Taiwan
b
Department of Physical Medicine and Rehabilitation, Chi Mei Medical Center, Tainan, Taiwan
ARTICLE INFO
Keywords:
Adult learning
Delphi method
Employee training
Training gamication
ABSTRACT
Gamication of training, the application of gamication to modify training content and methodology, has
become a hot topic in corporate training and school education in recent years. However, the relevant key ele-
ments have not been identied. Therefore, we aimed to engender and prioritize a list of key elements of effective
gamication in the course of corporate training.
Using the Delphi method, this study surveyed 14 corporate training gamication experts to evaluate the key
elements of gamication – 12 were identied and categorized into six design principles: integration with training
goals, rapid feedback, team competition, clear rules, goal-oriented challenges, and freedom to fail. Points and
leaderboards were also named as the keys to effective game mechanics.
The integration of educational objectives with gamication design, and the balancing between game me-
chanics and design principles were the conception for gamication in corporate training. This study also pro-
vided qualitative data, presenting the rich and differentiated views of corporate training experts regarding the
key elements of gamication, to provide some practical tips for teachers in the development of training
gamication.
1. Introduction
1.1. The concept of gamication
Gamication, dened as “the use of game design elements in non-
game contexts” [1], gradually became a hot topic after 2010 [2–4].
The idea is to turn tasks into game-playing to boost intrinsic motivation
[5], entice more participants and to increase their level of engagement in
the task [2]. Gamication has since been widely used in different elds
such as business operations, marketing, and education [6–8].
The eld of education and training has also generated a great deal of
interest in gamication [2,9–10]. The process of applying gamication
elements to modify training content and methodology is called gami-
cation of learning [11] or training gamication. This has become a
common choice in redesigning traditional training [12], and its appli-
cation in higher education and corporate employee training has also
become more prevalent. Moving forward, the gamication market
growth estimate is billions of U.S. dollars, and this new trend will likely
transform the workplace in the future [13–14].
Currently, a unied theoretical framework for academic research on
gamication is still lacking. Existing studies on gamication are based
on quantitative research methods that rely on survey data to infer user
behavior; there is very little qualitative research on gamication [15].
Even though there is growing application and literature on gamication
for adult education, many issues are unresolved. For example, the
effectiveness of gamication is unclear [11], and the key elements of
gamication await clarication [16]. From existing literature, it is
difcult to decipher the gamication elements or combinations with the
most impact [17]. Therefore, this study applied the Delphi method to
conduct surveys of expert opinions, targeting corporate training pro-
fessionals as survey subjects. The study focused on two goals:
(1) Exploration of key elements of successful gamication in corpo-
rate training practices. (2) Evaluate key gamication elements and their
degree of importance, and how corporate trainers apply such elements
in their practices.
2. Gamication in research and practice
2.1. Gamication
As gamication is still in the early stages of development, its de-
nition is often misunderstood [17]. Among related terminologies such as
* Corresponding author at: Department of Information Management, National Yunlin University of Science and Technology, Yunlin, Taiwan.
E-mail address: afuwang@gmail.com (Y.-F. Wang).
Contents lists available at ScienceDirect
Entertainment Computing
journal homepage: www.elsevier.com/locate/entcom
https://doi.org/10.1016/j.entcom.2021.100463
Received 29 May 2020; Received in revised form 5 July 2021; Accepted 7 October 2021

Entertainment Computing 40 (2022) 100463
2
games, serious games, game-based learning and gamication, there are
many similarities but also plenty of differences in principles and con-
cepts [11,17]. A game is a made-up imaginary world with very specic
rules, and often played at a specic time and place [18]. Usually for
entertainment purposes, a game consists of many elements, including
game functions or gameplay [3]. Serious games are also called educa-
tional games or learning games – their primary purpose is education and
not entertainment [18]. Even though very closely related to games,
serious games are often used in different learning environments [11],
such as military, government, education, business, and healthcare [19].
For example, “American’s Army”, considered the most successful mili-
tary serious games, is used to attract new recruits into the army [20].
Serious games can also be used as ight simulations to support pilot
training [21]. One of the focuses of game-based learning is about
learning inducement, while serious games focus on behavior modica-
tion of the trainees [22]. Nevertheless, game-based learning is often
used interchangeably in the literature with serious games [23].
Different from ordinary games, gamication doesn’t provide an in-
dependent experience, and doesn’t have a complete narrative or pro-
gram [24]. It differs from serious games [11], and has no relation to
software applications or computers [25]. The widely accepted denition
is “the use of game design elements in non-game contexts” [1], also
known simply as the “points, badges, leaderboard” method [5]. The goal
is to enhance participants’ motivation through the addition of gaming
elements [26], enticing users and encouraging specic types of
behavior, creating signicant driving force to induce desirable user
behavior [27].
2.2. Gamication of training
Gamication can attract the attention of students [16], thus drawing
interest towards its implementation in education [2,9–10]. Gamication
of learning is the process of applying gaming elements to modify training
content and methodology [11], to help develop student potential and
provide positive encouragement to boost their performance level [28].
Gamication is not about playing games in the classroom. It is the
insertion of a few gaming elements into an existing teaching program. It
is a type of educational design process, and not a type of teaching
method. The user learns from the educational content, they do not learn
from gamication [11]. Thus, gamication of training is dened here as
applying gamied elements in a training environment. The imple-
mentation of gamication elements in the process of learning, teaching
and training may elevate students’ interest and motivation to participate
in the subject content, increasing their concentration and enjoyment of
learning [24,29]. As students maintain a proactive attitude, improve
their learning process and overcome any potential psychological bar-
riers [30], they will be able to learn with efciency [15].
2.3. Key elements of successful gamication
Common gamication elements are points, scoring, leaderboards
[17,30–31], progress bars, ranks, rewards or incentives, all of which
often appear in the implementation of gamication [32]. The addition of
a story or narratives is also part of gamication [33]. Common gami-
cation design principles include: goals and challenges, personalization,
rapid feedback, visible feedback, freedom of choice, freedom to fail and
social engagement [9]. A review study of educational and learning
gamication found that gamication elements may be divided into ve
different types including: progressive achievement, social interaction,
immersion experience, non-digital elements, and others, while the most
common elements are points, challenges, badges and leaderboard [34].
Another way to categorize gamication elements is extrinsic moti-
vation vs. intrinsic motivation [35], which may be explained by
applying theories of psychology, including the theory of operant con-
ditioning, the expectancy-based theories, the theories of self-regulation
and the theory of self-determination [11]. For example, from the
perspective of the theory of operant conditioning, points and badges can
be considered as reinforcement of operant conditioning, prompting
students to respond to teaching expectations and complete specic tasks
or goals [36]. The self-regulation theories may be used to explain how to
apply the leaderboard to encourage students to attempt more chal-
lenging goals [37]. The theory of self-determination may explain how
badges can satisfy students’ need for positive feedback on their abilities
[38].
2.4. Possible gamication problems
Although past research showed that gamication achieved positive
effects in general, some gamication approaches may lead to inconsis-
tent results [15]. For example, the leaderboard and the badges may
cause interference with intrinsic motivation [35], or lower satisfaction
and reduce the momentum for action [39], or lead to discontentment
with the fairness of gamication [16]. If students view the gaming ele-
ments as redundant, then the implementation of gamication may be
counterproductive [17]. If the focus is on students’ erroneous behavior
or attitude, simply making the training program fun will not improve the
effect of the training [11]. Students will ultimately lose interest if the
implementation fails to adhere to essential gamication design guide-
lines [24].
3. Method
Originated from the RAND Corporation in 1950 [40–41], the Delphi
method uses an interactive iterative process to seek expert consensus
[42–43], in order to make decisions or evaluations [44], or conduct
predictive research [41]. The classical Delphi method includes the
following four key features: anonymity, iteration, controlled feedback
and statistical summary [42,44–48]. When a problem or phenomenon is
not fully understood, then the Delphi method is a suitable framework for
research [49–50]. The Delphi research process calls for a pre-round to
develop a key elements questionnaire for discussion during subsequent
rounds [51]. The Likert ve-point scale may be used to evaluate the
consistency of expert opinions [51]. In the rst round, experts are asked
to ll out a questionnaire, which are collected for statistical analysis.
Afterwards, experts are invited for a second-round evaluation where
they may reference the analysis result from the rst round, and revise or
expand on their answers in the rst round. Next, a similar analysis and
evaluation/selection process is used for the third round [42]. At least
two rounds of questionnaires are conducted iteratively to obtain expert
opinions. For most studies, two to three rounds of iterative discussions
using the Delphi methodology is enough to reach a consensus [45].
Even though the concept of consensus is the foundation of many
Delphi studies, the meaning of consensus is not yet clearly dened. The
most common denition is a certain percentage in agreement or a
certain ratio of scores within a designated range [52]. The Inter-quartile
range (IQR), the standard deviation (SD) and the frequency distribution
are often used as the basis of consensus assessment [45]. An IQR of less
than 1 indicates that over 50% of the expert opinions are focused on a
certain point. An IQR of zero indicates that the experts are in complete
agreement [53]. The SD is an indicator of the level of opinion dispersion.
An SD value of less than 1.5 indicates that the experts have reached a
consensus [45]. Moreover, if the average score for an element is greater
than 4 on the ve-point scale, and more than 51% of the experts assigned
a score between 4 and 5, it is an indication that experts view the item
being evaluated as a key element. When multiple indicators are used for
measurement at the same time, it is possible to obtain a more reliable
expert consensus [45].
Therefore, this study has set the consensus criteria as IQR≦ 1 and
SD≦1.5, which are regarded as the requirements for a consensus of
expert opinions. After a consensus has been reached, when an item
received an average score greater than 4 in the expert evaluation, and
more than 51% of the expert evaluations gave a score of 4 or above, the
Y.-F. Wang et al.

Entertainment Computing 40 (2022) 100463
3
item is considered a gamication key element.
Research related to educational gamication is still in the early
stages [54], thus this study adopted the Delphi method to explore the
key elements of training gamication. Fourteen corporate training ex-
perts with ample experience in training gamication were invited to
participate in three-rounds of expert opinion collections in accordance
with the Delphi method, to evaluate the key ingredients for successful
gamication of training. To reach consensus among the experts and to
select the key elements of gamication, this study conducted at least two
rounds of expert-opinion collection, and the Delphi research stopped
after the third round or earlier when consensus was reached. The data
for this research were collected between September 2019 through March
2020.
3.1. Expert participants
The selection of appropriate experts is essential to the Delphi
research method [41]. Suitable experts would be knowledgeable and
experienced, have the willingness and time to participate, and be
equipped with communication skills [50]. Therefore, experts were
chosen purposefully [55], relying on expert recommendations [45] or
the snowball method of sampling [42]. If the experts are homogeneous,
then 10 to 15 people would be enough to form a representative group
[42,56].
For this research, two gamication training experts served on the
nomination team. One of them is a coach specialized in training
corporate lecturers, and a published author of several books on educa-
tional training and gamication teaching methodology. The other
nominating team member is a gamication research scholar.
Furthermore, the following three criteria were considered for the
nomination of educational gamication experts: (1) The nominee has
utilized common gamication elements such as points, scoring, and
leaderboards within their training courses. (2) The nominee has more
than three years of experience with educational gamication. (3) The
nominee has been a lecturer at one of the top 500 listed companies in
Taiwan over the past year.
To ensure the completeness of the gamication expert nomination,
three sources were used to collect the list of gamication experts,
including: human resource executives of listed companies, executives of
management consulting rms that specialize in corporate training, and
the recommendation of fellow corporate lecturers. In the end, 27 qual-
ied gamication experts were nominated. Invitations were extended
via phone or email, and 16 of the 27 experts agreed to participate in the
research, and two of them withdrew in the middle of the process. The
remaining 14 experts completed all three rounds of survey question-
naires. The participants included 12 males (86%) and 2 females (14%).
Among them, they averaged about 7 years of gamication teaching
experience, and most of them (n =7, 50%) had more than 9 years of
gamication experience.
The content of courses taught by these experts included the following
elds: presentation skills, innovation and creativity, project manage-
ment, communication skills, problem analysis and resolution, sales
skills, language skills, online marketing, nance and nancial report,
and train the trainer. Within the last year of the survey, each expert had
multiple experiences performing gamication training at some of the
top 500 listed companies in Taiwan. The gamication approach used by
the experts in their own practices were also collected. A total of 29
gamication elements were used, including points, leaderboards, virtual
identities, puzzles, storylines, experiential activities, integration with
software apps, and so on. For detail information on the experts who
participated in this research, please refer to Table 1.
3.2. Pre-round of Delphi method
For the pre-round of this research, experts were interviewed via
email and telephone to explore their practical experiences of training
Table 1
Professional Qualications of Surveyed Experts.
Code Gender Years of
Experience
Fields of
Educational
Expertise
The approaches of
gamication that each
expert used in their own
practice
A Male Over 9 years Communication
Skills
Integration with training
goal, Game rules,
Fairness, Task with
challenging goals, Team
work, Points, Time
pressure, Experiential
activities, Competition,
Leaderboard, Prize or
bonus money, Storyline,
Virtual identity
B Male 6 to 9 years Problem Analysis
and Resolution
Integration with training
goal, Team work,
Experiential activities
C Female 3 to 6 years Sales Skills Integration with training
goal, Fairness, Team
work, Points,
Competition, Prize or
bonus money
D Male 3 to 6 years Presentation Skills Integration with training
goal, Game rules,
Fairness, Team work,
Points, Competition,
Prize or bonus money,
Board games, Virtual
identity
E Male 3 to 6 years Innovation and
Creativity
Integration with training
goal, Game rules,
Fairness, Team work,
Points, Competition,
Prize or bonus money
F Female Over 9 years Language Skills Integration with training
goal, Team work, Points,
Time Pressure,
Competition, Virtual
currency or chips,
Integration with software
apps
G Male 3 to 6 years Presentation Skills Integration with training
goal, Rapid feedback,
Task with challenging
goals, Team work,
Points, Time pressure,
Competition, Prize or
bonus money, Storyline,
Social network, Virtual
identity, Board games
H Male 3 to 6 years Online Marketing Integration with training
goal, Prize or bonus
money, Storyline, Virtual
identity
I Male Over 9 years Finance and
Financial Report
Integration with training
goal, Fairness Team
work, Points,
Competition,
Leaderboard, Prize or
bonus money, Virtual
currency or chips
J Male 3 to 6 years Presentation Skills Integration with training
goal, Game rules, Team
work, Points,
Competition, Clues,
Virtual identity
K Male Over 9 years Project
Management
Integration with training
goal, Game rules, Task
with challenging goals,
Team work, Points, Time
pressure, Competition,
Puzzles, Virtual identity
L Male Over 9 years Project
Management
Integration with training
goal, Rapid feedback,
Game rules, Task with
(continued on next page)
Y.-F. Wang et al.

Entertainment Computing 40 (2022) 100463
4
gamication, and the following three questions were asked: (1) training
gamication implementation examples, (2) the key elements of training
gamication, and (3) training gamication challenges and recommen-
dations. After the expert’s answers were transcribed, consideration was
given to key gamication elements of education and learning based on a
review of empirical literature by Majuri et al. (2018) and a review of
related gamication research [3,9,31–33,55–56]. Comparisons were
made of the research reviews and practical gamication experience
described by experts during the interviews, and the data were catego-
rized into 35 key elements of training gamication. Out of the 35 items,
19 items (59%) were recognized by both the literature and the experts
interviewed, 11 items (31%) were proposed by the literature and 5 items
(14%) were derived from the experts’ opinions,this would serve as the
basis for subsequent expert evaluation of key elements in training
gamication.
3.3. The three Delphi rounds
For the rst round, the experts were asked via email to evaluate 35
training gamication key elements using the Likert ve-point scale,
where points 1 through 5 indicated not at all important, not important,
neutral, important and very important, respectively. The order of the
questions in each expert questionnaire had been randomized to avoid
ordering bias [41]. At the end of the questionnaire, the experts were
invited to write in key elements that should have been on the list. The
consensus criteria were: IQR≦ 1 and SD≦ 1.5 [45].
At the rst round, 24 items met the criteria of consensus in terms of
IQR, SD. For the rst round, no experts had added any key element
items. Since the Delphi research method requires at least two iterative
rounds to obtain expert opinions [42], thus all gamication key ele-
ments evaluated were kept for the second round of Delphi research.
One of the special features of the Delphi method is to allow partici-
pants to reference other people’s opinions anonymously, to iteratively
rene their perspectives or revise their own ideas [42,45]. For the sec-
ond round, participating experts were again asked to evaluate the 35
gamication elements, and each item’s descriptive statistical data of
expert opinions in the previous round were also available to them,
including the mean, SD, and IQR. If an expert’s opinion exceeded the
upper or lower IQR, that expert was asked to provide an explanation. At
the end of the second round, the experts reached consensus for 33 items,
which was 9 more than the rst round. For the third round, the experts
were asked to evaluate only the remaining 2 items without consensus. In
the end, the experts met the criteria and arrived at consensus on all 35
items, thus ending the Delphi method investigation. The situation was in
line with most Delphi method research implementations of 2 to 3 rounds
of surveys [45]. Table 2 and Table 3 shows the three rounds of iterative
process.
4. Results and discussions
This study obtained consensus through three rounds of expert sur-
veys, and 35 gamication key elements were ranked based on the
importance mean value. Among them, 12 items (34%) had a score of
four or more in mean importance value, 13 items (37%) had a mean
score between 3.00 and 3.90, and 10 items (29%) had a mean score of
less than three. A closer examination of items with a mean score greater
than 4 shows that the ratio of experts who assigned a score of 4 or
greater all exceeded 51% (the range was 79% ~ 100%). In the end, the
experts reached consensus on 12 items which the mean score ≧4 as the
key elements of training gamication.
Currently, there isn’t a common method to classify gamication key
elements. For example, badges may be considered as game interface
design model [3], or gaming mechanics [9,57,58]. Dicheva el al. (2015)
conducted a comprehensive analysis research of educational gamica-
tion by incorporating the gamication element denition of Deterding
et al. (2011), and proposed two major categories of gamication ele-
ments: gamication design principles and game mechanics [3,9]. Based
on these two major categories, this study further explores effective
gamication elements obtained from the surveys via the method of
collection and organization.
4.1. Category I: Gamication design principles
Based on the expert evaluation score of four or more points (indi-
cating important or very important), see Table 2, this study gathered
expert opinions and organized them into six major gamication design
principles.
(1) Integration with Training Goals
The experts unanimously agreed that this “Integration with training
goals” was the most important key element of training gamication,
with an expert scoring mean value of 5.00, SD =0.00 and IQR =0.00.
The ratio of evaluation importance value of 4 points or above was 100%
(see Table 2). The experts stated that:
N: “The purpose of games is to serve the course material. Games are
the means, not the end goal. The connection between games and the
course content must be very strong. Do not gamify for the sake of
gamication.”
I: “Do not gamify just for gamication, it should be related to the
subject that is being taught.”
G: “Without clear goals, two situations may result: (1) The learner
does not know what to do. (2) After everyone experiences the high of
game-playing, there is a letdown. Let the learners understand clearly
what this activity is for, and what they would gain by doing a good
job.”
Table 1 (continued )
Code Gender Years of
Experience
Fields of
Educational
Expertise
The approaches of
gamication that each
expert used in their own
practice
challenging goals, Team
work, Points, Time
Pressure, Increasing
difculty, Experiential
activities, Competition,
Leaderboard,
Performance status
feedback, Storyline,
Clues, Social network,
Progress bar, Virtual
identity.
M Male Over 9 years Training of
Lecturers
Integration with training
goal, Rapid feedback,
Game rules, Fairness,
Task with challenging
goals, Team work,
Points, Time pressure,
Increasing difculty,
Competition, Prize or
bonus money
N Male Over 9 years Innovation and
Creativity
Integration with training
goal, Game rules, Team
work, Points, Time
pressure, Competition,
Puzzles, Virtual currency
or chips, Storyline, Board
games, Virtual identity,
Virtual Helper
O†Male 3 to 6 years Project
Management
Integration with training
goal, Fairness, Team
work, Points,
Competition, Prize or
bonus money
Note. †indicated this expert withdrew this study in the second round period.
Y.-F. Wang et al.

Entertainment Computing 40 (2022) 100463
5
Training instructors are advised to rst determine the knowledge,
skill set, or performance goals of the training, and diligently choose the
elements of gamication [17]. Just as the phrase - “Do not gamify for the
sake of gamication” - frequently mentioned by many experts in the
study, careful consideration of the course objectives is recommended
when gamication is used. Before starting the process of training
gamication, our experts suggested a re-evaluation or redesign of the
curriculum using the instructional design method or the ADDIE (Anal-
ysis, Design, Development, Implementation, Evaluation) model, to
integrate gamication and training goals while enhancing learning
effectiveness [11].
(2) Rapid Feedback
“Rapid Feedback” is a key element highly valued by the experts
(mean =5.00, SD =0.00, IQR =0.00, 100% of experts viewed it as
important or very important, see Table 2). The experts stated that:
G: “Whether it’s a race to answer questions, presentation or simu-
lation, immediate feedback is very important.”
D: “If you answer, you get extra points.” Strengthen the students’
“familiarity and approval of the scoring rules” so that they are even
more willing to answer questions.
Ample feedback would enhance the participants’ experience
[35,59]. Points, badges and leaderboards are all key elements of feed-
back that encourage users’ participation and their level of engagement
[51]. For this study, the participating experts mostly used points and
leaderboards to provide feedback in their practices, which will be dis-
cussed further in the section on gamication element categories.
(3) Team-based Competition:
“Competition” is an important key element of successful gamica-
tion (mean =4.30, SD =0.62, IQR =1.00, 93% of the experts viewed it
is important or very important, see Table 2), and the unit of competition
is “team work” (mean =4.40, SD =0.63, IQR =1.00, 93% of experts
viewed it as important or very important, see Table 2). Through team-
based competitions, learners become invested in the challenges and
maintain their participation and degree of engagement in the training
course. Experts stated that:
Table 2
The result of the rst and the second round of expert opinion collection.
Round 1 (n =15) Round 2 (n =14)
Elements Mean SD IQR ≧4 Ratio Rank Mean SD IQR ≧4 Ratio Rank
Integration with training goal 5.0 0.00 0.0 100% 1 5.0 0.0 0.0 100% 1
Rapid feedback 4.8 0.40 0.0 100% 4 5.0 0.0 0.0 100% 2
Game rules 4.8 0.40 0.0 100% 3 4.9 0.26 0.0 100% 3
Fairness 4.8 0.34 0.0 100% 2 4.7 0.59 0.0 93% 4
Task with challenging goals 4.6 0.49 1.0 100% 6 4.5 0.49 1.0 100% 5
Team work 4.5 0.73 1.0 86% 9 4.4 0.63 1.0 93% 6
Points or scoring 4.1 0.98 1.5* 73% 16 4.4 0.62 1.0 93% 7
Time Pressure 4.8 0.44 0.5 100% 5 4.3 0.45 0.75 100% 8
Increasing difculty 4.5 0.63 1.0 93% 8 4.3 0.48 1.0 100% 9
Experiential activities 4.4 0.73 1.0 86% 11 4.3 0.48 1.0 100% 10
Competition 4.6 0.60 0.5 93% 7 4.3 0.62 1.0 93% 11
Freedom to fail 4.5 0.62 1.0 93% 10 4.1 0.77 1.0 79% 12
Educational tools or props 4.4 0.47 1.0 100% 12 3.8 0.59 0.0 79% 13
Leaderboard 4.2 0.91 2.0* 67% 13 3.8 0.67 0.0 79% 14
Prize or bonus money 3.9 1.15 1.0 80% 14 3.8 0.88 1.0 57% 15
Adaptation to difculty 4.0 0.85 1.0 73% 15 3.7 0.86 1.0 64% 16
Performance status feedback 3.7 1.05 2.0* 60% 21 3.6 0.73 1.0 64% 17
Virtual currency or chips 3.7 1.25 2.0* 40% 17 3.5 0.97 1.0 43% 18
Storyline 3.5 1.25 2.0* 60% 23 3.4 0.62 1.0 50% 19
Peer rating 3.7 0.70 1.0 67% 18 3.3 0.59 1 36% 20
Clues 3.5 0.71 1.0 60% 22 3.3 0.72 1 36% 21
Social network 3.7 1.05 1.5* 67% 20 3.3 0.91 1.75* 36% 22
Customization or personalization 3.3 0.77 1.0 47% 24 3.2 0.7 0.75 29% 23
Puzzles 3.6 0.71 1.0 47% 19 3.1 0.70 0.75 29% 25
Badges 3.0 0.97 1.0 27% 27 3.0 0.64 0.75 14% 24
Board games 2.7 0.83 1.0 20% 31 2.9 0.67 0 7% 26
Progress bar 3.2 1.00 1.0 33% 25 2.9 0.74 0 14% 27
Integration with software apps 2.5 0.71 0.5 0% 32 2.8 0.72 1 7% 28
Virtual identity 3.0 1.10 1.5* 33% 29 2.7 0.82 1 14% 29
Virtual Helper 2.9 1.06 0.0 13% 28 2.7 0.91 1 14% 30
Virtual treasure 3.0 0.82 0.5 20% 26 2.4 0.72 1 0% 31
Hierarchy 2.8 1.00 1.5* 27% 30 2.3 0.73 1 7% 32
Virtual reality 2.5 1.15 1.5* 13% 33 2.3 0.94 1.75* 7% 33
Health score 1.8 0.93 1.5* 7% 34 1.8 0.77 1 7% 34
Check in 1.8 0.91 2.0* 0% 35 1.6 0.62 1 0% 35
Note. Total 15 experts participated in the rst round, while the 14 experts participated in the second round. At the rst round, there 24 items met the criteria of
consensus and 11 items did not meet. At the second round, there 33 items met the criteria of consensus and 2 items did not meet. * shown that did not meet the criteria
of consensus.
SD: standard deviation; IQR: inter-quartile range; ≧4 Ratio: The numbers was 4 points and above / total numbers ×100%.
Table 3
The result of the third round of expert opinion collection.
Elements Mean SD Q1 Q3 IQR ≧4 No ≧4 Ratio
Social network 3.10 0.86 3.00 4.00 1.00 5 36%
Virtual reality 2.20 0.69 2.00 3.00 1.00 0 0%
Note. Total 14 experts participated in the third round. Here, the 2 items which
did not meet at the second round were be met the criteria of consensus at the
third round.
SD: standard deviation; Q1: rst quartile; Q3: third quartile; IQR: inter-quartile
range; ≧4 No: The numbers was 4 points and above; ≧4 Ratio: The numbers was
4 points and above / total numbers ×100%.
Y.-F. Wang et al.

Entertainment Computing 40 (2022) 100463
6
J: “Set up a team-based competition, let the students compete to
answer questions. …this measure brings about healthy competition
and establishes a positive, autonomous cycle of team motivation.”
G: “Competition, especially team competition, can stimulate
learners’ degree of participation and enhance concentration. Indi-
vidual competition may evolve into a situation where some people
become very active while others slowly step back into the role of
“observers”.
This type of social oriented ability is also a category often seen in
gamication [34]. Our experts found that a gamication design prin-
ciple based on the concept of team competition would contribute to
gamication of training in a corporate setting.
(4) Clear and fair game rules
It is evident from the experts’ feedback that clear and simple “game
rules” is recommended (mean =4.90, SD =0.26, IQR =0.00, 100% of
experts viewed it as important or very important, see Table 2). “Fair-
ness” of the game (mean =4.70, SD =0.59, IQR =0.00, 93% of experts
viewed it as important or very important, see Table 2) is one of the keys
of training gamication implementation. Experts stated that:
I: “The design should consider fairness, timeliness, transparency, and
clear rules for the scoring system”
J: “The game rules must be very easy to understand, and any games
requiring too much time for explanation are really inappropriate.”
J:“When the fairness of the game is undermined, it is easy for some
learners to choose to ‘give up participation’ in the process.”
Without rules and limitations, games are just simple enjoyment [60].
For training gamication, rules are part of the key elements [61], and
fairness is an important link of game rules, as unfair games will cause
participants to become dissatised [16]. Our experts suggested that one
of the key elements of gamication was to determine how to motivate
learner participation through good rule design, taking fairness into
consideration while encouraging certain desirable behaviors from the
students.
(5) Increasing level of difculty in challenging goals
“Task with challenging goals” (mean =4.50, SD =0.49, IQR =1.00,
100% of experts viewed it as important or very important, see Table 2) is
an important element in training gamication design. During the pro-
cess of training gamication, it is possible to design different tasks
within the training course to let learners participate in the resolution.
Expert opinions also believed that “increasing difculty” (mean =4.30,
SD =0.48, IQR =1.00, 100% of experts viewed it as important or very
important, see Table 2) and “time pressure” (mean =4.30, SD =0.45,
IQR =0.75, 100% of experts viewed it as important or very important,
see Table 2) is an important element. This indicates that as the lessons
move forward, the task’s level of difculty may gradually increase and
the learners may be asked to complete the tasks within the time limit.
Experts stated that:
K: “Give learners challenges and ask them to come up with a solution
within a limited timeframe, and encourage team competition.”
I: “Gamication must have layers of difculties, from the easiest to
the hardest.”
G: “Effectively control the time from the perspectives of curriculum
design, adjustment of level of difculty, and rule descriptions.”
Difcult yet concrete goals can motivate people to take actions,
stimulate their performance level in pursuit of strategies for improve-
ment, which may be explained by the theories of self-regulation [19,11].
Our experts also suggested that as the skill sets of learners increased, a
gradual increment of task difculties helped trigger a state of ow and
creative happiness for the learners, optimizing user experience by
matching the challenges with upgraded abilities of the users.
(6) Experiential activities and freedom to fail
While conducting training gamication, experts also introduced
“experiential activities” to replace traditional descriptive teaching
(mean =4.30, SD =0.48, IQR =1.00, 100% of experts viewed it as
important or very important, see Table 2). Also, “freedom to fail” is an
important element to success (mean =4.10, SD =0.77, IQR =1.00, 79%
of experts viewed it as important or very important, see Table 2). Experts
stated that:
M: “Within the context of a game, experience the journey from non-
cooperation to cooperation.”
G: “Allow learners to try their best in a relatively safe environment –
thus it is very important to have low risk of failure. Design questions
without standard answers so that everyone can speak freely. Other-
wise, the trainer’s attitude and situational control skills must be
excellent such that students are not embarrassed when they answer
incorrectly.”
Through training gamication design, learners may look upon fail-
ures as opportunities; when they do not succeed at rst, they feel free to
make multiple attempts and practice repeatedly [63]. Our experts sug-
gested that the integration of experiential activities with curriculum
goals, the provision of meaningful feedback, and a risk-free learning
environment would be the key elements of gamication in corporate
training.
4.2. Category II: Game mechanics
Regarding gamication elements, many prior studies considered
points, badges, leaderboard as the most common game mechanics
[9,17,31,34]. For this study, which conducted interviews with experts of
corporate training, some key elements identied by the experts echoed
previous research results, while other expert-identied elements
differed from prior studies.
(1) Points
The rst element is points. Experts in this study had a high degree of
support for using gamication elements such as “points or scoring” in
the training course (mean =4.40, SD =0.62, IQR =1.00, 93% of experts
viewed it as important or very important, see Table 2). The experts
stated that:
M: “Each unit has points, which may be points for an interactive unit,
or discussions or case exercises. Each of these tasks would have
different points.”
E: “What I encourage is participation. What I encourage is not cor-
rectness. Wrong answers also get 100 points, but right answers get
300 points. My approach is to reward participation as well as correct
answers.”
This echoes previous research conclusions. Points may build a
connection between efforts, performance and results, thereby strength-
ening related behaviors [62]. Through attendance, participation in
tasks, students receive points which may further affect the students’
level of engagement [16,25] and enhance their learning results [17].
Care should be taken that the points themselves are not rewards, but
serve as the guidelines and reminders to achieving the training goals
[35]. Students should consider points or other gamication elements as
valuable or having psychological signicance, otherwise such measures
would not increase student participation [17]. Our experts suggested
that the use of a scoring mechanism would play an important role in
training gamication by enhancing learning effects while heightening
student reactions and expectations.
Y.-F. Wang et al.

Entertainment Computing 40 (2022) 100463
7
(2) Badges
Even though prior studies have identied badges as a part of gami-
cation mechanism, the importance of “badges” were not supported by
corporate training experts. After two rounds of evaluations by the ex-
perts, only 14% of the experts considered it important (mean =3.0, SD
=0.64, IQR =0.75, 14% of experts viewed it as important or very
important, see Table 2). Most experts in this study believed that badges
were not an important element to training gamication. After the
completion of three rounds of Delphi method research, experts were
invited to provide supplementary comments about the application of
badges. The experts stated that:
N: “Many games use badges, but I think they are more suitable for
educational games that are larger in size and more time-consuming,
in order to provide a driving force like “stamp collection” so that
players will continue to play and add to their collections. There are
very few large-size and continuous educational games like that in a
training environment.”
D: “Badges are usually for games where the users may upgrade their
ranks. For short-term courses, to explain clearly the rules for
upgrading ranks and badges to the learners may end up confusing
them instead.
F: “The essence of badges is the same as bonus points, which is really
a bonus point mechanism that has been exquisitely packaged and
looks nicer. Perhaps it would attract teenagers or younger students.
Since the cost of implementing badges is relatively low, trainers may
be particularly attracted to them [17]. Our experts found that the use of
badges in short-term training courses (e.g. duration of 1–3 days) would
not be effective gamication as the impact to adult learners would be
insignicant. In addition, badges used frequently in software supple-
mented gamication systems would not be easily applicable to most
corporate trainings conducted with face-to-face teaching. This is one
area for further research in the future.
(3) Leaderboard
The level of importance of the “leaderboard” changed in an inter-
esting manner through the course of this study. At the rst round of
expert evaluation, even though it received a mean value of 4.20 with
67% of the experts viewing it as important or very important, it did not
meet the consensus standard criteria (SD =0.91, IQR =2.00, see
Table 2). At the second round, consensus was reached (SD =0.67, IQR =
0.00, see Table 2), but its importance mean value dropped to 3.8, and
the ratio of experts viewing it as important or very important grew to
nearly 80% (see Table 2). The survey process showed that even though
many experts supported the use of leaderboard and believed it can
strengthen competitiveness and promote student participation, some
experts felt that excessive emphasis on competition would adversely
affect learning. Experts stated their very different views on the impor-
tance of leaderboard as follows:
A: “The score board is right in the front. Using the team scoring
method, the scores are announced and updated during each
intermission.”
L: “I personally do not believe in the importance of competition. In
some classes, I would even purposely minimize competition: we
would not have any rankings, and no reward mechanism either, not
even verbal complements.”
E: “The use of competition, is that actually good or bad? In order to
score points, we may even lose a little of the essence of learning. Of
course, we want to keep learning as the main objective, and not have
scoring points as the main task.”
Past research showed that in the process of training gamication, the
points and leaderboard mechanism can operate appropriate
competitiveness, but the effects of the leaderboard were mixed [25]. If
competition is over-emphasized, it could lower participants’ motivation
[24], or make students uncomfortable [16]. Our experts found that
appropriate application of the element of competition while empha-
sizing the essence of learning without excessive competition would be
important to training gamication.
(4) Other ndings
For corporate educational training, storylines or plots may be
interwoven in the process, but the importance of stories varied with our
experts. Some experts used stories as introductions prior to the begin-
ning of the teaching process or during case discussions. Other experts,
meanwhile, believed that stories were an important component to
attracting the students’ attention. Regarding the importance of a
storyline, the divergent views of the experts may be due to their indi-
vidual application experience and different focus in terms of course
planning. Since this study utilized the Delphi method, the number of
expert samples was relatively small. If the expert samples came from
different groups with different teaching experiences, the importance of a
storyline may change. This aspect may require further research in the
future. Moreover, the experts in this study seldom relied on the assis-
tance of information software systems or apps, possibly because the use
of software could increase teaching variables, distracting the attention
of the students or increasing the level of difculties. The experiences of
the experts in the study seemed to indicate that the use of software was
not necessary during the gamication of the teaching process – these
experiences may offer a point of reference for other instructors.
5. Conclusion
Many studies have shown that training gamication effectively en-
hances learners’ interest and participation. However, ambiguous gami-
cation directions may lead to erroneous results. This study used the
Delphi method to evaluate the importance of key gamication elements
according to 14 corporate training experts. Expert interviews and ap-
plications of gamication design principles/mechanics were also pre-
sented as a reference for future gamication research or application.
According to our experts, some gamication design principles would
enhance learner participation and learning results in corporate training.
The principles include: integration of gamication with the curriculum,
provision of rapid feedback when learners exhibit desirable behavior, a
team-based competition mechanism, and a gradual increase in the dif-
culties of tasks to be completed by the teams in a limited timeframe.
Also, immersion learning would result given the following: clear and fair
game rules, risk-free experiential activities allowing the freedom to fail,
and a training environment tolerant of trials and errors.
Secondly, our experts often relied on points and leaderboards as
game mechanics, planning different scoring components in the course.
When the learners exhibited desirable behaviors, such as answering
questions, participating in discussions, or taking on difcult challenges,
they would score points for accuracy as well as participation. Further-
more, leaderboards were used to display cumulative team scores, which
may boost participation by displaying the progress of the learners and
their peers.
Thirdly, our experts found that the gamication of training was not
about playing games in the class, and did not require the use of software
applications or board games as props. Several experts also mentioned:
“Do not gamify for the sake of gamication”. Thus, it is important to
focus on the design of key gamication elements to enhance learning
effectiveness for research and application in the future.
5.1. Limitations and future research
There are some limitations in this study. First, the number of
participating experts was limited, thus further research is required to see
Y.-F. Wang et al.

Entertainment Computing 40 (2022) 100463
8
whether these opinions are sufcient to represent a wide range of
educational gamication. Moreover, most of the gamication course
material were used by enterprise employees. When the intended target
users are students in an academic setting or elsewhere, different effects
may result. Also, corporate training sessions are often special seminars
or short-term classes, unlike school classes which may last a semester or
more. With different teaching timeframe, should gamication be
implemented differently? Furthermore, experts that implement training
gamication may not fully understand the underlying gamication
theories and research, which could limit their gamication design or
usage of elements. The individual course subject or practical experience
of each expert may reect their preference of different instructional
methods for training gamication. For example, some experts preferred
the use of rewards or storylines, while others viewed these mechanisms
as less important. Such different perspectives in the qualitative results
could be due to the selection and recruitment process to nd expert
participants for this study. Even with such research limitations, this
study integrated the practical experiences of the experts as well as ac-
ademic research, in hopes of offering some valuable references in the
research and application of training gamication.
Funding
The authors did not receive any funding for this study.
Declaration of Competing Interest
The authors declare that they have no known competing nancial
interests or personal relationships that could have appeared to inuence
the work reported in this paper.
Acknowledgements and Disclosures
None.
References:
[1] S. Deterding, M. Sicart, L. Nacke, K. O’Hara, D. Dixon, Gamication: using game-
design elements in non-gaming contexts, in: Proceedings of the 2011 CHI
Conference on Human Factors in Computing, ACM, Vancouver, BC, Canada, 2011,
pp. 2425-2428.
[2] K. Huotari, J. Hamari, Dening gamication: a service marketing perspective, in:
Proceedings of the 16th International Academic MindTrek Conference, ACM,
Tampere, Finland, 2012, pp. 17-22.
[3] S. Deterding, D. Dixon, R. Khaled, L. Nacke, From game design elements to
gamefulness: dening gamication, in: Proceedings of the 15th International
Academic MindTrek Conference: Envisioning Future Media Environments, ACM,
Tampere, Finland, 2011, pp. 9-–15.
[4] J. Hamari, V. Lehdonvirta, Game design as marketing: How game mechanics create
demand for virtual goods, Int. J. of Bus. Sci. & Appl. Manage. 5 (1) (2010) 14-29,
http://ssrn.com/abstract=1443907.
[5] M. Vesa, J.T. Harviainen, Gamication: concepts, consequences, and critiques,
J. Manage. Inquiry. 28 (2) (2019) 128–130, https://doi.org/10.1177/
1056492618790911.
[6] L.C. Stanculescu, A. Bozzon, R.-J. Sips, G.-J. Houben, Work and play: an
experiment in enterprise gamication, in: Proceedings of the 19th ACM Conference
on Computer-Supported Cooperative Work and Social Computing, ACM, San
Francisco, CA, USA, 2016, pp. 346-358.
[7] R.N. Landers, Developing a theory of gamied learning: Linking serious games and
gamication of learning, Simul. & Gaming. 45 (6) (2014) 752–768, https://doi.
org/10.1177/1046878114563660.
[8] A. Anderson, D. Huttenlocher, J. Kleinberg, J. Leskovec, Steering user behavior
with badges, in: Proceedings of the 22nd International Conference on World Wide
Web, ACM, Rio de Janeiro, Brazil, 2013, pp. 95-106.
[9] D. Dicheva, C. Dichev, G. Agre, G. Angelova, Gamication in education: a
systematic mapping study, Educ. Technol. Soc. 18 (3) (2015) 75–88. https://www.
jstor.org/stable/10.2307/jeductechsoci.18.3.75.
[10] S. De Sousa Borges, V.H. Durelli, H.M. Reis, S. Isotani, A systematic mapping on
gamication applied to education, in: Proceedings of the 29th Annual ACM
symposium on Applied Computing, 2014, pp. 216–222.
[11] R.N. Landers, E.M. Auer, A. Helms, S. Marin, M.B. Armstrong, Gamication of
adult learning: gamifying employee training and development, in: R.N. Lamders
(Ed.), Cambridge Handbook of Technology and Employee Behavior, Cambridge
University Press, Cambridge, 2019, pp. 271–295.
[12] P. Denny, The effect of virtual achievements on student engagement, in:
Proceedings of 2013 Association for Computing Machinery’s Special Interest Group
on Computer Human Interaction Conference, Paris, France, 2013, pp. 763-772.
[13] S. Smith-Robbins, This game sucks: how to improve the gamication of education.
Educau. Rev. 46 (1) (2011) 58–59, https://er.educause.edu/-/media/les/article-
downloads/erm1117.pdf.
[14] M. Burke, T. Hiltbrand, How gamication will change business intelligence. Bus.
Intell. J. 16 (2) (2011) 8–16, http://citeseerx.ist.psu.edu/viewdoc/download?
doi=10.1.1.476.6771&rep=rep1&type=pdf.
[15] J. Hamari, J. Koivisto, H. Sarsa, Does gamication work? a literature review of
empirical studies on gamication, in: Proceedings of the 47th Hawaii International
Conference on System Sciences, Waikoloa, HI, USA, 2014, pp. 3025-3034.
[16] L.A. Ribeiro, T.L. da Silva, A.Q. Mussi, Gamication: a methodology to motivate
engagement and participation in a higher education environment, Inter. J. of Educ.
& Res. 6 (4) (2018) 249-264, https://www.ijern.com/journal/2018/April-2018/
21.pdf.
[17] M.B. Armstrong, R.N. Landers, Gamication of employee training and
development, Int. J. Train. & Dev. 22 (2) (2018) 162–169, https://doi.org/
10.1111/ijtd.12124.
[18] D.R. Michael, S.L. Chen, Serious Games: Games that Educate, Train, and Inform,
Muska & Lipman/Premier-Trade, New York, 2005.
[19] R. Garris, R. Ahlers, J.E. Driskell, Games, motivation, and learning: a research and
practice model, Simul. Gaming. 33 (4) (2002) 441–467, https://doi.org/10.1177/
1046878102238607.
[20] D. Nieborg, America‟ s Army: more than a game, in: Proceedings of the 35th
Conference of the International Simulation and Gaming Association, Munich,
Germany, 2004.
[21] C. Perryer, B. Scott-Ladd, C. Leighton, Gamication: implications for workplace
intrinsic motivation in the 21st century, Asian Forum on Bus. Educ. J. 5 (3) (2012)
371–381. https://doi.org/10.14456/afbe.2012.13.
[22] B. Sawyer, P. Smith, Serious games taxonomy, in: Proceedings of the Serious
Games Summit at the Game Developers Conference, San Francisco, CA, USA, 2008.
[23] T.M. Connolly, E.A. Boyle, E. MacArthur, T. Hainey, J.M. Boyle, A systematic
literature review of empirical evidence on computer games and serious games,
Comput. Educ. 59 (2) (2012) 661–686. https://doi:10.1016/j.compedu.2012.0
3.004.
[24] I. Furdu, C. Tomozei, U. Kose, Pros and cons gamication and gaming in classroom,
Broad Res. Arti Intell. Neurosci. 8 (2017) 56–62. https://doi:10.5281/zenodo.
1045364.
[25] T.A. Papp, A. Theresa, Gamication effects on motivation and learning:
Application to primary and college students, Int. J. Cross-Discip. Subj. in Educ. 8
(3) (2017) 3193–3201, http://doi.org/10.20533/ijcdse.2042.6364.2017.0428.
[26] B.M. Sørensen, S. Spoelstra, Play at work: continuation, intervention and
usurpation, Organ. 19 (1) (2012) 81–97, https://doi.org/10.1177/
1350508411407369.
[27] G. Barata, S. Gama, J. Jorge, D. Gonçalves, Studying student differentiation in
gamied education: a long-term study, Comput. Hum. Behav. 71 (2017) 550–585,
https://doi.org/10.1016/j.chb.2016.08.049.
[28] J. Hamari, J. Koivisto, Why do people use gamication services? Int. J. Inf.
Manage. 35 (4) (2015) 419–431, https://doi.org/10.1016/j.
ijinfomgt.2015.04.006.
[29] R.N. Landers, M.B. Armstrong, A.B. Collmus, How to use game elements to enhance
learning: applications of the theory of gamied learning, in: M. Ma, A. Oikonomou,
L.C. Jain (Eds.), Serious Games and Edutainment Applications, Springer, New York,
2017, pp. 457–483.
[30] R.N. Landers, K.N. Bauer, R.C. Callan, M.B. Armstrong, in: Psychological theory
and the gamication of learning, Gamication in Education and Business, Springer,
New York, 2015, pp. 165–186.
[31] K. Seaborn, D.I. Fels, Gamication in theory and action: a survey, Int. J. Hum.
Comput. Stud. 74 (2015) 14–31, https://doi.org/10.1016/j.ijhcs.2014.09.006.
[32] F.F.-H. Nah, Q. Zeng, V.R. Telaprolu, A.P. Ayyappa, and B. Eschenbrenner,
Gamication of education: a review of literature, in: Proceedings of the
International Conference on Human–Computer Interaction in Business, Springer,
Heraklion, Crete, Greece, 2014. pp. 401-409.
[33] M.B. Armstrong, R.N. Landers, An evaluation of gamied training: using narrative
to improve reactions and learning, Simul. Gaming 48 (4) (2017) 513–538, https://
doi.org/10.1177/1046878117703749.
[34] J. Majuri, J. Koivisto, J. Hamari, Gamication of education and learning: a review
of empirical literature, in: Proceedings of the 2nd International GamiFIN
Conference, GamiFIN, Pori, Finland, 2018.
[35] C. Perryer, N.A. Celestine, B. Scott-Ladd, C. Leighton, Enhancing workplace
motivation through gamication: transferrable lessons from pedagogy, Int. J.
Manage. Educ. 14 (3) (2016) 327–335, https://doi.org/10.1016/j.
ijme.2016.07.001.
[36] K.M. Kapp, The Gamication of Learning and Instruction, John Wiley & Sons Inc,
San Francisco, 2012.
[37] R.N. Landers, K.N. Bauer, R.C. Callan, Gamication of task performance with
leaderboards: a goal setting experiment, Comput. Hum. Behav. 71 (2017) 508–515,
https://doi.org/10.1016/j.chb.2015.08.008.
[38] R.M. Ryan, C.S. Rigby, A. Przybylski, The motivational pull of video games: a self-
determination theory approach, Motiv. & Emot. 30 (4) (2006) 344–360, https://
doi.org/10.1007/s11031-006-9051-8.
[39] M.D. Hanus, J. Fox, Assessing the effects of gamication in the classroom: a
longitudinal study on intrinsic motivation, social comparison, satisfaction, effort,
and academic performance, Comput. & Educ. 80 (2015) 152–161, https://doi.org/
10.1016/j.compedu.2014.08.019.
Y.-F. Wang et al.

Entertainment Computing 40 (2022) 100463
9
[40] N. Dalkey, O. Helmer, An experimental application of the Delphi method to the use
of experts, Manag. Sci. 9 (3) (1963) 458–467, https://doi.org/10.1287/
mnsc.9.3.458.
[41] C. Okoli, S.D. Pawlowski, The Delphi method as a research tool: an example, design
considerations and applications, Inf. & Manage. 42 (1) (2004) 15–29, https://doi.
org/10.1016/j.im.2003.11.002.
[42] G.J. Skulmoski, F.T. Hartman, J. Krahn, The Delphi method for graduate research,
J. Inf. Technol. Educ. Res. 6 (1) (2007) 1–21. https://doi.org/10.28945/199.
[43] W. Vernon, The Delphi method: a review, Int. J. Ther. Rehab. 16 (2) (2009) 69–76.
https://doi.org/10.12968/ijtr.2009.16.2.38892.
[44] G. Rowe, G. Wright, The Delphi technique as a forecasting tool: issues and analysis,
Inter. J. Forecast. 15 (4) (1999) 353–375, https://doi.org/10.1016/S0169-2070
(99)00018-7.
[45] L. Giannarou, E. Zervas, Using Delphi technique to build consensus in practice.
Inter. J. of Bus. Sci. & Appl. Manage. 9 (2) (2014) 65-82, https://doi.org/10.5430/
jha.v3n4p1.
[46] J. Landeta, Current validity of the Delphi method in social sciences, Technol.
Forecast. & Soc. Change. 73 (5) (2006) 467–482, https://doi.org/10.1016/j.
techfore.2005.09.002.
[47] T. Shields, G. Silcock, H. Donegan, Y. Bell, Methodological problems associated
with the use of the Delphi technique, Fire Technol. 23 (3) (1987) 175–185, https://
doi.org/10.1007/BF01036934.
[48] N.C. Dalkey, The Delphi method: an experimental study of group opinion, in: D.
R. Lewis, D. Snyder (Eds.), Studies in the Quality of Life: Delphi and Decision-
Making, Lexington Books, Lexington, 1972, pp. 13–54.
[49] P. Kittell-Limerick, Perceived Barriers to Completion of the Academic Doctorate: A
Delphi Study, Texas A and M University Commerce, Bryan, 2005.
[50] M. Adler, E. Ziglio, Gazing into the Oracle: The Delphi Method and its Application
to Social Policy and Public Health, Jessica Kingsley Publishers, London, 1996.
[51] J. Greatorex, T. Dexter, An accessible analytical approach for investigating what
happens between the rounds of a Delphi study, J. Adv. Nurs. 32 (4) (2000)
1016–1024, https://doi.org/10.1046/j.1365-2648.2000.t01-1-01569.x.
[52] I.R. Diamond, R.C. Grant, B.M. Feldman, P.B. Pencharz, S.C. Ling, A.M. Moore, P.
W. Wales, Dening consensus: a systematic review recommends methodologic
criteria for reporting of Delphi studies, J. of Cli. Epide. 67 (4) (2014) 401–409,
https://doi.org/10.1016/j.jclinepi.2013.12.002.
[53] D. Ramos, P. Arezes, P. Afonso, Application of the Delphi Method for the inclusion
of externalities in occupational safety and health analysis, DYNA 83 (196) (2016)
14–20. https://dx.doi.org/10.15446/dyna.v83n196.56603.
[54] L.E. Nacke, C.S. Deterding, The maturing of gamication research, Comput. Hum
Behav. (2017) 450–454, https://doi.org/10.1016/j.chb.2016.11.062.
[55] A. Fink, J. Kosecoff, How to Conduct Surveys: A Step by Step Guide, Sage
Publications Inc., Newbury Park, 2015.
[56] A.L. Delbecq, A.H. Van de Ven, D.H. Gustafson, Group techniques for program
planning: a guide to nominal group and Delphi processes, Scott Foresman & Co.,
Glenview, 1975.
[57] G. Zichermann, C. Cunningham, Gamication by Design: Implementing Game
Mechanics in Web and Mobile Apps, O’Reilly Media Inc, Sebastopol, 2011.
[58] J. Antin, E.F. Churchill, Badges in social media: a social psychological perspective,
in: Proceedings of the 2011 CHI Conference on Human Factors in Computing
Systems, ACM, Vancouver, BC, Canada, 2011, pp. 1-4.
[59] J. Juul, A Casual Revolution: Reinventing Video Games and Their Players, MIT
Press, Cambridge, 2010.
[60] T. Fullerton, Game Design Workshop: A Playcentric Approach to Creating
Innovative Games, CRC Press, Boca Raton, 2014.
[61] R.W. Helms, R. Barneveld, F. Dalpiaz, A method for the design of gamied
trainings, in: Proceedings of the Pacic Asia Conference on Information Systems,
Singapore, 2015, pp. 59.
[62] L. Von Ahn, L. Dabbish, Designing games with a purpose, Commun. ACM. 51 (8)
(2008) 58–67, https://doi.org/10.1145/1378704.1378719.
[63] Kiryakova, G., Angelova, N., Yordanova, L. Gamication in education, in:
Proceedings of the 9th International Balkan Education and Science Conference,
Turkey, 2014.
Y.-F. Wang et al.