Content uploaded by Mohammed Saqr
Author content
All content in this area was uploaded by Mohammed Saqr on Aug 04, 2021
Content may be subject to copyright.
Computers & Education 173 (2021) 104296
Available online 31 July 2021
0360-1315/© 2021 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license
(http://creativecommons.org/licenses/by-nc-nd/4.0/).
Two decades of game concepts in digital learning environments –
A bibliometric study and research agenda
Soa Sch¨
obel
a
, Mohammed Saqr
b
,
d
, Andreas Janson
c
,
*
a
University of Kassel, Information Systems, Research Center for IS Design (ITeG), Pfannkuchstraße 1, 34121, Kassel, Germany
b
School of Computing, University of Eastern Finland, Joensuu Campus, Yliopistokatu 2, P.O. Box 111, -80100, Joensuu, Finland
c
University of St. Gallen, Institute of Information Management, Müller Friedbergstrasse 8, 9000, St. Gallen, Switzerland
d
School of Electrical Engineering and Computer Science, Media Technology & Interaction Design, KTH Royal Institute of Technology, Lindstedtsv¨
agen
3, SE-100 44 Stockholm, Sweden
ARTICLE INFO
Keywords:
Games
Simulations
Mobile learning
Gamication
Game concepts
Bibliometric study
ABSTRACT
In recent years, using game concepts for educational purposes in digital environments has become
continually more popular and relevant. Games can be used to motivate and engage users in
regular system use and, in the end, support learners in achieving better learning outcomes. In this
context, different kinds of game concepts exist, such as gamication or serious games, each with a
different perspective and usefulness in digital learning environments. Because developing and
using with game concepts in digital learning environments has recently become more important,
and developing them is still not fully established, questions arise about future research directions
involving games in digital learning. Therefore, this study aims to identify the state of the eld and
determine what is relevant when using game concepts in digital learning. To achieve this goal, we
present the results of a bibliometric analysis considering more than 10,000 articles between 2000
and 2019 and summarize them to develop a research agenda. This agenda supports researchers
and practitioners in identifying avenues for future research. We contribute to theory by providing
a detailed understanding of the relevance of game concepts in digital learning. We propose a
research agenda to assist researchers in planning future approaches with and about gamication
concepts in digital learning. Practical implications are proposed by demonstrating what should be
considered when using game concepts in learning environments.
1. Introduction
Gaming is a ubiquitous part of everyday life (Huizinga, 1949). Using game concepts has increased at an astounding pace and has
inspired trends, such as gamication and serious games (Hamari et al., 2016; Wouters, van Nimwegen, van Oostendorp, & Van Der
Spek, 2013). As game concepts have gained popularity, various well-established research elds have been using with games, including
human-computer interaction, information systems, and psychology (Burgers, Eden, van Engelenburg, & Buningh, 2015; Hamari &
Koivisto, 2015; Seaborn & Fels, 2015). Considering concepts from games is especially relevant for the digital learning domain, which
has received recognition from a multitude of high-impact publications and gained importance over the last decade (Cruz, Hanus, &
Fox, 2017; Liu, Li, & Santhanam, 2013; Mekler, Brühlmann, Tuch, & Opwis, 2017; Santhanam, Liu, & Milton-Shen, 2016). In digital
* Corresponding author.
E-mail addresses: soa.schoebel@uni-kassel.de (S. Sch¨
obel), mohammed.saqr@uef. (M. Saqr), andreas.janson@unisg.ch (A. Janson).
Contents lists available at ScienceDirect
Computers & Education
journal homepage: www.elsevier.com/locate/compedu
https://doi.org/10.1016/j.compedu.2021.104296
Received 8 November 2020; Received in revised form 22 July 2021; Accepted 24 July 2021
Computers & Education 173 (2021) 104296
2
learning environments, learners must self-regulate and monitor their own learning process (Wong, Khalil, Baars, Koning, & Paas,
2019). The notion in this context is that game elements in digital learning support learners in engaging in their learning activities on a
more regular basis (Domínguez et al., 2013) in order to ultimately achieve higher learning outcomes.
The use and effects of using games in digital learning have been extensively discussed in, for example, published meta-analyses
(Bai, Hew, & Huang, 2020; Putz, Hofbauer, & Treiblmaier, 2020; Zainuddin, Chu, Shujahat, & Perera, 2020). Nonetheless, ques-
tions arise about what to consider in future research to better understand how we can work with game concepts in digital learning to
make them more effective (Super, Keller, Betts, & Roach Humphreys, 2019). Discussing and debating research that considers games
has changed our perspective, allowing us to think in more detail about the effectiveness of game concepts, gaining a deeper under-
standing of how game concepts trigger engagement and evaluating if and how we can make game concepts more intelligent (Hamari
et al., 2016; Sch¨
obel et al., 2020). The volume of research on game concepts illustrates new trends and developments in design and
game concepts in digital learning, especially in how learners differ in their competitive behavior or in their goal orientation (San-
thanam et al., 2016; Super et al., 2019), underscoring the need for a comprehensive analysis of past research efforts leading to an
overview of future research directions.
The goal of our study is to provide an overview of the current state of the eld of game concepts in digital learning to provide future
research directions. To summarize the results of existing research and provide an overview of how this research area has developed so
far, most studies refer to literature reviews, generic and scoping reviews, or meta-analyses (Chen, Shih, & Law, 2020; Gao, Li, & Sun,
2020; Liu, Moon, Kim, & Dai, 2020). Typically, reviews focus on a limited and analyzable number of studies, and meta-analyses discuss
the effects of specic variables on an outcome of interest. Bibliometric studies complement these insights by enabling a holistic
synthesis of research streams and describing the distribution of patterns of research articles on a given topic over a given period, thus
also being able to identify trends for emerging and future research areas. In our study, we focus on the following research questions
(RQ):
RQ1: Which are the most relevant and cited journals and publications relating to game concepts in digital learning that provide a starting
point in identifying high-impact research in the eld?
RQ2: What are the major research streams of game concepts in digital learning literature?
RQ3: Which directions for future research can be offered to researchers to further explore the eld of game concepts in digital learning?
To answer our research questions, we consider research studies from nearly two decades and present a detailed keyword analysis
and research agenda for future research projects, highlighting current developments and trends concerning game concepts in digital
learning environments. We contribute to theory by presenting a summarized overview of the existing research on game concepts in
digital learning. We develop an agenda to assist researchers in aligning and exploring the future of game concepts in digital education.
We further contribute to theory by providing an overview of the development of different keyword clusters to understand where
research involving game concepts is heading in digital learning, enabling us to explore the concepts in more detail and with a different
level of understanding. With this study, we support practitioners and provide implications for how to use game concepts in digital
learning environments.
2. Theoretical background
2.1. Game concepts in digital learning
Generally speaking, we use games for fun and entertainment (Werbach & Hunter, 2012). Games today are an essential part of
everyday life (Lamb, Annetta, Firestone, & Etopio, 2018). The way we think about games reects the status games have in society,
meaning that they are included in daily life, but more as a secondary activity. Therefore, people do not pay much attention to the actual
signicance of this concept and its impact on other areas of life, for example, economics and psychology.
Due to the effectiveness of games in our private lives, the concept has been transferred to other parts of our lives, such as work and
digital learning. To better understand how we can transfer games to a serious context, such as digital learning, research typically relies
on two dimensions: gaming versus playing and parts versus the whole (Deterding, Dixon, Khaled, & Nacke, 2011a). Fig. 1 visualizes the
Fig. 1. Focus of this study based on Deterding et al. (2011a)
S. Sch¨
obel et al.
Computers & Education 173 (2021) 104296
3
four classes of games and highlights the focus of the present study.
In this study, we focus on game concepts (upper part of Fig. 1) while considering both the whole of game concepts as well as their
parts. We neglect playing (lower part) as something done just for fun, therefore falling beyond the scope of more serious contexts such
as learning and education, and focus on game concepts, namely serious games and gamication. These major concepts consider the
parts of gaming or the whole, that is, determining if only game elements are considered in a given context or if a complete game world
is used for a purpose. In considering the latter—a whole game that has more deliberate context—we refer to the concept of serious
games. A serious game can be dened as a “mental contest, played with a computer in accordance with specic rules, that uses
entertainment to further government or corporate training, education, health, public policy, and strategic communication objectives”
(Zyda, 2005, p. 26). Serious games are those in which the primary goal is education rather than entertainment (Landers, 2015). In
contrast, gamication (in other words, a game-like design) refers to single parts of a game that are used in a gaming context. Deterding,
Dixon, Khaled, and Nacke (2011a) dene gamication as an informal umbrella term for the use of video game elements in non-gaming
systems to improve user experience and user engagement (Deterding, Sicart, et al., 2011b). In gamication research, game elements
can be described and categorized using the mechanics, dynamics, and aesthetics (MDA) framework, which suggests categorizing game
elements into mechanics, dynamics, and aesthetics (Hunicke, LeBlanc, & Zubek, 2004). Mechanics are the functioning components of
the game; dynamics, on the other hand, are described as the user’s interactions with those mechanics; and aesthetics are characterized
as the emotional triggers felt by the player during the interaction in a given context (Hunicke et al., 2004). To sum up, we therefore
consider both serious games and gamication in our study and will refer to them as game concepts, dened as “concepts that are used
in digital environments and that consider parts of a game or a whole game in non-gaming contexts,” throughout the paper.
2.2. The relevance of game concepts in digital learning and avenues for future research
The relevance of game concepts derives from its purpose. Game concepts in digital learning are used to make learning more
engaging and more effective, promising better learning outcomes through engaging and motivating learning processes (Christy & Fox,
2014). Toward those ends, learning with games has two purposes (Ib´
anez, Di-Serio, & Delgado-): rst to encourage desired learning
behavior, and second, to engage users in learning through the use of materials and methods (tutorials, quizzes, videos, or other digital
documents) presented in digital learning environments (Bedwell, Pavlas, Heyne, ; Davis & Singh, 2015; De-Marcos, Garcia-Lopez, &
Garcia-Cabot, 2016; Hanus & Fox, 2015). Game concepts can support learners in recognizing that learning can be fun and entertaining,
which can stimulate them to perform better (Sailer & Homner, 2019).
To evaluate the effectiveness of game concepts in digital learning, research often relies on experiments or surveys (Hanus & Fox,
2015; Katsaliaki, 2013). To summarize and assess how the use of game concepts in digital learning has developed over time and to
identify avenues for future research, previous studies have typically relied on literature reviews, meta-analyses, or bibliometric ap-
proaches. Although literature reviews are an important instrument for summarizing existing literature, they are often limited by the
number of studies they comprise. Studies that provide a broader perspective on the development of topics follow a meta-analytic or
bibliometric approach. Both incorporate the use of statistical methods but with different aims. While meta-analytic approaches focus
on highlighting the strength (or absence) of effects of independent on dependent variables, bibliometric studies have become a popular
approach to discover patterns in collected knowledge and work that reveal emerging trends in research as it evolves (Trinidad, Ruiz, &
Calder´
on, 2021). We summarize the insights from a selected number of game concept studies in digital learning, that used a literature
review, meta-analysis or bibliometric approach in Appendix A.
Meta-analyses summarize existing research and help formulate conclusions about the effects of an independent variable on a
determined outcome variable. Consequently, they are supportive of better understanding an individual phenomenon of interest, but
they are limited if the intent is to gain a broader perspective on how research involving game concepts in digital learning has developed
over time. Nonetheless, they can help us rene our focus for future studies. Thus, meta-analyses related to game concept research in
digital learning call for a long-term analysis and more meticulous database research (Chian-Wen, 2014; Lamb et al., 2018), which we
address with a bibliometric study. Such research also broadens the perspective on game concepts in digital learning by considering not
only gamication but also serious games (Martí-Parre˜
no, M´
endez-Ib´
a˜
nez, & Alonso-Arroyo, 2016). In line with recommendations in
existing bibliometric studies that focus on this topic, future research is suggested that considers a longer period and that includes
proceedings along with journal publications (Martí-Parre˜
no et al., 2016; Trinidad et al., 2021). Additionally, other bibliometric studies
of learning contexts concentrate on the most prominent authors and sources in their work (Karakus, Ersozlu, & Clark, 2019). To
identify avenues for future research, bibliometric methodologies provide powerful tools, such as examining the metadata of research
papers, especially their keywords. Keywords provide more details about the development of any topic of interest, revealing more about
which kinds of outcomes have been analyzed in research. Studies conducted for a specic concept, for example gamication, without a
contextual framework, in this case learning, highlight the importance of games in education, calling for a more detailed analysis of
game concepts in digital learning (Trinidad et al., 2021).
In conclusion, this study acknowledges the recommendations of existing research and adopts a bibliometric approach to present a
holistic synthesis of research streams involving game concepts in digital learning to derive an agenda for future research avenues.
3. Methodology
The decision to rely on a bibliometric method is based on several factors. In general, bibliometric methods are concerned with the
study of scientic documents, authors, and publication venues. Bibliometrics have grown beyond frequency analysis and citation
counts to include a wide range of methods and techniques that incorporate, for example, network analysis, machine learning, advanced
S. Sch¨
obel et al.
Computers & Education 173 (2021) 104296
4
visualizations, and text mining. They allow the study of a large number of articles. Similarly, bibliometrics can help map numerous
authors and their impact, country, or institution (Abramo, D’Angelo, & Di Costa, 2011). The analysis of data was performed with R!
Programming language using the Bibliometrix library (Aria & Cuccurullo, 2017). The search was conducted on the ISI Web of Science
(WoS) database on April 6, 2020, using the search terms “game” OR “games” OR “gaming” OR “gamication” for all articles published
between 2000 and 2019, inclusive. It was performed in education and education-related research areas in the WoS database. The ISI
WoS offered a broad collection of articles, ensuring that all published articles were of good quality, and is a robust database. To ensure
that the results returned were all relevant, we included other sources
1
not included in the education collection on the WoS for several
important reasons: First and foremost, the eld of educational technology at the intersection of game concepts is rather interdisci-
plinary, for instance, as shown through seminal papers on gamication (Deterding, Dixon, et al., 2011a; Hamari, Koivisto, & Sarsa,
2014). Therefore, we included other relevant sources not covered by the initial WoS search, such as those from the information systems
discipline, because we were analyzing game concepts in digital learning (Santhanam et al., 2016). Here, interdisciplinary sources
provided extensive contributions to the elds of researching digital environments, as recommended by other studies (Wu, Hsiao, Wu,
Lin, & Huang, 2012). By expanding our search, we uncovered more reliable results (as also shown in other bibliometric studies on
gamication in general; see Martí-Parre˜
no et al. (2016). Second, by widening the scope to include major conferences explicitly,
especially those at the intersection of information systems, and computer sciences, we expected to cover a broad range of recent
research, as suggested by vom Brocke et al. (2015). Third, we included an open search (similar to the logic of a backward search, e.g.
Webster and Watson (2002)) to identify relevant sources that discuss game concept in educational contexts.
All articles matching the following inclusion criteria were included in the study:
•An empirical article matching the search keywords and time span
•A peer-reviewed original article (journal articles and conference papers)
•An article published in English to enable keyword comparison
Articles that did not match all the inclusion criteria were excluded, as were these types of publication: book chapter, letter to the
editor, editorial, and news documents. Articles containing the keywords “sport*” OR “physical educat*” OR athleti* OR “train*” were
manually inspected by reading the abstracts, and articles unrelated to digital game concepts were excluded. In the case of papers
without abstracts or with abstracts lacking clarity, we screened the full texts. The initial search yielded 11,528 records. After removing
the sports-related keywords and two duplicate records, four review articles and one editorial were excluded. The nal number of
articles was 10,273.
The data were cleaned, and author names, journals, and conferences with different spellings were checked and xed. The analysis
included descriptive statistics, in which the count of documents, authors, sources, article types, and other statistics were calculated. As
“country” is not a standard eld in the metadata provided by the WoS database, the country was extracted from the afliation of the
rst author. The Collaboration Index (CI) was calculated as the ratio of the sum of the authors of multi-authored articles to total multi-
authored articles. Annual growth rate was calculated as the average percentage increase in the number of articles. Citation analysis was
performed based on citations provided by WoS at the time of the search. The number of citations was proposed as a measure that
reects the impact of an article, author, or journal, as it is widely recognized as an important metric. However, it should be viewed with
caution and consideration—citation count is also used as a basis for other indices to reect the author’s contribution to a eld. We
calculated the H index and its variants (M index and G index). The H index is a metric that ranks the impact of an author according to
the highest number of articles that received h or more citations (Hirsch, 2005). To overcome the shortcomings of author-selected
keywords, which may be restricted by number and space and are not well standardized, similar keywords were grouped together,
for example, keywords like “game, games”; “coding, programming”; “collaboration, collaborative”.
The network of keywords was plotted using a force-directed algorithm. For the readability of relationships and labels, the network’s
plot was limited to the 30 most frequent keywords. The keywords were clustered into communities using the Louvain modularity
algorithm, an algorithm that has been shown to be computationally efcient and provides good quality community detection (Meo,
Ferrara, Fiumara, & Provetti, 2011). Keyword communities are designated by unique colors in the network plot. We also calculated
centrality measures of the keywords to assess their importance and interconnectedness with other research themes. Degree centrality
represents the total number of unique keywords to which a keyword was linked; Eigenvector centrality represents the strength of
connectedness of the connected keywords to the target and therefore represents the expressed inuence of the keyword beyond im-
mediate connections. H index and diffusion centralities were also calculated, and both were found to reect the diffusion of the
examined phenomena (Liao, Mariani, Medo, Zhang, & Zhou, 2017). The historiography map is a popular method developed by Eugene
Gareld to map the chronological direct citation network within a set of papers. The method maps the most relevant papers by ranking
1
Articles from the following journals and conferences were manually added to the education collection to ensure that the results include all
relevant articles: Computers in Human Behavior, Information Systems Research, European Journal of Information Systems, Journal of Management
Information Systems, Computers & Education, Information Systems Journal, Modern Language Journal, International Journal of Human–Computer
Studies, Simulation & Gaming, Journal of Management Inquiry, Informatics-Basel, Journal of Strategic Information Systems, Media and Commu-
nication, Journal of the Association for Information Systems, MIS Quarterly, Consumption Markets & Culture, Journal of Information Technology,
International Journal of Game-Based Learning, Games and Culture, Universal Access in the Information ociety, ACM Transactions on Computer-
–Human Interaction, Human–Computer Interaction, Information Technology & Management, Production Planning & Control, and Proceedings of
Information Systems Security.
S. Sch¨
obel et al.
Computers & Education 173 (2021) 104296
5
them according to their citations within the examined dataset, known as the local citation score (LCS), compared to the papers’ overall
citations, known as the global citation score (GCS). Papers with high local citation scores are more relevant to the examined topic,
which is shown on the graph.
The country network of collaboration was created for documents with two or more authors afliated with different countries. For
clarity, the top 30 country networks were represented in the network. To understand the structure of collaborating networks, we
applied the Louvain modularity (Meo et al., 2011) for community detection to cluster countries that collaborated frequently. Com-
munity detection was also applied to the keyword network to cluster the words that were mentioned together in the same abstracts.
Keywords belonging to the same community reect common research themes. Communities are color-coded in the network plots.
4. Results
4.1. General results
In our study, we present articles published from 2000 to 2019 (see Fig. 2).
Using game concepts in digital learning started early in 2000 when 44 articles were published. Until then, the use of game concepts
in learning environments had continuously increased. Aside from 2006 to 2009, the number of published articles increased each year,
resulting in the most published articles in 2017, with 1396 articles. After 2017, the number of articles published decreased.
2
Pro-
ceedings were increasing much faster than journal publications. Journal publications remained at a similar level from 2015 to 2019.
To answer RQ1, we identied the top sources publishing articles on the use of game concepts in digital learning environments. A
summary of the top sources and the citations of each type of publication is given in Table 1. According to the number of publications,
the most relevant source was the journal Computers in Human Behavior, in which studies also had a high number of citations. The
second source was Computers & Education, which had a lower number of publications but was cited more often than Computers in
Human Behavior. Two conferences were named more than once in our top 15 sources: the International Technology, Education, and
Development Conference and the European Conference on Game Based Learning. However, we noted that they were not cited as often
as the journal sources.
In a more detailed step, we focused on the most cited articles to better understand which topics were relevant for establishing game
concepts in digital learning. Therefore, we referred to the top 20 authors’ most-cited papers (Table 2). Aside from the study by Hanus &
Fox (2015) published in 2015, all others were published between 2000 and 2013. Consequently, we could assume that these studies
were the most inuential in making game concepts prominent the eld of digital learning. Three of our ten most-cited manuscripts
focused on gamication. Three other publications worked with serious games, whereas the other four manuscripts focused on games or
simulations. Although the four game-related studies claimed to work with games, they introduced serious gaming components. A game
is something that is only used to entertain without more serious intentions. Concerning the studies’ context, studies that worked with
serious games related to middle school (Dunleavy, Dede, & Mitchell, 2009) or elementary school (Barab, Thomas, Dodge, Carteaux, &
Tuzun, 2005). Learners from middle and elementary schools seem to learn better and more efciently when acting in a game-like
reality environment. All gamication studies related to a university context, which probably requires a more serious learning envi-
ronment by referring only to the components of a game.
4.2. Authors and countries
In the next step, we analyzed the most prominent authors and countries (Fig. 3). With 42 publications, Hainey was the most
productive author. The author’s work focused on reality games (Connolly, Stanseld, & Hainey, 2011), serious gaming (Hainey,
Connolly, Stanseld, & Boyle, 2011a; Hainey, Hakulinen, Connolly, Stanseld, & Boyle, 2011b), and animations and simulations
(Boyle et al., 2014), with the rst publication in 2007. The second most productive author was Connolly. With 36 publications, Hwang
was in third position among the most productive authors over time. His rst study was published in 2011, and most of his work focused
on serious games and on educating students on healthier eating behavior (Sung & Hwang, 2013; Yien, Hung, Hwang, & Lin, 2011). The
rst study published in 2003 focused on the differences between novices and experts regarding computer games (Hong & Liu, 2003).
Similar to this was the work of Chan, with a total of 26 publications, the rst in 2002. Again, the studies’ foci were on games and
serious games in education, and considered learning preferences and game design in the earliest published study (Chen, Liao, Cheng,
Yeh, & Chan, 2012; Yu, Chang, Liu, & Chan, 2002). Finally, the work of Kaufman, with 19 studies, was also about simulations and
digital games (Sauv´
e, Renaud, & Kaufman, 2010, pp. 1–26; Zhang & Kaufman, 2016). However, compared to the other two authors,
Kaufman focused not only on younger students but also on older adults (Zhang & Kaufman, 2016).
Next, we analyzed how the studies inuenced each other and contributed to the temporal development of the eld (Fig. 4).
3
Early inuential studies were authored by de Freitas and Oliver (2006), and Ebner and Holzinger (2007). The work presented by de
2
The last two years showed a marked drop in proceeding papers but not journal papers, reecting the slow process of indexing proceeding papers
rather than a drop in published articles.
3
This is only a fraction of prominent authors. Not all authors will show up in the graph. This graph shows authors who are more prominent locally
than universally, so those with high impact in technology/education/tech-ed will have little chance of showing up here. In other words, authors
with a more global impact (beyond the gaming community) may not show here. Juho Hamari, for example, is a well-known author on gamication
(and so far only that); most of Hamari’s work was developed after 2015, which is why he does not show up in this graph.
S. Sch¨
obel et al.
Computers & Education 173 (2021) 104296
6
Fig. 2. Articles published by year, differentiated by journal and proceedings.
Table 1
Basic data of analyzed studies.
Source Number published Times cited Percentage of all citations Citations per article
Computers in Human Behavior 623 14,784 17.16 23.73
Computers and Education 374 17,278 20.05 46.19
British Journal of Educational Technology 134 3580 4.15 26.71
Educational Technology and Society 130 2093 2.42 16.1
9th International Conference on Education and New Learning Technologies 129 21 0.02 0.16
10th European Conference on Games-based Learning 126 89 0.10 0.70
Journal of Chemical Education 125 803 0.93 6.42
11th European Conference on Games-based Learning 123 64 0.078 0.52
12th European Conference on Games-based Learning 113 24 0.02 0.21
11th International Technology, Education and Development Conference 110 50 0.05 0.45
8th European Conference on Games-based Learning 105 191 0.22 1.81
12th International Technology, Education and Development Conference 101 12 0.01 0.11
9th European Conference on Games-based Learning 101 118 0.13 1.16
International Journal of Engineering Education 96 504 0.58 5.25
5th European Conference on Games-based Learning 96 122 0.14 1.27
Table 2
Top-cited manuscripts.
Author Total
Citations
Keywords Category
Papastergiou
(2009)
597 Interactive learning environments, multimedia/hypermedia systems, applications in subject areas,
secondary education, gender studies
Serious Games
Dominguez A.
(2013)
472 Gamication, game-based learning, computer game, game mechanics, motivation, engagement, e-
learning
Gamication
Barab S. (2005) 422 NA Serious Games
Charlton J.P.
(2007)
395 Addiction, attitudes, impulse control disorders, taxonomies, computer games, computer Games
Dunleavy M. (2009) 355 Augmented reality, immersive participation, simulation, classroom technology practices, handheld
devices, GPS devices
Internet
Games
de Freitas (2006) 326 Addiction, attitudes, impulse control disorders, taxonomies, computer games, computer Simulation
Hanus and Fox
(2015)
317 Virtual reality, improving classroom teaching, human–computer interface, interactive learning
environments, teaching strategies
Gamication
Chou C. (2000) 303 Internet, internet addiction Games
Ebner M. (2007) 302 Game-based learning, e-learning, human technology interaction, usability, civil engineering, structural
concrete, theory of structures
Gamication
Rosas R. (2003) 299 NA Serious Games
S. Sch¨
obel et al.
Computers & Education 173 (2021) 104296
7
Freitas and Oliver (2006) was a kind of “how-to” publication guiding the creation of one’s own simulation or serious gaming world. All
studies that referred to the work of de Freitas and Oliver (2006), and Ebner and Holzinger (2007) demonstrated gamication research
(most presented the results of eld experiments exploring the effectiveness of gamication) (Domínguez et al., 2013; Hanus & Fox,
2015; Simoes, DíazRedondo, & Fern´
andezVilas, 2013).
We then analyzed which countries published their work together (Fig. 5).
4
Working with game concepts seems to be important and
relevant for many different countries. Our network graph demonstrates that only limited strong relationships exist between countries.
Authors from the United States seem to publish their studies more often with authors from China and Canada. Authors from the United
Kingdom publish their work with many other countries, including the United States, Spain, Netherlands, Italy, and Australia. Last,
German authors published work more often with researchers from Italy. Spain was involved with countries such as the United Kingdom
Fig. 3. Top authors’ production over time.
Fig. 4. Historical direct citation network.
4
We applied Louvain modularity for community detection to cluster countries that frequently collaborate. Countries that belong to the same
community reect common collaboration. Communities are color-coded in the network plots.
S. Sch¨
obel et al.
Computers & Education 173 (2021) 104296
8
and the Netherlands. All other countries had fewer publications and were not signicantly involved in publishing work with other
countries.
Interestingly, although researching game concepts in digital learning seems to be of international (Ernst, Janson, S¨
ollner, & Lei-
meister, 2016) relevance, cultural adaption of game concepts was not a key component of the studies we analyzed, nor was culture a
major part of our keyword analysis.
5
Work by Li and Kirkup (2007) analyzed cultural differences between Chinese and UK students
regarding their internet use. However, few studies considering cultural aspects in their research studies discussed the design of game
concepts and digital learning solutions in combination with cultural issues.
4.3. General keyword analysis
Table 3 presents the top 20 keywords among all studies and supports us in answering RQ2. A detailed keyword network can be
found in Appendix B. The keywords most often mentioned focused on the classication of different kinds of games: games, gamication,
and serious games. Games was by far the most popular keyword, which is easily explained because it is relevant in publications about
both serious games and gamication. Using parts of a game seemed to be more popular than implementing a serious game-like world.
Motivation was relevant in considering game concepts in digital learning; it was used 345 times as a keyword, and collaboration also
seemed to be of importance in digital learning environments. Although the goal of gamication and/or serious games is to support
learning, learning outcomes was not in the top 20 keywords. Outcomes that were considered were motivation and engagement. These two
are important variables for assessing the effectiveness of a game concept, but they do not necessarily characterize the effectiveness of
game concepts in digital learning. The effectiveness of game concepts could be supported by evaluating learning outcomes.
In the next step, we analyzed the yearly growth of our top keywords (Fig. 6). A detailed overview can be found in Appendix C.
Research involving gamication was rst published in 2010 (Bunchball, 2010). From 2011, the frequency increased markedly until
2017, when it began to decline. Other than serious games, games was mentioned earlier than gamication. Games seem to have been
relevant for digital learning since 2001. In 2008, the keyword game-based learning decreased to being named less than 50 times and
Fig. 5. Countries network.
5
The keyword “culture” was used in 18 of the publications we analyzed. Similar keywords connected to culture, such as cultural diversity or
intercultural education, were used in four or fewer publications.
S. Sch¨
obel et al.
Computers & Education 173 (2021) 104296
9
then began to regain its relevance, steadily increasing until 2017. The keywords collaboration, technology, programming, game design,
virtual world, simulation, and motivation occurred at a lower frequency compared to games, gamication, serious games, game-based
learning, learning, e-learning, and education.
6
This was very interesting and helpful in terms of identifying other research directions to
pursue and supporting the use of game concepts in education and learning.
4.4. Keyword Clustering—A detailed understanding of the development of game concepts
To better understand the relevance of various keywords, we next clustered our keywords to get an idea of how they had developed
over time. A denition of each cluster is presented in Appendix D. By clustering, we mean that we consolidated keywords that focused
on the same issue, such as motivation and intrinsic motivation or cooperation and collaboration. We categorized keywords that reect an
outcome variable and keywords that designate a specic device, technology or system. We considered game dynamics (e.g., collab-
oration and competition) and the kind of elements that were used as keywords—research suggests analyzing individual game elements
such as badges in more detail (Seaborn & Fels, 2015). Last, we considered surrounding realities, such as virtual reality or augmented
reality, and considered two other keywords that were important to consider for future research—game design and articial intelligence
(AI).
A summary of the keyword clusters, their frequency, and centrality measures of the keywords are given in Table 4.
The centrality measures of the keywords showed that game, gamication, and learning-related keywords were the most central
keywords with the highest inuence according to degree, Eigenvector, and H-index centrality. Diffusion centrality was also high but
with small differences between these keywords, highlighting the comparable range of spread of these research themes and their close
Table 3
Top 20 keywords.
Position Keyword Frequency Position Keyword Frequency
1 Games 1752 11 Virtual Worlds 311
2 Game-based learning 1441 12 Collaboration 272
3 Gamication 760 13 Technology 272
4 Serious Games 671 14 Programming 228
5 Learning 594 15 Game Design 193
6 E-Learning 537 16 Children 190
7 Education 497 17 Engagement 186
8 Stem 360 18 Higher Education 184
9 Motivation 345 19 Assessment 148
10 Simulation 327 20 Evaluation 133
Fig. 6. Yearly growth of the top 15 keywords.
6
Note that the decreasing number of publications from 2018 to 2019 does not necessarily indicate that fewer studies were published. This is an
articial effect, as ISI had not published all 2019 conferences and proceedings, causing these low values in the last year.
S. Sch¨
obel et al.
Computers & Education 173 (2021) 104296
10
Table 4
Keyword clustering.
Keyword Frequency Degree Eigenvector H-Index Diffusion
Outcomes
Motivation 1844 384 0.654166018 79 24,251
Performance 1280 290 0.544886025 76 20,180
Engagement 860 259 0.511034186 76 20,295
Flow 471 111 0.272894423 52 11,402
Enjoyment 216 88 0.213309383 46 8934
Cognitive Load 85 39 0.097880221 26 4406
Satisfaction 50 33 0.130779701 25 6468
Learning Environments and Devices
Simulation 2075 288 0.54041196 74 19,713
Virtual World 1188 213 0.43677801 67 16,754
Mobile 408 139 0.3192302 58 12,781
Active 215 86 0.25809592 45 8715
MOOC 188 68 0.19253663 39 8535
Computer-based 138 57 0.15825591 36 7754
Dynamics
Collaborative 1712 321 0.58528995 76 21,234
Competition 548 127 0.31744417 61 13,234
Elements
Avatar 111 24 0.07135122 20 3463
Badges 76 45 0.13817185 34 5768
Technology Trends
VR 1060 227 0.43173038 65 16,558
AR 505 133 0.30629453 58 12,286
Others
Game Design 1118 18 0.06367297 15 3149
AI 257 53 0.13700102 34 5875
Fig. 7. Keyword development.
S. Sch¨
obel et al.
Computers & Education 173 (2021) 104296
11
inuence. To further examine the inuence and diffusion of our categorized keywords, we calculated the centrality measures for these
themed keywords. The keywords motivation, performance, and engagement had very high H-index and diffusion centrality, indicating
their strong inuence and spread in the reported literature, like simulation, virtual worlds, and virtual reality. The values of diffusion and
H-index centralities indicated that these themes were associated with a wide range of applications of games, adoption, and
endorsement by diverse game communities.
In general, the outcome keyword most often mentioned was motivation, followed by performance and engagement. Neither cognitive
load nor satisfaction were included often as keywords. Cognitive load theory posits that learners have a working memory with a limited
capacity when it comes to dealing with new information (Sweller, 1994) and is discussed extensively when researching digital learning
(Janson, Sӧllner, & Leimeister, 2020) environments (Ayres, 2020; Sweller, 1994). Satisfaction is another important outcome regarding
the effectiveness of digital learning (Janson, S¨
ollner, & Leimeister, 2017; Gupta & Bostrom, 2013). The devices most often mentioned
were simulations and virtual worlds, which can be explained by the prevalence of serious games and gaming in the studies we
analyzed. Mobile learning seemed to be more relevant than computer-based learning and even more relevant than active learning (e.g.,
in a classroom without technical devices). The dynamic that was referred to most often was collaboration, which is important for
learning when it occurs in groups, considering not only the single learner but also a group of learners. Collaboration was a keyword
more often than competition (collaboration was used 1712 times and competition 548 times). This is supported by other researchers who
suggest that competition in learning needs to be further explored to make it more efcient and effective in designing game concepts
and game components (Burguillo, 2010; Dissanayake, Mehta, Palvia, Taras, & Amoako-Gyampah, 2019; Santhanam et al., 2016;
Scheiner, 2015). The surrounding reality most often used was VR. Creating VR requires a detailed design concept. Game design was
named as a keyword 1118 times. Game design seems to be more relevant in creating serious games and simulations for which a
complete virtual environment must be constructed and single elements are not enough to work with. Surprisingly, AI was referred to
Fig. 8. Individual keyword development.
S. Sch¨
obel et al.
Computers & Education 173 (2021) 104296
12
only 257 times. Accordingly, there is still much more to explore about how to construct intelligent and adaptive game designs,
regardless of whether we are using game concepts such as serious games or gamication.
Besides analyzing keyword frequency, we were interested in their development over time. Fig. 7 shows the proportion of articles
having the given keyword of all articles published at that year. It demonstrates that certain keywords are continuously relevant over
time, such as simulation, which was rst used in keyword combinations in 2006 and remained part of studies till 2018. Interest in AR
grew from 2015 to 2018. Discussing and researching game design began in 2008 and was then relevant in more studies throughout that
time.
To illustrate a more detailed overview of each of our keywords and its development from 2000 to 2018, we constructed Fig. 8. A
consolidated overview about the yearly growth of the keywords can be found in Appendix E.
The keyword development of the terms virtual worlds and augmented reality decreased directly after increasing. Other keywords
were just becoming part of research studies, such as satisfaction, which was rst mentioned in 2014 as a keyword. Similarly, cognitive
load was not referred to often. It started to appear more often in 2015.
The outcome variable motivation steadily increased as a keyword as did performance. In contrast, engagement as a keyword appeared
more often in 2012 and ow in 2015. Satisfaction, enjoyment, and cognitive load were listed as keywords more often from 2014 to 2019,
but at a lower rate compared to the motivation and performance. The most relevant environment seemed to be simulation, whose growth
as a keyword started earlier in 2007. Computer-based learning was referred to more often in 2013 and 2014. Comparing the two dy-
namics, we observed that collaboration started to grow in 2009 and grew markedly between 2014 (frequency less than 20) and 2016
(frequency close to 50), whereas in 2014, competition started to grow. Both game elements avatar and level were part of the studies’
keywords at a lower level. Both rose in frequency from 2014 on.
VR began to increase in 2011, decreased until 2014, and increased intensively in more recent years. The relevance of AR seemed to
decrease and increase more often than we observed for the development of VR. AI seemed more relevant during more recent years,
with a higher number of keyword mentions in 2013. The development of our keyword clusters supported us in deriving a research
agenda for research on game concepts in digital learning, which we refer to in the following subsection.
4.5. Research agenda
We used our keyword clusters to propose a research agenda for game concepts in digital learning environments to answer RQ3
(Fig. 9).
To derive our agenda, we categorized our clusters into three groups: emerging keywords that were just starting to be explored,
Fig. 9. Suggestion for research agenda.
S. Sch¨
obel et al.
Computers & Education 173 (2021) 104296
13
those with developing research topics considering keywords that had grown over time and that needed more detailed research articles
to explore them in more detail, and those with saturated research topics. We will discuss group in more detail, considering the
literature that supports the state of each topic.
4.5.1. Emerging research topics
Badges and avatars are among the emerging topics. Other researchers suggest analyzing the effectiveness of individual game el-
ements to be able to understand and adapt their individual designs and to get away from project-based learning (PBL) designs (Sch¨
obel,
Janson, & Mishra, 2019; Seaborn & Fels, 2015; Wouters et al., 2013). This holds especially true for gamied learning solutions
(Santhanam et al., 2016). In addition, three of our outcome variables were in the rst group. Research supports the idea of focusing on
specic variables to better understand how game concepts can support digital learning (Chian-Wen, 2014). Cognitive load could be
especially important in the future for game worlds, for example, for VR environments that consider complex animations, as they could
increase the risk of overwhelming a learner’s working memory (Ayres & Paas, 2007). This perspective is supported by several re-
searchers and warrants a more detailed analysis of how games in digital learning affect the reactions of learners due to mixed effects on
performance (Sailer & Homner, 2019; Santhanam et al., 2016; Silva, Rodrigues, & Leal, 2020). Long-term effects, especially (e.g. for
analyzing motivational effects of gamication), are still to be explored in their effects, for example, on specic outcome variables
(Sailer & Homner, 2019).
Three of our environmental keywords are also in the rst group. Since most studies on games focus on incorporating computers, a
combination of active and computer-based learning could be interesting to explore further. Finally, AI was grouped as an emerging
research topic because it has gained relevance over the last few years and enables individualizing game concepts in digital learning
(Sch¨
obel et al., 2020). Its relevance to better analyzing individual game concepts comes from observations that the effects on learning
outcomes still cannot be fully explained (Super et al., 2019). Learners, especially, differ in their learning behavior and their goals,
better known as learner goal orientation (Hakulinen & Auvinen, 2014). This theory clusters learners into performance- and
mastery-oriented learners. While one group totally supports competitive learning environments and wants to compete with other
learners, the other group is focused on their own individual performance success. Here, the rst group could be supported by
competitive game elements, whereas the other group would not be motivated by competitive game concepts. However, even when
using competitive elements, learners differ in how they experience them and require individualized designs (Santhanam et al., 2016).
Such aspects can be approached through AI methods.
4.5.2. Developing research topics
This group of keywords appeared at a higher frequency. All keywords increased in frequency in 2013 and 2014, except engagement,
which increased in 2012. Engagement is one of the most important constructs related to game concepts; it has been part of many
studies and is essential to making game concepts in digital learning effective (Kuo & Chuang, 2016). However, some researchers assert
that we still need to determine which constructs surround engagement in order to better understand how it develops (Hamari et al.,
2016). Consequently, we support the idea that the keywords in the second group should be considered for detailed research that
discusses each topic (keyword) in more depth, taking a closer look at how engagement is induced by game concepts in digital learning.
This is similar to observing how ow develops (Suh, Cheung, Ahuja, & Wagner, 2017) and how game concepts can be adapted to make
them more useful for mobile phones (Kurniawan, Sitohang, & Rukmono, ). Competition is another construct of interest. We understand
how we can design and consider competitive game elements; what is still being discussed and analyzed in research and in practice is
how we can adapt competition to individual game designs (Santhanam et al., 2016). Ultimately, AR was categorized as a developing
research topic.
4.5.3. Saturated research topics
The third group of keywords comprises saturated research topics and is the high frequency group. This frequency continuously
increased in 2006, 2007, and 2009. The keywords we considered have been extensively discussed in previous research, such as
collaboration or simulations, which have long been a part of serious games for learning and online education. It may be worthwhile to
think about how to connect these keywords to other areas of research or how to make them more applicable to increase the number of
keywords. Here, we can refer to methods of AI to consider adaptive and intelligent VR solutions for collaborative learning experiences.
Game design has been part of most serious gaming environments, but it has also become more important for gamication designs to
enable designers to construct, for example, more user-centered and meaningful game concepts (Hallifax, Serna, Marty, Lavou´
e, &
Lavou). Performance is especially worth considering when looking at the long-term effects of operating with game concepts, which
have recently become more important for research (Ahmad et al., 2020). Analyzing long-term effects would lead to other contributions
and a deeper understanding of what matters in digital learning in the long run. Additionally, the positive and negative effects of game
concepts on outcomes are worth comparing to offer veriable evidence (Trinidad et al., 2021) or to concentrate on effective outcomes
(Chian-Wen, 2014). This also holds true for motivation. Although we know that there are motivating effects when we work with game
concepts, it is still important to analyze and discuss how different kinds of motivations are guided by game concepts, by explaining, for
instance, how intrinsic motivating effects differ from extrinsic effects in terms of the elements involved (Sailer & Homner, 2019).
S. Sch¨
obel et al.
Computers & Education 173 (2021) 104296
14
With our research agenda in mind, we next discuss our results and posit the potential contributions, limitations, and implications
for future research.
5. Discussion and contributions
The results of our bibliometric analysis provide information on the most cited publications in journals as well as conferences (RQ1).
Game concepts in digital learning have not only been a part of conference publications but have also been extensively discussed in
journal articles. Our results highlight that different types of game concepts (e.g., gamication, serious games, etc.) have been
considered equally in research studies to date. While gamication and serious games both work with game elements, they differ
regarding how these elements are embedded—either in a simulated world or in a regular system involving only some game elements
without presenting a virtual world (Werbach & Hunter, 2012). Because both gamication and serious games seem to be equally
important, we can guess that they provide a useful concept for motivating learners in digital learning environments. Both journal-
s—Computers in Human Behavior and Computers and Education—make a signicant contribution, highlighting that game concepts in
digital learning have their origins at the interdisciplinary intersection of psychology and educational technology. The general statistics
in our bibliometric study demonstrate that game concepts in digital learning are of international relevance, supporting the idea that
concepts such as gamication or serious games can be culturally adapted. Furthermore, using games involves not only the concepts of
serious gaming and simulation; also relevant is the concept of gamication. However, we observed that each concept faces different
challenges. Gamication research, for example, calls for a more in-depth analysis of specic elements, whereas examining serious
games requires considering game design in more detail (Seaborn & Fels, 2015).
To answer RQ2, we need to learn more about the most important research streams in relation to game concepts in digital learning.
To better understand which streams of research exist, we clustered the keywords we identied into six groups. A group of keywords
that have not yet been listed often are those relating to outcomes—satisfaction, cognitive load, enjoyment, or motivation and performance.
Research shows that the effects of establishing game concepts in digital learning deliver mixed results regarding learner performance
(Sailer & Homner, 2019; Seaborn & Fels, 2015). Consequently, to better understand how performance is inuenced by game concepts,
it could also be helpful to analyze constructs surrounding learning performance from a long-term perspective (Silva et al., 2020). Our
clustering of technology trends supports the idea that AI could be a promising new approach to better individualize game concepts in
digital learning and to allow for greater personalization (Trinidad et al., 2021). Another cluster that requires more in-depth under-
standing is the cluster of elements (e.g., badges and avatars). Referring to the two game dynamics we identied, more could be learned
about the effectiveness of competition (Santhanam et al., 2016).
The aim of RQ3 was to identify streams for future research. We identied three research directions that demand different kinds of
research studies. Regarding the group of emerging research topics, we observed that for some constructs, we still need to determine
how they are formed in relation to using games and game-like elements in online learning environments (Hamari et al., 2016).
Constructs like satisfaction or cognitive load, especially, could be explored in more detail when implementing game concepts. (Sailer,
Hense, Mayr, & ). One interesting observation regarding emerging research areas was that AI and VR were in the most cited manu-
scripts. Game concepts in digital learning seem to be important in relation to specic devices, making a game more realistic and fun to
facilitate better learning processes through customized game-based learning interventions (Rowe, Shores, Mott, & Lester, 2011;
Sch¨
obel et al., 2020). Such customization could happen by referring to AI. Regarding our second research cluster, topics such as AR,
ow, and competition should be examined in more detail in future research, with special consideration given to empirical analyses.
With regard to saturated research topics, it could be worthwhile to consider rethinking the concepts of game design in digital learning.
Keywords such as collaboration are well-known in research on digital learning. For this group of keywords, researchers could discuss
how to further explore them from another perspective.
Changing the perspective on a group of keywords also implies changing how we think about game concepts. Designing a game
concept entails making a gaming experience meaningful to users. Design was part of our keyword analysis, which indicated that it is
most relevant for serious games. However, game design is becoming more important for gamied solutions, such as through the use of
functional affordances (Lowry, Petter, & Leimeister, 2020; Sch¨
obel et al., 2020). Our analysis also indicated that different constructs
and variables matter in designing gamication concepts in digital learning. For those groups of keywords especially, it could be
worthwhile to change our perspective on them. For example, motivation was among the keywords we grouped as saturated research
topics. When operating and analyzing the motivational effects of gamication concepts, most studies refer to self-determination theory
(Deci et al., 2001). However, in looking at recommendations given in the research, we should be more open-minded towards new and
alternative theories to bring concepts such as gamication to the next level of development (Lowry et al., 2020) and change the
perspective on specic outcomes like motivation. Furthermore, it may be worth stepping away from standardized game elements and
thinking about new elements and combinations of elements (Trinidad et al., 2021).
Our study contributes to theory and practice. From a theoretical perspective, we provide clarication on topics related to game
concepts in digital learning. In our research, we present a summarized overview of existing research on game concepts in digital
S. Sch¨
obel et al.
Computers & Education 173 (2021) 104296
15
learning to enable researchers to identify new areas for future research projects. By differentiating our keywords into clusters, we
provide precise implications for future research on different concepts and game-related outcomes. From our analysis, researchers can
conduct more specialized analyses by conducting, for example, follow up meta-analyses with specic keywords to discuss the rele-
vance and impact of gamied AI solutions in digital learning or by analyzing the role of satisfaction in more detail. Therefore, we
contribute to a better understanding of the relevant key terms and support practitioners by highlighting which topics and areas are
relevant for game concepts in digital learning. Hence, practitioners can refer to the results of our study to better adapt their game
concepts to the context of learning. In addition, we provide an overview of topics that are relevant to consider when designing game
concepts, thus enabling practitioners to better understand how game concepts have developed and will develop. This development is
important for practitioners to better understand what they need to focus on in the future, such as using AI, which can make game
concepts more intelligent but will require developing individualized game concepts that are better adapted to the needs of learners.
6. Limitations, future research, and conclusion
Our study has some limitations that may suggest areas for future research to complement what we present in our research agenda.
First, although we overcame the limitations of the work of Parreno et al. (2016), we do not encourage other researchers to consider
studies from other disciplines, such as management or marketing. Second, we only scratched the surface and provided an overview of
more than 10,000 studies. Future research should concentrate on exploring the studies in more detail, covering aspects such as
research designs and implications for research. Third, we provided a limited number of clusters. We encourage other researchers to
further consider other kinds of clusters to gain a more detailed understanding of how specic groups of keywords have developed over
time.
In this study, we present an overview of the state of the art of designing game concepts in digital learning. We present the most
important sources and authors that have contributed signicantly to theory and practice by establishing game concepts in digital
learning. With our keyword analysis and the clustering of keywords, we identied the most important streams of research and their
evolution and presented a research agenda. This enables us to identify directions for future research to contribute further to both
theory and practice. Finally, this article is limited to metadata and citation counts that relied on WoS. Therefore, the citation counts
and inclusivity are expected to be lower than in other databases (e.g., SCOPUS or Google Scholar) (Harzing & Alakangas, 2016;
Kulkarni, Aziz, Shams, & Busse, 2009). However, we made this decision based on the advantage that WoS offers in terms of a
well-maintained database, veried sources of articles, and rigorous quality control (Birkle, Pendlebury, Schnell, & Adams, 2020; Li,
Rollins, & Yan, 2018). Search options such as Google Scholar have been heavily criticized for lack of quality (Halevi, Moed, & Bar-Ilan,
2017; Memon, 2018), inclusion of predatory journals (Ross-White, Godfrey, Sears, & Wilson, 2019; Severin & Low, 2019),
non-scientic reports (Aguillo, 2012), and manipulation by bots and humans to inate citations and H-index (Delgado L´
opez-C´
ozar,
Robinson-García, & Torres-Salinas, 2014). A recent review of 91 studies comparing Google Scholar with other databases synthesized
the evidence that Google Scholar lacks a transparent indexing policy, is inated with duplications, has manipulated documents and,
more importantly, lacks quality. Authors are cautioned against the use of Google Scholar for scholarly benchmarking or citation counts
(Halevi et al., 2017). Mixing two databases for bibliometrics is not warranted, as it produces a faulty comparison. Since SCOPUS and
WoS implement a different citation count strategy, an article in WoS may be assigned more citations that the same article in SCOPUS.
Therefore, articles in either database alone will be at a disadvantage (Harzing & Alakangas, 2016; Kulkarni et al., 2009), and
furthermore, this will result in erroneous H-indexes and unfair comparisons of articles and authors. However, it is up to future re-
searchers to enlarge their search and include other databases.
Credit author statement
All authors were responsible for the conceptualization of the paper. The data and results were produced by Mohammed Saqr. Soa
Sch¨
obel was responsible for the original draft and the supervision. Andreas Janson as well as Mohammed Saqr and Soa Sch¨
obel were
responsible for writing, reviewing, and editing.
Acknowledgments
This work has been partially developed in the context of the projects KoLeArnwww.kolearn.de) Grant No.: 01BE17008A, and
Nudger (www.nudger.de), Grant No.: 16KIS0890K, both funded by the German Federal Ministry of Education and Research (BMBF)
The third author acknowledges funding from the Basic Research fund of the University of St.Gallen (GFF-IPF).
Appendix A. Related Work
The following table summarizes key publications in relation to our studies goal. We identied meta analyses, as well as existing
bibliometric studies and key literature reviews.
S. Sch¨
obel et al.
Computers & Education 173 (2021) 104296
16
Reference Kind of Study Goal of Study Strengthens &Limitations (compared to our study goal) Future Research
de Freitas (2018) Literature
Review
Problematize the current scope of education studies and
to reposition game science more critically within this
educational context and perspective.
(+) Nice overview about development of games in
education. Demonstration of contributions to other
disciplines.
- Summarize contributions along different disciplines.
- Analyze feedback system behind games in education.
- Balance between game playability and fun and solid
learning design. (−) Limited view on studies because of method of
literature review. Only a short discussion of different
game concepts.
Peterson (2010) Analysis of the psycholinguistic and sociocultural
constructs proposed as a basis for the use of games and
simulations in computer assisted language learning
(+) Sociocultural and psycholinguistic view on games
and their role in learning languages; demonstration of
examples of how to work with games and simulations
- Enhanced understanding of the complex processes
involved in language learning.
- Investigate how simulation-based language learning
may be facilitated in the dynamic learning
environments.
(−) It is a limited view on the state-of-the-art; what is
claimed to be a meta-analysis is not per se a meta-
analysis (when we think about it as comparing a large
number of studies and the effect sizes of variables
Gao et al. (2020) Understand the potential of mobile game-based learning
in STEM education by identifying the studies context,
research foci, research methodologies, features, research
instruments and measurements
(+) Focus on mobile learning that provide useful
insights about how to work with and consider mobile
devices in STEM
- How does the inclusion or exclusion of specic game or
mobile features affect the outcome of learning?
- Is a particular combination of game and mobile features
more effective to help learners to achieve their goals?
- In what situation are intrinsically integrated games
better than extrinsically integrated games?
- How can mobile games be designed to meet the needs of
students with different characteristics?
(−) With 30 studies considered it is only a limited view
on the state-of-the-art of research and its future
directions
Girard, Ecalle, and
Magnan (2013)
Review the results of experimental studies designed to
examine the effectiveness of video games and serious
games on players’ learning and engagement
(+) Good overview about role of serious games in
education and clarication on terms used
- Working with control groups to analyze effectiveness of
serious games.
- Analyze effects of serious games in more detail
regarding e.g. acquired knowledge.
- Consider various types of serious games.
(−) Only four years (2008–2011) are considered;
selection of rst 30 studies; only limited view on
learning effects of games
Wouters et al.
(2013)
Meta-analysis Investigation of effectiveness of serious games in learning
and if they are more motivating than conventional
instruction methods.
(+) Detailed overview on variables that support
motivation and outcomes when working with serious
games in learning
- More value-added research on specic game features
and their effectiveness (e.g. competitive elements).
- Analyze cognitive consequences.
- Examine denition of motivation. (−) Focus on only serious games and on specic
outcomes; because we have a meta-analysis it is only a
limited number of studies considered (compared to a
bibliometric study
Vogel et al. (2006) Analysis of computer games and simulations that are used
for learning compared to traditional approaches
(+) Large sample size in meta-analysis and comparison
of effectiveness between concepts
- Enlarge search (because many articles could not be
used).
- Working with control groups to analyze effectiveness.
- Compare games and simulation with traditional
methods of teaching.
(−) Focus on specic variables and limited discussion of
results
Clark, Tanner-Smith,
and
Killingsworth
(2016)
Systematic review on digital games and learning for K–16
students. Comparisons of game versus nongame
conditions (i.e., media comparisons) and comparisons of
augmented games versus standard game designs (i.e.,
value-added comparisons).
(+) Consideration of a large stream of literature; very
detailed analysis and also analysis of moderators;
supportive contributions for future research
- Deeper assessments of student learning should be
investigated in future research.
- Explore whether or not the simple gamication studies
(e.g., games that simply add contingent points and
badges to learning activities) more frequently focus on
lower order learning outcomes as compared to studies
with more sophisticated game mechanics.
- Analyze effectiveness of game characteristics.
(−) Focus on games and serious games; study covers
literature from 2000 to 2012
Wu et al. (2012) The study conducts a meta-analysis to investigate both
the development of the use of learning theory as a
foundation in game-based learning, or lack thereof, and
the nature of links between the use of learning theory and
game-based learning.
(+) Very detailed and supportive overview about
theories that are important to learning and education
(with a resulting strong contribution); good presentation
of game-based learning to theories of learning
- Combine different academic elds (i.e. education,
psychology, computer science and engineering or
management) and even include specialists from industry
to collaboratively develop a game-based learning system
(continued on next page)
S. Sch¨
obel et al.
Computers & Education 173 (2021) 104296
17
(continued)
Reference Kind of Study Goal of Study Strengthens &Limitations (compared to our study goal) Future Research
or platform to increase the proportion and use of
learning-theory foundations in game-based learning
- Apply different types of game elements to principles of
behaviorism and cognitivism (or combinations)
(−) Limited presentation of meta analytic outcomes;
very short cut results discussion
Lamb et al. (2018) Characterize and compare outcomes related to serious
educational games, serious games, and educational
simulations as they are presented in the educational
literature.
(+) Good overview about the effectiveness of serious
games in education; well done meta-analysis and
included moderator analysis
- Need to include more moderators.
- Differentiation between the more specic term of
educational simulation.
- Need for long-term analysis. (−) Focus on only a limited number of studies; focus on
serious games that are only one part of game concepts
Chian-Wen (2014) Analysis of effectiveness of digital games and how they
improve language learning and identication of
moderating variables in digital game-based learning
(+) Overview about existing meta-analyses; well done
meta-analysis and analysis of moderating variables;
demonstration of relationship between game-based
learning and learning languages
- Consider other databases for search.
- Consider more variables to understand effectiveness of
games in digital learning.
- Explore affective outcomes not only cognitive ones
- Analysis of long-term effects. (−) Limited to language learning so only a focused view
on the state-of-the-art; limited number of databases
considered
Karakus et al.
(2019)
Bibliometric
Study
Analysis of how AR has developed in education
considering studies from 1999 to 2018
(+) Nice overview about the establishment of AR along
the years and how they have been developed
- Analyze learner characteristics and experience in future
research.
(−) Keyword analysis is rather short, and the role of
games is not discussed in detail (at least to what we want
to achieve with our study)
Marti-Parreno et al.
(2016)
Provide useful up-to-date information to picture the state
of the art about current research and evolution of
gamication in education
(+) Helpful recommendations regarding effectiveness,
acceptance, engagement and social interaction
- Enlarge search to more than ve years and include
proceedings.
- More research about clarifying game-based learning,
serious games and gamication.
- Research about how to integrate games and game
elements in traditional learning frameworks.
(−) Only focus on journal articles; limited time period
(only from 2010 to 2014); comparison of only 139
articles
Trinidad et al.
(2021)
Conduct a bibliometric study to describe how
gamication as scientic perspective is structured and
how it has evolved over time.
(+) Very detailed overview about gamication in
general with all perspectives (authors, keywords,
countries)
- More knowledge about effective game designs (and new
elements).
- Personalized and adaptive data-driven gamication.
- Empirical research that helps to offer veriable evidence
of positive and negative effects of gamication.
- Ethical use of gamication.
(−) Limited time period (2011–2019) research involving
gamication has started around 2000; learning is
highlighted as important but not focused analysis is
done
S. Sch¨
obel et al.
Computers & Education 173 (2021) 104296
18
Appendix B. Keyword Network
The keywords that were most often used in combination with each other were games, game-based learning, gamication, and
serious games. They were also the keywords that were most often used among all keywords we identied. Motivation seemed to be
relevant when using with gamication, other than engagement, which was more often referred to in combination with motivation.
Simulations seemed to be of relevance for games and serious games but not as important for gamication. More interesting was that
game design was often not connected to gamication or serious games but rather to game-based learning. Furthermore, performance
was part of the keywords, but it was not intensively connected to the other keywords and did not have a strong relationship to other
keywords like the one we saw between games and game-based learning.
Appendix C. Detailed Overview Keyword Development
The following gure displays a detailed overview about the most relevant keywords and their development over time.
S. Sch¨
obel et al.
Computers & Education 173 (2021) 104296
19
S. Sch¨
obel et al.
Computers & Education 173 (2021) 104296
20
Appendix D. Description of Keyword Clusters
We identied and derived clusters from literature to consolidate the large number of identied keywords. E.g., we considered
different kind of outcomes describing different effects and results when working with game concepts in digital learning.
Cluster Description
Outcomes The effectiveness of working with games has intensively been discussed in literature. Some studies claim that we need to get a better
understanding about how games and its elements affect outcomes such as engagement (Hamari et al., 2016). This cluster considers all kind
of outcomes that can result from working with games and its elements such as engagement and motivation that are central outcomes to
analyze the effectiveness of games (Muntean, 2011; Vos, van der Meijden, & Denessen, 2011). As our study focusses on education and
learning, we considered performances as another outcome.
Environment and
Devices
Learning can happen in different context and with different kind of devices (e.g., mobile or computer-based learning (Bartel & Hagel,
2014; Fitz-Walter, Tjondronegoro, & Wyeth, 2012)). The device worked with and environment worked in, determine the design of a game
concept. This cluster considers different devices and environments to explore trends towards environments and devices.
Dynamics Dynamics describe describes the run-time behavior of game elements used in a concept, acting on users inputs and each other’s outputs
over time (Hunicke et al., 2004).They give an overview of how a game concept is grounded e.g., as competitive concept aiming to increase
competition between users.
Elements Elements are central components of a game (Seaborn & Fels, 2015). This cluster considers keywords that directly refer to game elements
such as badges or avatars.
Technology Trends These cluster considers both AR and VR as technology trends that have become more and more relevant during the last years and that
change a learner’s gaming experience (Zyda, 2005).
Others In a last cluster we consider two keywords that are relevant to better understand how games in digital learning have developed and will
develop. First, game design is important to make concepts of working with games more meaningful (Hong, Hwang, Tai, & Kuo, 2016;
Salen & Zimmerman, 2004). Lastly, working with AI allows us to develop new and innovative game concepts in the future to better adapt
and individualize such concepts to the needs of users (Sch¨
obel et al., 2020).
Appendix E. Consolidated Growth per Year
The following Figure summarizes the consolidated yearly growth of our keywords.
References
Abramo, G., D’Angelo, C. A., & Di Costa, F. (2011). National research assessment exercises: A comparison of peer review and bibliometrics rankings. Scientometrics, 89
(3), 929–941.
Aguillo, I. F. (2012). Is Google scholar useful for bibliometrics? A webometric analysis. Scientometrics, 91(2), 343–351.
Ahmad, A., Zeshan, F., Khan, M. S., Marriam, R., Ali, A., & Samreen, A. (2020). The impact of gamication on learning outcomes of computer science majors. ACM
Transactions on Computing Education (TOCE), 20(2), 1–25.
S. Sch¨
obel et al.
Computers & Education 173 (2021) 104296
21
Aria, M., & Cuccurullo, C. (2017). bibliometrix: An R-tool for comprehensive science mapping analysis. Journal of Informetrics, 11(4), 959–975.
Ayres, P. (2020). Something old, something new from cognitive load theory. Elsevier.
Ayres, P., & Paas, F. (2007). Making instructional animations more effective: A cognitive load approach. Applied Cognitive Psychology: The Ofcial Journal of the Society
for Applied Research in Memory and Cognition, 21(6), 695–700.
Bai, S., Hew, K. F., & Huang, B. (2020). Is gamication “bullshit”? Evidence from a meta-analysis and synthesis of qualitative data in educational contexts. Educational
Research Review.
Barab, S., Thomas, M., Dodge, T., Carteaux, R., & Tuzun, H. (2005). Making learning fun: Quest Atlantis, a game without guns. Educational Technology Research &
Development, 53(1), 86–107.
Bartel, A., & Hagel, G. (2014). Engaging students with a mobile game-based learning system in university education. Global Engineering Education Conference, 957–960.
Bedwell, W. L., Pavlas, D., Heyne, K., Lazzara, E. H., & Salas, E. (2012). Toward a taxonomy linking game attributes to learning: An empirical study. Simulation &
Gaming, 43(6), 729–760. https://doi.org/10.1177/1046878112439444
Birkle, C., Pendlebury, D. A., Schnell, J., & Adams, J. (2020). Web of Science as a data source for research on scientic and scholarly activity. Quantitative Science
Studies, 1(1), 363–376.
Boyle, E. A., MacArthur, E. W., Connolly, T. M., Hainey, T., Manea, M., K¨
arki, A., et al. (2014). A narrative literature review of games, animations and simulations to
teach research methods and statistics. Computers & Education, 74, 1–14.
Bunchball, I. (2010). Gamication 101: An introduction to the use of game dynamics to inuence behavior (Vols. 1–14). Bunchball Inc. (White Paper).
Burgers, C., Eden, A., van Engelenburg, M. D., & Buningh, S. (2015). How feedback boosts motivation in play in a brain-training game. Computers in Human Behavior,
48, 94–103.
Burguillo, J. C. (2010). Using game theory and competition-based learning to stimulate student motivation and performance. Computers & Education, 55(2), 566–575.
Chen, Ching-Huei, Shih, Chun-Chao, & Law, Victor (2020). The effects of competition in digital game-based learning (DGBL): a meta-analysis. Educational technology
research and development, 68(4), 1855–1873.
Chen, Z.-H., Liao, C. C. Y., Cheng, H. N. H., Yeh, C. Y. C., & Chan, T.-W. (2012). Inuence of game quests on pupils’ enjoyment and goal-pursuing in math learning.
Journal of Educational Technology & Society, 15(2), 317–327.
Chian-Wen, K. A. O. (2014). The effects of digital game-based learning task in English as a foreign language contexts: A meta-analysis. Education Journal, 42(2),
113–141.
Christy, K. R., & Fox, J. (2014). Leaderboards in a virtual classroom: A test of stereotype threat and social comparison explanations for women’s math performance.
Computers & Education, 78, 66–77.
Clark, D. B., Tanner-Smith, E. E., & Killingsworth, S. S. (2016). Digital games, design, and learning: A systematic review and meta-analysis. Review of Educational
Research, 86(1), 79–122.
Connolly, T. M., Stanseld, M., & Hainey, T. (2011). An alternate reality game for language learning: ARGuing for multilingual motivation. Computers & Education, 57
(1), 1389–1415.
Cruz, C., Hanus, M. D., & Fox, J. (2017). The need to achieve: Players’ perceptions and uses of extrinsic meta-game reward systems for video game consoles. Computers
in Human Behavior, 71, 516–524.
Davis, K., & Singh, S. (2015). Digital badges in afterschool learning: Documenting the perspectives and experiences of students and educators. Computers & Education,
88, 72–83.
De-Marcos, L., Garcia-Lopez, E., & Garcia-Cabot, A. (2016). On the effectiveness of game-like and social approaches in learning: Comparing educational gaming,
gamication & social networking. Computers & Education, 95, 99–113.
Deci, E. L., Ryan, R. M., Gagne, M., Leone, D. R., Usunov, J., & Kornazheva, B. P. (2001). Need satisfaction, motivation, and well-being in the work organizations of a
former eastern bloc country: A cross-cultural study of self-determination. Personality and Social Psychology Bulletin, 27(8), 930–942.
Delgado L´
opez-C´
ozar, E., Robinson-García, N., & Torres-Salinas, D. (2014). The G oogle scholar experiment: How to index false papers and manipulate bibliometric
indicators. Journal of the Association for Information Science and Technology, 65(3), 446–454.
Deterding, S., Dixon, D., Khaled, R., & Nacke, L. (2011a). From game design elements to gamefulness: Dening gamication. Mind, 9–15.
Deterding, S., Sicart, M., Nacke, L., O’Hara, K., & Dixon, D. (2011b). Gamication: Using game design elements in non-gaming contexts. Human Factors in Computing
Systems, 2425–2428.
Dissanayake, I., Mehta, N., Palvia, P., Taras, V., & Amoako-Gyampah, K. (2019). Competition matters! Self-efcacy, effort, and performance in crowdsourcing teams (pp.
103–158). Information & Management.
Domínguez, A., Saenz-de-Navarrete, J., de-Marcos, L., Fern´
andez-Sanz, L., P´
ages, C., & Martínez-Herr´
aiz, J.-J. (2013). Gamifying learning experiences: Practical
implications and outcomes. Computers & Education, 63, 380–392.
Dunleavy, M., Dede, C., & Mitchell, R. (2009). Affordances and limitations of immersive participatory augmented reality simulations for teaching and learning.
Journal of Science Education and Technology, 18(1), 7–22.
Ebner, M., & Holzinger, A. (2007). Successful implementation of user-centered game based learning in higher education: An example from civil engineering.
Computers & Education, 49(3), 873–890.
Ernst, S-J, Janson, A, S¨
ollner, M, & Leimeister, JM (2016). It’s about Understanding Each Other’s Culture - Improving the Outcomes of Mobile Learning by Avoiding
Culture Conicts. ICIS 2016 Proceedings.
Fitz-Walter, Z., Tjondronegoro, D., & Wyeth, P. (2012). A gamied mobile application for engaging new students at university orientation. Proceedings of the 24th
Australian Computer-Human Interaction Conference, 138–141.
Freitas, S. de (2018). Are games effective learning tools? A review of educational games. Journal of Educational Technology & Society, 21(2), 74–84.
Freitas, S. de, & Oliver, M. (2006). How can exploratory learning with games and simulations within the curriculum be most effectively evaluated? Computers &
Education, 46(3), 249–264.
Gao, F., Li, L., & Sun, Y. (2020). A systematic review of mobile game-based learning in STEM education. Educational Technology Research & Development, 68(4),
1791–1827.
Girard, C., Ecalle, J., & Magnan, A. (2013). Serious games as new educational tools: How effective are they? A meta-analysis of recent studies. Journal of Computer
Assisted Learning, 29(3), 207–219.
Gupta, S., & Bostrom, R. (2013). Research note—an investigation of the appropriation of technology-mediated training methods incorporating enactive and
collaborative learning. Information Systems Research, 24(2), 454–469.
Hainey, T., Connolly, T. M., Stanseld, M., & Boyle, E. A. (2011a). Evaluation of a game to teach requirements collection and analysis in software engineering at
tertiary education level. Computers & Education, 56(1), 21–35.
Hainey, T., Hakulinen, L., Connolly, T., Stanseld, M., & Boyle, E. (2011b). The differences in motivations of online game players and ofine game players: A
combined analysis of three studies at higher education level. Computers & Education, 57(4), 2197–2211.
Hakulinen, L., & Auvinen, T. (2014). The effect of gamication on students with different achievement goal orientations. International Conference on Teaching and
Learning in Computing and Engineering (IEEE), 9–16.
Halevi, G., Moed, H., & Bar-Ilan, J. (2017). Suitability of Google Scholar as a source of scientic information and as a source of data for scientic evaluation—review
of the literature. Journal of Informetrics, 11(3), 823–834.
Hallifax, S., Serna, A., Marty, J.-C., Lavou´
e, G., & Lavou´
e, E. (2019). Factors to consider for tailored gamication. Proceedings of the Annual Symposium on Computer-
Human Interaction in Play, 559–572.
Hamari, J., & Koivisto, J. (2015). Working out for likes": An empirical study on social inuence in exercise gamication. Computers in Human Behavior, 50, 333–347.
Hamari, J., Koivisto, J., & Sarsa, H. (2014). Does gamication work? A literature review of empirical studies on gamication (pp. 3025–3034). Hawaii International
Conference on System Science (HICSS).
S. Sch¨
obel et al.
Computers & Education 173 (2021) 104296
22
Hamari, J., Shernoff, D. J., Rowe, E., Coller, B., Asbell-Clarke, J., & Edwards, T. (2016). Challenging games help students learn: An empirical study on engagement,
ow and immersion in game-based learning. Computers in Human Behavior, 54, 170–179.
Hanus, M. D., & Fox, J. (2015). Assessing the effects of gamication in the classroom: A longitudinal study on intrinsic motivation, social comparison, satisfaction,
effort and, academic performance. Computers & Education, 80, 152–161.
Harzing, A.-W., & Alakangas, S. (2016). Google scholar, scopus and the Web of science: A longitudinal and cross-disciplinary comparison. Scientometrics, 106(2),
787–804.
Hirsch, J. E. (2005). An index to quantify an individual’s scientic research output. Proceedings of the National Academy of Sciences, 102(46), 16569–16572.
Hong, J.-C., Hwang, M.-Y., Tai, K.-H., & Kuo, Y.-C. (2016). Crystallized intelligence affects hedonic and epistemic values to continue playing a game with saliency-
based design. Computers & Education, 95, 75–84.
Hong, J.-C., & Liu, M.-C. (2003). A study on thinking strategy between experts and novices of computer games. Computers in Human Behavior, 19(2), 245–258.
Huizinga, J. (1949). Homo ludens. Boston: The Beacon Press.
Hunicke, R., LeBlanc, M., & Zubek, R. (2004). Methoden, Daten und Analyse: A formal approach to game design and game research, 4(1), 1–5.
Ib´
anez, M.-B., Di-Serio, ´
A., & Delgado-Kloos, C. (2014). Gamication for engaging computer science students in learning activities: A case study. IEEE Transactions on
Learning Technologies, 7(3), 291–301.
Janson, A, S¨
ollner, M, & Leimeister, JM (2017). Individual Appropriation of Learning Management Systems — Antecedents and Consequences. AIS Transactions on
Human-Computer Interaction, 9(3), 173–201.
Janson, A, Sӧllner, M, & Leimeister, JM (2020). Ladders for Learning: Is Scaffolding the Key to Teaching Problem-Solving in Technology-Mediated Learning Contexts?
Academy of Management Learning & Education, 19(4), 439–468. https://doi.org/10.5465/amle.2018.0078
Karakus, M., Ersozlu, A., & Clark, A. C. (2019). Augmented reality research in education: A bibliometric study. Eurasia Journal of Mathematics, Science and Technology
Education, 15(10), Article em1755.
Katsaliaki, K. (2013). Serious games for sustainable development. Journal of Management Education, 37(6), 889–894.
Kulkarni, A. V., Aziz, B., Shams, I., & Busse, J. W. (2009). Comparisons of citations in Web of Science, Scopus, and Google Scholar for articles published in general
medical journals. Jama, 302(10), 1092–1096.
Kuo, M.-S., & Chuang, T.-Y. (2016). How gamication motivates visits and engagement for online academic dissemination–An empirical study. Computers in Human
Behavior, 55, 16–27.
Kurniawan, H. C., Sitohang, B., & Rukmono, S. A. Gamication of mobile-based Japanese language shadowing. International Conference of Articial Intelligence and
Information Technology (ICAIIT), 215–219.
Lamb, R. L., Annetta, L., Firestone, J., & Etopio, E. (2018). A meta-analysis with examination of moderators of student cognition, affect, and learning outcomes while
using serious educational games, serious games, and simulations. Computers in Human Behavior, 80, 158–167.
Landers, R. N. (2015). Developing a theory of gamied learning: Linking serious games and gamication of learning. Simulation & Gaming, 45(6), 752–768. https://
doi.org/10.1177/1046878114563660
Liao, H., Mariani, M. S., Medo, M., Zhang, Y.-C., & Zhou, M.-Y. (2017). Ranking in evolving complex networks. Physics Reports, 689, 1–54.
Li, N., & Kirkup, G. (2007). Gender and cultural differences in internet use: A study of China and the UK. Computers & Education, 48(2), 301–317.
Li, K., Rollins, J., & Yan, E. (2018). Web of science use in published research and review papers 1997–2017: A selective, dynamic, cross-domain, content-based
analysis. Scientometrics, 115(1), 1–20.
Liu, D., Li, X., & Santhanam, R. (2013). Digital games and beyond: What happens when players compete. MIS Quarterly, 327(1), 111–124.
Liu, Zhichun, Moon, Jewoong, Kim, Byungjoo, & Dai, Chih-Pu (2020). Integrating adaptivity in educational games: a combined bibliometric analysis and meta-
analysis review. Educational technology research and development, 1–29.
Lowry, P. B., Petter, S., & Leimeister, J. M. (2020). Desperately seeking the artefacts and the foundations of native theory in gamication research: Why information
systems researchers can play a legitimate role in this discourse and how they can better contribute. European Journal of Information Systems, 29(6), 609–620.
Martí-Parre˜
no, J., M´
endez-Ib´
a˜
nez, E., & Alonso-Arroyo, A. (2016). The use of gamication in education: A bibliometric and text mining analysis. Journal of Computer
Assisted Learning, 32(6), 663–676.
Mekler, E. D., Brühlmann, F., Tuch, A. N., & Opwis, K. (2017). Towards understanding the effects of individual gamication elements on intrinsic motivation and
performance. Computers in Human Behavior, 71, 525–534.
Memon, A. R. (2018). Predatory journals spamming for publications: What should researchers do? Science and Engineering Ethics, 24(5), 1617–1639.
Meo, P. de, Ferrara, E., Fiumara, G., & Provetti, A. (2011). Generalized louvain method for community detection in large networks. International Conference on
Intelligent Systems Design and Applications, 88–93.
Muntean, C. I. (2011). Raising engagement in e-learning through gamication. International Conference on Virtual Learning ICVL, 323–329.
Peterson, M. (2010). Computerized games and simulations in computer-assisted language learning: A meta-analysis of research. Simulation & Gaming, 41(1), 72–93.
Putz, L.-M., Hofbauer, F., & Treiblmaier, H. (2020). Can gamication help to improve education? Findings from a longitudinal study. Computers in Human Behavior,
106–392.
Ross-White, A., Godfrey, C. M., Sears, K. A., & Wilson, R. (2019). Predatory publications in evidence syntheses. Journal of the Medical Library Association: JMLA, 107
(1), 57.
Rowe, J. P., Shores, L. R., Mott, B. W., & Lester, J. C. (2011). Integrating learning, problem solving, and engagement in narrative-centered learning environments.
International Journal of Articial Intelligence in Education, 21(1–2), 115–133.
Sailer, M., Hense, J. U., Mayr, S. K., & Mandl, H. (2017). How gamication motivates: An experimental study of the effects of specic game design elements on
psychological need satisfaction. Computers in Human Behavior, 69, 371–380.
Sailer, M., & Homner, L. (2019). The gamication of learning: A meta-analysis. Educational Psychology Review, 1–36.
Salen, K., & Zimmerman, E. (2004). Rules of play: Game design fundamentals. Massachusetts: MIT Press.
Santhanam, R., Liu, D., & Milton-Shen, W.-C. (2016). Research note-gamication of technology-mediated training: Not all competitions are the same. Information
Systems Research, 27(2), 453–465.
Sauv´
e, L., Renaud, L., & Kaufman, D. (2010). Games, simulations, and simulation games for learning: Denitions and distinctions. Educational gameplay and simulation
environments: Case studies and lessons learned. IGI Global.
Scheiner, C. (2015). The motivational fabric of gamied idea competition: The evaluation of game mechanics from a longitudinal perspective. Creativity and Innovation
Management, 24(2), 341–352.
Sch¨
obel, S., Janson, A., Jahn, K., Kordyaka, B., Turetken, O., Djafarova, N., … Adam, M. (2020). A research agenda for the why, what, and how of gamication designs
results on an ECIS 2019 panel. Communications of the Association for Information Systems.
Sch¨
obel, S, Janson, A, & Mishra, A (2019). A Congurational View on Avatar Design – The Role of Emotional Attachment, Satisfaction, and Cognitive Load in Digital
Learning. ICIS 2019 Proceedings.
Seaborn, K., & Fels, D. I. (2015). Gamication in theory and action: A survey. International Journal of Human-Computer Studies, 74, 14–31.
Severin, A., & Low, N. (2019). Readers beware! Predatory journals are inltrating citation databases. Springer.
Silva, R., Rodrigues, R., & Leal, C. (2020). Gamication in management education-A literature mapping. Education and Information Technologies, 25(3), 1803–1835.
Simoes, J., Díaz Redondo, R., & Fern´
andez Vilas, A. (2013). A social gamication framework for K-6 learning platform. Computers in Human Behavior, 29(2), 345–353.
Suh, A., Cheung, C. M. K., Ahuja, M., & Wagner, C. (2017). Gamication in the workplace: The central role of the aesthetic experience. Journal of Management
Information Systems, 34(1), 268–305.
Sung, H.-Y., & Hwang, G.-J. (2013). A collaborative game-based learning approach to improving students’ learning performance in science courses. Computers &
Education, 63, 43–51.
Super, J., Keller, R. H., Betts, T. K., & Roach Humphreys, J. (2019). Simulation games: Learning goal orientations and norms for knowledge sharing. Academy of
Management Proceedings, (1).
S. Sch¨
obel et al.
Computers & Education 173 (2021) 104296
23
Sweller, J. (1994). Cognitive load theory, learning difculty, and instructional design. Learning and Instruction, 4(4), 295–312. https://doi.org/10.1016/0959-4752
(94)90003-5
Trinidad, M., Ruiz, M., & Calder´
on, A. (2021). A bibliometric analysis of gamication research. IEEE Access, 9, 46505–46544.
Vogel, J. J., Vogel, D. S., Cannon-Bowers, J., Bowers, C. A., Muse, K., & Wright, M. (2006). Computer gaming and interactive simulations for learning: A meta-analysis.
Journal of Educational Computing Research, 34(3), 229–243.
Vom Brocke, J., Simons, A., Riemer, K., Niehaves, B., Plattfaut, R., & Cleven, A. (2015). Standing on the shoulders of giants: Challenges and recommendations of
literature search in information systems research. Communications of the Association for Information Systems, 37(9), 205–244.
Vos, N., van der Meijden, H., & Denessen, E. (2011). Effects of constructing versus playing an educational game on student motivation and deep learning strategy use.
Computers & Education, 56(1), 127–137. https://doi.org/10.1016/j.compedu.2010.08.013
Webster, J., & Watson, R. T. (2002). Analyzing the past to prepare for the future: Writing a literature review. MIS Quarterly, 26(2), 13–23.
Werbach, K., & Hunter, D. (2012). For the win: How game thinking can revolutionize your business. Wharton Digital Press.
Wong, J., Khalil, M., Baars, M., Koning, B. B. de, & Paas, F. (2019). Exploring sequences of learner activities in relation to self-regulated learning in a massive open
online course. Computers & Education, 140, 103595.
Wouters, P., van Nimwegen, C., van Oostendorp, H., & Van Der Spek, E. D. (2013a). A meta-analysis of the cognitive and motivational effects of serious games. Journal
of Educational Psychology, 105(2), 1–17.
Wu, W.-H., Hsiao, H.-C., Wu, P.-L., Lin, C.-H., & Huang, S.-H. (2012). Investigating the learning-theory foundations of game-based learning: A meta-analysis. Journal
of Computer Assisted Learning, 28(3), 265–279.
Yien, J.-M., Hung, C.-M., Hwang, G.-J., & Lin, Y.-C. (2011). A game-based learning approach to improving students’ learning achievements in a Nutrition course.
Turkish Online Journal of Educational Technology-TOJET, 10(2), 1–10.
Yu, F.-Y., Chang, L.-J., Liu, Y.-H., & Chan, T.-W. (2002). Learning preferences towards computerised competitive modes. Journal of Computer Assisted Learning, 18(3),
341–350.
Zainuddin, Z., Chu, S. K. W., Shujahat, M., & Perera, C. J. (2020). The impact of gamication on learning and instruction: A systematic review of empirical evidence.
Educational Research Review.
Zhang, F., & Kaufman, D. (2016). Physical and cognitive impacts of digital games on older adults: A meta-analytic review. Journal of Applied Gerontology, 35(11),
1189–1210.
Zyda, M. (2005). From visual simulation to virtual reality to games. Computer Society, 38(9), 25–32.
S. Sch¨
obel et al.
