ArticlePDF Available

Abstract and Figures

Although serious games are proven to serve as educational tools in many educational domains, there is a lack of reliable, automated and repeatable methodologies that measure their effectiveness: what do players know after playing serious games? Do they learn from them? Previous research shows that the vast majority of serious games are assessed by using questionnaires, which is in stark contrast to current trends in the video game industry. Commercial videogame developers have been learning from their players through Game Analytics for years via non-disruptive game tracking. In this paper, we propose a methodology for assessing serious game effectiveness based on non-disruptive in-game tracking. The methodology involves a design pattern that structures the delivery of educational goals through a game. This structure also allows one to infer learning outcomes for each individual player, which, when aggregated, determine the effectiveness of a serious game. We tested the methodology by having 320 students play a serious game. The proposed methodology allowed us to infer players’ learning outcomes, to assess the game effectiveness levels and to identify issues in the game design.
Content may be subject to copyright.
A methodology to assess the
effectiveness of serious games
and infer player learning outcomes
Ángel Serrano-Laguna1, Borja Manero1, Manuel Freire1, Baltasar Fernández-Manjón1
1 Department of Software Engineering and Artificial Intelligence, Complutense University of
Madrid. Madrid (Spain)
Corresponding Author:
Ángel Serrano-Laguna.
Department of Software Engineering and Artificial Intelligence
Complutense University of Madrid
Facultad de Informática
C. Profesor José García Santesmases, s/n.
28040 Madrid (Spain)
Email: angel.serrano@fdi.ucm.es
Abstract
Although serious games are proven educational tools in many educational domains, they lack
reliable, automated and repeatable methodologies to measure their effectiveness: what do
players know after playing a serious game? Did they learn with it? Literature research shows
that the vast majority of serious games are assessed through questionnaires, which strikes a
stark contrast with current trends in the video game industry. Commercial videogames have
been learning from their players through Game Analytics for years, using non-disruptive
game tracking. In this paper, we propose a methodology to assess serious games effectiveness
using non-disruptive in-game tracking. The methodology proposes a design pattern that
structures the delivery of educational goals within the game. This structure also allows
inferring learning outcomes for each individual player, which, when aggregated, would
determine the effectiveness of the serious game. We have tested this methodology with a
serious game that was played by 320 students. The proposed methodology allowed us to infer
playerslearning outcomes and assess game effectiveness and to spot issues in the game
design.
This research has been partially financed by the Ministry of Education, Culture and Sport of
Spain, through their FPU Programme (grant FPU12/04310), by the Regional Government of
Madrid (eMadrid S2013/ICE-2715), by the Complutense University of Madrid (GR3/14-
921340), by the Ministry of Education (TIN2013-46149-C2-1-R), by the RIURE Network
(CYTED 513RT0471) and by the European Commission (RAGE H2020-ICT-2014-1-644187,
BEACONING H2020-ICT-2015-687676).
Keywords
serious games, learning analytics, game design, learning outcomes analysis, educational games
1. Introduction
A serious game is a video game designed with purposes beyond pure entertainment [1]. Serious
games are multimedia tools by nature. As a subfamily of videogames, they combine different types
of media (animations, music, text…) to create immersive experiences for the players. Their
versatility allow them to be used as tools with many applications in different domains. One of the
main ones is education, where they have become proven learning tools: they are used across many
domains with multiple goals and formats, and their acceptance and effectiveness is almost always
positive [2, 3]. Traditionally, a large percentage of serious games has been both developed and
deployed by educational researchers, limiting their scope and reach. This trend is starting to change.
Nowadays, widespread use of Virtual Learning Environments (VLE) allows for the application of
serious games in unprecedented scales. To reach their full potential, serious games should adopt the
latest advances in education and commercial videogames [4].
On-line education has increased exponentially in recent years, and many students now learn through
Internet-connected devices. This vastly increases the amount of educational data available for
analysis. Disciplines such as Learning Analytics (LA) or Educational Data Mining (EDM) study the
patterns inside students’ interactions to better understand the underlying learning processes [5, 6].
This knowledge can be used by different stakeholders with diverse purposes: from university
administrators calculating dropout rates in each class, to teachers identifying students at risk of
school failure [7].
Serious games (and video games in general) are particularly well suited for data analysis. Their
highly interactive nature, based on a constant loop of user input followed by game feedback, pose
them as rich sources of interaction data. These interactions can be later analyzed to explore how
users play, and, in the case of serious games, understand how users learn.
The video game industry has been performing these types of analysis in commercial games for
years, via Game Analytics (GA) [8]. One of the main uses of GA is to measure balance in
gameplay: a balanced video game is one that keeps its players in the flow zone, a state where the
player feels challenged by the game, but neither bored nor frustrated [9]. GA helps to locate parts
inside games where players get stuck or quit; and moments where a games mechanics or internal
rules fall short. GA also provides clues on how to fix these problems.
Commercial video games usually collect data from their players in a non-disruptive way, with
tracking systems that go unnoticed by the players [10]. However, according to the literature [11],
the main method to assess any aspect of a serious game is the use of questionnaires filled by
players. There is a clear need to combine the emerging disciplines of LA and EDM with the non-
disruptive techniques of GA to provide reliable, automated and repeatable assessment for serious
games.
Serious game assessment can focus on many results, such as usability, engagement or motivation.
However, learning outcomes is the result most stakeholders want to obtain from serious games [12].
Learning outcomes are also the most frequent result assessed in recent serious games [11], and
some authors even believe that such outcomes could be used to replace standardized tests [13].
However, multiple issues with serious games must first be addressed. One of them is the lack of
methods to assess serious games effectiveness [14]: teachers, lecturers and policy-makers need
guarantees that serious games are effective enough to be used in the classroom. In this regard, the
use of GA techniques with serious games can provide stakeholders with objective and reliable data.
In this paper, we propose a methodology to infer learning outcomes and serious games effectiveness
based on non-disruptive tracking. The methodology targets two different phases in the life of a
serious game: 1) its design and implementation, where we propose a game-design pattern to shape
the delivery of the educational content throughout the game, and 2) its validation and deployment,
where we propose an analysis, based on the game-design pattern, to infer learning outcomes and
game effectiveness.
The paper is structured as follows: Section 2 presents a research review on serious game
assessment. Section 3 presents the methodology and section 4 describes an experimental case study
where the methodology was applied. Section 5 presents the results of the case study, which are then
discussed in Section 6. Finally, section 7 presents the conclusions, some limitations, and future
work.
2. Serious games assessment
Although the most common tool to assess serious games are questionnaires [11], several authors
have addressed the implications of using non-disruptive tracking for this task. Authors have
proposed a set of minimum requirements to enable automatic assessment in serious games [15], and
have addressed the game design implications of combining learning analytics and serious games
[16]. The project ADAGE [17] is a framework that defines several “assessment mechanicsthat
capture basic gameplay progression and critical achievements. Similarly, we have previously
proposed a set of universal “traces”, particularly interesting for serious games, that can be emitted
by any video game [18].
Other authors have implemented their own ad-hoc analytics to, for instance, analyze playerssteps
in a math puzzle to predict their movements based on current game state [19], assess learning
outcomes analyzing answers to quizzes integrated in the game [20], or analyze how players
progress in learning-language courses to create rich visualizations for teachers [21].
We consider that serious game designers must take into account analytics and assessment
constraints from the inception of the game and throughout the design phase [15]. Many authors
have defined methodologies and guides to design serious games [13, 22–25], however, this body of
research proposes methodologies that are applicable to any analytics-aware video game, serious or
not. In particular, these works usually do not address key serious-game aspects, such as how to
deliver knowledge and educational contents through gameplay or how to infer the corresponding
learning outcomes. Some work is starting to explore these issues, proposing a taxonomy of possible
elements that a serious game should include to be more effective [26].
To summarize, we found research that describes effective analytics-aware serious game design, but
lacks concrete methodologies to infer learning outcomes. On the other hand, there is research that
proposes ways to analyze serious game learning outcomes, either via general frameworks or ad-hoc
analysis, but without addressing the implications of that assessment in the game design. We propose
to combine both approaches to define a methodology that tackles all the phases in the development
of a serious game, from game design and implementation, to deployment and learning outcome
analysis.
3. Proposed methodology
Our methodology pursues two goals: 1) to ease the measurement of serious game learning outcomes
and 2) to provide a systematic way to assess the effectiveness of serious games as a whole. To
achieve these goals, our approach covers the complete lifecycle of the serious game (Figure 1). The
process starts in the design phase, where the learning goals and the target population are the basis to
create a learning and game design. These designs combined are used to implement the game, which
is then validated in a formative evaluation with a sample of the target population. This process is
repeated until the game is fully validated. Then, the game is ready to be used by the target
population (deployment). In the following subsections, we describe each step of the process in more
detail.
Figure 1. Serious game design and deployment process, with learning outcomes assessment.
3.1. Design and implementation
In the context of our methodology, we define “learning designas the transformation of learning
goals into game mechanics and learning outcomes observables, considering the characteristics of
the target population.
The chosen game mechanic should fulfill two requirements: 1) that is appropriate for the learning
goal content, using models like the one presented in [27], where learning mechanics are mapped to
game mechanics; and 2) that players’ gameplay can produce learning outcome observables (also
termed events) that attest the playersknowledge or skill.
During game design, these constraints, along with many other considerations for the game (such as
art style, storytelling or technologies), shape the implementation of the serious game. Additionally,
in this phase designers must define how the serious game should scaffold the delivery of the
learning goals. Although there is a lack of concrete methodologies to translate educational theories
into game design aspects [28], some authors have proposed models describing learning processes in
videogames. For instance, in the serious games domain, Kiili proposes the experiential gaming
model [29] and problem-based learning [28], both of them based on an iterative process in which
players form an strategy, experiment in the game world, receive feedback, and reflect on the results.
In the commercial videogames domain, there are similar proposals that split the experimentation in
two sub-steps: experimentation in a safe game environment”, where the level of the difficulty of
the challenge to overcome is low and mistakes are not punished, and experimentation in an “unsafe
game environment”, where the level of the difficulty is higher, and mistakes are punished (e.g.,
losing game lives, coins, score, etc.) [30].
For the purposes of our methodology, we have combined and extended these ideas into a game
design pattern that also considers the learning outcomes observables. Each learning goal is
presented to players throughout 3 phases, with 2 points of non-disruptive measurement (Figure 2):
1) Strategy: Players are first introduced to the learning goal. This can include knowledge they
might need in subsequent steps, as well as concrete instructions on how to interact with the
game world, for instance through non-interactive scenes or game tutorials. The player
receives information to understand the challenge behind the learning goal and start forming
some strategies to tackle it. This is also coherent with the initial exploration behavior that is
very common in games.
2) Practice: players start to apply the knowledge presented in the previous phase. This practice
must occur in a game environment where players' mistakes have either no consequence at all
Figure 2. Game design pattern phases. The phases would be repeated for each sub-goal.
Observables are emitted during practice and mastery phases.
or only mildly adverse consequences (“safe game environment”). This experimentation must
be designed in such a way that players can make deductions and test hypothesis on both the
knowledge presented in the previous phase and the game mechanics. In this phase students
test and practice their strategies. Strategies that work better will later be refined by the
player during the mastery phase.
In this phase, players apply the knowledge associated with the current learning goal for the
first time. This allows us to collect initial observables from which their initial knowledge
can be estimated.
3) Mastery: players are required to prove that they have acquired the intended knowledge,
facing challenges similar to those presented in the practice phase, but with increasing
difficulty, and greater in-game consequences, such as loss of score or in-gamelives
(“unsafe game environment”).
In this phase, players prove the degree to which they have acquired the intended skill or
knowledge – therefore, we can collect final observables that will allow us to measure their
final progress towards the learning goal.
These three phases can be iterated to deliver multiple learning goals, or to deliver a single goal with
increasing difficulty, adding a new related concept or skill in each cycle. Additionally, this game
pattern optimizes the time the players are in the flow zone [9], since it alternates moments where
players are learning new things in a safe environment (practice), with moments where they are
challenged to prove their skills (mastery), all with an incremental approach to avoid frustration.
3.2. Collecting observables
Players perform different interactions to advance in the game: they make choices, resolve puzzles,
beat bosses, etc. These events will be the core observables to perform the learning outcomes
analysis. The following principles (many of them shared with general GA) can facilitate this
analysis:
1. Observables data should be time-stamped events, representing simple interactions of the
player with the game [18]. These events should be sent to a central server, where all player
interactions will be stored for later access and analysis.
2. Events sent to the server should be raw interactions instead of opaque scores [18, 31]. For
instance, if the mastery phase contains two puzzles, the events to transmit would be the
interactions performed to resolve the two puzzles, instead of just a combined score of the
final result. This ensures flexibility, since scores can be later recalculated from interaction
data if the subsequent analysis identifies a need to do so.
3. Data collection should be as non-disruptive during gameplay as possible. Ideally, game flow
should never be interrupted to collect data – players should not be explicitly asked to stop
their play to pass an exam or to answer questions not integrated in the gameplay.
Once all interaction events (observables) are stored in a central location, analysis can begin.
3.3. Learning outcome analysis
We store all gameplay interactions in a single server. Following our design pattern, each interaction
is associated with a learning phase (strategy, practice or mastery) of a specific learning goal.
Interactions from the strategy phase are not used to infer learning outcomes (since this phase should
only contextualize the learning goal). Analyzing interactions from the other two phases, we can
calculate two assessment scores:
1. Initial assessment (IA), using the initial observables from the practice phase. It estimates
the learner’s initial degree of knowledge. A high value would indicate that the player likely
possessed this knowledge before starting to play, while a low value would mean the
opposite.
2. Final assessment (FA) using the final observables from the mastery phase. It estimates the
learning outcome. A high value would indicate the player succeeded in the learning goal,
while a low value would indicate she failed.
The specific steps to transform observables events into IA and FA will be different for each serious
game. However, they can generally be expressed with formulas that combine data from each
interaction. In section 4, we provide details of this process in a real case study.
We define two assessment thresholds: an initial threshold (IT) associated with the IA, and a final
threshold (FT) associated with the FA. These thresholds are used to determine whether a phase is
successfully accomplished or not. For instance, if FAs value ranges from 0 to 1, a possible value for
FT could be 0.5, so we consider that a player that reaches an FA value equal or greater than 0.5 have
successfully completed the mastery phase.
For serious games that includes multiple learning goals, we can calculate their global IA and FA
using a weighted average combining results from each learning goal: given a game with N
educational goals, each with two assessments (IAi, FAi), two thresholds (ITi, FTi) and a weight (Wi),
we can then calculate the global assessment value (A) for the initial and final assessments as:
=×
=1
=1
And the global threshold value (T) for initial and final thresholds as:
=×
=1
=1
With these values, we can now estimate learning outcomes and assess the serious games’
effectiveness.
3.3.1. Inferring playerslearning outcomes
The analysis of observables or signals provides two measures for each learning goal: FA and IA.
With these values, we can calculate two concrete learning outcomes:
FA as the final score of the player: We can use FA as a score or mark for the players when
considered as students – essentially scoring what they know after playing the game. We
should avoid using IA to calculate this mark. Although it represents players' knowledge,
using it to calculate final marks would be unfair, since IA takes into account mistakes
committed during the practice phase, while a fair grade should only consider what students
know at the end of the game, not what they ignored at the beginning.
The difference between accomplishments in the practice and mastery phase as game
effectiveness: If we compare IA and FA to their respective thresholds (IT and FT), we can
determine whether a player succeeded in the practice and mastery phase. The game is most
effective when players that failed in the practice phase ended up succeeding in the mastery
phase, as this indicates a knowledge gain. This difference is the base from which we
calculate the serious game effectiveness.
3.3.2. Assessing serious game effectiveness
In the context of our methodology, we consider that a serious game is effective if we find a positive
change in the knowledge level of the player. We can determine this change using the results of IA
and FA respect to IT and FT. Using these values we can classify each player in a different learning
category:
If FA FT, the players successfully completed the mastery phase and possess the skill
intended for the learning goal. Depending on the IA value, we can classify players as either:
o Learners, if IA < IT: players committed errors during the practice phase, indicating
that they did not possess the skill or knowledge before playing the game. However,
they ended up being successful in the master phase, which means there is an
educational gain during the gameplay.
o Masters, if IA IT: the players did not commit errors during the practice phase,
indicating that they probably possessed the skill or knowledge before playing.
If FA < FT, the players failed the mastery phase and do not possess the skill intended for the
educational goal. Depending on the IA value, we can classify players into two different
categories:
o Non-Learners, if IA < IT: the players also failed the practice phase, indicating that
they struggled throughout the game potentially with little or no benefit.
o Outliers, if IA IT: the players succeeded during the practice phase, but were unable
to apply the acquired knowledge in the mastery phase.
We determine serious game effectiveness by classifying each gameplay session according to these
criteria, and then comparing the total number of players in each category.
If the majority of players were learners, the game was highly effective: most players learned
something while playing. If the majority were masters, the game produced no learning effect since
most players already possessed the intended knowledge before playing. If the majority were non-
learners, the game was not effective at all, since most players were unable to success in any phase.
And finally, a majority of outliers indicates that the game and/or the chosen FA and IA formulas
probably had design flaws.
It is important to remark that most serious games will output different results for different
populations. A serious game could be highly effective with kids from 10 to 12, and not effective at
all with kids from 7 to 9. The key is to have a well-defined target population during the design
phase of the serious game, and to perform a validation process to ensure that effectiveness goals are
met.
3.4. Validation and deployment
Once we have the serious game implemented along with the infrastructure to track its observables,
hooked to the learning outcomes analysis, we need to validate it.
In the validation phase, domain experts and, ideally, a sample of the target population, play the
serious game, producing gameplays that are later assessed with the learning outcomes analysis,
yielding preliminary results, in a process usually called formative evaluation [32]. This process is
iterative and designed to detect aspects to fix, polish, tweak or improve in the serious game, which
can range from changing the game mechanic of a learning goal, because preliminary results suggest
low performance, to changing the how FA and IA are calculated because experimental results
contradict certain game design hypothesis.
Once the game is validated, it can be used in production in the final deployment. In this final phase,
the serious game and its learning outcomes analysis are used to assess the students that play with it
(final evaluation).
4. Case study
In previous sections, we have presented our methodology to model and infer learning outcomes and
effectiveness in serious games. This section describes a case study that illustrates how this
methodology works when applied. The case study starts with the following research questions:
RQ1.What are the implications of using our game-design pattern during the design and
implementation of a serious game?
RQ2. What results, regarding learning outcomes and effectiveness, can be obtained from a
serious game developed and analyzed with this methodology?
To answer them, we used the proposed methodology to implement and analyze “The Foolish Lady”,
a serious game1 based on the homonymous theater play by Spanish playwright Lope de Vega. In
this game, players are presented with several language and literature challenges. Its main learning
goal is to teach youngsters about Spanish Golden Century poetry. In the following subsections, we
describe the design and implementation process, the data collection and analysis process, and the
results of an experiment with 320 high school students that played the game.
4.1. Design and implementation
“The Foolish Lady” serious game [33, 34] is an adventure game based on a classical Spanish play.
In the game, players advance through scenes of the play making decisions that affect the overall
story and the final scene. Along the way, they find puzzles and mini-games where they need to
apply knowledge on language and literature. The game is designed to be completed in 30 to 40
minutes.
1 Available (in Spanish) at https://play.google.com/store/apps/details?id=es.eucm.androidgames.damaboba
One of its main learning goals is to teach poetry structure and rhymes, especially, the “redondilla”, a
Spanish poetic composition that uses a specific rhyming scheme and verse length. During the
learning design phase, the chosen game mechanic was point and click mini-games (drag-and-drop
puzzles and option selection in conversations with non-playable characters in the game), typical of
adventure games, due to the educational benefits of this genre [35]. During the game design, we
subdivided this goal into the three phases defined by our design pattern. Figures 3, 4, and 5 show in-
game screen captures for each of these phases.
Players are first presented with a written explanation with notions of rhymes and the “redondilla
structure (Figure 3). These instructions appear in two non-interactive scenes that can be skipped
(after reading the content or not) with a click. These scenes belong to the strategy phase.
Later on, players find a mini-game where they need to complete a poem composed as a “redondilla
(Figure 4). The poem is missing five words and the players can fill in the blanks by dragging words
from a container placed at the right side of the screen. Once filled, they can check the correctness of
the poem by clicking a check button. They can try as many times as they want, until they find the
right combination of words: as the practice phase of the goal, results in this mini-game are
irrelevant for the final score.
Finally, players encounter two mini-games (Figure 5). In the first one, players must fight a knight
by exchanging rhyming verses. Players win this battle if they choose three correct rhyming replies
in a row – or lose it if they fail three times in a row. Their score decreases with each error. In the
second and final mini-game, the foolish lady's father assesses the protagonist’s suitability as a son-
in-law by asking the player a series of questions on the “redondilla” poetic composition. Players can
answer these questions only once, and both the score and their protagonist’s marital prospects
decrease if they fail. Both mini-games belong to the mastery phase of the goal, and therefore their
results affect the final score.
Figure 3. The game exposes the basic concepts of the redondillathrough two screens with
written explanations.
Figure 4. In the first puzzle, players need to apply their knowledge of the redondilla”. They
can try as many times as they want.
Figure 5. In the final mini-games, players must prove their knowledge. In both cases, players
lose score with each error.
4.2. Collecting observables
To record and analyze the gameplay sessions of all students, we developed a framework composed
by a tracker, bundled within the game itself responsible for sending events (observables), and by a
collector server, responsible for receiving and storing the events. The type of events are fully
detailed in [18, 31]; here, we only highlight those events relevant for the learning outcomes
analysis:
Events representing that a new attempt to beat the “redondilla puzzlewas started. Every
time the player click the check” button and the result is incorrect a new attempt starts.
Events representing that a new attempt to beat the fight mini-gamewas started. Every
time the player loses the fight and restarts the mini-game a new attempt starts.
Answers chosen by the player during the final mini-game.
The game itself does not make any assessment calculation: only raw events are sent to the server.
4.3. Learning outcomes analysis
All players encounter the 3 mini-games during their playthroughs: the “redondillapuzzle mini-
game in the practice phase, and the fight mini-game and the test mini-game in the mastery phase.
For each mini-game we calculate a score between 0 and 1:
Redondilla Game score (RG): if A is the observable representing the number of attempts to
solve the “redondilla” puzzle mini-game, RG is computed using the formula RG = 1 - (MIN
(A - 1, AMAX) / AMAX), where AMAX is the reasonable number of attempts needed to solve the
game. The initial assessment will be 1 when the player beats the puzzle at the first attempt,
i.e., A = 1. The initial assessment will be 0 if the player does not complete any attempt on
the puzzle, or tries over AMAX times.
Fight Game score (FG): if E is the observable representing the number of erroneous
options chosen before completing the fight mini-game, FG is calculated by the formula FG
= MAX (0, 1 - (MIN (E, EMAX) / EMAX)), where EMAX is the maximum number of reasonable
errors needed to beat it.
Test Game score (TG): In the test mini-game, each question has four answers, and only one
of them is correct. Each answer has an associated score. The correct answer always has a
score of 0, and the rest of answers have scores that correspond to their distance from the
truth: 1 for answers that are almost right, 2 for answers that are wrong, and 3 for answers
that, due to their content or formulation, are clearly intended as jokes. If I is the observable
representing the accumulated score of incorrect answers after finishing the test mini-game,
TG = MAX (0, 1 – I / 4), since the number of questions asked is 4.
We set AMAX = 3 and EMAX = 6. These values were agreed between game designers and educators,
considering the educational and game challenge each mini-game entails for the players. However,
since we are going to track raw A and E values, AMAX and EMAX values can always be changed a
posteriori if, after running the validation process, the data suggest that more appropriate values
should be applied.
With these values, now we can calculate IA and FA:
IA = RG, since the “redondilla” puzzle mini-game is the only one in the practice phase.
FA = FG x 0.5 + TG x 0.5, since the fight and test mini-game are in the mastery phase, and
we have decided to give both equal weights in the final score.
For all mini-games, we set the assessment threshold to 0.5, making both the IA and FA thresholds
also 0.5.
Table 1 shows possible values for RG, FG, TG, IA and FA used in the analysis of this experiment.
Initial Assessment / Redondilla
game
AMAX=3
IA = RG= 1
( MIN( A - 1, AMAX) / AMAX)
Final Assessment (FA)
Fight game
EMAX=6
FG= 1 – (MIN( E, EMAX) / EMAX)
Test game
TG = MAX(0, 1 - I/4)
Attempts (A) RG/IA Errors (E) FG Incorrect score (I) TG
FINAL ASSESS-
MENT
0.5*FG+0.5*TG
1
1
0
1
0
1
1
2
.66
1
.83
1
.75
0.8
3
.33
2
.66
2
.5
0.58
4
0
3
.5
3
.25
0.375
5
0
4
0.33
4
0
0.165
6
0
5
0.16
5
0
0.08
7 or more
0
6 or more
0
6 or more
0
0
Table 1: Some illustrative values for IA and the components of FA.
4.4. Case study
To answer RQ2, we ran an experiment with high school students that played the serious game.
4.4.1. Experimental design
Before high school students (our target population) played the game, and as part of the validation
process, we first ran a formative evaluation with graduate students [34] and the teachers involved in
the experiment. Results from this validation helped to fix some implementation flaws, and to
improve the gameplay and the overall learning design. For instance, two questions from the final
mini-game were changed to improve its alignment with the learning goal.
After the validation, high school students played “The Foolish Ladyduring 30 to 40 minutes in a
PC, under the supervision of a researcher who did not provide any sort of assistance (only a brief
indication on how to start the game). We collected one gameplay per student (deployment phase).
We consider a gameplay session as the set of traces (interactions with the game) generated from the
first screen to the final screen of the game.
From each gameplay, we computed 3 values: RG, FG and TG. Those that did not complete a mini-
game scored 0. From these variables, we calculated IA and FA with the formulas presented above.
Using their results, we classified each student in a learning category (learner, master, non-learner or
outlier) to draw conclusions on the games effectiveness.
In order to gain insight into our methodology, we wanted to know if we could answer these case
study questions (CSQ) concerning “The Foolish Ladyserious game:
CS1: Did the students possess the intended skill at the end of “The Foolish Lady”
game? Given our demographic variables, were there differences between groups?
CS2: Is “The Foolish Lady” game effective at teaching its intended skill to our
population? Given our demographic variables, were there differences between groups?
4.4.2. Participants
The experiment involved N = 320 high school students from 8 different schools in Madrid. 32 of the
gameplay sessions were corrupted or incomplete, due to different technical problems while playing
the game (power cuts, Internet connection issues and computer malfunctions), and were therefore
discarded.
The gender proportion in the resulting population (N = 288) was 44.4% females and 55.6% males.
The participants were all between the ages of 12 and 16 (mean age was 13.70 ± 1.27), from high
schools of the Madrid area. By schools, 3 were charter or private schools (58% of the population),
and 4 were public schools (42% of the population). In terms of gender, age and school type, the
participants are a representative sample of the student population of Madrid for this age [36, 37].
Additionally, we collected participant game habits to classify each student in a player category,
evaluating what types of games they play and how often. According to the instrument developed by
[38], 14.9% were non-gamers (they never play any type of video games), 28.8% were casual
gamers (they play casual video games for short periods of times), 31.6 % were hardcore gamers
(they frequently play games such as FPS or MMORPG) and 24.6% were well-rounded gamers (they
play all types of games frequently). There is a more detailed explanation of each category in [38].
5. Results
In this section, we present the results of the learning outcomes analysis of the deployment phase,
i.e., results from the high school students.
5.1. Game completion
Figure 6 shows the number of players that completed each phase of The Foolish Lady”: all 288
players started the game and also completed the strategy phase; 281 completed the “redondilla
puzzle mini-game; 246 completed the fight mini-game; and 231 completed the test mini-game. The
largest drop of players (35) happens between the “redondillapuzzle and the fight mini-game.
In summary, 80.21% of players finished the game at least once.
Figure 6: Number of players that accomplished each phase of The Foolish Lady” game.
5.2. Learning outcomes
To answer whether the students reached the intended skill level at the end of the game, we
calculated the values of RG, FG and TG, and therefore, FA and IA. In total, 196 players (68.05% of
the total population, 84.84% of players that completed the game) scored more than 0.5 (adequacy
threshold set for the game during design) in FA. IAs mean value is greater than 0.5 in all age
groups.
On the other hand, the second part of CS1 led us to calculate FA and IA across the different
demographic groups: gender, age and gaming-habits. We first performed a one-way analysis of
variance (ANOVA) over IA, to find initial differences across groups. IA showed statistically
significant differences on groups by gender and game habits (Table 2).Therefore, to consider these
differences on IA, we applied an analysis of covariance (ANCOVA) to evaluate differences on the
FA score (dependent variable), across groups (independent variables) using the IA score as
covariate. Standard preliminary checks were conducted to confirm that there was no violation of the
assumptions of normality, linearity, homogeneity of variances and homogeneity of regression slopes
[39].
Table 3 shows the ANCOVA results for the 3 independent variables, showing statistically significant
differences (p<0.05) among groups by age and game habits: First ANCOVA [between-subjects
factor: age (12 to 16); covariate: IA scores] revealed main effects of age F(4,288) =7.28, p<0.01,
and a medium ηp²=.094; second ANCOVA [between-subjects factor: gender (male, female);
covariate: IA scores] showed no main effects on gender F (1,288) =.62, p=.43, ηp²=.002; and third
ANCOVA [between-subjects factor: game-habits (4 clusters); covariate: IA scores] revealed main
effects of game-habits F(3, 288) =2.880, p=.036, and a small ηp²= .030. Table 4 shows the adjusted
means for each demographic group.
Independent variable
One-way ANOVAs on IA
N
df
F
p
Age
288
288
288
4
2.5
.031
Gender
1
18.41
<.005
Game Habits
3
12.10
<.005
Table 2: ANOVA results on IA, showing significant differences among groups by gender and
game habits.
Independent variable
ANCOVAs on FA
N
df
F
p
Partial η²
Age
288
288
288
4
7.28
.000*
.094
Gender
1
.62
.43
.002
Game Habits
3
2.88
.036*
.030
*p<0.05
Table 3: Test scores and ANCOVA results by age, gender and gaming profile.
ANCOVA
Ind. Variable
Values
N
Adj. Mean*
Std. Err.
Age
12
69
.508
.038
13
50
.508
.044
14
96
.708
.032
15
43
.631
.048
16
30
.766
.057
Gender
Female
128
.603
.029
Male
160
.633
.026
Game Habits
Casual
83
.559
.035
Non-gamer
43
.554
.049
Well-rounded
gamer
71
.680
.038
Hardcore
91
.659
.034
*Adjusted mean using practice phase scores as covariate (ia=.6146)
Table 4: FA adjusted means by age, gender and gaming profile.
5.3. Serious game effectiveness
Figure 7 shows the total number of players grouped by learning category. Most players are masters,
followed by learners. The number of outliers is higher than that of non-learners.
Figure 7. Categorization of players according to their assessment category.
Figure 8 and 9 shows the playerscategorization grouped by learning category and segmented by
age and game-habits. In all groups, the number of masters exceeds that of other categories,
especially in the 14 year-old group. In all groups, the number of outliers is greater than the number
of non-learners, except in the group of students aged 16.
Figure 8. Distribution of players across assessment categories, segmented by age.
Learners
62
Masters
143
Non-learners
35
Outliers
48
15
11
16
11
9
23
18
63
22
17
14
9
6
3
3
17
12
11
7
1
12 Y/O 13 Y/O 14 Y/O 15 Y/O 16 Y/O
Learners Masters Non-Learners Outliers
16
9
20
17
43
16
35
49
11
8
9
7
13
10
7
18
CASUAL NON-GAMER WELL-ROUND
GAMER
HARDCORE
Learners Masters Non-Learners Outliers
Figure 9. Distribution of players across assessment categories, segmented by game-habits.
6. Discussion
In this section we first present the answers to the case study questions, and then further elaborate to
answer the methodology research questions.
CS1: Did the students possess the intended skill at the end of “The Foolish Lady” game?
Given our demographic variables, were there differences between groups?
Yes. 80.2% of students completed the game, which required, by design, a basic understanding of the
principles of the learning goal. Regarding the final score of these students, ANCOVA analysis (dep.
Variable: FA; covariate: IA) has shown statistical differences when data are segmented by age and
by gaming habits.
By age, students aged 12 and 13 obtained the lowest values (Adj. Mean= .508), and students aged
16 obtained the highest values (Adj. Mean=.766). It seems natural: older students found the game
easier.
By game-habits, the segment with the best results are well-round gamers (Adj. Mean= .680),
closely followed by hardcore gamers (Adj. Mean= .659). These two types of players are used to
playing games with complex mechanics. The Foolish Lady” is an adventure game, with fairly
simple mechanics, so these playersexpertise probably helped them to complete the game more
effectively.
CS2: Is “The Foolish Ladygame effective at teaching its intended skill to our population?
Given our demographic variables, were there differences between groups?
No. Not because the players did not learn, but because, according to the results, most of them were
categorized as Masters, i.e., many of them already knew most of the educational contents. This
could mean the game was too easy for most of the players. However, we think there is an additional
problem in the game design that prevented us from capturing a more accurate IA (and,
consequently, a more accurate learning profile): since we wanted to keep the game short —so that it
could be completed in 40-minute sessions—the practice phase was deliberately shorter than the
mastery phase. This forced us to keep it only to a single, comparatively easy mini-game, whose
score was not enough to fully measure the initial knowledge. This flaw went unnoticed during the
validation process because the players in this phase were domain experts, and therefore, were
classified as masters, which seemed natural. That is why the serious game should be also validated
with a sample of the target population.
Segmenting groups by age and game-habits there is no particular group in which the game was
more effective.
These results do not imply the game has no value as educational tool. Students playing this game
enjoyed other benefits, such as a measurable increase their motivation to go to the theatre, as
demonstrated in [33].
RQ1. What are the implications of using our game-design pattern during the design and
implementation of a serious game?
The methodology forced us to define a clear learning goal from the beginning, and to stick to it
during the game development process.
In cooperation with the educational experts, we designed the mini-games clearly defining which
role each of them covered in delivering the educational goal. We defined mini-game difficulty,
weight and placement guided by the proposed game-design pattern. The mini-games were
implemented in such a way the interactions and events involved in their resolution were clearly
identified, and since the beginning, we formulated how those events were converted into
assessment.
We also integrated a tracker into the game engine to capture all relevant interactions. This approach
is common in the games industry for any analytics-related tasks, although its difficulty varies
depending on the chosen game engine. In our case, we used an open source engine, where all the
required events and interactions were generated in a handful of locations within the code; wiring in
the tracker was relatively simple.
We also needed a service to collect all the data. Ours consisted of a REST back-end to process
HTTP requests with the events, a database to store these traces, and some Python scripts to query
the database. Although we used a custom solution, the serious game could be integrated in any other
VLE. This opens new interesting questions regarding sharing of data between these systems and the
serious game, which however fall out of the scope of this paper.
RQ2. What results, regarding learning outcomes and effectiveness, can be obtained from a
serious game developed and analyzed with our methodology?
Identifying relevant educational observables during game design simplified the task of calculating
learning outcomes and game effectiveness. These results were used to answer several interesting
questions of the case study.
By default, our methodology converts the serious game into an assessment tool: it relies on clear
assessment locations that are associated with both the learning design and goals, which are then
combined to infer learning outcomes. However, using our design pattern, we can also determine if
students actually learned playing the game, which is key to assess the games effectiveness within a
particular population group.
In our case study, we concluded that the initial assessment was higher than expected. The number of
outliers indicates a design flaw in the practice phase. We consider that this finding based on actual
data is very useful. Once this flaw is detected, we can iterate over the methodology again to
improve the game.
Finally, if student demographics data is available, we can use statistical analysis to identify and
characterize those groups in which the learning outcomes are better or where they game effect is
higher. This helps to narrow down and better identify the ideal target population for the game.
7. Conclusions
In this paper we present a methodology to structure the design and assessment of serious games at
two levels: inferring learning outcomes and assessing serious games effectiveness as educational
tools. We think this is a contribution to systematize serious games development that improves some
of the aspects of the methodologies found in the literature review: our methodology is fully
integrated with the production cycle of a serious game (from design to deployment), and proposes a
non-disruptive assessment alternative to questionnaires, the most common assessment method for
serious games. It poses extra requirements during game development (a tracker in the game engine
and a server to collect the data), but with today’s big data technologies this becomes an affordable
task.
We tested our methodology developing a serious game that was played by 320 students. The
methodology clearly guided our steps in the design process, and later in the analysis that we would
use to determine if The Foolish Lady” was an effective serious game. The game proved to be an
effective assessment tool (i.e., we were able to give a mark to each student), however, it was unable
to fully capture the initial knowledge of the students.
One of the conclusions of the experiment is that the design of the practice phase is key to implement
an effective serious game. However, the balance in the practice phase can be hard to keep: the
designer wants the player to advance flawlessly, while capturing their mistakes to obtain an accurate
assessment of initial knowledge. Additionally, the implementation result derived from the case
study (i.e., a tracker and a basic server infrastructure to receive and analyze traces) is going to be
used in the RAGE European Project [40] as main infrastructure to assess games.
Although our case study is focused on a serious games designed to deliver knowledge and teach
several skills, we think the methodology could be applied to any serious game whose goal can be
measured in a quantitative way. For instance, a serious game designed to help diabetics to control
their blood glucose levels could ask for the playerslevels to determine whether the goal was
achieved (instead of relying on puzzle or mini-games, as our game does).
In summary, we consider that the methodology presented in this paper provides a richer and a more
understandable assessment analysis for serious games. One major point is that, once the game starts
sending observable events, everything is automated, and all assessments are based on how learners
interact with the game, instead of using traditional out-of-game questionnaires. Additionally, the
assessment model is adaptable to researchers’ needs since it is not hardwired to game signals: the
way each dependent variable (FA and IA) is calculated can be changed a posteriori, allowing the
constants used in assessment model to be updated if required (for instance AMAX and EMAX, in our
case study). Additionally, results obtained by this methodology could complement formal
experiments to measure serious games effectiveness, which is still an open issue [41].
We believe that this methodology opens up new research venues. In this paper we have limited the
studentsassessments to 3 particular points for clarity reasons (the 3 mini-games). In the future, we
plan to enrich our game design pattern with more observables in of both phases. These data will
provide us with more information on students’ progression, enabling researchers to build a more
precise photo of what is exactly going on during the learning process. We plan to go one step further
by analyzing other gameplay data (such as the time spent in each phase), that may shed light into
the reasons that make some players struggle in certain areas of the game. We also want to explore
further the transformation from game observables to assessment scores, by identifying and
addressing common patterns in different game mechanics.
Finally, the integration of serious games following our proposed methodology inside VLE also
raises interesting questions. What standards should be used in the communication? What
visualizations should be provided to the different stakeholders? Addressing this integration will be
an important step towards realizing the full potential of combining serious games and learning
analytics.
8. References
1. Liu G, Fu L, Rode A V., Craig VSJ (2011) Water Droplet Motion Control on
Superhydrophobic Surfaces: Exploiting the Wenzel-to-Cassie Transition. Langmuir. doi:
10.1021/la104669k
2. Squire K (2003) Video games in education. International Journal of Intelligent. Simulations
and Gaming 2:49–62.
3. Connolly TM, Boyle E a., MacArthur E, et al (2012) A systematic literature review of
empirical evidence on computer games and serious games. Comput Educ 59:661–686. doi:
10.1016/j.compedu.2012.03.004
4. Loh CS, Sheng Y, Ifenthaler D (2015) Serious Games Analytics: Theoretical Framework. In:
Serious Games Anal. Springer International Publishing, Cham, pp 3–29
5. Elias T (2011) Learning Analytics : Definitions , Processes and Potential. Learning 23. doi:
10.1.1.456.7092
6. Chatti MA, Dyckhoff AL, Schroeder U, Thüs H (2012) A reference model for learning
analytics. Int J Technol Enhanc Learn 4:318–331. doi: 10.1504/IJTEL.2012.051815
7. Ferguson R (2012) The state of learning analytics in 2012: a review and future challenges.
Tech Rep KMI-12-01. doi: 10.1504/IJTEL.2012.051816
8. El-Nasr MS, Drachen A, Canossa A (2013) Game Analytics: Maximizing the Value of Player
Data. doi: 10.1007/978-1-4471-4769-5
9. Chen J (2007) Flow in games (and everything else). Commun ACM 50:31. doi:
10.1145/1232743.1232769
10. Santhosh S, Vaden M (2013) Telemetry and Analytics Best Practices and Lessons Learned.
In: Game Anal. Maximizing Value Play. Data. pp 85–109
11. Calderón A, Ruiz M (2015) A systematic literature review on serious games evaluation: An
application to software project management. Comput Educ 87:396–422. doi:
10.1016/j.compedu.2015.07.011
12. All A, Nuñez Castellar EP, Van Looy J (2015) Towards a conceptual framework for assessing
the effectiveness of digital game-based learning. Comput Educ 88:29–37. doi:
10.1016/j.compedu.2015.04.012
13. Annetta LA (2010) The “I’s” have it: A framework for serious educational game design. Rev
Gen Psychol 14:105–112. doi: 10.1037/a0018985
14. Vargas JA, García-Mundo L, Genero M, Piattini M (2014) A Systematic Mapping Study on
Serious Game Quality. In: Proc. 18th Int. Conf. Eval. Assess. Softw. Eng. EASE 2014. pp 1–
10
15. Moreno-Ger P, Burgos D, Martínez-Ortiz I, et al (2008) Educational game design for online
education. Comput Human Behav 24:2530–2540. doi: 10.1016/j.chb.2008.03.012
16. Hauge JB, Berta R, Fiucci G, et al (2014) Implications of Learning Analytics for Serious
Game Design. In: Proc. 14th Int. Conf. Adv. Learn. Technol. IEEE, pp 230–232
17. Owen VE, Ramirez D, Salmon A, Halverson R (2014) Capturing Learner Trajectories in
Educational Games through ADAGE ( Assessment Data Aggregator for Game
Environments ): A Click-Stream Data Framework for Assessment of Learning in Play. Am
Educ Res Assoc Annu Meet 1–7.
18. Serrano Á, Marchiori EJ, Blanco Á del, et al (2012) A framework to improve evaluation in
educational games. In: IEEE Glob. Eng. Educ. Conf. IEEE, pp 1–8
19. Lee SJ, Liu Y, Popovic Z (2014) Learning Individual Behavior in an Educational Game : A
Data-Driven Approach. In: Proc. 7th Int. Conf. Educ. Data Min. pp 114–121
20. Dudzinski M, Greenhill D, Kayyali R, et al (2013) The Design and Evaluation of a
Multiplayer Serious Game for Pharmacy Students. In: Proc. 7th Eur. Conf. Games Based
Learn. Vols 1 2. pp 140–148
21. Ye F (2014) Validity, reliability, and concordance of the Duolingo English Test.
https://s3.amazonaws.com/duolingo-certifications-data/CorrelationStudy.pdf. Accessed 27
Nov 2016
22. Marne B, Wisdom J, Huynh-Kim-Bang B, Labat J-M (2012) The six facets of serious game
design: a methodology enhanced by our design pattern library. In: 21st Century Learn. 21st
Century Ski. pp 208–221
23. Dickey MD (2006) Game design and learning: a conjectural analysis of how massively
multiple online role-playing games (MMORPGs) foster intrinsic motivation. Educ Technol
Res Dev 55:253–273. doi: 10.1007/s11423-006-9004-7
24. Denis G, Jouvelot P (2005) Motivation-driven educational game design. In: Proc. 2005 ACM
SIGCHI Int. Conf. Adv. Comput. Entertain. Technol. - ACE ’05. ACM Press, New York,
New York, USA, pp 462–465
25. Dondlinger M (2007) Educational video game design: A review of the literature. J Appl Educ
Technol 4:21–31. doi: 10.1108/10748120410540463
26. Carvalho MB, Bellotti F, Berta R, et al (2015) An activity theory-based model for serious
games analysis and conceptual design. Comput Educ 87:166–181. doi:
10.1016/j.compedu.2015.03.023
27. Arnab S, Lim T, Carvalho MB, et al (2015) Mapping learning and game mechanics for
serious games analysis. Br J Educ Technol 46:391–411. doi: 10.1111/bjet.12113
28. Kiili K, Ketamo H (2007) Exploring the learning mechanism in educational games. In: Proc.
Int. Conf. Inf. Technol. Interfaces, ITI. pp 357–362
29. Kiili K (2005) Digital game-based learning: Towards an experiential gaming model. Internet
High Educ 8:13–24. doi: 10.1016/j.iheduc.2004.12.001
30. Nutt C, Hayashida K (2012) The Structure of Fun: Learning from Super Mario 3D Land’s
Director. In: Gamasutra.
http://www.gamasutra.com/view/feature/168460/the_structure_of_fun_learning_.php?page=4
. Accessed 27 Nov 2016
31. Serrano-Laguna Á, Torrente J, Moreno-Ger P, Manjón BF (2012) Tracing a little for big
improvements: Application of learning analytics and videogames for student assessment. In:
Procedia Comput. Sci. pp 203–209
32. Fuchs LS, Fuchs D (1986) Effects of Systematic Formative Evaluation: a Meta-Analysis.
Except Child 53:199–208. doi: 10.1177/001440298605300301
33. Manero B, Torrente J, Serrano Á, et al (2015) Can educational video games increase high
school studentsinterest in theatre? Comput Educ 87:182–191. doi:
http://dx.doi.org/10.1016/j.compedu.2015.06.006
34. Manero B, Fernández-Vara C, Fernández-Manjón B (2013) E-learning a escena: De La Dama
Boba a Juego Serio. Vaep Rita 1:51–58.
35. Dickey MD (2006) Game design narrative for learning: Appropriating adventure game
design narrative devices and techniques for the design of interactive learning environments.
Educ Technol Res Dev 54:245–263. doi: 10.1007/s11423-006-8806-y
36. Comunidad de Madrid (2011) Datos y Cifras de la Educación.
http://www.madrid.org/cs/Satellite?blobcol=urldata&blobheader=application/pdf&blobheade
rname1=Content-
Disposition&blobheadervalue1=filename=DATOS+Y+CIFRAS+2010_2011.pdf&blobkey=i
d&blobtable=MungoBlobs&blobwhere=1271936872331&ssbinary=true. Accessed 27 Nov
2016
37. Ministerio de Educación (2008) Escolarización y población.
http://www.mecd.gob.es/dctm/ievaluacion/indicadores/2011-
e1.2.pdf?documentId=0901e72b810b4d41. Accessed 27 Nov 2016
38. Manero B, Torrente J, Fernández-Vara C, Fernández-Manjón B (2015) Gaming preferences
and habits, gender and age on educational videogames effectiveness: An exploratory study
(In press). IEEE Trans. Learn. Technol.
39. Pallant J (2013) SPSS survival manual: a step by step guide to data analysis using IBM
SPSS. Open Univ Pr
40. Hollins P, Westera W, Manero B (2015) Amplifying applied game development and uptake.
41. All A, Nuñez Castellar EP, Van Looy J (2016) Assessing the effectiveness of digital game-
based learning: Best practices. Comput Educ 92–93:90–103. doi:
10.1016/j.compedu.2015.10.007
... Game design strategy is a crucial element while designing any serious game. We followed the well-structured methodology proposed by Serrano-Laguna et al. [52] while developing CodAR. The numerous advantages of this methodology are summarized at the end of this subsection. ...
... Practice: After getting an overview of the game in the previous phase, players are required to apply and improve their knowledge by solving game levels in this phase. As suggested by Serrano-Laguna et al. [52], we kept this phase as a safe game environment. Players can try various combinations of programming cards until they eventually solve the level. ...
... Thus, our game had a total of 3 levels in this phase. Splitting the game into phases has the following advantages as suggested by Serrano-Laguna et al. [52]: (a) it ensures a good balance between player enjoyment and learning, (b) it gives cues about the measurement of serious game learning outcomes, and (c) it provides a systematic way to assess the effectiveness of serious games. ...
Article
The ubiquity of smartphone and tablet devices, combined with the increasing availability of serious games, has enabled students to learn various abstract concepts in an appealing and convenient manner. While several researchers have explored the use of Augmented Reality (AR) in serious games, many of these games have not been critically explained or evaluated. To that end, we employed game-based learning methodologies and Game Learning Analytics (GLA) to systematize the design and evaluation of an AR-based serious game to teach programming. We evaluated our game for usability and effectiveness by conducting a user study on twenty-seven undergraduate students. The evaluation primarily consisted of a learning test conducted twice–before and after playing the game–along with a usability questionnaire that players completed after playing the game. Our results showed that players made significant progress after playing the game. The game helped players improve their basic programming skills, especially for the group having lower prior programming skills. The results highlighted various ways in which GLA can be used to benefit different stakeholders in the game. Based on players’ qualitative responses, we also identified several areas of improvement, most prominently the trade-off between ease of use and game complexity. We provide suggestions and discuss implications for future work.
... They are a subset of the DCVGs family that combine different types of media (animation, music, text, etc.) to create immersive experiences for players. The versatility of SGs allows them to be used as a tool with many applications in various fields [32]. Initially, they were used to train people to perform tasks in particular occupations, such as training army personnel. ...
Conference Paper
Full-text available
Games in their various forms in the world today have become both a growing market in the commercial industry and are being developed and studied in various disciplines. While in Iran, most researchers in this field, especially newcomers who want to work on games, do not know the terms of different concepts or use terms that are incorrect and irrelevant to their work. To address this issue, a literature review was conducted and definitions of different types of game concepts were reviewed. Then these definitions (digital, computer and video games / digital, computer and video learning games / serious games / serious games based on instructional content & training / gamification) and their relationship and distinction are expressed and finally their boundaries are specified in the form of a diagram. In general, the purpose of this article is to introduce and define different concepts in the field of gaming for researchers in this field.
... The ECD framework asks fundamental questions common in any assessment: "what, where, and how are we measuring, and how much do we need to measure" [8, p. 3]. This form of assessment can also provide some clues on other learner attributes such as critical thinking, challenges in processing information and problem solving rather than scores alone [26], [27]. Using Python powerful data analytics libraries like Pandas Visualisation and Seaborn, it is possible to analyse and visualise data for easier understanding to the parents, teachers, and other stakeholders. ...
Conference Paper
Full-text available
School readiness predicts both school and life success, so measuring it effectively is extremely important. Current school readiness tests focus on pre-academic skills; however, mastery motivation (MM: persistent, focus on trying to do a task, and executive functions (EF; planful self-control) are also crucial. The purpose of this paper is to provide an overview of the chronological development of Finding Out Children’s Unique Strengths (FOCUS App) which was initially designed for Hungarian and US cultures. FOCUS App is a game-like tablet-based assessment of two Approaches to Learning (ATL) domains, Mastery Motivation (MM) and Executive Functions (EF) and pre-academic skills for 3-8-year-old children. To measure these competencies, FOCUS has a total of seven tasks. Tasks 1 and 2 assess two pre-academic skills. Tasks 3-5 are letter and number search tasks designed to assess MM, operationalised as the child’s persistence during moderately challenging tasks. Tasks 6 and 7 are primarily designed to assess EFs and provide MM measures based on a modified version of the dimensional card change sorting tasks (DCCS). Task 3-7 assess Approaches to Learning. The development of the FOCUS App started with face to face testing using paper-based prototypes designed to simulate the computer tablet screens. These pilot tests were successful in assessing number and letter search but not emotions. The level of difficulty was improved, and computer-based tasks were developed. The first version of FOCUS was released in 2017 for testing, piloting and evaluation. This version was web-based, developed on the .net platform and collected data on pre-academic skills, MM and EF. This version was in English and Hungarian. Due to internet connectivity challenges in schools, the second version was released in 2018, which was offline, tablet-based, and built on the Android platform. This version collected video recordings that enable the assessment of participants’ task persistence and emotional expressions. Two other adaptations have been designed, the Hebrew and Kenyan versions. We are now designing a feedback module that will provide real-time results to the user during school readiness tests. FOCUS App has several advantages over other similar apps. First, it assesses more than one domain. Second, it has adopted the Evidence Centred Design which provides evidence for tasks undertaken by children. Third, it is computer game-based to make it fun for the children and to automate the process of data collection using a narrator that gives instructions. Moreover, the tasks are individualised, thus increasing the ability of the app to offer individualised interventions. Lastly, unlike other tools that have adopted the Dimensional Change Card Sort task (DCCS) the FOCUS app does not use reaction time to measure difficulty at higher levels.
... We have chosen a basic form of visual artefacts-images for our MINC design to meet the requirement of low threshold of technical abilities and mitigate side effects reported in previous studies [42]. However, the growing popularity of story-based content would mean different interventions involving serious games, immersive virtual reality, and videos are worthy of exploration [14,64]. Future studies involving diverse visual methods may help determine the effectiveness of different serious storytelling applications. ...
Article
Full-text available
Serious storytelling as a media genre has the potential to accentuate the benefits of narrative interventions in health and education. To inform its application, it is necessary to identify effects of sensory inputs. Here, we focus on visual stimuli and observe their effects on an anxiety condition. We examine whether serious storytelling incorporating images, a type of basic visual stimuli, may reduce interview performance anxiety. In a double-blind randomised control trial, 69 participants with matched levels of anxiety received serious storytelling interview training and were allocated to exposure (image-based preparation) and control (standard preparation) groups. A week later, participants attended individual interviews with two independent interviewers and reported their interview anxiety. Analyses revealed a positive relationship between generalised anxiety and some dimensions of interview anxiety, but serious storytelling with images predicted a reduction in interview performance anxiety (effect size at the median value of covariates on a visual analogue scale with the range 0–100: -36.7, 95% CI [−54.7, −2.5]). Low participation burden in the brief intervention was confirmed through a deductive thematic analysis. The images were analysed based on format type and origin to inform further inquiries. This study yielded empirical findings with implications of media and technology development for serious storytelling. Seeing images of experiences during interview preparation was associated with a relief of interviewees’ anxiety towards interview performance, but further studies are necessary to consolidate the evidence for visual narrative applications in health and education.
Presentation
Full-text available
Výskumným zámerom štúdie je overenie efektivity edukačných videí vo vzdelávacom procese. Videá sú populárnou formou trávenia voľného času u mladistvých. V tejto súvislosti by videá mohli byť považované za účinnú školskú pomôcku. Výskumy poukazujú na to, že edukačné videá napomáhajú vytvoreniu pamäťovej stopy, ale i na sociálny presah, kedy študenti a študentky viac interagujú vďaka videám zadaným od pedagógov/pedagogičiek. Výskum je zameraný na zisťovanie informácií o používaní edukačných videí študentmi a študentkami stredných škôl vo veku od 15 do 19 rokov v ČR a SR. Zaujíma nás kvalita ich exekutívnych funkcií, motivácia, triedna klíma a prospech respondentov/respondentiek vo vzťahu s používaním edukačných videí.
Article
Full-text available
Precise assessment of school readiness is critical because it has practical and theoretical implications for children's school and life success. However, school readiness assessment mainly relies on teacher reports and a few direct evaluations requiring a trained examiner. Studies indicate that 80% of games and apps target preschool children and education, suggesting that apps are familiar and fun for this age group. Previous reviews have fo-cused on these apps' training capability but not on their assess-ment of school readiness. This Scoping review examines 31 studies published from 2011-2019. The Evidence Centred Design (ECD) framework was used to evaluate game-based assessment (GBA) suitability to assess school readiness domains. Results show that it is possible to assess school readiness using GBA. Most studies assessed the cognitive domains in school settings and adopted an external assessment of the quasi-experimental design tasks. However, most studies only evaluated one competency, and few intervention strategies targeted the enhancement of school readiness. Besides, few studies followed the ECD framework strictly. Implications include expanding the assess-ment to other school readiness domains with a real-time inbuilt assessment that conforms to the ECD framework. GBA provides a new approach to assess school readiness outside or inside the school settings in this online era.
Article
Full-text available
Serious Games have been used in professional training to increase employee engagement and improve training initiatives. This review intends to identify the application of Serious Games in professional training and how these games have been developed, evaluated, and adapted to relate to the learning outcomes. It aims to map the use of Serious Games in professional training, verifying which forms of adaptation are used, learning outcomes, and models and frameworks of Serious Games that include game elements. Different learning outcomes associated with Serious Games were identified, in the general context and professional training, with knowledge acquisition being the most investigated result and questionnaires the most used type of assessment. It was found that several technologies are used in the adaptation of Serious Games. This literature review highlighted gaps in Serious Games research, especially in adaptive games applied in the context of professional training, such as the absence of frameworks for adaptive Serious Games and the lack of a framework for Serious Games that relates game elements to learning outcomes.
Article
Full-text available
—This study examines the influence of players’ age, gender, and gaming preferences and habits (from now on, “gaming preferences”) on the effectiveness of a specific videogame that has been designed to increase the interest towards classical theater among teenagers. Using a validated instrument, participants were divided into four groups based on their gaming preferences: (1) Wellrounded (WR) gamers, who play all types of games often; (2) Hardcore players, who frequently tend to play first-person shooter (FPS) and sports games; (3) Casual players, who play moderately and tend to play music, social, and puzzle games; and (4) Non-gamers, who barely play videogames at all. Among all of the participants’ personal factors (age, gender, and type of player) that were measured, only gaming preferences seemed to have a significant (p<.05) positive influence on students’ interest in theater-going. Neither age nor gender seemed to affect the outcomes. Casual and Well-rounded gamers scored higher in the game than Non-gamers and Hardcore players. Due to these results, we also explored whether the gaming profile affected traditional educational approaches. Traditional education worked better than videogames only for students who do not usually play videogames. This study suggests that gaming preferences may influence the effectiveness of different educational approaches. Knowing students’ gaming preferences in advance may help educators find the best educational approach for each student.
Conference Paper
Full-text available
The established (digital) leisure game industry is historically one dominated by large international hardware vendors (e.g. Sony, Microsoft and Nintendo), major publishers and supported by a complex network of development studios, distributors and retailers. New modes of digital distribution and development practice are challenging this business model and the leisure games industry landscape is one experiencing rapid change. The established (digital) leisure games industry, at least anecdotally, appears reluctant to participate actively in the applied games sector (Stewart et al., 2013). There are a number of potential explanations as to why this may indeed be the case including ; A concentration on large-scale consolidation of their (proprietary) platforms, content, entertainment brand and credibility which arguably could be weakened by association with the conflicting notion of purposefulness (in applied games) in market niches without clear business models or quantifiable returns on investment. In contrast, the applied games industry exhibits the characteristics of an emerging, immature industry namely: weak interconnectedness, limited knowledge exchange, an absence of harmonising standards, limited specialisations, limited division of labour and arguably insufficient evidence of the products efficacies (Stewart et al., 2013; Garcia Sanchez, 2013) and could, arguably, be characterised as a dysfunctional market. To test these assertions the Realising an Applied Gaming Ecosystem (RAGE) project will develop a number of self contained gaming assets to be actively employed in the creation of a number of applied games to be implemented and evaluated as regional pilots across a variety of European educational, training and vocational contexts. RAGE is a European Commission Horizon 2020 project with twenty (pan European) partners from industry, research and education with the aim of developing, transforming and enriching advanced technologies from the leisure games industry into self-contained gaming assets (i.e. solutions showing economic value potential) that could support a variety of stakeholders including teachers, students, and, significantly, game studios interested in developing applied games. RAGE will provide these assets together with a large quantity of high-quality knowledge resources through a self-sustainable Ecosystem, a social space that connects research, the gaming industries, intermediaries, education providers, policy makers and end-users in order to stimulate the development and application of applied games in educational, training and vocational contexts. The authors identify barriers (real and perceived) and opportunities facing stakeholders in engaging, exploring new emergent business models ,developing, establishing and sustaining an applied gaming eco system in Europe.
Article
Full-text available
This paper addresses the implications of combining learning analytics and serious games for improving game quality, monitoring and assessment of player behavior, gaming performance, game progression, learning goals achievement, and user's appreciation. We introduce two modes of serious games analytics: in-game (real time) analytics, and post-game (off-line) analytics. We also explain the GLEANER framework for in-game analytics and describe a practical example for off-line analytics. We conclude with a brief outlook on future work, highlighting opportunities and challenges towards a solid uptake of SGs in authentic educational and training settings.
Chapter
Full-text available
“Serious Games” is a unique industry that is concerned with the training/learning performance assessment of its clients. It is one of three digital technology industries (along with digital games, and online learning) that are rapidly advancing into the arena of analytics. The analytics from these industries all came from the tracing of user-generated data as they interacted with the systems, but differed from one another in the primary purposes for such analytics. For example, the purpose of game analytics is to support the growth of digital (entertainment) games, while that of learning analytics is to support the online learning industries. Although some game and learning analytics can indeed be used in serious games, they lack specific metrics and methods that outline the effectiveness of serious games—an important feature to stakeholders. Serious Games Analytics need to provide (actionable) insights that are of values to the stakeholders—specific strategies/policies to improve the serious games, and to (re)train or remediate play-learners for performance improvement. Since the performance metrics from one industry are unlikely to transfer well into another industry, those that are optimal for use in the Serious Games industry must be properly identified as Serious Games Analytics—to properly measure, assess, and improve performance with serious games.
Article
Full-text available
The value of educational video games in education is undeniable and the benefits of using video games in classroom instruction have been proved by many researchers. Nevertheless, these benefits have not been proved sufficiently for some domains, such as artistic disciplines. In this paper we explore the effects of an educational video game on high school students' interest towards classical theatre. The game covers the story of " The Foolish Lady " (La Dama Boba) based on the homonymous classic theatre play by Spanish playwright Lope de Vega. A mixed experimental design was followed, whereby researchers conducted pre-tests and post-tests to estimate the effect of playing the video game on student interest (within-subjects factor) towards theatre. We also measured changes in linguistic knowledge and knowledge about the play. The experiment was carried out with 754 students from 8 different schools in the Madrid region in Spain, divided into experimental group and two control groups. With the objective of improving the comparative power of the study, two control groups were used: (1) traditional teaching with the usual teacher and, (2) as the best educative case we could implement, teaching with a professional actor who had played the male protagonist of the theatre play. The experimental group played the video game. Results show that the videogame was more effective in incrementing students' interest in theatre than the traditional class, but slightly less effective than the class with the actor. On the other hand, game and teacher approaches obtained similar results in the improvement of students' knowledge about the play's plot and some linguistic concepts. These results open up a new horizon in using videogames as motivators in different artistic domains.
Article
Recently, interest in how this data can be used to improve teaching and learning has also seen unprecedented growth and the emergence of the field of learning analytics. In other fields, analytics tools already enable the statistical evaluation of rich data sources and the identification of patterns within the data. These patterns are then used to better predict future events and make informed decisions aimed at improving outcomes (Educause, 2010). This paper reviews the literature related to this emerging field and seeks to define learning analytics, its processes, and its potential to advance teaching and learning in online education.
Chapter
In this chapter, the process of embedding metrics collection within Sony Computer Entertainment will be outlined and discussed. We (the authors), are engineering development managers in an internal group within Sony Computer Entertainment that provides networking technology for games that Sony publishes. Our group provides a networking SDK and several online gaming services that benefit game studios in all Sony Computer Entertainment territories. We help integrate online services with games running on the PlayStation consoles, social networking and franchise websites.
Article
Educational computer games are increasingly being used in higher education and offer the potential of greater engagement, improved results and simpler, centralised updating of teaching material. However the evidence for the usefulness of such technologies is not yet conclusive. Consequently there is a need for improved design and evaluation of educational games. The aim of this study is to identify a successful game design for a multiplayer serious game to be used in learning. The design is being developed and evaluated through the creation of a game called 'Pharmacy Challenge' to allow small groups of pharmacy students at Kingston University (KU) to simultaneously revise certain aspects of the pharmacy curriculum in timed quiz-based challenges. The game is a web application with both single and multiplayer modes that can be run from a web browser on phones, tablet devices and PCs. All activities performed by players including time of access, time to answer and questions answered can be stored in data logs for future analysis. A pre-intervention survey conducted on students' perceptions on educational gaming informed the design of the game, which indicated that most students tend to play games on mobile devices. The game was then trialled on a group of around 60 mostly female students on a module running on years 3 and 4 of the pharmacy course over a week long period which could be played at any time of the day. Following the trial a post-intervention survey was used to assess the students' perception of the game. Students found the game interesting, stimulating and helpful and they identified its potential to motivate them and to facilitate their learning Positive responses indicate that games can be a valuable addition to pharmacy curriculum. The successful introduction of the game into the pharmacy curriculum demonstrates the value of education games in learning and student engagement.
Article
Training that future practitioners receive in software project management is a topic of great importance. The objective of this systematic literature review is to summarize the current state of the art of the different methods and procedures used to assess serious games. The review follows a predefined procedure that involves automatically searching well-known digital databases. 1199 papers were found by the automatic searches in the digital databases and 102 papers were selected as primary studies. The process was complemented with manual searches using author and backward snowballing techniques. Our systematic literature review identified the main methods followed to assess serious games, the application domains in which the assessments took place, the categories of serious games assessed, the main features considered to assess the educational effectiveness of serious games, the procedures followed for the assessments and the size of the population that participated in the assessments. The results are useful to researchers and practitioners willing to assess serious games in different fields, but specially to those interested in assessing serious games in the area of software project management.