Content uploaded by Dietrich Albert
Author content
All content in this area was uploaded by Dietrich Albert
Content may be subject to copyright.
A Psycho-Pedagogical Framework for
Multi-Adaptive Educational Games
Michael D. Kickmeier-Rust, Elke Mattheiss, Christina M. Steiner, & Dietrich Albert
Department of Psychology, University of Graz, Austria
ABSTRACT
One of the trump cards of digital educational games is their enormous intrinsic motivational
potential. Although learning game design is often understood on a one-fits-all level, the actual
motivational strength of an educational game strongly depends on the individual learners, their
very specific goals, preferences, abilities, strength and weakness, personality, and experiences
with gaming. Considering motivation being a fragile and constantly changing state, it is
important to continuously assess learning and gaming processes and the oscillations of
motivation and immersion within a game. With this premise in mind, we developed a psycho-
pedagogical approach to a non-invasive, embedded assessment of motivational states and
learning progress, feeding into a dynamic, ontology-driven learner (and gamer) model. To
evaluate the approach, the demonstrator games were subject to intensive quantitative and
qualitative experimental research. The results show that a meaningful personalization and an
individual support are key factors of the success of learning games.
Keywords: Digital educational games, non-invasive assessment, motivation, micro adaptivity,
macro adaptivity, adaptive storytelling
INTRODUCTION
Computer games are an incredibly successful genre that captivates children as well as adults and
that instantly mirrors the spirit of a time and the state-of-the-art in computer technology.
Computer games combine art and technology in a fascinatingly natural and convincing way. The
games’ success is reflected in enormous sales figures, economic growth, and numbers of users.
Particularly, Massively Multiplayer Online Games (MMOGs), brings together millions of
players in a single virtual world, and have become a market and technology leader. Thus, it is no
surprise that computer games spill over into more serious applications beyond pure entertainment
- and the hype over serious games and especially games for learning exists, with a great many
initiatives, projects, and even products.
The core strengths of computer games, distilled to their essence, are fun, fantasy,
curiosity, challenge, and control (Malone & Lepper, 1987), leading to an enormous intrinsic
motivational potential. The idea of utilizing those strengths for educational purposes is amongst
the most exciting developments in the area of educational technologies in the past decades. It is
thrilling and challenging for educators, researchers, developers, and designers - educators and
parents are struck by “the quality of engagement that stands in stark contrast to the half-bored
watching of many television programs and the bored performance exhibited with school
homework” (Kafai, 2006). Of course, the idea is not new. The attempt to utilize technological
trends for education has a long history. Technologies such as radio, television, computers, or the
Internet were quickly – and successfully – adopted for fostering learning. The motivational
potential along with the high level of interactivity and the large degrees of freedom in computer
games for educational purposes may open entirely new horizons for educational technology (de
Freitas, 2006).
Playing games, in general, is not only one of the most natural forms of human activity but
also one of the most natural forms of learning. Children learn to talk by playing with sounds and
learn collaboration and strategic thinking when playing Cowboys and Indians. Already Johan
Huizinga in 1938 ventilated the view that the Homo ludens, the playing man, develops abilities
through play. Thus it is no surprise that educational computer games have a long history. An
early example is the educational game Oregon Trail, a resource management game released first
in 1971 and re-released by the educational publisher Brøderbund for the Apple II in 1985. So in
conclusions, the essence of game-based learning is the attempt to utilise the strengths and
educationally beneficial aspects of computer games, for example, the high level of intrinsic
motivation to play and proceed in the game, a meaningful yet rich and appealing learning
context, immediate feedback, or a high level of interactivity, challenge, and competition. It is
clear, digital educational games (DEG) can be way more than just “chocolate covered broccoli”
(Jacob Habgood, 2009).
According to many researchers in the field of game-based learning, however, DEGs are
still in their infancy from a scientific and pedagogical perspective (e.g., Fu, Su, & Yu, 2009;
Oblinger, 2006). Major challenges for research, design, and development are seen, for example,
in finding an appropriate balance between gaming and learning activities (Van Eck, 2006) or
finding an appropriate balance between challenges through the game and abilities of the learner
(e.g., Kickmeier-Rust et al., 2007). We see the most important challenges for research on
educational games in relation to their core strength, which can be summarized with their
enormous intrinsic motivational potential. On the one hand, maintaining a high level of
motivation requires an intelligent and continuous real-time adaptation of the game to the
individual learner, for example, a continuous balancing of challenge and ability, of problems and
learning progress. This adaptation and level of responsiveness must occur in the context of
learning progress but also in the context of gaming and story. As important the intrinsic
motivation is, equally difficult is it to maintain that level of motivation and equally fragile is a
suitable balance between challenges and abilities. Essentially, this idea is covered by the concept
of flow – a highly immersed experience when a person is engaged in a mental and/or physical
activity to a level where this person loses track of time and the outside world and when
performance in this activity is optimal (Csikszentmihalyi, 1990).
On the other hand, another challenge is the enormous quality of today’s commercial, non-
educational computer games and, associated with that, the skyrocketing development costs.
Modern triple-A games have a development budget of tens of millions of Euros. Unfortunately,
the target audience for a non-educational computer game is way larger than the target audience
of an educational game that is usually developed in a specific language, for a specific limited
target age, and according to a specific curriculum.
PERSONALIZING LEARNING AND GAMING
As outlined above, one of the most crucial factors for successful educational games can be seen
in the game’s ability to maintain an individual learner’s motivation and interest by adapting the
individual learning and gaming experiencing the this very learner’s needs, preferences, goals,
and abilities. On the one hand, this certainly is a matter of a suitable and creative learning game
design. But sheer design cannot cover individual differences, thus we need mechanisms to assess
we the learners need and what they want and, subsequently, to adjust the learning game
accordingly. This attempt is not trivial.
Generally, the idea comes from the field of adaptive/intelligent tutoring in conventional
technology-supported teaching and learning, basically inspired by Benjamin Bloom in 1984 who
stated that students who received one-to-one tutoring performed on average as well as the top
two percent of those receiving classroom instructions. Ever since psychologists, instructors, and
technicians attempted to develop technology that is able to take the role of a private teacher and
to intelligently provide individual learners with suitable tutoring. The spectrum of approaches,
methods, frameworks, and applications is quite broad (De Bra, 2008; Kinshuk, Lin, & Patel,
2006). Adaptivity refers to three major concepts: (a) Adaptive presentation, which means
adjusting the look and feel of a learning environment according to individual preferences or
needs; for example, different colour schemes, layouts, or amount of functionality; (b) adaptive
curriculum sequencing, which means providing the learner with learning tailored to individual
preferences, goals, learning styles, or prior knowledge; (c) adaptive problem solving support,
which means providing the learner with feedback, hints, or solutions in the course of problem
solving processes.
Generally speaking, those rough classes of adaptation have in common that they require
an assessment of knowledge and learning progress and that adaptation significantly influences
the presented learning objects. While in conventional learning environments such approach
works well, in game environments it is not applicable. On the one hand, conventional assessment
methods such as popping up queries or multiple choice items would most likely destroy
immersion and flow experience. On the other hand, it is not possible to add or skip specific
learning objects because this substantially harms the story and red thread through the game.
In the framework of the ELEKTRA project a new approach was introduced, addressing
those problems. The new concepts, which are tailored to learning environments with large
degrees of freedom, are adaptivity on macro and micro levels (Kickmeier-Rust & Albert, 2010).
Macro adaptivity refers to rather traditional techniques of adaptation such as adaptive
presentation and adaptive navigation on the level of learning objects (or learning situations in an
educational game). Generally, macro adaptive interventions are based on a fixed learner model
(e.g., traits) or adaptation model (e.g., pedagogical implications) and on typical (knowledge)
assessments (via test items). Micro adaptive interventions, on the other hand, are non-invasive
(meaning that an overall narrative is not compromised) and affect the characteristics of a specific
learning object or learning situation. Techniques of micro adaptive interventions are, for
example, adaptive hinting, adaptive feedback, or an adaptive adjustment of the environment.
Micro Level Adaptation
Non-invasive Assessment of Cognitive and Motivational States
In the first instance, micro adaptation relies on an embedded method to the assessment of
learning progress, cognitive as well as motivational states. The basic idea is to monitor and
interpret the learner’s behavior in the game. To achieve this, we utilize the formal framework of
Competence-based Knowledge Space Theory (CbKST), which is a cognitive framework,
extending the originally behavioural Knowledge Space Theory (Doignon & Falmagne, 1999),
where a knowledge domain is characterised by a set of problems and prerequisite relations
among them, establishing a knowledge space. The basic idea of CbKST is to separate observable
performance and underlying latent skills or competencies (e.g., Albert & Lukas, 1999; Korossy
1999). The relationships between the skills and problems (or learning objects) are established by
skill and problem functions. By associating skills with the problems of a domain, a knowledge
structure on the set of problems is induced. CbKST provides an internal cognition-based logic
that is quite similar to the logic of ontologies: well-defined entities (the skills) are in a well-
defined relationship (a so-called prerequisite relation). The domain model, the set of meaningful
skill states, and the resulting set of meaningful learning paths are combined with a model of tasks
and problems within certain parts, so-called learning missions, of the game (equivalent to
conventional “learning objects”), the so-called problem space (cf. Newell, 1990; Newell &
Simon, 1972). A simple example for such mission might be the task to fly with the space ship to
a certain city and to take a picture. The learning objective of this task might be (among others) to
learn about the location of the city on the map. In this situation are various manipulable objects,
for example the space ship. The learner can perform certain actions to achieve the goal, in this
example primarily changing the directions of the flying space ship or controlling speed and
altitude. The aim of micro level assessment is in the first instance to assign a problem solution
state from the problem space to each action (e.g., pressing an arrow key). This mapping is done
by classifying actions according a set of rules. An example for such rule might be “the distance
between space ship and target location is increasing”. The second aim is to assign a set of
available and a set of lacking skills to each problem solution state; for example, flying in the
right direction indicates that the learner knows the wind direction towards the city. Of course, a
single observation is not very convincing. Thus, CbKST provides a probabilistic approach to
assessment. We have a probability distribution over all possible skill states and with each action
we update the probabilities of those states that include the relevant skills and we decrease those
states that include the lacking skills (for details on the probabilistic updating procedure refer to
Falmagne & Doignon, 1988). Recently Augustin, Hockemeyer, Kickmeier-Rust, and Albert
(2010) have elaborated and most importantly simplified the probability update procedure to
reduce the computational load in the real-time assessment context. At the end of this procedure
stands a more or less well-founded assumption about the skills the learners have, the skills they
don’t have, and their position in the problem solving process.
Similarly, we can assign specific motivational assumptions to specific classes of actions,
again based on a set of rules. The rules were derived from the large body of research in the area
of motivation psychology and aggregated into a novel framework to motivational assessment. In
essence, the framework builds upon the expanded model of motivation to learn (Heckhausen &
Heckhausen, 2006) attribution theory (Weiner, 1974) and the concept of self-efficacy (Bandura,
1977), and Keller’s ARCS model (Keller, 1987). Motivational interventions may provide the
learner with information about the learning progress or the game, provide or announce incentives
or rewards, may address attention or confidence, but may also involve emotionally focused
feedback. An example is to interpret the density of actions, that is, the number of actions
performed in a specific time interval. The continuously gathered and updated assumptions on the
skills and motivational state throughout the game serve the provision of adaptive psycho-
pedagogical interventions tailored to the learner’s current needs.
Interventions on the Micro Level
It is important to avoid comprising the game’s flow by assessing learning progress or
motivational state, but it is equally important to interventions be convincingly embedded in the
game and, more importantly, suitable for the individual learners in their very gaming situations.
Micro level interventions may be hints, suggestions, warnings, or feedback. We propose the
following general types of interventions:
• Educational interventions provide the learner with specific information (i.e.,
skills) if the system concludes that the related skills are lacking. In the game
context such interventions can come for example from non-player characters.
• Problem solving support provides the learner with information about his/her
current state in the game-related problem solving process. To give an example, if
the system detects that a number of actions did not decrease the distance between
the present problem solution state and the target state, the system can trigger a
hint that perfectly suits the present problem solving state.
• Meta-cognitive interventions are supposed to foster reflection about the learner’s
own abilities, confidence, or self-esteem. A typical realization of such
intervention type is to let a non-player character ask specific questions like “are
you sure?” or “why did you do that?”.
• Assessment interventions are a special form of intervention. If the probabilistic
assessment (of either learning progress or motivation) does not lead to clear
results after a certain number of actions, the system can trigger interactions to
improve the assessment. Typically this can be realized by providing the learner
with different problems/tasks or by specific questions through a non-player
character.
• Dissolving interventions are a further form to provide the learner with specific
information. The purpose of this intervention type is to provide the learner with
the solution of a problem/task if the learner wasn’t able to do so within a
reasonable number of actions. Such interventions, ultimately, shall assure that the
game can continue even if the learner is not able to solve a problem/task. Of
course, for didactical reasons, this intervention type might not be used for all
problems/tasks.
• Motivational interventions are supposed to retain the learner’s motivation on a
high level or to intervene when the system detects that the motivational states
(potentially) decreases. Forms of such interventions are feedback, praise,
incitation, encouragement, or directing attribution of success or failure (from a
motivational point of view the learner should attribute success to his/her own
abilities and failure to external components such as bad luck).
All interventions of a game require a manifestation in form of game assets (e.g., a sound file with
a specific sentence). Of course, not all possible interventions can be realized. In general, we
propose an approach of using interventions conservatively and sparsely. One must be aware that
repeated inadequate interventions due to misinterpretations of a situation (e.g., assuming a lack
of motivation on the basis of no actions for longer period of time while the learner just has gone
to the fridge) are a significant harm to motivation, engagement, and the game flow. The
conditions under which a certain adaptive intervention is given are to be developed on the basis
of psycho-pedagogical rules, as briefly referenced above.
Macro Level Adaptation
So far our concept of personalization and adaptation for adaptive educational computer games
just concerned assessment and interventions within specific limited and pre-defined learning
situations. Educationally important techniques for personalization and adaptation such as
adaptive sequencing of learning units (curriculum sequencing) or adaptive presentation,
however, cannot be addressed reasonably. To extend and enrich the approach to in-game
personalization and adaptation, we conceptualized a fusion of the micro adaptivity concept with
techniques of interactive digital storytelling. In that way, we can realize a personalized
sequencing of learning situations and units according to educational aspects as well as
personalized adjustments of the game according to individual needs and preferences. In other
words, we can shift in-game adaptation to the macro level.
In the literature several techniques for interactive or adaptive storytelling are described,
varying in the openness of story generation and in their operational reliability. The approaches
range from a recombining of self-contained story elements to an open-ended automated
generation of “new” stories. For our goal of adaptation we rely on a robust approach based on the
specification of atomic story-related entities (ranging from single spoken sentences to self-
contained story units). In this context, a crucial aspect of interactive storytelling is to find an
appropriate storyline on the basis of a pool of given atomic story or game elements. These
entities can be compared to the rooms of a house and the furniture in those rooms, each entity
has a specific goal (e.g., providing the learner with information, assessing internal states, or
contributing to story and gameplay), specific characteristics and properties. During a gaming
episode the single game entities must be adaptively re-combined and re-assembled into a
meaningful storyline and a meaningful environment. The assembly is driven by specific sets of
rules which refer to aspects of the game genre, the story model, educational aspects, and
individual aspects.
The story model underlying our approach relies on a formalization of the classical three-
act structure of Aristotle providing an arc model with ‘exposition’, ‘rising action to climax’, and
‘denouement’. The related set of rules, in combination with additional annotations such as
importance for the game or the educational aim, establishes a set of meaningful paths through the
story. This story space can be overlaid with the domain model on the basis of CbKST (the
competence structure). This combination generates so-called game paths possible and
educationally meaningful paths through the game accounting for story model, learning
objectives, and pedagogical interventions (cf. Figure 1; see also Göbel at al., 2009 for details).
Interventions on the macro level now can be either system-driven adjustments to the overall
storyline or adjustments to the game’s pace or intensity (for example, a mission can be
accomplished calmly without any time pressure or, on the other hand, driven and fast and with
time pressure (e.g., because being chased by virtual opponents).
THE REALIZATION
The introduced approach was developed and realized in two European project, ELEKTA
(www.elektra-project.org) and 80Days (www.eightydays.eu). From a technical perspective, the
realization is based on a complex interplay of various specialized software components and
engines.
The game is traditionally realized with a state-of-the art game engine (Nebula 2 engine by
Radon Labs in this case). The learner interacts solely with this game engine. The game passes
Figure 1. A combination of knowledge and story paths as a basis for adaptation.
information about the game progress to a central adaptation control engine. This engine passes
the relevant data to engines for the real-time assessment of the present knowledge state and for
the motivational assessment. The results coming from those engines are analyzed. On the one
hand, the results are transcribed into recommendation regarding micro adaptive interventions, as
described above. The ultimate decision about triggering micro level interventions comes from
the game engine and includes information about the intervention history and the game progress
(in order to avoid annoying repeated or similar interventions or interventions in inappropriate
situations). On the other hand, the results are considered for macro adaptive adjustments of the
entire narrative (including the alteration of the game’s speed and intensity). The relevant
information for the engines comes from an OWL ontology that serves as a comprehensive
database (cf. Kickmeier-Rust & Albert, 2007).
Case Study 1: ELEKTRA
The ELEKTRA project (www.elektra-project.org), funded by the European Commission, ran
from 2006 through 2008 and had the ambitious goal to utilize the advantages of computer games
and their design fundamentals for educational purposes and to address and eliminate the
disadvantages of game-based learning as far as possible. Nine interdisciplinary European
partners contributed to the development of a sound methodology for designing educational
games and the development of a comprehensive game demonstrator based on a state-of-the-art
3D adventure game teaching physics according to national curricula. In this context the
approaches to non-invasive assessment and embedded interventions were developed. The
research efforts were realized in form of a compelling demonstrator game which was realized as
a classical 3D adventure game in first-person view and it is supposed to teach physics (see Figure
2 for screenshots).
Very briefly, the aim of the ELEKTA game is to save the girl Lisa and her uncle Leo who
have been kidnapped by the evil Black Galileans; moreover, the learner has to stop the evil
forces from taking over the entire world. During this journey, the learner needs to acquire
specific, curriculum-related knowledge, concretely, the learner learns about 8th grade optics. The
learning occurs in different ways, ranging from hearing or reading to freely experimenting. After
finding a magic hour glass, the learner is in company of the ghost of Galileo Galilei, who is the
learner’s (hidden) teacher. In addition, the learner can interact with Lisa via a headset, which is
indicated in the upper left corner of the screen. Those non-playing characters also play a
significant role for intelligent, non-invasive educational and motivational interventions. For
example, Galileo tells the learner specific facts, which are needed for certain events in the game,
or he intervenes by providing the learner with hints or feedback.
A concrete example for a LeS is the so-called “slope device” situation (Figure 2). In this
LeS the students experiment with a machine where several balls of different materials (solid and
hollow iron, wood, and plastic) are running down a slope and also a laser can beam down this
slope. This machine has a fan and a strong magnet. The learners’ task is to make the balls fall
into a hole by setting appropriate values for fan and magnet. In addition they should estimate the
trajectory of the laser beam in dependence fan, gravity, and magnetic force. This experiment is
supposed to visualize the effects of fan, gravity, and magnet on different material and, in the first
instance that the laser beam is not influenced by such external forces and independently
propagates in a straight line. The approach to solution value indicates how fast a learner finds
the correct settings of fan and magnet and how well s/he can estimate the trajectory of the laser
beam.
Experimental Results
The demonstrator game was evaluated in-depth with children from French schools. In this
context a large amount of empirical quantitative and qualitative data were recorded. The most
Figure 2. Screenshots from the ELEKTRA demonstrator game. The right image shows the so-
called slope device learning situation.
essential results concern the learning and the impact of micro adaptive interventions.
Prototypically we present the results for the slop device learning situation. In this context we
distinguished two dependent variables, first, the so-called ‘approach to solution’ (ATS) variable,
which states how many action were performed following a certain type of intervention/feedback
that were (a) closer to the final solution, (b) farther from that, or (c) without an effect. The value
of this variable depends on the number of interventions of a type each learner received. For the
analyses we used an ATS value relative to the base line o receiving no interventions. Second, we
analyzed the response time that is, the time the learners needed after receiving an
intervention/feedback to perform their next actions in the experiments. Since this type of analysis
compares intervention/feedback types and not participants (each of them got several of different
types), the total experimenting time is not a meaningful measure.
The results (on the basis of 40 students, 17 female, 23 male, with an average age of 13.08
years (SD = 1.08)) of these analyses are summarized in Figure 3. Appropriate interventions
resulted in an average relative ATS of 0.33, neutral in an average relative ATS of .06, and
inappropriate in a relative ATS of 4.00 (SD = 15.21), and not receiving any interventions or
feedback resulted in an average approach of -.01. These differences are statistically not
significant. However, they clearly indicate that appropriate interventions/feedback result in a
quicker problem solving progress that needs fewer steps. Interestingly, neutral interventions
resulted in a slightly better performance while – quite reasonably – inappropriate interventions
(interventions that did not fit to the situation) reduced the performance in comparison to the no
intervention baseline. Somewhat different results were found for the response times after each
intervention/feedback. Appropriate interventions/feedback resulted in an average response time
of 3.90s (SD = 1.16), neutral in an average response time of 4.03s (SD = 1.08), inappropriate in
an average response time of 3.94s (SD = 0.84), and not receiving any interventions or feedback
resulted in an average response time of 3.06s (SD = 0.90). An analysis of variance (ANOVA)
yielded that receiving no interventions or feedback resulted in statistically significant shorter
response times (F(3)=33,86; p<01) than receiving interventions or feedback; the type of
feedback, however, did no influence response times.
The evaluation of a highly adaptive system is difficult in general since each learner
potentially receives different interventions at different points in time. Thus, we performed
analyses on the learner level by comparing ATS and the average response time for participants
Figure 3. The left panel shows the relative approach to the correct solution of the slope device
problem, the right panel shows the corresponding response times.
who received (i) (almost) no inappropriate interventions and feedback with such participants who
received a (ii) large portion of inappropriate interventions. These extreme groups included 10%
of participants who had received the most inappropriate interventions and the least inappropriate
interventions respectively. The results are summarized in Figure 4. The average relative ATS
was 0.38 in the appropriate intervention extreme group and 0.01 in the inappropriate intervention
extreme group. This means that tailored interventions increased the learners’ performance while
inappropriate interventions did not affect performance in comparison to the no interventions
baseline. Similarly, the average response times were 3.99s (SD = 0.91) in the appropriate
intervention extreme group and 3.64s (SD = 1.02) in the inappropriate intervention extreme
group. According to an ANOVA, the differences between the extreme groups were statistically
significant for both approach to solution (F(1)=0,31, p<0,01) and response time (F(1)=5,05;
p<0,05).
Finally, we analyzed overall learning performance with the demonstrator game with and
without interventions/feedback using a 34 item knowledge test before and after playing the
demonstrator. The results are summarized in Figure 4 (right panel). The group with adaptive
interventions clearly performed better in the knowledge test than the group without any
interventions although these results yielded no statistically significant difference.
In conclusion, the idea of assessing learning performance by monitoring and interpreting
the learners’ behavior in the context of a game environment with a large degree of freedom and
the subsequent personalized support by tailored interventions such as hints or feedback appeared
being a promising approach to enrich educational games with adaptive educational measures on
an individual level.
Case Study 2: 80Days
The European research project 80Days (www.eightydays.eu) funded by the European
Commission and inspired by Jules Verne’s novel “Around the world in eighty days” is a direct
successor project of ELEKTRA and runs from 2008 through 2010. Basically, the project’s
endeavors include addressing motivational assessment and adaptation, on the one hand, and the
realization of macro adaptation as described above, on the other hand.
Figure 4. The left panel shows the relative approach to the correct solution in the slope device
problem and the middle panel shows the corresponding response times. The right panel shows
the absolute learning performance for the slope device problem in dependence of adaptation.
Figure 5. Screenshots from the 80Days demonstrator game.
As in ELEKTRA, the research endeavors of the seven European partner organizations are
realized in form of a demonstrator game. The game is teaching geography for a target audience
of 12 to 14 year olds and follows European curricula. The game design includes premises,
concepts, metaphors, structures, gameplay, learning objectives, contents, background story, game
characters, visual design and game props. In concrete terms, an adventure game was realized
within which the learner takes the role of an Earth kid at the age of 14. The game starts when a
UFO is landing in the backyard and an alien named Feon is contacting the player. Feon is an
alien scout who has to collect information about Earth. The player wants to have fun by flying a
UFO and in the story pretends to be an expert in the planet earth. He or she assists the alien to
explore the planet and to create a report about the Earth and its geographical features. This is
accomplished by the player by means of flying to different destinations on Earth, exploring them,
and collecting and acquiring geographical knowledge. The goal is to send the Earth report as a
sort of travelogue about Earth to Feon’s mother ship. Finally, the player sees through the alien’s
game (of preparing the conquest of the earth) and reveals the “real” goal of the game: The player
has to save the planet and the only way to do it is to draw the right conclusion from the traitorous
Earth report. Therefore the game play has got two main goals: (1) to help the alien to complete
the geographical Earth report, and (2) to save the planet, which is revealed in the course of the
story, when the player realizes the true intention of the alien. Figure 5 gives some illustrations of
the game.
Experimental Results
The demonstrator game was evaluated in Austrian as well as British school classes. During the
ongoing evaluation activities a broad spectrum of questions is addressed. In this work we can
only present a minor cutout of the preliminary results. These results are based on 69 Austrian
children (27 boys and 42 girls) at an average age of 12 years and 40 British children (36 boys
and 4 girls) at an average age of 11. In this regard we want to concentrate on learning
performance with the adaptive demonstrator game, similar to the results presented for
ELEKTRA.
The most distinct results concern the learning performance; as independent measure we
computed the relative average performance increase in a 13-items knowledge test questionnaire
covering the knowledge and skills relevant for a terra forming mission (Figure 6, right image),
which is supposed to teach the effects of different constructible and cultivation measures on the
risk of floods and severity of flood damages. The measure indicates the amount of knowledge
gained from playing the game in comparison to a pretest, computed for the entire sample. As
shown in Figure 6, the Austrian children showed an increase of 19.97 (SD = 11.54), the British
children an increase of 49.47 (SD = 15.66). The reason for the clearly better performance of the
British children is lies in all likelihood in a language disadvantage of the Austrian sample since
the demonstrator game is in English language. The performance increase yielded statistical
significance for both Austrian (t=-2.19, df=44, p<.05) and British children (t=-4.93, df=27,
p<.001). An interesting aspect of evaluation concerns the motivational adaptation and the macro
adaptation (as describe above), which is novel to 80Days. We compared, as an example, different
adaptation groups, that is, a group with motivational interventions as well as macro adaptation, a
group with macro adaptive interventions only, and a group with no interventions at all. As shown
in Figure 6, right panel, the combined adaptation group yielded the highest learning performance.
Interestingly, macro adaptation only yielded even somewhat weaker results than no interventions
at all. For the presented sample, however, the differences are not statistical significant.
Figure 6. The left panel shows the learning performance in the terra forming problem for
Austrian and British children; the right panel shows the mean test scores in a knowledge test in
dependence of the adaptation group.
CONCLUSIONS
The key strength of educational computer games is usually seen in their tremendous motivational
potential. The motivation to play – and therefore to learn – however, is a fragile construct and
heavily relies on the preferences, abilities, the goals, and even the taste of individual
players/learners. This holds for commercial, non-serious games and it is even more important for
games with a well planned educational purpose. Today, learning game design is often understood
on a one-fits-all level, which does not account for the individual learners, their very specific
goals, preferences, abilities, strength and weakness, personality, and experiences with gaming.
With this idea in mind, in two projects we developed a psycho-pedagogically sound approach to
a non-invasive, strongly embedded assessment of motivational states as well as learning
progress, feeding into a dynamic, ontology-driven learner (and gamer) model. On this basis, the
game system responds to the learners’ demands in terms of motivation and in terms of didactical
support in a smart way and in real time. To collect empirical evidence on the effects and efficacy
of micro and macro adaptive assessment and interventions, we conducted in-depth evaluations
with the demonstrator games, focusing on different aspects of game-based learning, assessment,
and particularly interventions and feedback. The results provide some evidence that our idea of
personalization is key to a learning game’s impact and success. We could show that micro
adaptive interventions lead to a faster approach to the correct solution, meaning to a faster
problem solving process, in problem solving situation than neutral, inappropriate, or no
interventions. In addition, we could demonstrate that providing the learner with appropriate,
personalized interventions resulted in a better learning performance with the demonstrator game
in comparison to providing no interventions at all. On the basis of 80Days’ first results, we could
also show that motivational and macro adaptive interventions have a highly positive impact on
learning performance.
Future work will not only strengthen the experimental foundations of educational games,
important aspects of assessment and adaptation, namely those of the highly successful genre of
multiplayer games, must address increasingly.
ACKNOWLEDGEMENTS
The research and development introduced in this work is funded by the European Commission
under the sixth framework programme in the IST research priority, contract number 027986
(ELEKTRA, www.elektra-project.org) as well as under the seventh framework programme in the
ICT research priority, contract number 215918 (80Days, www.eightydays.eu).
REFERENCES
Albert, D., & Lukas, J. (Eds.). (1999). Knowledge Spaces: Theories, empirical research, and
applications. Mahwah, NJ: Lawrence Erlbaum Associates.
Augustin, T., Hockemeyer, C., Kickmeier-Rust, M. D., & Albert, D. (2010). Individualized skill
assessment in digital learning games: Basic definitions and mathematical formalism.
Accepted for publication in IEEE Transactions on Learning Technologies.
Bandura, A. (1977). Self-efficacy: Toward a unifying theory of behavioral change.
Psychological Review, 84, 191–215.
Bloom, B. (1984). Taxonomy of educational objectives. Boston, MA: Allyn and Bacon.
Csikszentmihalyi, M. (1990). Flow: The psychology of optimal experience. New York:
Harper and Row.
De Bra, P. M. E. (2008). Adaptive hypermedia, In H. H. Adelsberger, Kinshuk, J.
M. Pawlowski, & D. Sampson, (Eds.), Handbook on Information Technologies for
Education and Training (pp. 29-46), Berlin: Springer.
de Freitas, S. (2006). Learning in immersive worlds. A review of game-based learning.
Retrieved August 28, 2007 from http://www.jisc.ac.uk/media/documents/programmes/
elearning_innovation/gaming%20report_v3.3.pdf
Doignon, J.-P., & Falmagne, J-C. (1999). Knowledge spaces. Berlin: Springer.
Falmagne, J.-C., &Doignon, J.-P. (1988). A class of stochastic procedures for the assessment
of knowledge. British Journal of Mathematical and Statistical Psychology, 41, 1-23.
Fu, F.-L., Su, R.-C., & Yu, S.-C. (2009). EgameFlow: A scale to measure learners’
enjoyment of e-learning games. Computers & Education, 52(1), 101-112.
Göbel, S., Mehm, F., Radke, S., & Steinmetz, R. (2009). 80Days: Adaptive Digital
Storytelling for Digital Educational Games. In R. Klamma, N. Sharda, B. Fernández-
Manjòn, H. Kosch, & M. Spaniol (Eds.), In Proceedings of the First International
Workshop on Story-Telling and Educational Games (STEG'08), September 18-19, 2008,
Maastricht, The Netherlands.
Habgood, J. (2009). Wii don’t do edutainment. In Proceedings of Game-based Learning
2009, March 19-20, 2009, London, UK.
Heckhausen, J., & Heckhausen, H. (2006). Motivation und Handeln: Einführung und
Überblick [Motivation and action: Introduction and overlook]. In J. Heckhausen & H.
Heckhausen (Eds.) Motivation und Handeln [Motivation and action] (p. 1-8). Heidelberg:
Springer Verlag.
Huizinga, J. (1938). Homo ludens. Proeve eener bepaling van het spelelement der cultuur.
[Homo ludens: A study in the play-elements in culture] (tr. R.F.C. Hull, 1947). New
York: Roy Publishers.
Kafai, Y. B. (2006). Playing and making games for learning. Instructionist and constructionist
perspectives for game studies. Games and Culture, 1(1), 36-40.
Keller, J. M. (1987). Development and use of the ARCS model of motivational design.
Journal of Instructional Development, 10(3), 2-10.
Kickmeier-Rust, M. D., & Albert, D. (2008). The ELEKTRA ontology model: A learner-
centered approach to resource description. Advances in Web Based Learning – ICWL
2007 (pp. 78-89). Lecture Notes in Computer Science, 4823/2008. Berlin: Springer.
Kickmeier-Rust, M. D., & Albert, D. (2010). Micro Adaptivity: Protecting Immersion in
Didactically Adaptive Digital Educational Games. Journal of Computer Assisted
Learning, 26, 95-105.
Kickmeier-Rust, M. D., Peirce, N., Conlan, O., Schwarz, D., Verpoorten, D., & Albert, D.
(2007). Immersive digital games: Interfaces for next-generation e-learning. Universal
Access in HCI, Part III, HCII 2007 (pp. 647–656). Lecture Notes in Computer Science,
4556. Berlin: Springer.
Kinshuk, Lin, T., & Patel, A. (2006). User adaptation in supporting exploration tasks in
virtual learning eEnvironments. In J. Weiss, J. Nolan, J. Hunsinger, & P. Trifonas (Eds.),
The International Handbook of Virtual Learning Environments (Vol. 1) (pp. 395-425).
Dordrecht, The Netherlands: Springer.
Korossy, K. (1999). Modelling knowledge as competence and performance. In D. Albert &
J. Lukas (Eds.), Knowledge Spaces: Theories, Empirical Research Applications (pp. 103–
132). Mahwah, NJ: Lawrence Erlbaum Associates.
Malone, T. W., & Lepper, M. R. (1987). Intrinsic motivation and instructional effectiveness
in computer-based education. In R. E. Snow & M. J. Farr (Eds.), Conative and Affective
Process Analyses (pp. 255-286). Hillsdale, NJ: Laurence Erlbaum Associates.
Newell, A. (1990). Unified theories of cognition. Cambridge, MA: Harvard University
Press.
Newell, A. & Simon, H. (1972). Human problem solving. Englewood Cliffs, NJ: Prentice-
Hall.
Oblinger, D. (2006). Games and learning. Educause Quarterly Magazine, 29(3), 5-7.
Van Eck, R. (2006). Digital game-based learning. It’s not just the digital natives who are
restless. Educause Review, March/April 2006, 17-30.
Weiner, B. (1974). Achievement motivation and attribution theory. Morristown, NJ.: General
Learning Press.
