Content uploaded by Owen Conlan
Author content
All content in this area was uploaded by Owen Conlan
Content may be subject to copyright.
Realtime Knowledge Space Skill Assessment for Personalized Digital
Educational Games
Owen Conlan1, Cormac Hampson1, Neil Peirce1, Michael Kickmeier-Rust2
1Trinity College, Dublin; 2University of Graz
{owen.conlan, hampsonc, peircen}@cs.tcd.ie; michael.kickmeier@uni-graz.at
Abstract
Digital Educational Games offer immersive
environments through which learners can enjoy
motivational and compelling educational experiences.
Applying personalization techniques within these
games can further enhance the educational potential,
but the often realtime and narrative-driven focus of
games presents many challenges to traditional
adaptation approaches. This paper describes an
approach to the realtime assessment of learner skills
for personalization that was implemented and
evaluated as part of the ELEKTRA European
Commission funded project.
1. Introduction
Digital Educational Games (DEG) are an emerging
area in which personalization techniques, traditionally
developed within the Adaptive Hypermedia (AH)
research domain, are being applied. A major issue that
has plagued online learning solutions for quite some
time has been the high levels of drop out [1] often
precipitated by poor intrinsic motivation and relevance
in the material presented. Non-adaptive DEGs seek to
address the motivation issue by presenting the learner
with a compelling and engaging environment and
backdrop in which to learn. Through rich narratives
[2], engaging gameplay [3] and a fidelity to real world
situations [4] these games strive to engage and
motivate the learner. Combining personalization
techniques with such educational games has the
potential to further improve the relevance of what is
offered to the learner. A broad range of adaptation
axes, such as Cognitive feedback, Meta-cognitive
feedback, Affective/motivational feedback, Knowledge
based hinting and Progression hinting [5] may be
considered.
In order to offer appropriate adaptive interventions
three challenges must be overcome: 1) modeling of the
learner’s knowledge acquisition (also referred to as
cognitive gain) must be achieved in realtime; 2)
adaptive hypermedia techniques, which are typically
applied to web-based systems, also need to operate in
realtime; 3) the personalizations offered must not
adversely impact the flow [6] of the game. The
challenges of realtime adaptation and the maintenance
of flow [6] stem from the need to maintain a learner’s
immersion within the gaming experience.
This paper focuses on the first of these challenges,
while referencing the others, by presenting how the
Knowledge Space Theory (KST) [7] [8] was adopted
as a realtime, probabilistic approach to progressively
modeling a learner’s skills and knowledge whilst
engaged with an immersive DEG. It provides a brief
overview of the current state of the art in DEGs and
Adaptive Hypermedia, along with the basics of KST.
The paper will also introduce the ELEKTRA research
project, its architecture and the Skill Assessment
Engine, a realtime KST-based modeling engine for
personalized DEGs.
2. Background
DEGs have reported successful outcomes by
integrating adaptation and strong storylines with
inherent motivational qualities. The DARPA funded
Tactical Language and Cultural Training System
(TLCTS) [2] has shown effective learning outcomes
achieved through the application of adaptation. Façade
[4] and the Virtual Team Collaborator (VTC) [9] have
shown that a strong narrative, an adaptive narrative in
the second instance, can provide immersive
experiences. Whilst both of these showed benefits,
their technical approach was highly complex and
involved the authoring of several narrative strands. The
Adaptive Learning In Games through Non-invasion
(ALIGN) system [5] eases this authoring burden and is
an expansion on the proven APeLS multi-model,
metadata driven approach [10], but it does not
specifically focus on narrative issues.
Adaptive Hypermedia Systems have typically dealt
with narrative from a different perspective. Most
prevalent examples come from the adaptive eLearning
domain where narrative usually refers to the flow of a
2009 Ninth IEEE International Conference on Advanced Learning Technologies
978-0-7695-3711-5/09 $25.00 © 2009 IEEE
DOI 10.1109/ICALT.2009.150
538
piece of coursework [11] [12]. However, as these
systems are web-based and the narratives are usually
constructed periodically they do not suffer from the
severe realtime restraints of DEGs.
Knowledge Space Theory (KST), introduced by
Doignon and Falmagne [13], provides a theoretical
framework within which the knowledge or skill state of
a learner can be determined. It is based on a
prerequisite competence structure that describes the
relationships between different skills. For example, a
learner should typically be able to perform fraction
addition before they can multiply them. If the learner
exhibits evidence of fraction multiplication it may be
assumed that they can also add fractions. Such
probabilistic reasonings enable a system to infer a
learner’s skill state based on partial evidence [7]. The
fundamental approach taken in KST is to reduce the
number of possible pieces of evidence needed about a
learner to an optimal set. In this way the Knowledge
State of a learner may be assessed through the
minimum number of inferences, thus achieving
maximum efficiency. This is only possible by
examining the domain in which the learning is
occurring and identifying the underlying prerequisite
relationships that exist between concepts. This is a time
consuming and expert task that involves describing a
learning domain, such as mathematics, in terms of
formal prerequisite relationships. Specific educational
tasks, such as the learner interacting with a virtual
experiment, are broken down into specific sub-tasks.
Success or failure in these sub-tasks forms evidence
that facilitates the probabilistic update of the learner’s
model. The certainty is dependent on the level of
inference required. However, as only partial evidence
is needed to assess a skill state it can be done very
efficiently. When applied to DEGs KST has the
potential to provide the basis of a time sensitive
approach to modeling a learner’s acquisition of
knowledge and skills [8].
3. The ELEKTRA Project
One of the core design strategies of the ELEKTRA
project [14] was to separate the gaming environment
from the learning adaptation [15]. This was realized
through the two core components the Game Engine
(GE), responsible for graphics, audio, and gameplay,
and the Learning Engine (LE), which is responsible for
the adaptation of the educational experience. The
communication from the GE to the LE provides the
evidence on which adaptation is performed, and
conversely the LE to GE communication contains the
game adaptations to be executed. In this service-driven
approach to adaptation [16] the LE reasons over
educational concerns that have been abstracted and
inferred from the basic game evidence.
The nature of the game evidence sent from the GE
is game specific and consists of player actions,
movements, and task successes or failures. This
information however is not immediately useful for
educational adaptation, requiring a degree of inference
by the LE. Inference within the LE is the first step in
the four stage process employed to provide effective
non-invasive adaptation. The four stages employed are
inference, context accumulation, adaptation constraint,
and adaptation selection. Further details of the four
stage adaptation process are detailed in [5].
The design of the LE and the four stage adaptation
process allows for the educational adaptation to be
performed without regard for the game specifics. The
LE effectively infers and abstracts game actions into
educational evidence that can be reasoned over in a
generic manner, thus enabling it to be employed for
different games with minimal alteration. A key
example of this is the abstraction of skills provided
through the Skill Assessment Engine (SAE). The SAE
effectively maps user actions within the game to skill
evidence, and further generates a probabilistic skill
model for the learner.
The second stage of the adaptation process involves
accumulating game and learner evidence. In
consideration of the large quantity of evidence
accumulated, potentially dozens of items per second,
the use of XML based models, a traditional approach
in many Adaptive Hypermedia Systems such as
APeLS [10], becomes impractical due to manipulation
and reasoning speed. Consequently all data is
accumulated in a working memory provided by the
Drools rule engine. The use of the Drools rule engine
provides an efficient means to reason over large data
sets using declarative logic.
In order to perform adaptation within the GE the LE
must have an a priori abstracted understanding of the
adaptations possible. Within the LE these adaptations
are represented as Adaptive Elements. An Adaptive
Element consists of an adaptation identifier used by the
game and associated metadata indicating the probable
outcome of the adaptation and when it can be suitably
used. An example Adaptive Element in the ELEKTRA
game would be the Non Player Character (NPC)
Galileo giving an encouraging hint such as, “Yes. It
isn’t easy, and I’m not sure that I would do any better
in your position, but you must persevere.” Such an
Adaptive Element would have an abstracted outcome
description of “encouragement”, and a suitability
requirement of the Galileo NPC being present.
The following are the benefits of using Adaptive
Elements:
539
• Educational adaptation does not need to be
concerned with realizing adaptations
• Facilitates the independent authoring of the
game engine and the adaptation logic
The third LE stage of adaptation constraint is
concerned with ensuring that only appropriate
Adaptive Elements are used. By using constraint rules,
only feasible and appropriate Adaptive Elements are
made available for selection in the final LE stage. The
selection of adaptation is achieved through adaptation
rules that examine the accumulated learner data and the
available Adaptive Elements.
Through an authentic evaluation using secondary
school physics students the ELEKTRA game proved to
be effective and successful. The ELEKTRA game is a
narrative-driven adventure game where the
learner/gamer must overcome several physics-oriented
puzzles. They are guided by an NPC representing the
ghost of Galileo who advises and encourages them as
they interact with experimental apparatus. The skills
they acquire are directly relevant to tasks they will face
whilst playing the game. Through the evaluation of
ELEKTRA the realtime and appropriate nature of the
adaptation was favorably received [DIGITEL-ref].
4. Skill Assessment Engine
Interpreting evidence sent by the Game Engine (GE) is
central to the first inference step of the four stage
ELEKTRA process [5]. The Skill Assessment Engine
(SAE), a component of the Learning Engine (LE), is
responsible for translating each learner’s actions within
the game into a list of probabilities that show the
likelihood of each relevant skill having been acquired
by the learner. This assessment of a learner’s skills
must be done in an implicit fashion so as not to
negatively impact their flow through the game.
The domain specific skills to be acquired in the
ELEKTRA game were organized according to KST
into a prerequisite knowledge structure, which was
encoded as an OWL ontology and parsed by the SAE
at design time. The parsing process had a dual purpose:
firstly it extracted each valid skillstate (a unique
combination of skills a user could have at any one
time) from the ontology; secondly it converted these
skillstates into a binary matrix, which could be more
efficiently processed by the SAE at runtime, than the
more verbose OWL representation. The runtime
performance of ontologies, even quite small ones, is
poor and insufficient for use in time sensitive DEGs.
During the game, the user faces various learning
challenges, with specific educational rules triggered
depending on their interactions with learning objects,
such as virtual experimental apparatus (Fig. 1).
Learning objects are traditionally seen as static pieces
of content, usually HTML, with associated metadata.
In ELEKTRA a learning object was an interactive
experience that was woven into the game narrative.
Each learning object has skills associated with it, thus
if a rule relating to a learning object is fired through a
learner’s interaction the SAE must run its algorithm to
determine which skillstates (and subsequently which
individual skills) have increased or decreased in
probability. Once the thresholds for skill probabilities
have exceeded a pre-determined value, the user is said
to have acquired this skill. These calculations must be
done in less than 200ms [17] so that the delay in the
LE selecting an appropriate intervention for the GE is
not noticeable to the user. For the purposes of the work
presented here below 200ms is the definition of
realtime. The adjustment in skill probabilities is taken
into account in stage two of the ELEKTRA process
where all evidence from the game and user is
accumulated. Thus any change in skill probabilities
has influence over which adaptive interventions are
eventually presented to the user within the game
environment.
Figure 1. The Slope Device
The initial ontology created for the ELEKTRA game
contained 83 skills and had over 10 million
corresponding skillstates. Because of the large number
of skillstates it meant that there would be latency
issues when applying the SAEs algorithm at runtime.
Any such delay would be detrimental to selecting
appropriate adaptive interventions in a timely fashion.
Thus a reduced version of the skill list and its
corresponding prerequisite relation was developed,
which contained 25 skills of a lesser granularity.
These skills contained 12,414 skillstates, which was a
number that could be processed at runtime with
minimal delay by the SAE.
Because of the issue regarding the maximum
amount of skillstates that can be efficiently processed
by the SAE, the scalability of the solution is in
question. For ELEKTRA this was not an issue due to
540
the limited scope of the game, however for larger
games with many more learning situations (and
associated number of skillstates), it would not be a
viable technique to process the entire skill structure as
a single entity at runtime. The next iteration of the
SAE, currently being researched and developed as part
of the European Commission 80Days project [18], will
tackle this precise problem. By working with partitions
(with a correspondingly reduced number of skillstates)
and not the complete ontology, the next version of the
SAE will provide a scalable and practical solution for
the runtime calculation of a user’s current skills.
5. Evaluating Skill Assessment
The evaluation of the SAE relied on the comprehensive
log files generated by the Learning Engine. These log
files detailed every action performed by the learner, the
corresponding skill probability changes, and any
adaptations sent to the Game Engine. The following
graphs illustrate how a learner’s skill probabilities
change after successive task attempt. The large circles
indicate skills that were targeted with adaptations
following a task attempt, i.e. each circle indicates a
personalized adaptation that was presented to assist the
learner.
Figure 2. Skill probability change with task
attempts, student A.
The graphs shown in figures 2 and 3 plots of ten skill
probabilities against the number of learning task
attempts in a learning object. It shows ten of the
twenty-five possible skills. The remaining skills have
been omitted as they were not relevant to the specific
task and so showed negligible change. Through
comparing the final skill probabilities of the student A
(Fig. 2) and student B (Fig. 3) it is evident that the
SAE has effectively identified skills deficiencies and
provided adaptations accordingly. This is particularly
noticeable in task attempts 14 and 15 in figure 2, and in
task attempts 13-15 in figure 3.
By way of example, consider the plotline with
square markers (with a starting probability of
approximately 0.8) in Figure 2. This line indicates the
learner’s knowledge about gravity. The line with small
circles starting at a probability of about 0.7 represents
their knowledge of magnetism. The experiment they
are interacting with was referred to as the ‘slope
device’ and enabled a learner to experiment with the
effect gravity has on a falling object. They could
attempt to impact the objects trajectory by
manipulating a magnet and fan. In the case of the
learner shown in figure 1, they initially exhibited
slightly poor control over these mechanisms, by
altering the magnet when the falling object was made
of wood. This is exhibited in drops in the skill
probability of both the gravity and particularly the
magnetism skills. From task attempt six onwards the
learner receives adaptive hints and exhibits an
improvement in both skills. The learner shown in
figure 2, however, did not improve after the adaptive
hinting and a drop in their skills (corresponding to
weaker performance in the task) is seen.
Figure 3. Skill probability change with task
attempts, student B.
Although it appears that the relatively high probability
skills receive more frequent adaptations, this is a result
of the learning task in question dealing predominantly
with these skills. Additionally it should be noted that
not all hints are explanatory, such an example would
be the adaptation given for task attempt four for both
students. This adaptation was the following
congratulatory dialogue for the recent skill increases,
“I knew you were up to this challenge”.
Due to the finite number of Adaptive Elements
available, skills with dropping probabilities are not
always targeted for adaptations. This is by design as it
541
is not always feasible or appropriate in a game context
to select a particular adaptation.
The starting probability of each skill is determined
from the distribution of a particular skill across all
skillstates. Initially all skillstates are assigned a
uniform probability, thus making no assumptions of a
learner’s prior knowledge. This was again a design
decision as it was felt that an explicit pre-test would
adversely impact the game experience. However,
paper-based pre-tests were used as part of the
evaluation to investigate cognitive gain.
6. Future Work and Conclusion
This paper has presented the Skill Assessment Engine,
a component of the ELEKTRA DEGs Learning
Engine. This component is responsible for the realtime
evaluation of a learner’s skills through interpreting
evidence from a Game Engine. By using the
probabilistic-based Knowledge Space Theory the SAE
can determine a learner’s probable skillstate with a
minimum of evidence. However, this approach, whilst
effective within the limited scope of the ELEKTRA
game, does not seem to scale well. This is due to the
large number of possible skillstates that can exist with
even just a limited number of skills. As part of the
80Days project [18], a continuation of ELEKTRA, a
solution is being proposed that involves partitioning
the Knowledge Space. This approach will enable the
SAE to function much as it did in ELEKTRA, but a
solution for mapping the boundaries of partitions needs
to be derived. That said, the approach proposed in this
paper shows much potential for effectively and
efficiently assessing learners’ skills when there is a
realtime consideration.
7. References
[1] Frankola, K. (2001). Why online learners dropout.
Workforce, 10, 53- 63.
[2] A. Stern and M. Mateas, "Build It to Understand It:
Ludology Meets Narratology in Game Design Space," in
International DiGRA Conference Vancouver, British
Columbia, Canada 2005.
[3] P. Moreno-Ger, D. Burgos, J. L. Sierra, and B.
Fernández-Manjón, "A Game-Based Adaptive Unit of
Learning with IMS Learning Design and <e-Adventure>," in
Second European Conference on Technology Enhanced
Learning (EC-TEL 2007), Crete, Greece., 2007.
[4] W. L. Johnson, N. Wang, and S. Wu, "Experience with
serious games for learning foreign languages and cultures,"
in SimTecT Conference., Australia, 2007.
[5] Peirce, N., Conlan, O., Wade, V., "Adaptive Educational
Games: Providing Non-invasive Personalised Learning
Experiences," Digital Games and Intelligent Toys Based
Education, 2008 Second IEEE International Conference on ,
vol., no., pp.28-35, 17-19 Nov. 2008.
[6] M. Csikszentmihalyi, Flow: The psychology of optimal
experience. NY: New York: Harper and Row., 1990.
[7] Conlan, O., Hampson, C., O'Keeffe, I., Heller, J., Using
Knowledge Space Theory to support Learner Modeling and
Personalization, World Conference on E- Learning in
Corporate, Government, Healthcare, and Higher Education,
Hawaii, USA, 2006.
[8] D. Albert, C. Hockemeyer, M. D. Kickmeier-Rust, N.
Peirce, and O. Conlan, "Microadaptivity within Complex
Learning Situations – a Personalized Approach based on
Competence Structures and Problem Spaces," in 15th
International Conference on Computers in Education, ICCE
2007 Hiroshima, Japan, 2007.
[9] R. M. Carro, A. M. Breda, G. Castillo, and A. L.
Bajuelos, "A Methodology for Developing Adaptive
Educational-Game Environments," in AH 2002:Adaptive
Hypermedia and Adaptive Web-Based Systems Malaga,
Spain: Springer, 2002.
[10] O. Conlan and V. P. Wade, "Evaluation of APeLS - An
adaptive eLearning service based on the multi-model,
metadata-driven approach," Adaptive Hypermedia and
Adaptive Web-Based Systems, Proceedings, vol. 3137, pp.
291-295, 2004.
[11] Specht, M., Kravcik, M., Klemke, R., Pesin, L.,
Huttenhain, R.: Adaptive Learning Environment for
Teaching and Learning in WINDS. LECTURE NOTES IN
COMPUTER SCIENCE (2002) 572-575
[12] De Bra, P., Aerts, A., Berden, B., de Lange, B.,
Rousseau, B., Santic, T., Smits, D., Stash, N.: AHA! The
adaptive hypermedia architecture. Proceedings of the
fourteenth ACM conference on Hypertext and hypermedia
(2003) 81-84
[13] Doignon, J.-P., & Falmagne, J.-C. (1985). Spaces for the
assessment of knowledge. International Journal of Man-
Machine Studies, 23, 175-196.
[14] "ELEKTRA - Enhanced Learning Experience and
Knowledge TRAnsfer", Retrieved 25th June 2008 from
http://www.elektra-project.org
[15] Kickmeier-Rust, M. D., Peirce, N., Conlan, O., Schwarz,
D., Verpoorten, D., Albert, D. (2007) Immersive digital
games: The interfaces for next-generation e-learning.
Proceedings of the HCI International 2007, 22-27 July 2007,
Beijing, Lecture Notes in Computer Science. New York,
Heidelberg. Springer. pp647 – 656.
[16] Brusilovsky, P., Wade, V., Conlan, O.: From Learning
Objects to Adaptive Content Services for E-Learning. In:
Pahl, C. (ed.): Architecture Solutions for E-Learning
Systems. IGI Global (2007) 243 – 261.
[17] I. S. MacKenzie and W. Colin, "Lag as a determinant of
human performance in interactive systems," in Proceedings
of the INTERACT '93 and CHI '93 conference on Human
factors in computing systems Amsterdam, The Netherlands:
ACM, 1993.
http://portal.acm.org/citation.cfm?doid=169059.169431
[18] “80Days - around an inspiring virtual learning world in
eighty days” Retrieved 16th January 2009 from
http://www.eightydays.eu
542
