Conference PaperPDF Available

Challenges and Opportunities in Evaluating Learning in Serious Games: A Look at Behavioural Aspects

Authors:

Abstract and Figures

In this paper, we describe an approach to modelling competences as learning resources in a serious game environment, where competences are described in detail to identify observable behavioural indicators. This enables the evaluation and assessment of learning, where specific behaviours indicate if a player does or does not have a competence. We have used the OKEI Competence Modelling Framework to describe the competences, where the application of a competence in a specific situation or within a context can be modelled. The main focus of this paper is to analyse and discuss the opportunities and challenges that we have experienced during this work. While the approach is resources intensive to describe the competences in sufficient level of detail, it provides a reusable set of Behavioural Indicators that can be used both in designing and evaluating other Technology Enhanced Learning applications. Most importantly, the work provided important input for the design of the game scenarios in describing situations and relevant contextual information as well as input for improving the believability of the avatars in the game.
Content may be subject to copyright.
A preview of the PDF is not available
... On one hand, basic KPIs could be: ¿did the player reached the final stage?, ¿did the player get an item?, etc. On the other hand, more complex KPIs could be: ¿did the player complete a quest talking to non playable characters in a specific order?, ¿did the player lose more than 3 "lifes" between two specific enemies?, etc. KPIs are also called Behaviour Indicators or just Performance indicators and they are widely used to assess players in serious games [32,33]. In our architecture, events logged are defined by in-game KPIs. ...
Conference Paper
Full-text available
Serious games are video games with educational purposes. Players interact in many points during a gameplay. These interactions can be registered, producing data sets with sequence of events which provide relevant information about player's skills. Unfortunately, traditional skill assessment methods present limitations to carry out a detailed analysis of large data sets. Sequence Analysis is a group of techniques which allow to analyze data sets consisting of sequence of events. These techniques have been successfully implemented in different fields, and we consider that they can help overcome these limitations. In this paper, we propose an architecture of skill assessment in learning experiences based on serious games using a set of Sequence Analysis techniques known as Process Mining. First, several in-game events are stored in a log. These events are produced by player's interactions with Key Performance Indicators included in the game. Second, event log is used as input for a Process Mining tool. Discovery process is executed and a behavioural model is provided. Third, an assessment metric must be carried out over the model. Finally, a synthetic experiment is conducted and promising results are obtained.
... The last choice we can meet in the related bibliography is about serious games. They are computer systems that simulate situations based on real life to safely integrate these experiences in university curricula [11], [12]. ...
Conference Paper
Full-text available
The Learning Management Systems provide a set of facilities for the lecturer to manage his courses. Unfortunately, they have limitations when it comes to assessing generic skills. In most of them, every activity is assessable but just with a simple grade and there is not a direct link between activities and generic skills. In this work we present two alternatives to solve this issue: an assisted method based on a Model-driven architecture approach and a Rest Web service that facilitates the assessment of generic skills. We apply both approaches to a case study consisting in a Moodle-based course where we assess the ability of plan and manage time of each student. Results show that the approaches are complementary, the Web service provides more detailed formative feedback, but the Model-driven approach seems more scalable for courses with a high number of students, where it is more difficult to assess their generic skills.
... Another choice we can see in the related bibliography is about serious games. They are computer systems that simulate situations based on real life to safely integrate these experiences in university curricula [9], [10], [11], [12]. ...
Conference Paper
Full-text available
Learning Management Systems provide a set of facilities for the lecturer to create courses based on learning activities. However, assessing skills is rather limited because activities must are usually assessed with simple grades without considering the links between the activities and the skills aimed to be developed. As a consequence, limited feedback can be provided to the students, thus losing relevant information of the learning process. In this work we present a software architecture for web-based learning management systems to mitigate this issue. It consists of a web service that facilitates the assessment of skills and an extension to the browser that enables to mark and compile the evidences of assessments on web activities. The system has been applied to courses in a Moodle box where different skills that students must develop in an external wiki can be assessed while assessment evidences are tracked. All the software of this experience is fully functional and available as free software.
... However, factual knowledge in itself does not necessarily equate to attitudinal or behavioural change. Attempts to incorporate competency assessment in a game-based context have noted the high resource demands in creating competency metrics, though their reusability is of value [2]. Such metrics for cultural learning frequently focus on specific cultures and contexts, rather than providing a ubiquitous solution; however, generalizable themes exist: cultural competence is not about an individual being able to emulate another culture, rather, it is about their capability to recognize cultural differences and respond effectively. ...
Article
Full-text available
As digital games continue to be explored as solutions to educational and behavioural challenges, the need for evaluation methodologies which support both the unique nature of the format and the need for comparison with other approaches continues to increase. In this workshop paper, a range of challenges are described related specifically to the case of cultural learning using digital games, in terms of how it may best be assessed, understood, and sustained through an iterative process supported by research. An evaluation framework is proposed, identifying metrics for reach and impact and their associated challenges, as well as presenting ethical considerations and the means to utilize evaluation outcomes within an iterative cycle, and to provide feedback to learners. Presenting as a case study a serious game from the Mobile Assistance for Social Inclusion and Empowerment of Immigrants with Persuasive Learning Technologies and Social Networks (MASELTOV) project, the use of the framework in the context of an integrative project is discussed, with emphasis on the need to view game-based learning as a blended component of the cultural learning process, rather than a standalone solution. The particular case of mobile gaming is also considered within this case study, providing a platform by which to deliver and update content in response to evaluation outcomes. Discussion reflects upon the general challenges related to the assessment of cultural learning, and behavioural change in more general terms, suggesting future work should address the need to provide sustainable, research-driven platforms for game-based learning content.
... In the assessment of both players and trainers, it continues to be relatively difficult to monitor and keep track of what happens, to objectify the observations and to compare them to other sessions, as well as to provide authoritative feedback on the information in order to enhance learning or support a judgement. Stealth assessment in SG can therefore serve several functions (Kickmeier-Rust et al., 2009;Petersen & Bedek, 2012;Seitlinger, Bedek, Kopeinik, & Albert, 2012;Shute & Kim, 2010;Shute et al., , 2009Shute, 2011;Zapata-Rivera, VanWinkle, Doyle, Buteux, & Bauer, 2009): ...
Chapter
Full-text available
In this chapter, the authors present a methodology for researching and evaluating Serious Games (SG) and digital (or other forms of) Game-Based Learning (GBL). The methodology consists of the following elements: 1) frame-reflective analysis; 2) a methodology explicating the rationale behind a conceptual-research model; 3) research designs and data-gathering procedures; 4) validated research instruments and tools; 5) a body of knowledge that provides operationalised models and hypotheses; and 6) professional ethics. The methodology is intended to resolve the dilemma between the “generality” and “standardisation” required for comparative, theory-based research and the “specificity” and “flexibility” needed for evaluating specific cases.
Article
Full-text available
Learning outcomes in higher education are defined as competencies, which are either specific or generic. The former refers to those skills specific to the subject studied. The latter, also known as transferable skills, refer to other capabilities that a capable professional has to put into practice to work in real-world. Generic competencies are usually shared amongst most science fields, and include teamwork, leadership and self-criticism. These skills are inherently difficult to assess, as they are not usually considered in traditional engineering assessment procedures. As a result, lectures usually have to assess them using subjective information. This is especially problematic when the number of students increases. In this paper, we introduce Simple Assessment-Specific Query Language (SASQL), a Domain Specific Language (DSL) to alleviate this issue. It is a formal language that can be automatically processed to execute SASQL sentences using an assessment-specific vocabulary. SASQL has a simple syntax, oriented to learning assessments. Using our proposed workflow, a course coordinator can extract different objective indicators to assess competencies defined in the syllabus using simple queries. Such indicators are automatically extracted from the activity logs generated by the Learning Management System (LMS). Two case studies with Moodle LMS-based courses are carried out to explain how such indicators can be obtained and how to interpret the results obtained for student assessments.
Book
Full-text available
What do we mean by the word ‘context’ in education and how does our context influence the way that we learn? What role can technology play in enhancing learning and what is the future of technology within learning? Re-designing Learning Contexts seeks to re-dress the lack of attention that has traditionally been paid to a learner’s wider context and proposes a model to help educators and technologists develop more productive learning contexts. It defines context as the interactions between the learner and a set of inter-related resource elements that are not tied to a physical or virtual location. Context is something that belongs to an individual and that is created through their interactions in the world. Based on original, empirical research, the book considers the intersection between learning, context and technology, and explores: the meaning of the concept of context and it’s relationship to learning the ways in which different types of technology can scaffold learning in context the Learner-Centric ‘Ecology of Resources’ model of context as a framework for designing technology-rich learning environments the importance of matching available resources to each learner’s particular needs the ways in which the learner’s environment and the technologies available might change over the coming years the potential impact of recent technological developments within computer science and artificial intelligence This interdisciplinary study draws on a range of disciplines, including geography, anthropology, psychology, education and computing, to investigate the dynamics and potential of teacher-learner interaction within a learning continuum, and across a variety of locations. It will be of interest to those teaching, researching and thinking about the use of technology in learning and pedagogy, as well as those involved in developing technology for education and those who use it in their own teaching.
Article
Full-text available
This paper describes a survey undertaken to discover students' wishes concerning the contents, interaction and form of open learner models in intelligent learning environments. It was found that, in general, students are receptive to the idea of using open learner models to support their learning. Several different kinds of open learner model are presented, which illustrate a range of approaches and issues relating to the use of open learner models.
Article
Defines personal space (PS) and distinguishes it from its companion research areas and reviews and evaluates current PS theories and measurement techniques. The stop-distance and unobtrusive techniques appear best for experimental and naturalistic studies, respectively. It is suggested that considerable reevaluation of research on PS is necessary. Since only age, physical or psychological situation, stigmatizing conditions, and psychological disorders hold up as factors producing differences. Flight or withdrawal appears to result from intrusions and culturally appropriate interpersonal spacings enhance liking and message persuasiveness. It is concluded that more research is needed on theory construction, on comparative evaluation of the various measurement techniques, on specification of the 3-dimensional shape of PS, on specification of the intrusion–discomfort function, on the flexibility of PS, and on the subjective experience of PS. (4 p ref) (PsycINFO Database Record (c) 2012 APA, all rights reserved)