Figure 1 - uploaded by Baltasar Fernández-Manjón
Content may be subject to copyright.
An xAPI Statement representing a learning activity. Specifically, that "John Doe completed a Programming Course".
Source publication
Learning Analytics is an emerging field focused on analyzing learners’ interactions with educational content. One of the key open issues in learning analytics is the standardization of the data collected. This is a particularly challenging issue in serious games, which generate a diverse range of data. This paper reviews the current state of learni...
Context in source publication
Context 1
... format also derives from Activity Streams, and the main attributes in a Statement are actor, verb (action), and object. Figure 1 illustrates its structure. The statements can contain additional attributes with more information about the experience: result, containing the outcomes of the statement; context, representing the learning environment; or authority, specifying who assures the truthfulness of the statement. ...
Similar publications
Learning Analytics have become an indispensable element of education, as digital mediums are increasingly used within formal and informal education. Integrating specifications for learning analytics in non-traditional educational mediums, such as serious games, has not yet reached the level of development necessary to fulfil their potential. Though...
Serious Games (SGs), defined as a game in which education (in its various forms) is the primary goal rather than entertainment, have been proven as an effective educational tool for engaging and motivating students. However, more research is needed to sustain the suitability of these games to train users with cognitive impairments. This empirical s...
Citations
... Game metrics are quantitative measures that can range from simple measures such as point scores or reaction time to more complex measures that combine various parameters [51]. Game metrics have the potential to improve personalization and individualized progression of training according to users' performance, provide (real-time) feedback, and monitor gameplay [51][52][53]. However, there are currently no standards or guidelines for what data should be collected and for what specific purpose [52,54]. ...
... Game metrics have the potential to improve personalization and individualized progression of training according to users' performance, provide (real-time) feedback, and monitor gameplay [51][52][53]. However, there are currently no standards or guidelines for what data should be collected and for what specific purpose [52,54]. ...
BACKGROUND:
Exergame-based training enhances physical and cognitive performance in older adults, including those with mild neurocognitive disorder (mNCD). In-game metrics generated from user interactions with exergames enable individualized adjustments. However, there is a need to systematically investigate how well such game metrics capture true cognitive and motor-cognitive performance to provide a more robust basis for personalized training.
OBJECTIVE:
The primary objective was to identify valid game metrics as indicators for in-game domain-specific cognitive performance during exergaming in individuals with mNCD. We also aimed to explore game metric performance changes over time during exergame-based training.
METHODS:
Data were analyzed from individuals with mNCD who completed a 12-week home-based, exergame-based intervention following the Brain-IT training concept. A cross-sectional analysis was conducted by correlating game metrics with standardized neurocognitive reference assessments. To confirm the alternative hypothesis, we predetermined the following criteria: (1) statistically significant correlation (P≤.05; uncorrected; 1-sided) with (2) a correlation coefficient (Pearson r or Spearman ρ) of ≥0.4. Visual and curve-fitting longitudinal analyses were conducted to explore game performance changes over time.
RESULTS:
Data were available from 31 participants (mean age 76.4, SD 7.5 y; n=9, 29% female). In total, 33% (6/18) of the game metrics were identified as valid indicators for in-game cognitive performance during exergaming. In the neurocognitive domain of learning and memory, these metrics included the mean reaction time (ρ=–0.747), the number of collected items (ρ=0.691), and the precision score (r=–0.607) for the game Shopping Tour (P<.001 in all cases), as well as the point rate (P=.008; r=0.471) for the game Simon. In addition, point rate was a valid indicator for executive function (P=.006; r=0.455) and visuospatial skills (P=.02; r=0.474) for the games Targets and Gears, respectively. The exploratory longitudinal analysis revealed high interindividual variability, with a general trend of the expected typical curvilinear curves of rapid initial improvements followed by a plateau in performance.
CONCLUSIONS:
This study demonstrated that metrics reflecting the precision of responses generally performed better than metrics reflecting the speed of responses. These observations highlight the importance of selecting valid game metrics for implementation in exergame designs. Further research is needed to explore the potential of game metrics and identify factors contributing to individual variability in in-game performance and performance progression, as well as identifying and adopting strategies that facilitate individual learning success and thus promote effectiveness in improving health outcomes.
... Using a standard format to gather user interaction data not only simplifies data collection but also streamlines later analysis of the collected data. The Experience API for Serious Games (xAPI-SG) profile allows for the collection of GLA interactions from serious games using an xAPI-based vocabulary that represents the most common interactions present in these games [28]. Each player interaction is captured as an xAPI-SG trace (a so-called statement), usually represented in JSON. ...
Serious games are powerful interactive environments that provide more authentic experiences for learning or training different skills. However, developing effective serious games is complex, and a more systematic approach is needed to create better evidence-based games. Learning analytics—based on the analysis of collected in-game user interactions—can support game development and the players’ learning process, providing assessment information to teachers, students, and other stakeholders. However, empirical studies applying and demonstrating the use of learning analytics in the context of serious games in real environments remain scarce. In this paper, we study the application of learning analytics throughout the whole lifecycle of a serious game, in order to assess the game’s design and players’ learning using a serious game that introduces basic programming concepts through a visual programming language. The game was played by N = 134 high school students in two 50-min sessions. During the game sessions, all player interactions were collected, including the time spent solving levels, their programming solutions, and the number of replays. We analyzed these interaction traces to gain insights that can facilitate teachers’ use of serious games in their lessons and assessments, as well as guide developers in making possible improvements to the game. Among these insights, knowing which tasks students struggle with is critical for both teachers and game developers, and can also reveal game design issues. Among the results obtained through analysis of the interaction data, we found differences between boys and girls when playing. Girls play in a more reflexive way and, in terms of acceptance of the game, a higher percentage of girls had neutral opinions. We also found the most repeated errors, the level each player reached, and how long it took them to reach those levels. These data will help to make further improvements to the game’s design, resulting in a more effective educational tool in the future. The process and results of this study can guide other researchers when applying learning analytics to evaluate and improve the educational design of serious games, as well as supporting teachers—both during and after the game activity—in applying an evidence-based assessment of the players based on the collected learning analytics.
... The system captures events related to in-game activities and learning progress. Our logging system draws inspiration from the works of Carvalho and colleagues (2015) and Serrano-Laguna and colleagues (Serrano-Laguna et al., 2017). Carvalho and colleagues introduced the Activity Theory-based Model of Serious Games (ATMSG) to define and deconstruct explicit content for the logging system to collect. ...
... xAPI, a model proposed by Serrano-Laguna and colleagues, is a high-level standard for logging system design. This model aims to ensure that the data collected is efficient and effective for measuring learning goals across various game environments (Serrano-Laguna et al., 2017). The xAPI model has inspired our design for high-level data collection processes. ...
With the guidance of the Evidence-Centered Design (ECD) framework, we systematically implement stealth assessment for the serious game Mission HydroSci (MHS) and empirically evaluate its effectiveness using combinations of features derived from gameplay and their analysis using computational models. We show that adaptive stealth assessment implemented in a game-based learning environment can provide sufficient and reliable information about learning growth, game-related performance, and the associated human behavioral patterns. Stealth assessment unobtrusively collects information about students' in-game behaviors and progress, which teachers and game designers can leverage to sustain engagement and decrease test anxiety. This paper describes the process of implementing and evaluating the stealth assessment step by step, including raw game log generation, feature engineering process, computational model construction, and model outcome interpretation. The model performance indicates that the computational model using Random Forest with the full feature set outperforms other candidates with an accuracy rate of 86\%. This acceptable threshold empirically proves the reasonability of our workflow of stealth assessment implementation. We interpret the model outcome to summarize the findings by making model inferences with the Bayesian Generalized Linear model (BGL), an interpretable "White-box" model that approaches the second-best performance according to the model performance matrix. The discussion section lists promising practical applications inspired by the model outcome and limitations for future studies.
... Streams, which represent sequences of actions performed, being an action equal to an activity (SERRANO-LAGUNA et al., 2017). The xAPI specification represents the activities with a JSON-based syntax, adjusted to provide outcomes and context of the activity 5 . ...
... Learning activity representation. Source:Serrano-Laguna et al. (2017) structures, activity definition, and activities resource. LRS is also responsible for checking if the data corresponds to the xAPI statement(SIMAO et al., 2018).During the process, some stakeholders are involved, as summarized inFigure 2.4. ...
... The system uses the xAPI open-source standard [33] to format the in-game traces during the learning process, using the xAPI for Serious Games profile [34], which includes a vocabulary to identify all the in-game significant moments like progressed or completed. All GLA traces are anonymized at source, being solely tagged by an anonymous identifier provided for each student, with teachers being the only participants with access to the correspondence between identifiers and specific students. ...
By incorporating elements such as storytelling, problem-solving, and rewards, serious games can appeal to varied users, making the learning process more experiential, engaging, and enjoyable. Gender stereotyping is a prevalent social problem that occurs to a greater or lesser extent in all countries and cultures around the world. However, it is more present in certain places like Latin American countries. This study presents the evaluation of Kiddo, a serious game to raise awareness about gender stereotypes among school-aged children (10–13 years old). After its validation with teachers, this evaluation study was carried out with 210 students in a public school in Mexico. By conducting a pre-post experiment while collecting game learning analytics data, we explore how Kiddo can raise students’ awareness of gender stereotypes. Results show a statistically significant increase in awareness of all gender stereotypes addressed in the game. In addition, we explore the acceptance of the video game, the feelings that it causes in players, and the validity of its design as an educational tool including requirements such as class duration and participants’ reading ability. Kiddo provides a valuable real-world experience in a safe and controlled environment.
... User interaction data in this case refer to all user activities, behaviour and user preferences collected during interaction with an AR serious game. The model is based on event-based tracking as one of the most widely used methods to track user interactions in serious games [58]. This model aims to enable easier data analysis for understanding the data to draw relevant conclusions, such as how to improve and adapt an AR-based serious game in the context of accessibility for users. ...
... The interaction model proposed in [58] derived from the analysis of how learners' interactions are tracked in serious games. In this interaction model, events are used to represent players' interactions in a game where an interaction event is defined with the following attributes: (1) a timestamp, representing the instant the event was generated in the game; (2) a user ID, identifying the player that generated the event; (3) an action, representing the type of interaction performed by the player; (4) a target, representing a game element that is the objective of the player's action; and (5) an optional value, representing the parameters of the action [58]. ...
... The interaction model proposed in [58] derived from the analysis of how learners' interactions are tracked in serious games. In this interaction model, events are used to represent players' interactions in a game where an interaction event is defined with the following attributes: (1) a timestamp, representing the instant the event was generated in the game; (2) a user ID, identifying the player that generated the event; (3) an action, representing the type of interaction performed by the player; (4) a target, representing a game element that is the objective of the player's action; and (5) an optional value, representing the parameters of the action [58]. The interaction model defined like this suits the needs of tracking user interactions in AR-based serious games in the context of accessibility as well. ...
Serious games combined with augmented reality (AR) can positively impact user’s motivation and learning experience. To ensure that users with different needs can use them effectively, we must ensure the accessibility of such solutions. The goal of this paper is to show that game-learning analytics can be utilized to identify issues for people with various needs and define ways to improve the accessibility of AR-based serious games. This paper presents a framework based on analysing automatically collected user interaction data in handheld AR-based serious games to adapt them to the user in the context of accessibility. An experiment was conducted with people with disabilities and people of different ages using the established framework. The focus of the experiment results analysis in this paper is on the evaluation of interaction mechanisms adapted to users based on their interaction data and preferences. The results showed that for most participants the assigned interaction mechanisms for virtual object transformation tasks (translation, scaling, rotation) in AR were satisfactory. The framework proposed in this paper provides the basis for further research in this field concerning the use of advanced analytics that enable intelligent adaptation to the individual user.
... The analysis and interpretation of data generated by SGs can provide valuable information for learners and instructors in educational settings. For example, instructors can follow a student's progression in real-time while playing and take action on any identified learning problems (Serrano-Laguna et al. 2017). This data analysis would enable the application of JIT interventions in classrooms. ...
In recent years, serious games (SGs) have emerged as a powerful tool in education by combining pedagogy and entertainment, facilitating the acquisition of knowledge and skills in engaging environments. SGs enable the collection of valuable interaction data from students, allowing for the analysis of student performance, with artificial intelligence (AI) playing a key role in processing this data to make informed inferences about their knowledge and skills. However, the lack of explainability in AI models represents a significant challenge. This research aims to develop an interpretable model for predicting students' performance in real‐time while playing an SG by: (1) calculating the performance of an interpretable prediction model of task completion in an SG and (2) demonstrating the application of the interpretable model for just‐in‐time (JIT) classroom interventions. Our results show that we are able to predict students' task completion in real‐time with a balanced accuracy result of 77.21% after a short playtime has elapsed. In addition, an explainable artificial intelligence (XAI) approach has been applied to ensure the interpretability of the developed models. This approach supports personalised learning experiences, unlocks AI benefits for non‐technical users, and maintains transparency in education.
... In their search for dynamic data collection solutions for Learning Analytics, [Cooper, 2014], Ángel Serrano-Laguna et al. [2017] developed an API for serious games that allows educators to customize predefined actions. Their platform, known as xAPI Profile, provides a method to implement Learning Analytics within an application, enabling data collection from user interactions. ...
Improving education through virtual reality is challenging due to the high costs of developing systems and the lack of teacher engagement in project conception. This project introduces and evaluates a system that enables teachers to create their own lessons in virtual reality without prior programming knowledge, addressing these challenges. By integrating Learning Management tools, Learning Analytics, and Multi-user Connection Servers, we developed a system that allows multiple users to coexist in the same virtual environment and perform various tasks. Our system also enables educators to evaluate students based on their behavior. In EduVR, teachers can configure activities in advance and retrieve results. Heuristic evaluations of EduVR’s integrated environment design have demonstrated its potential as a model for developing new educational platforms, setting a new standard for virtual reality educational systems.
... Another step towards enhancing virtual patients could be more complex design. The data could be enriched by incorporating a technological addition: the use of xAPI (Serrano-Laguna et al., 2017), a method for learning analytics that records data about learner activity, allowing comparison of their actions across several versions of a virtual patient more comprehensively than is currently possible. Additionally, in narrative game design, the concept of storylets is discussed as an alternative approach to branched narrative, which is sometimes perceived as missing a dimension of dynamics and expression (Reed, 2017). ...
Interactive stories for learning (ISL) are a powerful pedagogical approach, well supported by learning theory and scholarly research. Learners traverse a story which reflects their real-life environment, make decisions and explore diverse narrative paths, learning from the consequences of their actions. It is a safe space for learners to practice, where failures function as learning opportunities. Despite their potential, ISL often fail to engage learners effectively due to poor execution. Learning designers face the challenge of ensuring narrative engagement while enhancing learner capability, but may lack the necessary skills to craft high-quality interactive stories. This gap is particularly clear when the ISL deals with intricate human interactions, such as healthcare provider-patient conversations. Scholars advocate for better narratives to enhance the potential of ISL in healthcare-referred to as virtual patients-for teaching non-technical skills, including empathy and compassionate care. However, crafting advice is scarce and fragmented, and too focused on learning from linear, not interactive storytelling. This study endeavoured to enhance ISL by learning the craft from narrative design in video games, where expertise and innovation in producing high-quality interactive narratives has been fostered since the earliest games, more than 50 years ago. In the first phase of this research, disseminations from narrative design experts were collected, analysed and synthesised. The expert advice emphasised the pivotal role of emotions and player self-expression in crafting interactive narrative, along with the importance of designing believable characters and meaningful choices. A comprehensive heuristics framework to craft ISL was developed based on the insights from this phase. Through iterative prototyping and reflection, the heuristics framework was evaluated and refined, and subsequently applied to the recrafting of a virtual patient for compassion training. The recrafted and original version were presented to nurses in the final phase of this study. An online survey measured the participants' narrative transportation in the virtual patient story and asked about their learner experience. Additionally, their decision-making during playthroughs was recorded. While no statistically significant differences for narrative transportation were found, the results from the playthrough data and open-ended questions demonstrated that incorporating emotional depth into virtual patient design significantly impacted learner engagement and empathy. Participants exhibited more compassionate care when interacting with the recrafted virtual patient, showing highly improved decision-making to promote patient outcomes. This study contributes valuable insights into leveraging game narrative techniques to enhance the crafting of virtual patients for compassionate care training. By bridging the gap between learning design and game narrative expertise, educators can create more immersive and effective ISL experiences, ultimately enhancing learner outcomes and experiences. 3
... The limitations of this research includes: (i) the bias of GLA specialists, who belong to the same research group, which may have affected the integrity of the evaluations; (ii) the analysis of the study, since it only contemplates the perspective of specialists (not yet of students) about the use of the agent; (iii) the tests conducted, which superficially analyze the agent's responses, not performing a statistical comparison with other strategies (Chat-GPT's versions, other LLMs, etc.); (iv) the emphasis on GLBoard, whose agent proposes structures compatible with this model but not with other tools -such as x-API-SG [Alonso-Fernández et al. 2021, Serrano-Laguna et al. 2017. As an initial study, this work aimed to present the first steps towards building a GLA specialist agent, focusing on analyzing the perspective of GLA specialists when using it and not yet comparing the generated structures, whose limitations will be minimized in subsequent works. ...
Game Learning Analytics (GLA) involves capturing and analyzing data from educational games, enabling the identification of evidence of learning. A fundamental step before implementing GLA techniques is data modeling, which is not trivial. Using large language models (LLMs) can help in this context, as they can generate text like humans. Therefore, considering Chat-GPT and its customizable functionality, “MyGPTs,” this work proposes creating a specialist agent to assist learning designers in data modeling and implementing GLA techniques based on the GLBoard system. Preliminary results with GLA specialists were positive, indicating the agent’s potential.
