Content uploaded by Stylianos Asteriadis
Author content
All content in this area was uploaded by Stylianos Asteriadis
Content may be subject to copyright.
A preview of the PDF is not available
Fusing Visual and Behavioral Cues for Modeling User Experience in Games
Abstract and Figures
Estimating affective and cognitive states in conditions of rich human-computer interaction, such as in games, is a field of growing academic and commercial interest. Entertainment and serious games can benefit from recent advances in the field as, having access to predictors of the current state of the player (or learner) can provide useful information for feeding adaptation mechanisms that aim to maximize engagement or learning effects. In this paper, we introduce a large data corpus derived from 58 participants that play the popular Super Mario Bros platform game and attempt to create accurate models of player experience for this game genre. Within the view of the current research, features extracted both from player gameplay behavior and game levels, and player visual characteristics have been used as potential indicators of reported affect expressed as pairwise preferences between different game sessions. Using neuroevolutionary preference learning and automatic feature selection, highly accurate models of reported engagement, frustration, and challenge are constructed (model accuracies reach 91%, 92%, and 88% for engagement, frustration, and challenge, respectively). As a step further, the derived player experience models can be used to personalize the game level to desired levels of engagement, frustration, and challenge as game content is mapped to player experience through the behavioral and expressivity patterns of each player.
Figures - uploaded by Stylianos Asteriadis
Author content
All figure content in this area was uploaded by Stylianos Asteriadis
Content may be subject to copyright.
... The tasks of agents therefore differ from other affective game mechanisms that mostly try to adapt the game world to player affect [9,26]. Procedural content generation (PCG) based on affective information has been shown to successfully increase enjoyment and offer personalized, immersive experiences in video games (see for example the work of Shaker et al. [27,28]). This is often done by fine-tuning certain mechanics shown to be associated with a target player emotion in order to increase the probability for that emotion [26]. ...
... The basis of the approach is rooted in traditional PCG research, particularly in studies integrating affect-adaptation (see for example [26,28]). User input (which in this case is text input) typically gets parsed into the game logic to adapt the content in a meaningful way, for example to better elicit a target experience [21]. ...
The development of believable, natural, and interactive digital artificial agents is a field of growing interest. Theoretical uncertainties and technical barriers present considerable challenges to the field, particularly with regards to developing agents that effectively simulate human emotions. Large language models (LLMs) might address these issues by tapping common patterns in situational appraisal. In three empirical experiments, this study tests the capabilities of LLMs to solve emotional intelligence tasks and to simulate emotions. It presents and evaluates a new Chain-of-Emotion architecture for emotion simulation within video games, based on psychological appraisal research. Results show that it outperforms control LLM architectures on a range of user experience and content analysis metrics. This study therefore provides early evidence of how to construct and test affective agents based on cognitive processes represented in language models.
... The tasks of agents therefore differ from other affective game mechanism that mostly try to adapt the game world to player affect [65,16]. Procedurally generated content (PCG) based on affective information has been shown in video games to successfully increase enjoyment and offer personalized, immersive experience (see for example the work of Shaker et al. [56,55]). This is often done by fine-tuning certain mechanics shown to be associated with a target player emotion to increase the probability for that emotion [65]. ...
... The basis of the approach is rooted in traditional PGC research, especially integrating affect-adaptation (see for example [65,55]). User input (which in this case is text input) typically gets parsed into the game logic to adapt the content in a meaningful way, for example to better elicit a target experience [64]. ...
The development of believable, natural, and interactive digital artificial agents is a field of growing interest. Theoretical uncertainties and technical barriers present considerable challenges to the field, particularly with regards to developing agents that effectively simulate human emotions. Large language models (LLMs) might address these issues by tapping common patterns in situational appraisal. In three empirical experiments, this study tests the capabilities of LLMs to solve emotional intelligence tasks and to simulate emotions. It presents and evaluates a new chain-of-emotion architecture for emotion simulation within video games, based on psychological appraisal research. Results show that it outperforms standard LLM architectures on a range of user experience and content analysis metrics. This study therefore provides early evidence of how to construct and test affective agents based on cognitive processes represented in language models.
... Another study on multimodal methods exploited facial features and interaction data to predict engagement and frustration. The authors used a multilayer perceptron (MLP) and achieved an accuracy of 78% (Shaker et al. 2013). A similar study by Monkaresi et al. (2016) utilized facial features and heart rate to detect engagement. ...
... There is a widespread assumption that models with multiple-sensor or multimodal methods give better accuracy than models with a few sensors or single-sensor methods (Aslan, et al. 2014;Asteriadis et al. 2009;Shaker et al. 2013). A previous study on this assumption shows that the multimodal approach is not always the best. ...
Detecting the attention of children with autism spectrum disorder (ASD) is of paramount importance for desired learning outcome. Teachers often use subjective methods to assess the attention of children with ASD, and this approach is tedious and inefficient due to disparate attentional behavior in ASD. This study explores the attentional behavior of children with ASD and the control group: typically developing (TD) children, by leveraging machine learning and unobtrusive technologies such as webcams and eye-tracking devices to detect attention objectively. Person-specific and generalized machine models for face-based, gaze-based, and hybrid-based (face and gaze) are proposed in this paper. The performances of these three models were compared, and the gaze-based model outperformed the others. Also, the person-specific model achieves higher predictive power than the generalized model for the ASD group. These findings stress the direction of model design from traditional one-size-fits-all models to personalized models.
... Figure 1 illustrates the protocol for SNS and PNS activity data acquisition through ECG, GSR and RESP sensors. Video games have recently gained recognition for their ability to improve social skills, stimulate mental activity and increase stress resilience [5]. Aliyary et al. studied the beneficial and harmful effects of video games on cognitive functions [6]. ...
Stress is induced in response to any mental, physical or emotional change associated with our daily experiences. While short term stress can be quite beneficial, prolonged stress is detrimental to the heart, muscle tissues and immune system. In order to be proactive against these symptoms, it is important to assess the impact of stress due to various activities, which is initially determined through the change in the sympathetic (SNS) and parasympathetic (PNS) nervous systems. After acquiring physiological data wirelessly through captive electrocardiogram (ECG), galvanic skin response (GSR) and respiration (RESP) sensors, 21 time, frequency, nonlinear, GSR and respiration features were manually extracted from 15 subjects ensuing a baseline phase, virtual reality (VR) roller coaster simulation, color Stroop task and VR Bubble Bloom game. This paper presents a comprehensive physiological analysis of stress from an experiment involving a VR video game Bubble Bloom to manage stress levels. A personalized classification and regression tree (CART) model was developed using a novel Gini index algorithm in order to effectively classify binary classes of stress. A novel K-means feature was derived from 11 other features and used as an input in the Decision Tree (DT) algorithm, strong learners Ensemble Gradient Boosting (EGB) and Extreme Gradient Boosting (XGBoost (XGB)) embedded in a pipeline to classify 5 classes of stress. Results obtained indicate that heart rate (HR), approximate entropy (ApEN), low frequency and high frequency ratio (LF/HF), low frequency (LF), standard deviation (SD1), GSR and RESP all reduced and high frequency (HF) increased following the VR Bubble Bloom game phase. The personalized CART model was able to classify binary stress with 87.75% accuracy. It proved to be more effective than other related studies. EGB was able to classify binary stress with 100% accuracy, which outperformed every other related study. XGBoost and DT were able to classify five classes of stress with 72.22% using the novel K-means feature. This feature produced less error and better model performance in comparison to using all the features. Results substantiate that our proposed methods were more effective for stress classification than most related studies.
Providing an appropriate difficulty level in a game is critical for keeping players engaged. Dynamic Difficulty Adjustment (DDA) is a common approach for optimizing player experience by automatically modifying game aspects. This paper reviews literature addressing mechanisms for adjusting video game difficulties in response to players’ performance, emotions, or personality. For this purpose, we examined DDA studies using employed machine-learning techniques, player modeling approaches, data types used to assess players’ states, testbed game genre, and application. Journal and conference articles published up to September 2022 served as the data sources in this review. The findings reveal that most studies have shown significant effects of DDA on parameters such as enjoyment, flow, motivation, engagement, and immersion. In addition, machine-learning and player modeling techniques have recently received more attention in the DDA design. However, given the ever-increasing use of games in various domains, more research is needed to understand player preferences better to adjust game parameters efficiently. By conducting further research into players’ cognitive characteristics, such as visual attention, working memory, and response time, it will be possible to understand players’ preferences better.
Accurately quantifying player experience is challenging for many reasons: identifying a ground truth and building validated and reliable scales are both challenging tasks; on top of that, empirical results are often moderated by individual factors. In this article, we present a study on the rogue-like game Hades designed to investigate the impact of individual differences in the operationalisation of player experience by cross-referencing multiple modalities (i.e., questionnaires, gameplay, and heart rate) and identifying the interplay between their scales.
This article surveys the current state-of-the-art in affective computing (AC) principles, methods, and tools as applied to games. We review this emerging field, namely affective game computing, through the lens of the four core phases of the affective loop: game affect elicitation, game affect sensing, game affect detection, and game affect adaptation. In addition, we provide a taxonomy of terms, methods, and approaches used across the four phases of the affective game loop and situate the field within this taxonomy. We continue with a comprehensive review of available affect data collection methods with regard to gaming interfaces, sensors, annotation protocols, and available corpora. This article concludes with a discussion on the current limitations of affective game computing and our vision for the most promising future research directions in the field.
Emotionally evocative games hold great potential for both industry and our basic understanding about human affect. The question of how best games elicit and respond to player emotion brings together research from psychology, design, and human-computer interaction (HCI). However, there is much confusion regarding the nature of emotions, emotional game-player interaction, and the process of creating emotional games. This paper reviews important work from relevant fields to provide an interdisciplinary model that integrates current theories and highlights conceptual gaps for relevant fields. Based on this model, we propose a methodological framework that provides a step-by-step process of developing an emotion-adaptive game. This framework is open to different models of emotion and its modular nature allows games to adapt to developments in the science of emotion and technological innovations in emotion detection. Together, the theoretical overview and framework can help reduce the confusion that surrounds creating and researching emotions in games.
This paper presents some results from our research using human-computer interactions to study the dynamics and interactive nature of emotional episodes. For this purpose, we developed the Geneva Appraisal Manipulation Environment (GAME; Wehrle 1996), a tool for generating experimental computer games that translate psychological theories into specific micro world scenarios (for details about theoretical and technical embedding of GAME see Kaiser & Wehrle, 1996). GAME allows automatic data recording and automatic questionnaires. While playing the experimental game, subjects are videotaped and these tape recordings enable an automatic analysis of the subject's facial behavior with the Facial Expression Analysis Tool (FEAT; Kaiser & Wehrle, 1992; Wehrle, 1997). With FEAT, facial actions are categorized in terms of FACS (Ekman & Friesen, 1978). These facial data can be automatically matched to the corresponding game data (using vertical time code as a refererence for both kinds of data).
Procedural content generation (PCG) is an increasingly important area of technology within modern human-computer interaction (HCI) design. Personalization of user experience via affective and cognitive modeling, coupled with real-time adjustment of the content according to user needs and preferences are important steps toward effective and meaningful PCG. Games, Web 2.0, interface, and software design are among the most popular applications of automated content generation. The paper provides a taxonomy of PCG algorithms and introduces a framework for PCG driven by computational models of user experience. This approach, which we call Experience-Driven Procedural Content Generation (EDPCG), is generic and applicable to various subareas of HCI. We employ games as an example indicative of rich HCI and complex affect elicitation, and demonstrate the approach's effectiveness via dissimilar successful studies.
In this paper, we show that personalized levels can be auto- matically generated for platform games. We build on previ- ous work, where models were derived that predicted player experience based on features of level design and on playing styles. These models are constructed using preference learn- ing, based on questionnaires administered to players after playing different levels. The contributions of the current pa- per are (1) more accurate models based on a much larger data set; (2) a mechanism for adapting level design parameters to given players and playing style; (3) evaluation of this adap- tation mechanism using both algorithmic and human players. The results indicate that the adaptation mechanism effectively optimizes level design parameters for particular players.
We consider supervised learning of a ranking function, which is a mapping from instances to total orders over a set of labels (options). The training information consists of examples with partial (and possibly inconsistent) information about their associated rankings. From these, we induce a ranking function by reducing the original problem to a number of binary classification problems, one for each pair of labels. The main objective of this work is to investigate the trade-off between the quality of the induced ranking function and the computational complexity of the algorithm, both depending on the amount of preference information given for each example. To this end, we present theoretical results on the complexity of pairwise preference learning, and experimentally investigate the predictive performance of our method for different types of preference information, such as top-ranked labels and complete rankings. The domain of this study is the prediction of a rational agent's ranking of actions in an uncertain environment.
We consider supervised learning of a ranking function, which is a mapping from instances to total orders over a set of labels (options). The training information consists of examples with partial (and possibly inconsistent) information about their associated rankings. From these, we induce a ranking function by reducing the original problem to a number of binary classification problems, one for each pair of labels. The main objective of this work is to investigate the trade-off between the quality of the induced ranking function and the computational complexity of the algorithm, both depending on the amount of preference information given for each example. To this end, we present theoretical results on the complexity of pairwise preference learning, and experimentally investigate the predictive performance of our method for different types of preference information, such as top-ranked labels and complete rankings. The domain of this study is the prediction of a rational agent's ranking of actions in an uncertain environment.
In search of suitable methods for measuring the affective state of video-game players, this study investigates the hypothesis that the player's state of arousal will correspond with the pressure used to depress buttons on a gamepad. A video game was created that would detect the force of each button press during play. It was found that as the difficulty level of the game increased, players would hit the gamepad buttons significantly harder.
We consider supervised learning of a ranking function, which is a mapping from instances to total orders over a set of labels (options). The training information consists of examples with partial (and possibly inconsistent) information about their associated rankings. From these, we induce a ranking function by reducing the original problem to a number of binary classification problems, one for each pair of labels. The main objective of this work is to investigate the trade-off between the quality of the induced ranking function and the computational complexity of the algorithm, both depending on the amount of preference information given for each example. To this end, we present theoretical results on the complexity of pairwise preference learning, and experimentally investigate the predictive performance of our method for different types of preference information, such as top-ranked labels and complete rankings. The domain of this study is the prediction of a rational agent’s ranking of actions in an uncertain environment.
More than a decade after the early research efforts on the use of artificial intelligence (AI) in computer games and the establishment of a new AI domain the term ``game AI'' needs to be redefined. Traditionally, the tasks associated with game AI revolved around non player character (NPC) behavior at different levels of control, varying from navigation and pathfinding to decision making. Commercial-standard games developed over the last 15 years and current game productions, however, suggest that the traditional challenges of game AI have been well addressed via the use of sophisticated AI approaches, not necessarily following or inspired by advances in academic practices. The marginal penetration of traditional academic game AI methods in industrial productions has been mainly due to the lack of constructive communication between academia and industry in the early days of academic game AI, and the inability of academic game AI to propose methods that would significantly advance existing development processes or provide scalable solutions to real world problems. Recently, however, there has been a shift of research focus as the current plethora of AI uses in games is breaking the non-player character AI tradition. A number of those alternative AI uses have already shown a significant potential for the design of better games.
This paper presents four key game AI research areas that are currently reshaping the research roadmap in the game AI field and evidently put the game AI term under a new perspective. These game AI flagship research areas include the computational modeling of player experience, the procedural generation of content, the mining of player data on massive-scale and the alternative AI research foci for enhancing NPC capabilities.