ArticlePDF Available

Assessing gameplay emotions from physiological signals: A fuzzy decision trees based model


Abstract and Figures

As video games become a widespread form of entertainment, there is need to develop new eval-uative methodologies for acknowledging the various aspects of the player's subjective experience, and especially the emotional aspect. Video game developers could benefit from being aware of how the player reacts emotionally to specific game parameters. In this study, we addressed the possibility to record physiological measures on players involved in an action game, with the main objective of developing adequate models to describe emotional states. Our goal was to estimate the emotional state of the player from physiological signals so as to relate these variations of the autonomic nervous system to the specific game narratives. To achieve this, we developed a fuzzy set theory based model to recognize various episodes of the game from the user's physiological signals. We used fuzzy decision trees to generate the rules that map these signals to game episodes characterized by a variation of challenge at stake. A specific advantage to our approach is that we automatically recognize game episodes from physiological signals with explicitly defined rules relating the signals to episodes in a continuous scale. We compare our results with the actual game statistics information associated with the game episodes.
Content may be subject to copyright.
KEER2010� PARIS | MARCH 2-4 2010
Joseph Onderi Orero a, Florent Levillainb, Marc Damez-Fontainea, Maria Rifqia
and Bernadette Bouchon-Meuniera
aLaboratoire d’Informatique de Paris 6 �LIP6), Universit´
e Pierre et Marie Curie, France
bLaboratoire Cognitions Humaine et Artificielle �CHART), Universit´
e Paris 8, France
As video games become a widespread form of entertainment, there is need to develop new eval-
uative methodologies for acknowledging the various aspects of the player’s subjective experience,
and especially the emotional aspect. Video game developers could benefit from being aware of
how the player reacts emotionally to specific game parameters. In this study, we addressed the
possibility to record physiological measures on players involved in an action game, with the main
objective of developing adequate models to describe emotional states. Our goal was to estimate
the emotional state of the player from physiological signals so as to relate these variations of the
autonomic nervous system to the specific game narratives. To achieve this, we developed a fuzzy
set theory based model to recognize various episodes of the game from the user’s physiological
signals. We used fuzzy decision trees to generate the rules that map these signals to game episodes
characterized by a variation of challenge at stake. A specific advantage to our approach is that we
automatically recognize game episodes from physiological signals with explicitly defined rules
relating the signals to episodes in a continuous scale. We compare our results with the actual game
statistics information associated with the game episodes.
Keywords: Emotion Recognition, Video Games, Physiological Signals, Fuzzy Sets.
In the recent years video games have enjoyed a dramatic increase in popularity, the growing
market being echoed by a genuine interest in the academic field, with an attempt to justify the
existence of video game theory in academia [1]. In this context of flourishing technological and
Corresponding author:104, Avenue du Pr´
esident Kennedy, 75016 Paris, France.
hal-00589941, version 1 - 2 May 2011
2010, Paris : France (2010)"
theoretical efforts, it is regrettable that effective methods of evaluation of the gaming technology
still lag behind. Until now, the evaluation of video games has mainly relied on traditional methods
through subjective self-reports such as questionnaires [2], interviews and focus groups [3]. These
subjective reports have limitations, one of them being that they only generate data at the moment
the question is asked, and not through a continuous process. Also, subjects are known to be poor
in self reporting their behaviors during game situations [4]. Since the main goal of video games
is to entertain through a continuous renewal of the user’s interest, controlling the emergence of
certain affective states is a key determinant in achieving a truly immersive experience. Therefore,
in evaluating video games, it would be more appropriate to assess dynamically the emotional
responses in relation to variations in challenge difficulty and to specific tasks at hand in the game.
Secondly, it has a particular interest regarding the field of developing affective human-computer
interaction applications. The video games industry appears as a laboratory where we can develop
innovative devices of interaction that could be used as a standard for designing affective-oriented
interfaces keeping track of users’s emotional states.
It is thus important to develop algorithms to automatically compute the affective state during
the play process. In this study, we addressed the possibility of recording physiological measures
on players involved in an action game, with the main objective of developing adequate models
to describe emotional states. Specifically, our main goal was to estimate the emotional state of
the player from physiological signals so as to: (i) relate these variations of the autonomic nervous
system to the specific game narratives and (ii) differentiate these game narratives according to their
level of challenge from physiological signature independent of the player.
The rest of the paper is organized as follows: In Section 2 we give an overview of affective
computing. In Section 3, we outline the methods of classification and physiological measures used.
We then describe the experimental setup in Section 4 and give our results in Section 5. Finally we
give conclusions and future work in Section 6.
Ever since Picard’s highlight [5] on the subject, affective computing has received considerable
interest from many researchers in Human Computer Interaction (HCI). In particular the recog-
nitions of emotions from physiological signals has become a major research focus in the recent
past [6]. Among a vast range of possible ways to access a user’s emotional responses such as
facial gesture or voice recognition through video and audio recording, subjective self report mea-
sures, physiological measures stand out. They grant an access to non conscious and non reportable
processes [7] and may to a certain extent be unobtrusively monitored [8]. In addition, physiolog-
ical measures may provide a fine-grained account of the rise, surge or extinction of an emotional
Considerable research progress has been made in the use of physiological measures to reliably
predict specific emotions [9, 10, 11]. However, recognition of emotions through physiological
signals is not yet a clear systematic task. There is lack of standardized methodology in the analysis
hal-00589941, version 1 - 2 May 2011
of physiological data for affective system in terms of models, measures, features and algorithms
to use [12]. The difficulty to uniquely map physiological patterns onto specific emotion types
is due to the nature of these bio data. Physiological data tends to vary considerably from one
person to another and may even display considerable differences within individuals on different
occasions [9, 10, 11]. The aim of this work was to address some of these difficulties.
First, one of the main difficulty in emotion recognition is to ensure that people are really in
the expected emotions. Researchers have employed various methods of eliciting emotions such as
guided imagery technique, movie clips, math problems and music songs [9, 13, 11]. However, use
of video games to elicit emotions is more advantageous in many ways. Particulary, due to their
interactive nature, video games tend to promote a natural sense of immersion, the player being
concerned by the consequences of its own actions, especially when mastering a skill is at stake. In
this work, we took advantage of this propensity to immerse the user with the hope that we could
drive specific physiological reactions corresponding to variations in the challenge proposed by the
game at different points.
Secondly, emotions are better represented in fuzzy terms rather than discrete categories. As
Goulev [14] already pointed out, even human ability to identify emotions from physical appearance
is rarely hundred percent and as such, it is necessary to express in fuzzy terms mapping emotions
from physiological data. Moreover, the physiological data from sensors is itself imperfect such
that it is difficult to express the results in crisp terms [15]. Also, one could express more than one
emotional state category at a particular time. A fuzzy set theory based approach could better
represent both the multi-state complexity and uncertainties as well as imperfections in emotion
3.1. Classification Methods
In emotion recognition from physiological signals, various methods of classification have been
used in the past such as linear discriminant analysis, k-nearest-neighbor (KNN), multilayer percep-
tron network and decision trees (DT) [9, 13, 10, 16, 11]. A comparison of various methods using
optimal features [10] seems to suggest that results depend on the method chosen and the nature
of the experiment. In particular, decision trees have been found to perform comparably well [16].
A major superiority characteristic of decision trees is that they give explicitly defined rules used
in classification. This is important in our case to explicitly know the relationship between the
physiological signals attributes and the emotional states.
Secondly, as we have already noted, it is preferable to use a fuzzy sets theory based approach
in classification. In fuzzy set theory [17], a fuzzy set is represented by a membership function,
µ:A[�1], indicating the degree to which an element belongs to a given set A. This is a
contrast to 1}in a crisp set, in which an element can only belong to a given set (membership
value of 1) or not (membership value of 0). It is interesting for this kind of recognition to express
the emotional states in a continuous scale of values [�1]. Therefore, we choose to use fuzzy
hal-00589941, version 1 - 2 May 2011
decision trees (FDT) in this work. Fuzzy approach has been used in the past to model emotions
from physiological signals [14, 18]. Particularly, Mandryk and Atkins [18], developed a fuzzy
logic model of rules that were used to transform arousal and valence based on psychophysiology
literature. The aim of our experiment was to automatically generated these rules that define the
psychophysiology relations from the fuzzy decision trees based on the players’ physiological data.
We used Marsala’s Salammbˆ
o Fuzzy Decision Tree [19] and compared our results with Quilan’s
C4.5 decision trees [20] and KNN [21] .
3.2. Physiological Signals and Features
Based on previous literature, we chose to collect galvanic skin response (GSR) which is a
measure of the conductivity of the skin. GSR is considered as an effective correlate to arousal
[22, 23, 24] and has been extensively used in the domain of affective computing [8, 25, 18]. We
also collected heart rate (HR) through a measure of cardiovascular activity. HR may differentiate
between positive and negative emotions [26, 27, 23] although the possible correlation between
heart rate and valence remains debated [28, 29].
After collecting the signals, the raw signals were expressed in values [�1] for each subject’s
data. Then as in [9], we calculated from each of these signals, six statistical features (the mean, the
standard deviation, and the mean of the first and the second absolute difference of the raw signal
and the normalized signal) for each game episode segment. However, in our case, for each feature,
we subtracted the subject’s baseline value. The baseline value was calculated by taking the value
of each player’s signal from the two minutes period preceding the beginning of the game. This was
to account for the variations from subject to subject in signal values and also to inform us whether
the attribute value increases or decreases on playing the particular game episode.
3.3. Feature Selection
A key research question is to determine which set of measures and features of these measures
provide the optimal combination for discriminating the different emotional categories [5]. There
is no agreement on a particular feature selection method [12], it highly depends on the nature of
the data and method used in classification. In this work we tested Sequential Backward Search
(SBS), Sequential Forward Search (SFS) and Sequential Floating Forward Search (SFFS) which
have been found to perform well in this kind of data [9, 10, 30, 11]. However, there was no
difference in results between them, the optimal feature set was the same for all the three. This
could be expected since, unlike other classifiers, decision trees also perform feature selection by
selecting the best attribute to discriminate the classes in each node of the tree. Nevertheless, to be
able to rank features in order of their relevance in classification, we utilized SFS.
4.1. Game Episodes
In this study we tested a game belonging to a popular genre in the game industry, Halo3, which
is a First-Person Shooter (FPS). This game appears as one of the most immersive genre in that it
propels the player at the heart of action through a first-person perspective. To segment the game
hal-00589941, version 1 - 2 May 2011
session in different episodes we used specific events scripted inside the game. Scripted events
are triggered by certain actions from the player, they are mandatory and are typically used by
game designers either to mark transitions between different periods of the game, to inform the
player about a certain state of the world or to draw her attention toward key information. They
manifest through dialogs, visual and/or sound effects (e.g. the rumor of a battleship afar indicating
the proximity of the enemy). By taking advantage of these scripted events we could delimitate
three different combat episodes. Each episode is characterized by distinct geographical cues and
corresponds approximatively to a game level in the traditional acceptation where boundaries of a
level are defined by the completion of a goal (in our case the complete ”cleaning” of a zone).
Before playing the game, participants were told to rest for two minutes during which a movie
was presented, depicting abstract forms evolving smoothly and slowly so as to help the participant
to relax. This period was considered as a baseline for the subsequent physiological recording. The
first combat sequence (Combat1) is characterized by a succession of short skirmishes. The overall
difficulty in this episode is low as it is almost possible to progress without shooting while fellow
soldiers take care of the enemies. This is followed by the second combat session (Combat2) which
is globally a bit more difficult than Combat1. The last combat (Combat3) begins when the player
arrives at an enemy site. This is the most intense episode with enemies outnumbering and taking
cover. In order to contrast combat episodes with more relaxing periods, we distinguished two
additional sessions consisting in transitory periods (REST), triggered by the death of the last enemy
in a level, wherein the player is able to rest, collect ammunitions and explore the environment.
4.2. Participants and Setting
The experiment was conducted at the LUTIN (Laboratoire des Usages en Technologies d’ In-
formation Num´
erique) in the Cit´
e des Sciences et de l’Industrie, Paris, France. Twelve male
participants aged between 18 to 40 were recruited from visitors of the Cit´
e des Sciences et de
l’Industrie to participate in the experiment. No specific expertise in the field of video games was
required, although we selected participants able to manipulate a gamepad and to orient themselves
in a virtual environment. Halo3 was played on a Microsoft Xbox 360 on a 32-inch LCD television.
A camera captured the TV screen. Participants were seated at approximatively one meter from the
screen, they were explicitly told not to move and keep the game pad onto their laps in order to
avoid any muscular artefact in the physiological recording. All participants played to the game
Halo3, at the easiest to difficult level. Participants were allowed to play at their own pace, until
the experimenter told them to quit the game. The game ended when participants reached a specific
point in the game (end of Combat3).
To collect the physiological measures we used the Biopac MP35 acquisition unit and the soft-
ware BSLPro to visualize the data. To record the HR, we measured the electrocardiography (ECG)
through a Einthoven derivation II placing pre-gelled surface electrodes on the ankles and on the
wrist. We recorded GSR using surface electrodes attached with Velcro Straps that were placed
on two fingers of the left hand. The fingers wearing the electrodes remained wedged under the
gamepad. In order to synchronize the video with the physiological data, we used sound markers
emitted at the beginning and at the end of the baseline video which were sent to the acquisition
hal-00589941, version 1 - 2 May 2011
unit. ECG and GSR data were collected at 200 Hz. As noisy ECG data may produce failures in
computing the HR, we inspected the HR data and corrected manually every erroneous samples.
  
Figure 1: Numbers of Red Warnings during Combat1, Combat2 and Combat3 for each participant
As we have already noted, our main goal in this experiment was to automatically discriminate
episodes of the game characterized by high challenge from those characterized by low challenge
based on physiological signals. The three different combat episodes could be characterized in term
of their level of difficulty, based on certain information provided by the player activity. Specifi-
cally, we took advantage of a warning signal triggered each time the player is in danger, i.e. when
his energy bar is fully depleted. This kind of information is a fair indicator of the level of intensity
of an episode, as it reflects the amount of stress placed upon the player in a circumstance of likely
death. As can be seen in Figure 1, there is a global and significant increase of the number of warn-
ing signals in Combat3 when compared to Combat1. However, there is only a slight increase of
warning signals in Combat2 when compared to Combat1. As a consequence, we chose to contrast
between Combat1 and Combat3.
As earlier explained, we calculated 6 features from each physiological signals and utilized these
attribute values as input to the classifiers. Although the output from the fuzzy decision tree was
in continuous values [�1], we deffuzzified the values into 1}for quantitative analysis and to
compare the results with other classifiers. In the first attempt, a classifier was trained by leaving
out one subject’s data (test sample) against the rest of the subjects’ data (training sample) using the
aggregate of the entire episode (3 to 5 minutes). This was to give us a global recognition model
of various game episodes. But in a related way, we considered a second alternative, in which we
subdivided each game episode into 20 segments (approximately 10 seconds length). This was to
hal-00589941, version 1 - 2 May 2011
validate if the model could apply on a continuous scale at various times during the game episodes.
Half (six) of the subjects’ samples was used for training and the other half for testing. In the
analysis, we give the results of both alternatives.
First, we performed machine learning to differentiate between game combats of high challenge
from those of low challenge with reasonable success as shown in Table 1 and Table 2. In dis-
criminating these episodes, the GSR (mean amplitude and the mean of the second difference of
the normalized) signal was the most relevant when compared across the classifiers using both the
aggregate of the entire episode and the 10 seconds segments samples.
Table 1: Low/High Challenge Episodes Pre-labeled Vs Predicted Results (Aggregate of Entire
- Low High Salammbˆ
o FDT Quilan C4.5 DT KNN�k=1)
Low 8 4 66.67% 50.00% 66.67%
High 1 11 91.67% 83.33% 58.33%
Average 79.17% 66.67% 62.50%
Table 2: Low/High Challenge Episodes Pre-labeled Vs Predicted Results (10 Seconds Segments)
- Low High Salammbˆ
o FDT Quilan C4.5 DT KNN�k=10)
Low 202 38 84.17% 85.00 % 80.00%
High 49 191 79.58% 77.08% 86.67%
Average 81.88% 81.04% 83.33%
Secondly, we tried to discriminate between episodes wherein the player is involved in com-
bat and transitory periods (REST). As already noted, the transitory periods (REST) came after
the death of the last enemy in the combat episode and thus index the moments the players feels
relieved to have accomplished a goal. The player is not involved in any shooting while moving
forward to start the next combat episode. During our machine learning process, we managed to
differentiate these episodes (combat and REST) with 68.75% and 70.4% success as shown in Ta-
ble 3 and Table 4 respectively. It turned out that, in addition to GSR (standard deviation) signal,
the HR (the mean of the first and second difference) signal was very relevant in discriminating the
REST/combat categories across the classifiers using both the aggregate of the entire episode and
the 10 seconds segments samples.
Table 3: Combat/REST Episodes Pre-labeled Vs Predicted Results (Aggregate of Entire Episode)
- Combat REST Salammbˆ
o FDT Quilan C4.5 DT KNN�k=1)
Combat 19 5 79.17% 41.70% 70.08 %
REST 10 14 58.33% 54.20% 70.08 %
Average 68.75% 47.95% 70.08 %
Generally, segmentation of episodes seems to have increased the classification rate across all
the classifiers. Although using the aggregate of the entire episode gives us a global classification
model, it drastically reduces the sample size for good machine learning. The performance of
classifiers is also comparable especially when episodes are segmented, if judged by classification
hal-00589941, version 1 - 2 May 2011
Table 4: Combat/REST Episodes Pre-labeled Vs Predicted Results (10 Seconds Segments)
- Combat REST Salammbˆ
o FDT Quilan C4.5 DT KNN�k=10)
Combat 377 103 78.54% 76.0% 77.3 %
REST 181 299 62.29% 66.5% 69.8 %
Average 70.42% 71.25% 73.55 %
into discrete categories. However, fuzzy decision trees will be more applicable as they can be used
on a continuous scale.
In this study we tested a model to automatically recognize specific episodes in an action game.
Our aim was to relate a set of physiological signals features to specific game episodes. By taking
advantage of the game narratives and the corresponding variations in the challenge proposed by
the game through its different levels, we trained fuzzy decision trees to predict the episodes and
generate rules that map the features to these periods of the game.
Although we couldn’t differentiate clearly between combat episodes and transitory episodes
where the player is not involved in a fight, we managed to identify with considerable success
different combat episodes characterized by a variation in the challenge at stake. Our results point
to galvanic skin response as a very relevant measure of the level of arousal of a player since it best
discriminated between very intense episodes and not so intense episodes. Similarly, we found that,
in addition to galvanic skin response, heart rate was also very relevant measure in discriminating
between shooting and the subsequent rest episodes. We anticipate that variations of these signals
could index the moments the players feels relieved when a goal has been accomplished.
However, several issues inhibited us from obtaining optimal results in this experiment. One
issue regards to the players level of expertise. As we couldn’t control properly the level of expertise
of our participants, we may have failed to obtain homogeneous samples. Confronted to the same
level of difficulty, players with sensibly different playing background may react very differently.
For instance a well trained player would react positively to an especially intense episode, as it may
represent a fair challenge to him. Inversely, a beginner would consider that the same episode is
outrageously punishing and thus, frustrating. Therefore, we would hope to obtain an even better
level of recognition by testing people with the same amount of experience, and possibly the same
kind of reaction confronted to an increase in intensity.
Also, the indexation method we used to differentiate the game episodes may have failed to be
the most accurate in differentiating clearly between combat episodes and transitory episodes where
the player is not involved in a fight. It is possible that psychological boundaries of an episode (i.e.
when a period of the game is subjectively felt as starting and ending) only partially coincide with
objective markers such as scripted events. In consequence, it would be useful to consider the
possibility to index events according to psychological markers.
Much is still to be done before getting access to the structure of the player’s emotional pro-
hal-00589941, version 1 - 2 May 2011
cesses. In particular, we need to improve our methods in order to define episodes with respect
to psychological states and their variations. As a next step toward recognizing levels of affective
involvement in a game, we would first need to consider more physiological measures. The present
study did not exhaust all the possible signals available and improvements can be expected from the
consideration of a large amount of physiological features. Additionally, we would need to differ-
entiate physiological signatures not only from a level of intensity we assume they elicit, but from
the players’ direct assessment. We could therefore evaluate episodes based on various subjective
scales (e.g. how frustrated I am, how concentrated I am, how fun is my experience, etc.) so as
to define the episodes on several dimensions. Altogether, the road map for the forthcoming inves-
tigation of affective states in video game will get through a clear definition of the most relevant
dimensions to account for the emotional response we target, as well as a thorough examination
of the best physiological features. This way, we hope for a truly systematic affective recognition
procedure to be incorporated to the games evaluation routines.
[1] Wolf, M and Perron, B. Introduction. In The Video Game Theory Reader. Routledge, 2003.
[2] Nel, F, Damez, M, Labroche, N, and Lesot, M.-J. Automated video games evaluation based
on the template formalism. In International Conference on Information Processing and Man-
agement of Uncertainty in Knownledge-Based Systems, IPMU, 2008.
[3] Grassioulet, Y. A cognitive ergonomics approach to the process of game design and de-
velopment. Master’s thesis, Educational Technologies (STAF), TECFA, Psychology And
Educational Sciences Department, University of Geneva, 2002.
[4] Pagulayan, R. J, Keeker, K, Wixon, D, Romero, R, and Fuller, T. User-centered design
in games. In Handbook for human-computer interaction in interactive systems. Lawrence
Erlbaum Associates, Inc. Mahwah, NJ, 2002.
[5] Picard, R. W. Affective Computing. The MIT Press, Cambridge, Massachusetts, London,
England, 1997.
[6] Fairclough, S. Fundamentals of physiological computing. Interacting With Computers,
21:133–145, 2009.
[7] Cacioppo, J. T, Tassinary, L. G, and Bernston, G. Psychophysiological science: Interdisci-
plinary approaches to classic questions about the mind. In Handbook of Psychophysiology.
Cambridge University Press, 2007.
[8] Scheirer, J, Fernandez, R, Klein, J, and Picard, R. Frustrating the user on purpose: A step
toward building an affective computer. Interacting With Computers, 14:93–118, 2002.
[9] Picard, R, Vyzas, E, and Healey, J. Toward machine emotional intelligence: Analysis of
affective physiological state. IEEE Transactions Pattern Analysis and Machine Intelligence,
23:1175–1191, 2001.
[10] Wagner, J, Kim, J, and Andr´
e, E. From physiological signals to emotions: Implementing and
comparing selected methods for feature extraction and classification. In IEEE International
Conference in Multimedia and Expo, 2005.
[11] Kim, J and Andr ´
e, E. Emotion recognition based on physiological changes in music listening.
IEEE Transactions on Pattern Analysis And Machine Intelligence, 30:2067–2083, 2008.
hal-00589941, version 1 - 2 May 2011
[12] Arroyo-Palacios, J and Romano, D. M. Towards a standardization in the use of physiological
signals for affective recognition systems. In Measuring Behavior, 2008.
[13] Nasoz, F, Alvarez, K, Lisetti, C, and Finkelstein, C. Emotion recognition from physiological
signals for presence technologies. International Journal Of Cognition, Technology And Work,
6, 2003.
[14] Goulev, P. An Investigation into the Use of AffectiveWare in Interactive Computer Applica-
tions. PhD thesis, Imperial College London, 2005.
[15] Bouchon-Meunier, B. Aggregation and Fusion of Imperfect Information. Physica-Verlag,
Spring-Verlag Company, 1998.
[16] Rani, P, Sarkar, N, and Adams, J. Anxiety-based affective communication for implicit human
machine interaction. Advanced Engineering Informatics, 21:323–334, 2007.
[17] Zadeh, L. Fuzzy sets. Information Control, 8:338–358, 1965.
[18] Mandryk, R. L and Atkins, M. A fuzzy physiological approach for continuously model-
ing emotion during interaction with play technologies. International Journal of Human-
Computer Studies, 65:329–347, 2007.
[19] Marsala, C. Fuzzy decision trees to help flexible querying. KYBERNETICA, 36:689–705,
[20] Quinlan, R. C4.5: Programs for Machine Learning. Morgan Kaufmann Publishers, San
Mateo, CA, 1993.
[21] Cover, T and Hart, P. Nearest neighbor pattern classficiation. IEEE Transactions Information
Theory, 13:21–27, 1967.
[22] Lang, P. The emotion probe studies of motivation and attention. American Psychologist,
50(5):372–385, 1995.
[23] Bradley, M, Greenwald, M. K, and Hamm, A. Affective picture processing. In The Structure
of Emotion. Hogrefe and Huber Publishers, Toronto, 1993.
[24] Detenber, B, Simons, R, and Bennett, G. Roll’ em� the effects of picture motion on emotional
responses. Journal of Broadcasting and Electronic Media, 21:112–126, 1998.
[25] Ward, R and Marsden, P. Physiological responses to different web page designs. Interna-
tional Journal of Human-Computer Studies, 59:199–212, 2003.
[26] Winton, W, Putnam, L, and Krauss, R. Facial and autonomic manifestations of the dimen-
sional structure of emotion. Journal of Experimental Social Psychology, 20:195–216, 1984.
[27] Papillo, J. F and Shapiro, D. Principles of psychophysiology: Physical, social, and inferential
elements. In Tassinary, L, editor, The Cardiovascular System. Cambridge University Press,
[28] Neumann, S and Waldstein, S. Similar patterns of cardiovascular response during emotional
activation as a function of affective valence and arousal and gender. Journal of Psychosomatic
Research, 50:245–253, 2001.
[29] Ritz, T and Th¨
ons, M. Airway response of healthy individuals to affective picture series.
International Journal of Psychophysiology, 46:67–75, 2002.
[30] Yang, R and Liu, G. Emotion feature selection from physiological signal based on bpso. In
ISKE-2007 Advances in Intelligent Systems Research, 2007.
hal-00589941, version 1 - 2 May 2011
... To this aim, I use Orero et al.'s data [2] in which subjects had to play a highly immersive first-person shooter game (FPS). 1 First, using the gamer's self-reported concentration enjoyment along with her concentration level, we build a gaming experience classification accounting for its self-rewarding component. In a standard way, the classification ranges from boredom to flow, to anxiety and relaxation. ...
... Video games are proving to be very popular with 2.2 billion casual or regular gamers across the globe in 2017. 2 Many studies [4][5][6][7] have examined so far the reasons why people play video games, which include the desire for competition, challenge, social interactions, entertainment, excitement, and escape [8]. However, the very primary goal of games is enjoyment. ...
... Csikszentmihalyi's work also strongly stresses that for the gamer to experience flow, not only does the game have to stretch the gamer's skills, but the gaming experience must also be self-rewarding. However, the common feature of the studies mentioned so far is that they share a narrow view of game enjoyment, which is mostly based on the pleasure gained from the game as measured through self-reported assessment, 3 GEQ questionnaires [24], or physiological measures [2,18,25,26]. Two drawbacks can also be mentioned here when using questionnaires. ...
Full-text available
Applied to video games, Csikszentmihalyi’s work on flow evidences that a positive gaming experience is intrinsically self-rewarding and primarily determined by the skill/challenge balance. A multi-layered measure of enjoyment is built to take these components into account. Gamers were asked to report the concentration-enjoyment they experienced during a first-person shooter game, and to better assess the gap between skill and challenge, the challenge enjoyment was also rated. Along with concentration level, concentration enjoyment is used to build a gaming experience typology that accounts for the self-rewarding component. An enjoyment-based challenge mapping is also drawn up, crossing challenge enjoyment and challenge level. The results show that this integrative enjoyment measure strengthens the causal link between challenge and gaming experience. Most importantly, the findings suggest that challenge or concentration-based enjoyment measures outweigh the standard concentration and difficulty measures as they are more likely to ensure a pleasant and positive experience (flow or relaxation) for the gamers. Indeed, regardless of the reported level of challenge, a gamer is more likely to have a positive experience when challenged at a level she perceives as pleasant. This article emphasizes the importance for game publishers of gathering enjoyment-based concentration and challenge assessments to ensure a positive gaming experience and gamers’ commitment.
... The imprecise and subjective nature of the problem is such that no deterministic algorithms are known to solve it exactly, and it is even hard to find heuristics that produces good results. Some works used Fuzzy Logic [49,64,78,81] to cope with the lack of precision in data. Other approaches have adopted linear or partial nonlinear methods (such Support Vector Machine, K-Nearest Neighbor and Principal Component Analysis) to solve the multi-label emotion classification problem [12,17,40], but these methods did not work properly on the nonlinear psychophysiological dataset and they did not consider all important aspects of features interactions. ...
... The authors in [34,50,64] explored the correlation between physiological changes (using GSR, ECG and EMG) in a game under different settings, and they discovered some important points, such as players who experience different emotional states while playing the same game with different settings. ...
Full-text available
Games User Research (GUR) is a relevant field of research that exploits knowledge on human-computer interaction, game design, and psychology, with a focus on improving the player experience (PX) and the quality of the game. Games form an environment of rich interactions which can lead to a variety of experiences for the player. Researchers employ new ways to assess PX over time with some degree of precision, while avoiding the interruption of gameplay. A possible way of attaining great PX evaluation can be using psychophysiological data. It is a source that can provide relevant details about the emotional states and a potential information in the context of GUR. This paper presents a process for classifying PX in games based on psychophysiological data acquired from the user during the gameplay. Biosensors and a webcam were employed to capture three signals: Galvanic Skin Response (GSR), Blood Volume Pulse (BVP) and Facial Expression. Our artificial neural network was trained with a dataset formed by psychophysiological data and human-annotated emotional expressions derived from assessment and judgment of players’ face and behavior with the help of an emotion annotation tool. Four classes of emotions, derived from the most significant game events, are considered for classification: Anger, Calm, Happiness and Sadness. The experimental results indicate that the proposed method leads to good human emotion recognition, and an accuracy score of 64%. The automatic assessment of player experience was compared with a traditional evaluation based on self-report, corroborating the effectiveness of the method.
... The studies in [33], [34], [35] investigated the correlation between physiological changes and a same game under different settings, and presented some interesting findings using Galvanic Skin Response, electroencephalogram and electromyography, confirming that players feel differently while playing a same game with different settings. ...
... Kansei measurement is classified into physiological measures and psychological measures. " (e.g., for physiological measures [Orero, Levillain, Damez-Fontaine, Rifqi, & Meunier, 2010], for psychological ones [Lee, Harada, & Stappers, 2002], and for psychophysiological ones [Lévy, Yamanaka, & Tomico, 2011]). KE will be explored further in the text. ...
Full-text available
For over three decades, kansei engineering has expanded greatly and has become a significant discipline both in the industrial and the academic worlds. In this paper, I present the current situation of kansei engineering, and plead for the emancipation of other disciplines, as part of kansei research as well. By reconstructing the historical path of kansei research and exploring the variety of disciplines within kansei research, I point out the opportunities for kansei design to emerge. Whereas kansei engineering and kansei science have found their roots in scientifically established approaches (respectively engineering and brain science), kansei design intends to return to earlier Japanese philosophical or cultural works to rediscover the essence of kansei, and to use them as inspirational means for design. This new discipline certainly needs to be elaborated further. Therefore, this paper aims to contribute to the elaboration of a more expansive point-of-view in design research regarding the relationship between human beings and their immediate environment.
... The same method was applied to the RR. In respect to physiological feature extraction, for each signal, we chose to calculate the features shown in Table I that we belief are very relevant based on results from past research and our earlier preliminary work [26]. ...
Conference Paper
Full-text available
In the recent years video games have enjoyed a dramatic increase in popularity, the growing market being echoed by a genuine interest in the academic field. With this flourishing technological and theoretical efforts, there is need to develop new evaluative methodologies for acknowledging the various aspects of the player's subjective experience, and especially the emotional aspect. In this study, we addressed the possibility of developing a model for assessing the player's enjoyment (amusement) with respect to challenge in an action game. Our aim was to explore the viability of a generic model for assessing emotional experience during gameplay from physiological signals. In particular, we propose an approach to characterize the player's subjective experience in different psychological levels of enjoyment from physiological signals using fuzzy decision trees.
We propose a real-time system that continuously recognizes emotions from body movements. The combined low-level 3D postural features and high-level kinematic and geometrical features are fed to a Random Forests classifier through summarization (statistical values) or aggregation (bag of features). In order to improve the generalization capability and the robustness of the system, a novel semisupervised adaptive algorithm is built on top of the conventional Random Forests classifier. The MoCap UCLIC affective gesture database (labeled with four emotions) was used to train the Random Forests classifier, which led to an overall recognition rate of 78% using a 10-fold cross-validation. Subsequently, the trained classifier was used in a stream-based semisupervised Adaptive Random Forests method for continuous unlabeled Kinect data classification. The very low update cost of our adaptive classifier makes it highly suitable for data stream applications. Tests performed on the publicly available emotion datasets (body gestures and facial expressions) indicate that our new classifier outperforms existing algorithms for data streams in terms of accuracy and computational costs.
Conference Paper
Physiology-based emotionally intelligent paradigms provide an opportunity to enhance human computer interactions by continuously evoking and adapting to the user experiences in real-time. However, there are unresolved questions on how to model real-time emotionally intelligent applications through mapping of physiological patterns to users' affective states. In this study, we consider an approach for design of fuzzy affective agent based on the concept of typicality. We propose the use of typicality degrees of physiological patterns to construct the fuzzy rules representing the continuous transitions of user's affective states. The approach was tested on experimental data in which physiological measures were recorded on players involved in an action game to characterize various gaming experiences. We show that, in addition to exploitation of the results to characterize users' affective states through typicality degrees, this approach is a systematic way to automatically define fuzzy rules from experimental data for an affective agent to be used in real-time continuous assessment of user's affective states.
Full-text available
A reliable condition monitoring is needed to predict faults. Pattern recognition technologies are often used for finding patterns in complex systems. Condition monitoring can also benefit from pattern recognition. Many pattern recognition technologies, however, only output the classification of the data sample but do not output any information about classes that are also very similar to the input vector. This paper presents a concept for pattern recognition that output similarity values for decision trees. The concept can be used on top of any normal decision tree algorithms and is independent of the learning algorithm. Performed experiments showed that the concept is reliable and it also works with decision tree forests to increase the classification accuracy.
Subjective information is very natural for human beings. It is an issue at the crossroad of cognition, semiotics, linguistics, and psycho-physiology. Its management requires dedicated methods, among which we point out the usefulness of fuzzy and possibilistic approaches and related methods, such as evidence theory. We distinguish three aspects of subjectivity: the first deals with perception and sensory information, including the elicitation of quality assessment and the establishment of a link between physical and perceived properties; the second is related to emotions, their fuzzy nature, and their identification; and the last aspect stems from natural language and takes into account information quality and reliability of information.
Full-text available
In the last decade, academic studies on games were mainly focusing on gamers, usually from sociological, educational and/or psychological perspectives. Here, in this qualitative study, the main features for a foundational framework of game design and development process will be described, using cognitive ergonomics methods such as semi-structured interviews, critical incidents gathering, and free mind mapping. Two European-leading games development studios were asked to be involved in this exploratory study. One lead game designer, one writer, two tech leads, one external lead artist, and one CEO/internal producer were interviewed about their work, during a 2-hours session. Analysis of these qualitative data helped us better understand how practices and representations can lead to a new vision of video games’ production, while integrating user-centered design. In brief, results show that the iterative process of game design and development can be formalized from a cognitive ergonomics point of view. Besides, “real” organizational, management, design, and technical critical incidents are found during the whole production process. Useful recommendations on game production and task environment/organization will then be suggested. Ultimately, it could help both the product to be improved in quality and the game design and development process to be more considered from a player/customer-centered perspective.
Full-text available
In emotion recognition, many irrelevant and redundant features will affect recognition results, so feature selection is necessary. Aimed at emotion physiological signal feature selection, this paper proposed with improved discrete binary particle swarm optimization(BPSO) to increase the correct classification rate of emotion state. When recognizing four emotional states with nearest classifier by four physiological signals, the whole correct recognition rate is up to 85%. Experimental results demonstrate that the BPSO is an effective way to emotion physiological signals feature selection.
Full-text available
Provides an introduction to the contents of this handbook as well as a brief history of psychophysiology and the current state of the field. This chapter aims to define psychophysiology, briefly reviews major historical events in the evolution of psychophysiological inference, outlines a taxonomy of logical relationships between psychological constructs and physiological events, and specifies a scheme for strong inference within each of the specified classes of psychophysiological relationships. (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Full-text available
This paper proposes a new approach for video game evaluation, based on a hierarchical model of quality cri- teria, in the framework of the tem- plate formalism: the latter oers a principled aggregation methodology to combine elementary and directly evaluable criteria into more complex properties until the global quality of a game can be assessed. We propose a structured organisation of existing video game quality crite- ria, and its implementation in the template formalism. The first ex- periments performed with real data show this model constitutes a rele- vant and powerful tool that provides interpretable assessment results con- sistent with expert evaluations.
Conference Paper
Full-text available
The implementation of physiological signals, as an approach for emotion recognition in computer systems, is not a straight forward task. This paper discusses five main areas that lack of standards and guided principles, which have led Human- Computer Interaction (HCI) researchers to take critical decisions about (i) models, (ii) stimulus, (iii) measures, (iv) features and (v) algorithms with some degree of uncertainty about their results. Methodology standardization would allow comparison of results, reusability of findings and easier integration of the various affective recognition systems created. The background theory is given for each of the five areas and the related work from psychology is briefly reviewed. A comparison table of the HCI common approaches of the five discussed areas is presented, and finally some considerations to take the best decisions are discussed. The aim of this paper is to provide directions on which the future research efforts for affective recognition in HCI should be focused on.
Aggregation Operators for Fusion under Fuzziness: S. Ovchinnikov: On robust aggregation procedures R. Mesiar, M. Komornikova: Triangular norm-based aggregation of evidence under fuzziness J. Fodor, T. Calvo: Aggregation functions defined by t-norms and t-conorms.- Aggregation in Decision Making and Control: M. Grabisch: Fuzzy integral as a flexible and interpretable tool of aggregation A. Kelman, R.R. Yager: Using priorities in aggregation connectives V. Cutello, J. Montero: Aggregation operators for fuzzy rationality measures M.T. Lamata: Aggregation in decision making with belief structures J. Kacprzyk: Multistage fuzzy control with a soft aggregation of stage scores.- Fusion of Complementary Information: S. Benferhat, D. Dubois, H. Prade: From semantic to syntactic approaches to information combination in possibilistic logic S. Moral, J. del Sagrado: Aggregation of imprecise probabilities I. Bloch, H. Maitre: Fusion of image information under imprecision G. Mauris, E. Benoit, L. Foulloy: Fuzzy linguistic methods for the aggregation of complementary sensor information A. Appriou: Uncertain data aggregation in classification and tracking processes M. Sato, Y. Sato: A generalized fuzzy clustering model based on aggregation operators and its applications.