Conference PaperPDF Available

Affective Design Patterns in Computer Games. Scrollrunner Case Study

Authors:

Abstract and Figures

A new dimension in human-computer interaction, that can be used to improve the user experience, is the emotional state of the user. This is the domain of the affective computing paradigm. In our work we focus on the applications of affective techniques in the area of the design of video games. We assume that a change in the affective condition of a player can be detected based of the monitoring of physiological signals following the James-Lange theory of emotions. We propose the use of game design patterns introduced by Bjork and Holopainen to build games. We identify a set of patterns that can be considered affective. Then we demonstrate how these patterns can be used in a design of a scroll-runner game. We address the problem of the calibration of measurements in order to reflect responses of individual users. We also provide results of practical experiments to verify our approach.
Content may be subject to copyright.
Affective Design Patterns in Computer Games.
Scrollrunner Case Study
Grzegorz J. Nalepa
Jagiellonian University,
ul. Goł˛ebia 24, 31-007 Kraków, Poland
AGH University of Science and Technology,
al. Mickiewicza 30, 30-059 Kraków, Poland
e-mail: gjn@agh.edu.pl
Barbara Gi˙
zycka
Jagiellonian University,
ul. Goł˛ebia 24, 31-007 Kraków, Poland
e-mail: barbarag.cka@gmail.com
Krzysztof Kutt
AGH University of Science and Technology,
al. Mickiewicza 30, 30-059 Kraków, Poland
e-mail: kkutt@agh.edu.pl
Jan K. Argasi´
nski
Jagiellonian University,
ul. Goł˛ebia 24, 31-007 Kraków, Poland
e-mail: jan.argasinski@uj.edu.pl
Abstract—The emotional state of the user is a new dimension
in human-computer interaction, that can be used to improve the
user experience. This is the domain of affective computing. In
our work we focus on the applications of affective techniques
in the design of video games. We assume that a change in the
affective condition of a player can be detected based on the
monitoring of physiological signals following the James-Lange
theory of emotions. We propose the use of game design patterns
introduced by Björk and Holopainen to build games. We identify
a set of patterns that can be considered affective. Then we
demonstrate how these patterns can be used in a design of a
scroll-runner game. We address the problem of the calibration
of measurements in order to reflect responses of individual users.
We also provide results of practical experiments to verify our
approach.
I. INT ROD UC TI ON
IN ORDER to improve usability and provide superior user
experience we seek new sources of information that can
be used in human-computer interaction. A new dimension
of such an information is related to the emotional state of
the user. The affective condition can be determined based on
number of data, e.g. heart rate, expression of face, etc. This is
the domain of the Affective Computing (AfC) paradigm, that
was originally introduced 20 years ago [1]. We believe, that
information regarding the affective condition of a user, can
lead to a better understanding of interactions of users with
machines, and interfaces.
In our work we focus on the applications of AfC techniques
in the area of the design of video games. We assert that these
techniques can improve the gaming experience of the player. In
fact, the affective component can, and should be incorporated
into the design process of video games. In order to do so, we
propose the use of game design patterns as introduced in [2].
There are number of models of human emotions considered
in psychology and adopted in AfC. In our work, we assume
that a change in the affective condition of a player can be de-
tected and identified based on the monitoring of physiological
signals, following the James-Lange theory of emotions [3].
The new generation of miniaturized computer sensors can be
used to measure such signals and deliver sensor data in real
time to a gaming system. In fact, our priority is to focus on
sensors that can be worn by a user, e.g. wearable wristbands.
While such devices do not offer high quality of measurement,
they are non intrusive and can be easily used by a player during
a video game. Furthermore, they are not very expensive which
makes them accessible to many players.
The original contributions of this paper are as follows:
First we identify a set of game design patterns that can be
considered affective, which means that they elicit an emotional
response from the player. Then we demonstrate how these
patterns can be used in a design of a simple yet illustrative
game. Next, we address the problem of the proper calibration
of measurements in order to reflect differences in responses
of individual users. Finally, we provide results of practical
experiments supporting our claims. Our work follows our early
experiments with virtual reality discussed in [4].
The rest of the paper is composed as follows. In Section II
we briefly introduce the affective computing paradigm, and
selected methods and models we use in our work. In Sec-
tion III we discuss the concept of patterns in game design.
We introduce our motivation in Section IV. Then in Section V
we demonstrate a practical design example of a scroll-runner
game with affective components. To verify our assertions,
we planned and conducted practical experiments described
in Section VI. We briefly discuss other works related to our
research in Section VII. The paper ends with summary and
plans for future works in Section VIII.
II. AFF EC TI VE CO MP UT ING
Affective computing is a paradigm originally proposed in
1997 by Rosalind Picard from MIT Media Lab [1]. It uses
results of biomedical engineering, psychology, and artificial
intelligence. It aims at allowing computer systems to detect,
Communication papers of the Federated Conference on
Computer Science and Information Systems, pp. 345–352
DOI: 10.15439/2017F192
ISSN 2300-5963 ACSIS, Vol. 13
c
2017, PTI 345
use, and express emotions [5]. It is a constructive and practical
approach oriented mainly at improving human-like decision
support as well as human-computer interaction. AfC puts
interest in design and description of systems that are able
to collect, interpret, process (ultimately – simulate) emotional
states (affects). Assuming that emotions are both physical and
cognitive, they can be studied interdisciplinary by computer
science, biomedical engineering and psychology. For affective
computing there are two crucial elements to be considered:
modes of data collection and ways of interpreting them in
correlation with affective states corresponding to emotions.
The first is carried out by selection of methods for de-
tecting information about emotions, using sensors capturing
data about human physical states and behaviors. Today most
often processed information is about: speech, body gestures
and poses, facial expressions, physiological monitoring (blood
pressure, blood volume pulse, galvanic skin response). In our
research we focus on the last source of signals, and we assume
a range of wearable physiological sensors.
The second crucial element of affective computing paradigm
relies on application of selected algorithms on acquired data
to develop models of interpretation for affective states. State
of the art methodologies assume the use of the full range of
available methods of data classification and interpretation [5].
Computational models of emotions derive from our interpre-
tation of psychological states and cognitive processes and are
in essence a way to describe relation between those two phe-
nomena. Affective computing makes its own use of models by
applying them to bio-physiological data obtained from sensors
in such a way that they can be used in specific software. Defin-
ing ”emotion” is a challenge. Modern theories of emotions
have their origin in 19th century. William James theorized
about affects in terms of reactions to stimuli. He was precursor
to appraisal theory which is among most popular in the
community of computational emotional modeling [6], [7], [8].
One of the most popular appraisal theories is OOC [9] which
categorizes emotion on basis of appraisal of pleasure/displea-
sure (valence) and intensity (arousal). These are quantifiable
values that can be measured and processed ascribing different
kinds of emotions (i.e. positive self-attribution of intensive
value might be interpret as "pride"). Another important set
of theories of emotions is less about discrete states and more
about relational affect states tracked on a number of continuous
dimensions [10]. Dimensional models similarly to appraisal –
map emotion-evoking impulses and states triggered. Popular
PAD theory [11] considers Pleasure, Arousal and Dominance
dimensions. Different theories of emotion lead to various
models which in turn lead to variety of emotion-oriented
systems. Good example is EMA [10] – a system implemented
on the Soar cognitive architecture which explains dynamic
affects through a sequence of triggers. Another system is
WASABI [12] believed to be one of most general models of
emotion (mainly for simulation).
Nowadays most popular GUIs are based on windows, menus
and forms. Natural User Interfaces (NUI) are meant to reduce
the barriers for users even further while empowering them to
perform more complicated tasks smoothly. In Adaptive User
Interfaces (AUI) paradigm the assumption is that interfaces
are subjects to modification as a result of interaction with the
user. Adaptation can take place on many levels. Most often it
means dynamically scalable quantity of displayed information
depending on users demands and capabilities.
User interfaces based on emotion processing are in a way
fusion of NUI and AUI. They are more ”natural” than classic
metaphor-based UIs because for humans ”emotive” means
natural. First, we have to keep in mind that in today’s practice
of UI design "emotive interface" means above all the use of
techniques that cause emotions in receiver by using classical
means of expression and communication. For us much more
interesting will be the inclusion of affective processes at
interfaces at the level of affective feedback for user.
In our work, we assume it is possible to identify a high-
level emotional state from low-level sensory data. We believe,
that Jesse Prinz’s Embodied Appraisal Theory [13] may be
useful to reach this goal. According to it, emotions are build
up by two parts: (a) form – bodily changes perception (as
in the classical James-Lange theory [3]) and (b) content
relationship between agent and environment. As an example,
faster heart rate (form) and perception of a loud sudden noise
(content) build up fear.
In the next section we will discuss the design patterns in
video games. Extending them with affective information will
lead to new design methods for computer games.
III. GAM E DES IG N PATTER NS
One of the most important elements when it comes to game
design are mechanics. They are basic building elements of the
whole game structure. If we define the game as "the voluntary
attempt to overcome unnecessary obstacles" [14], there are two
main factors required to establish "gaming situation". These
are the mental attitude of a player entering the play ("lusory
attitude") and rules prohibiting the use of more efficient for
less effectual means ("lusory means"). The role of the designer
of the game is to create the frames (game design) that make
this kind of activity (play) possible and pleasurable. It takes
place mostly through creating constitutive rules (mechanics)
which activated by the player (dynamics) result in his or her
affects (user’s experience). Another basis for the engagement
are storytelling and interface design, leading to the interactivity
of play that causes player’s immersion (see: [15]), but these are
less important from our point of view. In conclusion mechanics
are main factor constructing gameplay resulting in player’s
affective experience.
We can express rules of play as the verbs describing what
player can do inside designed game system: he can "jump" the
avatar, "hit" the enemy, "collect" the coins, "solve" the puzzles.
Usually, the player is aware of most of the rules. Mechanics
are very similar "verbal" constructs, but widely they include
invisible principles of the game system, those implemented
deeper in the software, frequently not displayed to the user
interface. For example when we play an arcade game we are
aware of the rules that tell us to shoot the enemies, but we
346 COMMUNICATION PAPERS OF THE FEDCSIS. PRAGUE, 2017
usually do not know about details of the algorithms governing
enemy A.I. Both shooting and A.I.’s behaviors are mechanics.
Particular game mechanics are a repeating phenomena that
strongly depend on the genre. An interesting fact is that the
dynamics (groups of mechanics activated by player) tend to
have emergent properties – it means that different gameplays
cannot be directly derived from basic mechanics. This case
seems to be the main source of diversity in the gaming domain.
Repetitive nature of the mechanics is also an origin of the
idea of game design patterns. There are few formulations
of the concept of modeling and aligning basic elements of
lusory structures. One of the most interesting is that pro-
posed by S. Björk and J. Holopainen [2]. It is an approach
to creating a language for talking about gameplay. As the
authors state: "Essential to the discussion of gameplay are the
different aspects of gameplay that can exist. Understanding
these aspects is important if one wants to go into details
about a specific game one is playing [...] there is a lack of
terminology associated with the elements of gameplay. We
offer a solution to this [...]" [2]. In Björk’s and Holopainen’s
framework besides the description of the structural elements
of overall gaming situation, we find a semi-formal exposition
of particular instances of gameplay.
For example components such as boundary (rules, modes,
goals of play), temporal (actions, events, closures), holistic
(game instance, session, play session), structural (interface,
game elements, players, facilitator) are described in details.
Game design patterns rely, as authors state: "on general
descriptions of particular areas of gameplay without using
quantitative measures" [2]. That means characterizations are
simple definitions with detailed descriptions of relationships
to other patterns. There are three basic types of relations:
instantiation (patterns tend to be present together with others,
some naturally imply the presence of others), modulation
(patterns change or are changed by the presence of others),
conflict (patterns render the presence of others impossible).
Basic design pattern template has the following form: Name
(arbitrary); Core definition; General Description; Clues for
Using the Pattern; Consequences; Relations; References. In [2]
authors identified and described in detail over 290 basic
patterns. The Game Design Patterns framework is specially
designed to facilitate the work of designers and analysts.
They allow for easy diagnose of potential problems and
identification of nodes where emergent properties can occur.
One of the goals of our proposal is to distinguish an arbitrary
set of design patterns which, we believe, can cause affective
reaction of the player. Our intention is also to test if embodied
dynamics resulting from this emotion-invoking mechanics can
be observed on a physiological level and included in the main
game loop. In the case of success, such operation should
allow for creating more complex feedback loops in order of
suppressing or amplifying evoked emotions.
IV. MOTIVATION
In our work we aim at demonstrating, how methods of
affective computing can improve design of video games. We
base our work on the game design patterns [2]. Specifically, we
identify certain patterns that in our opinion can be considered
”affective”. We expect an emotional response of a player
of a video game that contains these patterns. We believe
that a thoughtful application of these patterns can lead to
more immersive games and improve the gaming experience.
The motivation of our work is to provide a method for
identification, personalization, and application of the affective
design patterns in video games.
Our objective is to provide integration of a sensory frame-
work with a gaming environment. The framework will use
wearable physiological hardware sensors for detection of user
affects. Ultimately we aim at extending the game design
process by the use of affective patterns, and introduce an
affective loop in the gameplay. Based on our experience
with various hardware, we selected two most promising ones.
Empatica E4 is an advanced sensory wristband based on the
technologies previously developed in the Affective Computing
division of MIT Media Lab. Blood volume pulse and gal-
vanic skin response sensors, as well as infrared thermopile
and accelerometer are on board [16]. Microsoft Band 2 was
developed mainly for tracking fitness goals. Equipped with
optical heart rate, skin temperature and galvanic skin response
(GSR) sensors as well as accelerometer, available through well
documented Software Development Kit.
To verify our hypothesis we provide a two phase exper-
iment. We assume, that the emotional responses of a player
can be detected by the analysis of his physiological responses,
such as heart rate. We expect there are individual differences
of the responses of different players. To address this challenge
we provide an initial calibration phase. During it, the player
is exposed to a series of pictures evoking emotional response.
These pictures are selected from the NAPS [17] data base1.
The second phase of the experiment uses a simple video
game. In this game a series of design patterns is identified.
We consider two variants of the game. The first basic variant
does not include the patterns that we consider affective. In the
affective variant we augment the game with design patterns
resulting in stronger emotional responses of the player.
In the next section we discuss a design of a video game to
be used in our experiments, with the use of design patterns.
V. PRAC TI CA L DES IG N OF A N AFFE CT IV E GAM E
In order to implement our ideas, we started with the design
of a simple video game. In the game some basic design
patterns selected from Björk’s and Holopainen’s collection can
be observed. The selected design patterns, both affective and
non-affective ones, that were implemented in the game design
are described in part V-B.
A. Bridge Scroll-runner
For the purposes of the study, a simple video game was
designed. As video games are already well known as a mul-
1NAPS stands for Nencki Affective Picture System (http://lobi.nencki.gov.
pl/research/8) and is a set of affective images. The dataset consists of
standardized images, as well as normative ratings of valence and arousal.
It is freely available for noncommercial use by request.
GRZEGORZ J. NALEPA ET AL.: AFFECTIVE DESIGN PATTERNS IN COMPUTER GAMES. SCROLLRUNNER CASE STUDY 347
timodal, interactive tool for providing entertainment [18], the
game design was inspired by platformer games. In this sub-
genre of action games, the player’s goal is to control the
character in order to traverse through the game world and
complete subsequent levels. The character usually moves by
running and jumping on the platforms placed along the way,
hence the name ”platformer” game. The player is challenged
by various obstacles that are placed in the game world in order
to prevent him from completing the game, such as holes, gaps,
enemies, etc. Some popular examples of platform games are
Donkey Kong (1981) or Super Mario Bros (1985).
The reason we chose a platformer game and not some other
genre is that platform games are relatively easy to design and
to control all their parameters, which includes applying and
subtracting design patterns. Some patterns, such as collectibles
in case of platform games, are already an inherent feature of
the genre [19]. What is more, the game sessions can be short,
but nonetheless entertaining for the player.
The concept of the game that was designed for the purpose
of this study refers to traditional English nursery rhyme,
„London Bridge is Falling Down”. The player’s avatar (the
character that the player controls in the game), is an English
gentleman that needs to rush past the bridge that is falling
apart. To control the avatar, the player uses left and right
arrow keys on keyboard to navigate (the character proceeds
from left to right), and the spacebar to jump. Therefore, the
goal of the game is to complete the level within the specific
limited time, and completing all of the five designed levels
results in completing the game. In the basic variant of the
game, the player completes the level by collecting required
number of points, which increases per level (1000 points for
the first level, 2000 points for the second level, etc.). The
player collects the points, thereby increasing his score, by
picking up diamonds that are placed along the way through
the bridge. Each diamond increases the player’s score by 10.
Current score is showed in the top left corner of the screen.
To make the game more challenging, each subsequent level
has its time limit narrowed (90 seconds the first level, 75
seconds for the second level, 60 seconds for the third, etc.).
The remaining time is showed in the top center part of the
screen. If the player fails to collect the objective of collecting
the number of points required for the certain level, the game
is over – the bridge falls down, and the player has to start the
game from the first level. Besides the diamonds that are worth
10 points each, the player can pick up the little clocks that
grant additional 5 seconds to the remaining time. To enable the
new player to familiarize with the game mechanics, a tutorial
level is provided at the beginning of each game session. The
tutorial level serves as a teaching ground for the player to
learn the controls and objectives of the game. The player is
given 5 minutes, within which he explores the level and the
game elements. The tutorial level is completed when the score
reaches 1000 points. The tutorial level appears each time the
game is started (after pressing Enter key), but it can be skipped
by pressing [P] on the keyboard when it is entered.
In the affective variant of the game, the gameplay is en-
hanced by several elements that instantiate some game design
patterns, which will be specified in the following section.
Firstly, the score and time indicators are replaced by narrative
description („bad”, „nice”, „good”, and „excellent”) and the
remaining time is represented by a circular indicator. Secondly,
once every few seconds (ranging from 5 to 15), an event
will occur that will decrease the current score (a brick falling
from the top of the screen) or the remaining time (a crow
approaching the character from the right part of the screen).
The game was designed using Game Maker Studio version
1.4. software (YoYo Games). All of the game sprites and
materials (background, music) were either created by us (some
sprites, using GraphicsGale Free Edition) or acquired from
free online resources (e.g. http://freepik.com). In Figure 1
an example of design session of the game is shown. Actual
gameplay of the affective variant of the game is presented in
Figure 2.
B. Identification of Patterns
In the basic variant of the game that is used in this study,
a subset of game design patterns from those proposed by [2]
was identified by us. These are:
1) Alarms: the notifications that appear as a red text in
the top center part of the screen (below the remaining
time indicator) that inform the player that half of the
remaining time has passed.
2) Avatar: the character that is controlled by the player.
3) Collecting: the objectives of the game are reached by
collecting the diamond that increase the player’s score.
4) Deadly Traps: events that result in game over. In case
of this game it is the falling off the platform.
5) Dexterity Based Actions, Evade and Maneuvering: these
patterns refer to the mode of play that poses some
challenge for the player’s dexterity and eye-hand coor-
dination (moving and jumping to collect the diamonds,
evading falling bricks and incoming enemies).
6) Levels: spatial structures within the game world. The
player traverses through the level in order to complete
the objective of the game.
7) Movement: pattern refers to the action of moving within
the game world.
8) Pick-Ups: items (clocks) that the player may optionally
gather while playing, which grant him additional time
to complete the level.
9) Rhythm-Based Actions: the collectibles in the game are
placed in a manner (clusters) that resembles the rhythm
of the tune played in the background.
10) Score: the numerical representation of the player’s
progress in the game.
11) Single-Player Games: this pattern refers to the general
mode where there is only one player in a game instance.
12) Time Limits: some games, such as "London Bridge" that
is used in this study, rely on the fact that the time within
which the player can complete some action or reach
a certain state in the game is somehow restricted.
348 COMMUNICATION PAPERS OF THE FEDCSIS. PRAGUE, 2017
Fig. 1. Design of the scroll-runner game in the Game Makers Studio
Fig. 2. Gameplay of the affective variant of the game. One can see the
descriptive score („nice”) and the crow approaching the character
C. Affective Patterns
In the affective variant of the game, some additional patterns
are implemented:
1) Emotional Immersion: a higher level pattern that de-
scribes the player state that is hopefully reached when
he becomes emotionally engaged in the gameplay.
2) Enemies and Obstacles: considered together, are ele-
ments that hinder the player trying to complete the game
objective. Bricks and crows that consume score points
and remaining time respectively.
3) Imperfect Information: occurs when some aspect of in-
formation about the game state is hidden from the player,
for example – the exact score and time remaining.
In this study, player’s affective responses – reflected by
changes in his arousal, especially in heart rate and galvanic
skin response – are anticipated in both variants of the game.
However, we hypothesize that the responses in the affective
variant of the game will be stronger. We predict that ex-
ceptionally strong responses will appear when "Enemies and
Obstacles" pattern occurs.
VI. EX PE RI ME NT S AN D EVALUATI ON
A. Outline of the Calibration Phase
Valence and arousal are two of several dimensions often
used to characterize emotional experience (for overview of
see [20]). Especially, valence differentiates states of pleasure
and displeasure, and arousal contrasts states of low activa-
tion/relaxation and excitation [17], [21]. These dimensions
are revealed in Autonomic Nervous System (ANS) activity,
the part of nervous system responsible for unconscious au-
tonomous functions like respiration and reflex actions. Re-
search indicates that they can be measured by the use of ANS
measures, inter alia, by the use of Heart Rate (HR) and Skin
Conductance/Galvanic Skin Response (GSR) measures (for
meta-analysis see [22]). Based on this research, the calibration
phase was prepared. During it subjects are exposed to affective
pictures from NAPS dataset. At the same time current levels
of HR and GSR are collected by the wristband. Participants
are also asked to evaluate the perceived arousal of each of the
pictures. The goal of this phase is to combine physiological
data with pictures’ valence-arousal scores in order to prepare
HR and GSR patterns as functions of them. These patterns
will be used in the Gaming Phase as a reference to evaluate
if affective design patters work as intended.
Experimental procedure was designed in the PsychoPy-
Builder as shown in Figure 3. It was then detailed using the
PsychoPy 2 (v 1.84.2) environment and executed on notebook
with four cores processor 2.50GHz, 4GB RAM working under
GRZEGORZ J. NALEPA ET AL.: AFFECTIVE DESIGN PATTERNS IN COMPUTER GAMES. SCROLLRUNNER CASE STUDY 349
the control of Windows 7 Professional OS2. Stimuli were
presented on 15,6” notebook screen with 1366x768 resolution.
Physiological data was collected by MS Band 2 and Empatica
E4 bands. Bands were paired over Bluetooth with Samsung
Galaxy J3 smartphone, on which custom application for data
acquisition, developed by the authors, was installed. Data
from smartphone and notebook is synchronized using the Lab
Stream Layer3, a protocol for time-synced data transmission
over local network, developed at the Swartz Center for Com-
putational Neuroscience, University of San Diego.
Affective stimuli was a subset of Nencki Affective Picture
System. Pictures in this set have assigned valence and arousal
scores on 7 point scale [1-7] [17]. It was arbitrarily divided
into three same-length intervals: low [1,3), neutral [3,5] and
high (5,7]. Then, based on these intervals, group of pictures
were selected in order to cover the groups of interest: (a)
neutral valence and neutral arousal – 10 pictures (+ 6 pictures
for training session), (b) low valence and high arousal – 15
pictures, (c) high valence and low arousal – 15 pictures, (d)
high valence and high arousal – 15 pictures. NAPS has few
low valence and low arousal pictures, therefore this group was
not represented in the selected subset.
The experiment is done individually and takes about 22
minutes. At the beginning, subject seats comfortably in front
of the notebook. The wrist band is then placed on the less
used hand, to minimize muscle artifacts associated with user
interaction with procedure. PsychoPy procedure starts with
instruction and training session with 6 neutral images. Each
stimuli is presented for 3 seconds, then it disappears and
subject has 5 seconds to evaluate the arousal on 7-levels scale
[1,7]. After that another image appears without any pause.
Training session ends with time for questions to experimenter.
Then main session starts with 30 seconds of blank screen to
get baseline HR and GSR recordings during inactivity. After
that subjects is exposed to 3 series of 18 pictures. Series are
separated by the 30 seconds breaks.
B. Outline of the Gaming Phase
After the subject completes the task in the calibration phase,
the experimenter immediately runs the „London Bridge” game
on the same machine for the subject. The subject still has the
wristband on while the synchronization between the smart-
phone and the notebook is maintained. Before the Gaming
Phase, the subject is randomly ascribed either to control group,
where subjects play the basic game variant, or experimental
group, where subjects play the affective game variant. After
pressing Enter key on the notebook to start the game, a 30
seconds of blank screen is presented to get the baseline HR and
GSR recordings for the beginning of the Gaming Phase. After
30 seconds, tutorial level appears and the subject is allowed to
freely explore the game mechanics, and then to play the game
for the desired time, but not exceeding 20 minutes.
2PsychoPy (http://psychopy.org) is a standard software framework in Python
to support a wide range of neuroscience, psychology and psychophysics
experiments.
3See: https://github.com/sccn/labstreaminglayer.
C. Practical Experiments
The procedure of the Calibration Phase was applied in
the first experiment that was conducted in late April 2017
(Eurokreator lab, Krakow, Poland). The subjects (6 persons)
were two male and 4 female participants of the workshops
which were held in the lab. The full procedure, consisting
of Calibration Phase and Gaming Phase, has been conducted
as the second experiment in mid-June 2017. The subjects (9
persons) were Polish students from AGH-UST.
At the time of experiments the Lab Stream Layer synchro-
nization method was not running yet. Therefore, the synchro-
nization between the phone and the notebook was acquired by
the subject simultaneously pressing spacebar on the notebook
and „START RECORDING” button on the phone. It should
be noted that due to technical issues a total of 52 pictures was
used in the first experiment instead of 54.
D. Evaluation of Results
During the calibration phase, the following data is acquired:
NAPS Valence and arousal scores – shared with the
NAPS set.
Baseline HR and GSR – the average value of 30 seconds
recordings of HR and GSR at the beginning of main
session.
Gamers arousal score – the rating given by the player
after seeing each of the stimuli.
HR and GSR reactions – the HR and GSR levels recorded
from the appearance of the stimulus to the appearance of
the next stimulus.
The ultimate goal of this phase is to prepare HR and GSR
patterns as a function of valence-arousal scores. In order to
achieve this, the following analyses are being conducted:
Gamers arousal scores are compared with NAPS arousal
scores as a simple verification of NAPS data. If these
scores will be significantly different, further analyses will
be conducted for both sets of values.
Average absolute HR and GSR reactions are calculated
for different valence, arousal and valence*arousal values
(as described in Section VI-A). The statistical significance
of the differences between them is checked.
The same analysis are conducted for relative values:
instead of taking HR and GSR values, the difference
between actual reactions and baseline are considered.
It is also checked “if change appears” in the moment
of affective stimuli appearance: it is possible that band
sometimes record growth and sometimes falls and there
are no significant differences between reactions’ levels,
but there is always a change that can be used within game
context.
Preliminary analyses were carried out. HR and GSR re-
sponses to the presented stimuli are depicted on the plots
presented in Figure 4. Heart Rate varies as a function of
arousal: higher arousal values are indicated by steeper HR
changes than in the neutral condition and lower arousal values
are associated with flatter HR changes. This is consistent
350 COMMUNICATION PAPERS OF THE FEDCSIS. PRAGUE, 2017
Fig. 3. PsychoPyBuilder Design of the experimental procedure for calibration
Fig. 4. HR and GSR responses with regard to the group to which the stimulus belonged (Val = Valence, Aro = Arousal, H= High, L= Low).
with both the general knowledge and the results of other
research [20]: if a participant is more excited than his heart
beats faster. On the other hand, Galvanic Skin Response varies
as a function of valence. It shows changes to high valence
values and no changes for others. These are important results
because they point to the real utility of the budgetary wrist
bands for differentiating emotional states.
As the data was collected just before the submission of the
final version of this paper, a more exhaustive analyses leading
to more accurate HR and GSR patterns identification are still
being performed.
VII. REL ATED RE SE AR CH
In recent years there has been a lot of work related to
the introduction of affective components into video games.
Selected works are presented next.
In an early paper [23] an idea of enhancing biofeedback by
placing it within a competitive virtual gaming environment was
introduced. In fact, it was one of the first works to introduce
affective feedback (loop). The virtual environment affects the
players’ level of relaxation while the levels of relaxation
determine the outcome of the game. The most relaxed person
wins the game. The study [24] investigates the hypothesis that
the player’s arousal will correspond with the pressure used to
depress buttons on a gamepad. A practical video game was
created to detect the force of each button press during play.
In [25] the authors examined phasic psychophysiological
responses indexing emotional valence and arousal to different
game events during the video game Monkey Bowling 2.
Event-related changes in skin conductance, cardiac interbeat
intervals, and other signals were recorded. Furthermore, game
events elicited reliable valence- and arousal-related phasic
physiological responses. The authors proposed to use the
information on emotion-related physiological responses to
game events or event patterns, to guide choices in game design.
The paper [26] describes an investigation into how real-
time yet low-cost biometric information can be interpreted by
computer games to enhance gameplay. The primary benefit of
incorporating this technique into computer games is that game
developers can offer control over direction of game play and
game events to the player.
The paper [27] discusses how the emerging discipline
of affective computing contributes to important elements of
affective game design, with emphasis on the importance of
modeling, including sensing and recognition of the players’
emotions, generating ’affective behaviors’.
The paper [28] presents two studies that aim to realize
an emotionally adaptive game. It investigates the relations
between game mechanics, a player’s emotional state and
his/her emotion-data. In an experiment, one game mechanic
(speed) was manipulated. Emotional state was self-reported in
GRZEGORZ J. NALEPA ET AL.: AFFECTIVE DESIGN PATTERNS IN COMPUTER GAMES. SCROLLRUNNER CASE STUDY 351
terms of valence, arousal and boredom-frustration-enjoyment.
A number of (mainly physiology-based) emotion-data features
were measured. Significant correlations were found between
the valence/arousal reports and the emotion-data features.
As our work is not directly comparable in detail to any of
these works, certain objectives are similar. What seems novel
in our approach wrt to the state of the art is addressing specific
affective game design patterns.
VIII. FU TU RE WO RK S
In this paper we discussed applications of affective com-
puting techniques to the design of video games. We em-
ployed Jesse Prinz’s Embodied Appraisal Theory that follows
the assumptions of James-Lange theory of emotions. In this
approach a change in the affective condition of a player
can be detected and identified based of the monitoring of
physiological signals of the player. In fact, we focuses on heart
rate and galvanic skin response. We used game design patterns
introduced by Björk and Holopainen.
Our contribution consists in the identification of a set of
affective game design patterns. We demonstrated how these
patterns can be used in a design of a scroll-runner game. We
addressed the problem of differences in responses of individual
users by the introduction of a proper calibration phase using
NAPS pictures. Our approach is novel wrt to the state of the
art, by addressing and using specific affective game design
patterns.
Our objective was to provide integration of a sensory
framework with a specific gaming environment to detect
emotions. In this paper we discuss the use of the Game
Maker environment. In the future we plan to support the Unity
environment. We aim at extending the game design process by
the practical use of affective patterns. In this way, the designer
could introduce and control the affective loop in the game.
Moreover, new series of experiments are planned. Finally,
we are aiming to incorporate this work with a platform [29]
combining affective computing with context-aware systems for
ambient intelligence applications [30].
ACK NOW LE DG ME NT
This work is supported by the Jagiellonian University and
AGH University grants.
REF ER EN CE S
[1] R. W. Picard, Affective Computing. MIT Press, 1997.
[2] S. Björk and J. Holopainen, Patterns in Game Design. Charles River
Media, 2005.
[3] W. James, “What is an emotion?” Mind, no. 34, pp. 188–205, 1884.
[4] G. J. Nalepa, J. K. Argasinski, K. Kutt, P. Wegrzyn, S. Bobek, and
M. Z. Lepicki, “Affective computing experiments in virtual reality
with wearable sensors. methodological considerations and preliminary
results,” in Proceedings of the Workshop on Affective Computing and
Context Awareness in Ambient Intelligence (AfCAI 2016), ser. CEUR
Workshop Proceedings, M. T. H. Ezquerro, G. J. Nalepa, and J. T. P.
Mendez, Eds., vol. 1794, 2016.
[5] R. A. Calvo, S. K. D’Mello, J. Gratch, and A. Kappas, Eds., The Oxford
Handbook of Affective Computing, ser. Oxford Library of Psychology.
Oxford University Press, 2015.
[6] M. B. Arnold, Emotion and personality. Columbia University Press,
1960.
[7] R. S. Lazarus, Psychological stress and the coping process. McGraw-
Hill, 1966.
[8] S. K. R., “Apprasial theory,” in Handbook of cognition and emotion,
D. T. and P. M., Eds. John Wiley and Sons, 1999, pp. 637–663.
[9] A. Orthony, G. Clore, and A. Collins, The cognitive structure of
emotions. Cambridge University Press, 1988.
[10] S. C. Marsella and J. Gratch, “Ema: A process model of appraisal
dynamics,” Cognitive Systems Research, vol. 10, pp. 70–90, 2009.
[11] A. Mehrabian and J. A. Russell, An approach to environmental psychol-
ogy. MIT Press., 1974.
[12] C. Becker-Asano and I. Wachsmuth, “Affective computing with primary
and secondary emotions in a virtual human,” Autonomous Agents and
Multi-Agent Systems, vol. 20, pp. 32–49, 2009.
[13] J. Prinz, “Which emotions are basic?” in Emotion, evolution, and
rationality, D. Evans and P. Cruse, Eds. Oxford University Press New
York, 2004, pp. 69–88.
[14] B. Suits, The Grasshopper: Games, Life and Utopia. Broadview Press,
2005.
[15] B. Winn, “The design, play and experience framework,” in Handbook
of Research on Effective Electronic Gaming in Education. Hershey,
2009, pp. 1010–1024.
[16] R. W. Picard, “Recognizing stress, engagement, and positive emotion,”
in Proceedings of the 20th International Conference on Intelligent User
Interfaces. ACM, 2015, pp. 3–4.
[17] A. Marchewka, Ł. ˙
Zurawski, K. Jednoróg, and A. Grabowska, “The
Nencki Affective Picture System (NAPS): Introduction to a novel, stan-
dardized, wide-range, high-quality, realistic picture database,Behavior
Research Methods, vol. 46, no. 2, pp. 596–610, 2014.
[18] S. Swink, Game Feel. A game designer’s guide to virtual sensation.
Elsevier, 2009.
[19] E. Andersen, Y.-E. Liu, R. Snider, R. Szeto, S. Cooper, and Z. Popovi´
c,
“On the harmfulness of secondary game objectives,” in Proceedings
of the 6th International Conference on Foundations of Digital Games.
ACM, 2011, pp. 30–37.
[20] I. B. Mauss and M. D. Robinson, “Measures of emotion: A review,”
Cognition and Emotion, vol. 23, no. 2, pp. 209–237, 2009.
[21] J. A. Russell, “A circumplex model of affect,” Journal of Personality
and Social Psychology, vol. 39, no. 6, pp. 1161–1178, 1980.
[22] J. T. Cacioppo, G. G. Berntson, J. T. Larsen, K. M. Poehlmann, and
T. A. Ito, “The psychophysiology of emotion,” in Handbook of emotions.
Guildford Press, 2000, pp. 173–191.
[23] D. Bersak, G. McDarby, N. Augenblick, P. McDarby, D. McDonnell,
B. McDonald, and R. Karkun, “Intelligent biofeedback using an immer-
sive competitive environment,” 2001, paper presented at the Designing
Ubiquitous Computing Games Workshop at UbiComp 2001.
[24] J. Sykes and S. Brown, “Affective gaming: Measuring emotion through
the gamepad,” in CHI ’03 Extended Abstracts on Human Factors in
Computing Systems, ser. CHI EA ’03. New York, NY, USA: ACM,
2003, pp. 732–733.
[25] R. Niklas, S. Timo, L. Jani, K. Kari, and S. Mikko, “The psychophysi-
ology of video gaming: Phasic emotional responses to game events,
in DiGRA &#3905 - Proceedings of the 2005 DiGRA International
Conference: Changing Views: Worlds in Play, 2005.
[26] K. M. Gilleade and A. Dix, “Using frustration in the design of adaptive
videogames,” in Proceedings of the 2004 ACM SIGCHI International
Conference on Advances in Computer Entertainment Technology, ser.
ACE ’04. New York, NY, USA: ACM, 2004, pp. 228–232.
[27] E. Hudlicka, “Affective computing for game design,” in Proceedings of
the 4th International North American Conference on Intelligent Games
and Simulation (GAMEON-NA), 2008.
[28] T. Tijs, D. Brokken, and W. Ijsselsteijn, “Creating an emotionally
adaptive game,” in Proceedings of the 7th International Conference on
Entertainment Computing, ser. ICEC ’08. Berlin, Heidelberg: Springer-
Verlag, 2009, pp. 122–133.
[29] S. Bobek and G. J. Nalepa, “Uncertain context data management in
dynamic mobile environments,Future Generation Computer Systems,
vol. 66, pp. 110 – 124, 2017.
[30] G. J. Nalepa, K. Kutt, S. Bobek, and M. Z. Lepicki, “AfCAI systems:
Affective computing with context awareness for ambient intelligence.
research proposal,” in Proceedings of the Workshop on Affective Com-
puting and Context Awareness in Ambient Intelligence (AfCAI 2016),
ser. CEUR Workshop Proceedings, M. T. H. Ezquerro, G. J. Nalepa,
and J. T. P. Mendez, Eds., vol. 1794, 2016.
352 COMMUNICATION PAPERS OF THE FEDCSIS. PRAGUE, 2017
... To address the need for comparing those devices and using them in studies, we created a software solution called BandReader, described in detail in [35] (see Figure 1), which enables us to collect data from wristbands and other non-medical platforms using Bluetooth technology and then broadcast the acquired statistics to all listening applications and devices. Our previous preliminary research in field of AfC [36,37] led us to specify which software will be convenient for laboratory experiments. Among others, we established that such technology should facilitate recording data from several different tools simultaneously, take care of the fact that every wearable device provides another set of sensors, and save files in a convenient format (such as .CSV). ...
... In order to evaluate the ideas related to the detection of emotions using mobile sensors, we decided to take up the area of computer games. Our approach is based on an earlier study, described in [37], which was our first attempt at using affective computing techniques in the design of video games. This was also the time when our original idea of the proposed concept for facilitating games with an affective component first sparked. ...
... As in the case of the pilot study, we gathered GSR and HR/BVP data, as well as timestamps of visual stimuli appearance and of in-game events. The results from the first series and from the pilot study were published and discussed in [37]; these allowed us to configure our initial hardware setup and test the BandReader application. ...
Article
Full-text available
In this paper, we consider the use of wearable sensors for providing affect-based adaptation in Ambient Intelligence (AmI) systems. We begin with discussion of selected issues regarding the applications of affective computing techniques. We describe our experiments for affect change detection with a range of wearable devices, such as wristbands and the BITalino platform, and discuss an original software solution, which we developed for this purpose. Furthermore, as a test-bed application for our work, we selected computer games. We discuss the state-of-the-art in affect-based adaptation in games, described in terms of the so-called affective loop. We present our original proposal of a conceptual design framework for games, called the affective game design patterns. As a proof-of-concept realization of this approach, we discuss some original game prototypes, which we have developed, involving emotion-based control and adaptation. Finally, we comment on a software framework, that we have previously developed, for context-aware systems which uses human emotional contexts. This framework provides means for implementing adaptive systems using mobile devices with wearable sensors.
... Then, we demonstrate the experimental procedure prepared for testing of created prototypes. Our work follows our early ideas with affective design patterns discussed in [22] and in [11]. ...
... They can be understood as solutions to frequently recurring issues regarding gameplay. What is more, as shown in [22,8,6], they can be cross-culturally associated with certain emotions, which is a development of Hudlicka's ideas about game design. For example, the player character's unexpected death will invariably be associated with negative emotions, and a sudden loud sound can, with high probability, elicit fear. ...
... In this paper, we describe the games developed using Unity tutorials, although the narrative of Freud me Out is our original creation. [22] discusses the role of selection of affective patterns in base stage of games creation. It follows the ideas of Björk and Holopainen [4] about universal mechanics, which can be identified in almost every game, and enriches them with emotional factor. ...
Chapter
The use of emotions in the process of creating video games is still a challenge for the developers from the fields of Human-Computer Interaction and Affective Computing. In our work, we aim at demonstrating architectures of two operating game prototypes, implemented with the use of affective design patterns. We ground our account in biological signals, i.e. heart rate, galvanic skin response and muscle electrical activity. Using these modalities and the game context, we reason about emotional states of the player. For this purpose, we focus on defining rules with linguistic terms. What is more, we address the need for explainability of biological mechanics and individual differences in terms of reactions to different stimuli. We provide a benchmark, in the form of a survey, to verify our approach.
... Review of current software status and our previous preliminary research in AfC field [5], [6] led us to specification of a set of use cases and requirements for the software to be a convenient asset for studies: ...
... The Bandreader application was developed in order to support our practical research in the area of affective computing. We used BandReader to record data collected by Empatica E4 and MS Band2 wristbands in a research project regarding the development of affective design patterns in computer games [6]. Thanks to the portabilty of the developed solution, we are further extending it to provide the biofeedback loop. ...
... After each picture, the task is to rate subjectively felt arousal in response to the presented stimuli, on a 7-level scale from 1 to 7. The actual experiment is preceded by a short training session with neutral images. The second and last phase involved playing a simple platform game, designed specifically for research purposes [6]. The wristband continued to collect the subject's physiological data throughout the game and the BandReader recorded it for each participant individually. ...
Conference Paper
As technology becomes more ubiquitous and pervasive, special attention should be given to human-computer interaction, especially to the aspect related to the emotional states of the user. However, this approach assumes very specif c mode of data collection and storage. This data is used in the affective computing experiments for human emotion recognition. In the paper we describe a new software solution for mobile devices that allows for data acquisition from wristbands. The application reads physiological signals from wristbands and supports multiple recent devices. In our work we focus on the Heart Rate (HR) and Galvanic Skin Response (GSR) readings. The recorded data is conveniently stored in CSV files, ready for further interpretation. We provide the evaluation of our application with several experiments. The results indicate that the BandReader is a reliable software for data acquisition in affective computing scenarios.
... We described the design and the preliminary work on this platform in [15]. Later on, we turned our attention to computer games [8], as an interesting environment to design aective experiments [16]. As we focus on wearable devices, in our recent paper we provided results of a critical analysis of measurements using the selected devices [9]. ...
... This time, a second phase, which followed the Calibration Phase, was involved. Directly after the arousal evaluation of presented aective pictures, the subjects played a simple platform game (created specically for research purposes [16]), using the same computer. After pressing Enter key on the notebook to start the game, a 30 seconds of blank screen is presented to get the baseline HR and GSR recordings for the beginning of the Gaming Phase. ...
Conference Paper
Full-text available
In this overview paper we focus on our recent progress in the work on the mobile platform for AfC. We provide the main assumptions about the platform, as well as describe affective data acquisition and interpretation. We discuss our most recent experiments and provide an outlook of our future works.
... That is why in our previous works we started with creating games based on Affective Game Design Patterns [25] and then extended the idea with prototypes implementing the Affective Loop [13,27]. Simultaneously, we investigated into the area of context-aware systems that can be used for ambient intelligence [26]. ...
Conference Paper
With mobile games becoming best sellers in app stores of Google and Apple, more and more research interest is put in this field. Practical applications of games, however, may surpass sole entertainment and bring educational value to gaming experience. Such undertaking, under the name of serious games, has been actively explored in the academia and beyond. Numerous previous attempts at educational games brought significant insights into the edutainment area, which inspired us to garner the advantages of their ideas. In our work, we demonstrate an architecture of a mobile game with location data enabling context-aware play, which implements affective game design patterns for greater experience of cultural heritage of Cracow. Additionally, we identify several research questions and propose methodological approach to verify them.
... According to [20] the design concept that suits users with disabilities is challenging as we have to adhere to their affective needs. Studies on affective design have proven that the value of outcome can be increased especially on the interface design [21], [22] and interaction behavior [23]. The concept of affective design is based on [20], [21], [23], [24] that address the inclusion of feelings, thought (perception), emotion and actions. ...
Article
Full-text available
In this digital era, low vision children should learn Mathematics by using similar learning content provided for mainstream students. Teachers need to express their feeling, thought, emotions, and actions to ensure the low vision learners can grasp the mathematics content. However, it is too tough for low vision children to adapt to the common learning content which is in contrast with their needs. Consequently, this study aims to develop Affective 4- Dimensional Mobile Mathematics for Low Vision Alpha Generation. This study adapted the 3-Phases Development Model. Accordingly, the findings of this study are the design and development of the abovementioned application.
... As such, the studies so far were conducted in April 2017 (only calibration phase, 6 participants), June 2017 (calibration phase and gaming phase, 9 participants), as well as January and March 2018 (all three phases, 107 participants). Results of the first attempt, together with initial outline of our approach, have been described and published in [36]. ...
Chapter
In this paper we present results of an experiment designed to measure correlation between Galvanic Skin Response – GSR (also: Electrodermal Activity – EDA) and Oculographic (Eye Tracking) data. In the experiment we analyze the participants reactions on changing gameplay modes (changing dynamic and pace of user interaction with the game mechanics). The goal of the analysis is to demonstrate what influence of changing gameplay modes have on physiology (especially spontaneous body response). The gameplay modes used in experiment were chase mode (dynamic) and sneak mode (satisfying). In the experiment we have used Pupil Lab eye-tracker and EDA sensor based on BITalino platform. The game Serial Cleaner by iFun4all was used to conduct the experiment. The experiment resulted in multiple observations. Substantial correlation between participant failure and change of EDA pattern were observed as well as over 10% change in fixations durations.
Chapter
Affective computing gained a lot of attention from researchers and business over the last decade. However, most of the attempts for building systems that try to predict, or provoke affective state of users were done for specific and narrow domains. This complicates reusing such systems in other, even similar domains. In this paper we present such a solution, that aims at solving such problem by providing a general framework architecture for building affective-aware systems. It supports designing and development of affective-aware solutions, in a holistic and domain independent way.
Article
We discuss affective serious games that combine learning, gaming and emotions. We describe a novel framework for the creation and evaluation of serious affective games. Our approach is based on merging pertinent design patterns in order to recognize educational claims, educational assessment, best game design practices, as well as models and solutions of affective computing. Björk's and Holopainen's game design patterns have been enhanced by Evidence Centered Design components and affective components. A serious game has been designed and created to demonstrate how to outline a complex game system in a communicative way, and show methods to trace how theoretically-driven design decisions influence learning outcomes and impacts. We emphasize the importance of patterns in game design. Design patterns are an advantageous and convenient way of outlining complex game systems. Design patterns also provide favorable language of communication between multidisciplinary teams working on serious games.
Article
Full-text available
Selecting appropriate stimuli to induce emotional states is essential in affective research. Only a few standardized affective stimulus databases have been created for auditory, language, and visual materials. Numerous studies have extensively employed these databases using both behavioral and neuroimaging methods. However, some limitations of the existing databases have recently been reported, including limited numbers of stimuli in specific categories or poor picture quality of the visual stimuli. In the present article, we introduce the Nencki Affective Picture System (NAPS), which consists of 1,356 realistic, high-quality photographs that are divided into five categories (people, faces, animals, objects, and landscapes). Affective ratings were collected from 204 mostly European participants. The pictures were rated according to the valence, arousal, and approach–avoidance dimensions using computerized bipolar semantic slider scales. Normative ratings for the categories are presented for each dimension. Validation of the ratings was obtained by comparing them to ratings generated using the Self-Assessment Manikin and the International Affective Picture System. In addition, the physical properties of the photographs are reported, including luminance, contrast, and entropy. The new database, with accompanying ratings and image parameters, allows researchers to select a variety of visual stimulus materials specific to their experimental questions of interest. The NAPS system is freely accessible to the scientific community for noncommercial use by request at http://naps.nencki.gov.pl. Electronic supplementary material The online version of this article (doi:10.3758/s13428-013-0379-1) contains supplementary material, which is available to authorized users.
Article
Building systems that acquire, process and reason with context data is a major challenge. Model updates and modifications are required for the mobile context-aware systems. Additionally, the nature of the sensor-based systems implies that the data required for the reasoning is not always available nor it is certain. Finally, the amount of context data can be significant and can grow fast, constantly being processed and interpreted under soft real-time constraints. Such characteristics make it a case for a challenging big data application. In this paper we argue, that mobile context-aware systems require specific methods to process big data related to context, at the same time being able to handle uncertainty and dynamics of this data. We identify and define main requirements and challenges for developing such systems. Then we discuss how these challenges were effectively addressed in the KnowMe project. In our solution, the acquisition of context data is made with the use of the AWARE platform. We extended it with techniques that can minimize the power consumption as well as conserve storage on a mobile device. The data can then be used to build rule models that can express user preferences and habits. We handle the missing or ambiguous data with number of uncertainty management techniques. Reasoning with rule models is provided by a rule engine developed for mobile platforms. Finally, we demonstrate how our tools can be used to visualize the stored data and simulate the operation of the system in a testing environment.
Conference Paper
An intelligent interaction should not typically call attention to emotion. However, it almost always involves emotion: For example, it should engage, not inflict undesirable stress and frustration, and perhaps elicit positive emotions such as joy or delight. How would the system sense or recognize if it was succeeding in these elements of intelligent interaction? This keynote talk will address some ways that our work at the MIT Media Lab has advanced solutions for recognizing user emotion during everyday experiences.