CSEI: The Constructive Sensual Evaluation Instrument

Abstract

This paper introduces a constructive and collaborative digital, multi-touch instrument for tactile evaluation of user experiences with interactive systems. We are developing a tool that allows individuals or groups to digitally construct affective objects that represent their own personal mappings of emotions to shape characteristics. Our approach builds upon prior work with nonverbal assessments of affect, and stems from an interest in flexibly evaluating dynamic gestalt during individual and group interactions by using a wider variety of potential shapes. In this paper we provide background for our approach, and present a new valence-arousal model for identifying and constructing affective shapes and the Constructive Sensual Evaluation Instrument (CSEI).

Full text

CSEI: The Constructive Sensual
Evaluation Instrument
Abstract
This paper introduces a constructive and collaborative
digital, multi-touch instrument for tactile evaluation of
user experiences with interactive systems. We are
developing a tool that allows individuals or groups to
digitally construct affective objects that represent their
own personal mappings of emotions to shape
characteristics. Our approach builds upon prior work
with nonverbal assessments of affect, and stems from
an interest in flexibly evaluating dynamic gestalt during
individual and group interactions by using a wider
variety of potential shapes. In this paper we provide
background for our approach, and present a new
valence-arousal model for identifying and constructing
affective shapes and the Constructive Sensual
Evaluation Instrument (CSEI).
Author Keywords
Affect; Emotion; Nonverbal; Construction;
Collaboration; Ambiguity; Evaluation; Dynamic Gestalt;
Embodiment; Gesture-based Interaction.
Introduction
In recent years, HCI researchers and practitioners have
recognized the importance of emotion and pleasure in
user experience [3]. However, traditional affective
measurement tools and models of emotion have found
difficulty in exploring the full complexity and time
Copyright is held by the author/owner(s). Workshop on Tactile User
Experience Evaluation Methods at CHI 2014.
Edward Melcer
New York University
5 MetroTech Center,
Brooklyn, NY 11201
em1679@nyu.edu
Katherine Isbister
New York University
5 MetroTech Center,
Brooklyn, NY 11201
katherine.isbister@nyu.edu

sensitive nature of affect in an interactive experience.
Such tools are also generally unsuitable for trans-
cultural application, measuring affect among a group,
and users that deviate from the expected norm of
verbal, intellectual, or sensory abilities.
In the past two decades, a variety of affective
measurements have been developed to address these
issues such as the Sensual Evaluation Instrument (SEI)
[4,5,7], Self-Assessment Manikin (SAM) [1], and the
uses of biometric and sensor-based traces of emotion
[12]. While these measurements take distinctly
contrasting approaches — tangible objects vs.
anthropomorphic pictorials vs. interpretation of sensor
data — they all utilize nonverbal mapping from a
complex emotion space to a delimited set of objects,
images, or emotions. The SEI, in particular, is novel by
virtue of its ambiguous design intended to evoke
unique personal taxonomies of affect for the SEI
objects, rather than forcing all users to obey predefined
affective mappings. These unique taxonomies in turn
lead to a much richer dialogue between designers and
users relating to time specific affective events [7].
While the SEI has proven effective in measuring the
dynamic gestalt of non time-critical interactive stories,
it is not without issues. Some users felt the limited
amount of shapes in SEI was missing affective
dimensions such as ‘droopiness’ [4], while others found
certain objects difficult to interpret and map to affect
[5]. We believe this is where the large shape space of a
constructive approach may prove more effective than
mapping affect to a limited number of shapes.
To aid in uncovering the complex, time sensitive
emotions experienced during interactions, we draw
inspiration from the ambiguous affective objects of SEI,
its mapping of individual taxonomy to shape, and its
focus on generating dialogue. Additionally, we draw
inspiration from participatory design — bringing
children into the design process through construction of
physical objects representative of a co-created
language of evaluation [2,10]. We have also been
influenced by art therapy — the process of expression
and creation of products through art media that are
perceived predominantly through the tactile-haptic,
visual sensory, and perceptual channels, which are then
processed for their affect, associations, and meaning
through cognitive and verbal channels [9].
Ultimately this has led to our design of the Constructive
Sensual Evaluation Instrument (CSEI). The CSEI is a
multi-touch application that allows multiple users to co-
construct abstract shapes representative of emotional
states. The CSEI works by providing "digital" clay and
distributing the cognition of clay manipulation between
users and device. When users first interact with the
clay they see a neutral sphere. To perform a
manipulation, users draw a circle around the portion, or
whole shape, that they wish to manipulate (see Figure
1A). Users can then employ gestures for quick and
simple manipulations such as pinching fingers to
squash the selection, spreading fingers to stretch the
selection, and dragging four fingers to roll the selection
(see Figure 1B). Additionally, upon selection an edit
menu pops up for complex and precise manipulations
such as pushing the selection out from the shape or
pulling it inwards, smoothing or sharpening, twisting,
increasing size of the selection, and symmetrically
mirroring the manipulations over different axes (see
Figure 1C).
Figure 1. The CSEI tool. A) Users
circle portion of shape to edit. B)
Users can make gestures to edit
selection. C) Edit menu appears
with sliders for more manipulations.

The CSEI utilizes an underlying construct for the
dynamic relation among emotions called the Valence-
Arousal Cube (see Figure 2). We constructed the VA
Cube to address a common issue with many models of
emotion where positive and negative valence are
treated as opposite ends of the same dimensional
spectrum. By instead separating the valence dimension
into distinct positive and negative dimensions, a more
expressive model encapsulating mixed emotions and
contextual/cultural differences is constructed. This type
of model also more accurately represents current
research in neuroscience where highly interconnected
regions in the brain dealing with negative or positive
affect run concurrently and handle complex cognitive-
emotional interactions [11].
The VA Cube model is extremely useful for abstracting
a set of shape manipulations to their related affective
dimensions (see Table 1). By placing these
transformations along their corresponding dimensions
of the VA Cube, we create the Affective Shape Cube
which guides creation and analysis of objects in the
CSEI (see Figure 3). Such a model also captures the
dynamics and persistence of emotions over time since
all manipulations, and corresponding dimensions, can
be represented simultaneously. For instance, in the
game Minecraft one common interaction among players
is to go on a mining expedition. At the start of such an
experience the cube would likely represent excitement
for all players with lots of rounded extrusions (see
Figure 4A). However, over time the players experiences
could diverge. Such as one player maintaining
excitement after finding an area with rich resources,
another feeling bored after finding nothing, and a third
angry after dying and losing supplies. This could lead to
a shape that is squashed to show boredom, spiky to
show anger, and rounded to show excitement in order
to encapsulate the full experience (see Figure 4B).
Later on, the bored player may become excited after
finding a material rich area and remove the squashing
transformation, but the third player may remain angry
from the permanent loss of resources (see Figure 4C).
We envision the CSEI being used for evaluation of non
time-critical individual and group interactions. The CSEI
may be especially useful for highly collaborative and
non time-critical games such as Minecraft. While CSEI
is similar to SEI in the breadth of its use, CSEI offers
several gains with regard to the information it can
obtain. CSEI allows us to collect rich data about how
people vary individually and in group settings when
constructing emotional objects. These objects can also
be analyzed for the persistence and dynamics of
emotion/shape over time by using the Affective Shape
Cube. Ultimately, we could create a database of shape
dynamics from the constructed objects and use mass
Figure 2. Our proposed Valence-
Arousal Cube accounts for the
experience of mixed emotions.
Negative valence, positive valence,
and arousal are placed at right
angles to form a cube.
Arousal
Positive
Valence
Negative
Valence
Emotion
Manipulation
Low
Low
Low
Calm, Neutral
Smooth Surface [4]
High
Low
Low
Surprise
Extrude [4], Stretch [6],
Pull, Increase Size
Low
High
Low
Content
Roll
Low
Low
High
Depressed
Squash [6], Fold
High
High
Low
Excitement
Smooth (Bumpy) [4]
High
Low
High
Anger
Sharpen (Spiky) [4]
Low
High
High
Distant,
Dissonance
Symmetry or Asymmetry
(opposite current shape
valence) [4], Twist
High
High
High
Interest,
Engagement
Number of Extrusions, High
Surface Deformation [4]
Table 1. Edges of the Valence-Arousal Cube, corresponding emotions, and related manipulations.

surveying sites, such as MTURK, to obtain cross ratings
of each object's emotional signaling in order to verify
the Affective Shape Cube dimensions.
We also posit CSEI provides key advantages over
physical construction approaches such as art therapy:
 Preserves pleasure of constructing objects, but with
significantly more speed and precision due to a
simple gesture and slider based tactile interface.
 Provides a detailed, digital trace of dynamic gestalt
by representing and understanding shape changes
over time as a combination of affective
manipulations along the Affective Shape cube.
 Effective tool for use with children and adults.
 Provides a shared digital space and/or object for
collaborative, real-time construction and
communication among users.
 Evokes ambiguous communication of affect among
multiple users and designers through the co-
creation of mixed emotion objects.
The Valence-Arousal Cube
Typical models of emotion are primarily concerned with
mapping an affective response to a single valence
emotion. This approach to affect has mainly resulted in
circumplex models, such as Russell’s valence-arousal
model [13] (see Figure 5), that treat negative and
positive valence as opposite ends of the same
spectrum. While such models are sufficient for many
affective measurement tools, the complex, multi-
layered, and time sensitive nature of an interactive
experience (or story) requires a more expressive
model. Specifically, we need a model that accounts for
the simultaneous experience of opposite-valence
emotions (i.e. mixed emotions [8]) and
cultural/contextual differences. For instance, a user
could enjoy manipulating interface controls of some
system but greatly dislike the sound effects associated
with their use. Traditional models and tools would likely
map the overall experience to only a positive or
negative emotion rather than grasp the concurrent,
highly positive and negative affect occurring throughout
the experience. To address this issue, we altered
Russell’s circumplex valence-arousal model by splitting
the valence dimension into separate positive and
negative dimensions. We then placed arousal, negative
valence, and positive valence at right angles to form
the Valence-Arousal Cube (see Figure 2).
The Valence-Arousal Cube offers three distinct
advantages for construction of the CSEI. First, primary
emotions from Russell’s model can still be reached
along the cube’s edges. Second, mixed emotions fall
towards the center of the cube and can be reached
through combinations of positive and negative valence.
These combinations represent affect that is ambiguous,
eliciting a context and culture based interpretation from
the user. Third, separating valence into its atomic
dimensions allows us to associate distinct affective
manipulations of shape with each dimension, resulting
in an Affective Shape Cube (see Figure 3 and Table 1).
The Affective Shape Cube quantifies ways in which
shapes can be manipulated to evoke affective response.
By transforming a shape along affective dimensions of
the cube, a variety of primary and mixed emotion
objects can be created. It is important to note that the
Affective Shape Cube only maps shape to affective
response. The actual interpretation of emotion evoked
by shape is left to the individual, context, and culture.
Figure 3. Our proposed Affective
Shape Cube couples manipulations of
shape with the affective dimensions
of arousal, negative valence, and
positive valence. Combining differing
affective transformations produces
different affective shapes.
Figure 4. Objects created during a
mining experience in Minecraft. A)
Initial excitement is formed at
beginning of expedition. B)
Frustration from dying, boredom
from lack of events, and excitement
from success are experienced
during play. C) Boredom lessens as
new resources to mine are found.

However, shapes that fall along points closer to the
faces of the cube are far more likely to have a universal
interpretation than ambiguous shapes that lay closer to
the center of the cube. E.g. the anteater and spiky SEI
objects correlate to anger and surprise trans-culturally
because the majority of shape transformations are
along arousal and negative valence dimensions, placing
them near the arousal/negative valence face. On the
other hand, the barba papa SEI object has conflicting
dimensional transformations (asymmetry vs. smooth
extrusion) which places it towards the center of the
cube and leaves interpretation more to context and
culture (see Figure 6). This may explain why the barba
papa was confusing and unused by Swedish test
subjects since there was no cultural equivalent [5].
Current Status and Next Steps
The exact details of how the CSEI tool should function
or be used during tactile evaluation scenarios is still
unclear, and even what scenarios may be best for such
a tool needs further exploration. Ultimately, we want to
examine what people create with the CSEI tool to first
refine the instrument, and then focus on exploring the
proper usage scenarios. As such, we are currently
conducting a study employing a Wizard of Oz mockup
of the CSEI system using clay. Our goal in this study is
to verify and refine the Affective Shape Cube, and
determine manipulations/gestures necessary for
individual and group utilization of the CSEI. We then
plan to prototype the initial CSEI and conduct user
evaluations of the CSEI in tests with individuals,
groups, and finally children. We should have results
from the current study, and the preliminary prototype,
to present at this workshop.
References
[1] Bradley, Margaret M., and Peter J. Lang.
"Measuring emotion: the self-assessment manikin and
the semantic differential." Journal of behavior therapy
and experimental psychiatry 25.1 (1994): 49-59.
[2] Druin, Allison. "The role of children in the
design of new technology." Behavior and information
technology 21.1 (2002): 1-25.
[3] Höök, Kia, et al. "Evaluation of Affective
Interactive Applications." Emotion-Oriented Systems.
Springer Berlin Heidelberg, 2011. 687-703.
[4] Isbister, Katherine, et al. "The sensual
evaluation instrument: developing an affective
evaluation tool." Proceedings of the SIGCHI conference
on Human Factors in computing systems. ACM, 2006.
[5] Isbister, Katherine, et al. "The sensual
evaluation instrument: Developing a trans-cultural self-
report measure of affect." International journal of
human-computer studies 65.4 (2007): 315-328.
[6] Johnston, Ollie, and Frank Thomas. "The
illusion of life: Disney animation."Disney Editions
(1995).
[7] Laaksolahti, Jarmo, Katherine Isbister, and
Kristina Höök. "Using the sensual evaluation
instrument." Digital creativity 20.3 (2009): 165-175.
[8] Larsen, Jeff T., Yen M. To, and Gary Fireman.
"Children's understanding and experience of mixed
emotions." Psychological Science 18.2 (2007): 186-
191.
[9] Lusebrink, Vija B. "Art therapy and the brain:
An attempt to understand the underlying processes of
art expression in therapy." Art Therapy 21.3 (2004):
125-135.
[10] Muller, Michael J. "Participatory design: the
third space in HCI." Human-computer interaction:
Development process (2003): 165-185.
Figure 6. Top - anteater, spiky.
Bottom - barba papa.
Figure 5. Russell’s valence-arousal
model.

[11] Pessoa, Luiz. "On the relationship between
emotion and cognition." Nature Reviews
Neuroscience 9.2 (2008): 148-158.
[12] Picard, Rosalind W., and Shaundra Bryant
Daily. "Evaluating affective interactions: Alternatives to
asking what users feel." CHI Workshop on Evaluating
Affective Interfaces: Innovative Approaches. 2005.
[13] Russell, James A. "A Circumplex Model of
Affect." Journal of Personality and Social Psychology
39.6 (1980): 1161-178.