ResearchPDF Available

Abstract and Figures

This paper introduces a constructive and collaborative digital, multi-touch instrument for tactile evaluation of user experiences with interactive systems. We are developing a tool that allows individuals or groups to digitally construct affective objects that represent their own personal mappings of emotions to shape characteristics. Our approach builds upon prior work with nonverbal assessments of affect, and stems from an interest in flexibly evaluating dynamic gestalt during individual and group interactions by using a wider variety of potential shapes. In this paper we provide background for our approach, and present a new valence-arousal model for identifying and constructing affective shapes and the Constructive Sensual Evaluation Instrument (CSEI).
Content may be subject to copyright.
CSEI: The Constructive Sensual
Evaluation Instrument
This paper introduces a constructive and collaborative
digital, multi-touch instrument for tactile evaluation of
user experiences with interactive systems. We are
developing a tool that allows individuals or groups to
digitally construct affective objects that represent their
own personal mappings of emotions to shape
characteristics. Our approach builds upon prior work
with nonverbal assessments of affect, and stems from
an interest in flexibly evaluating dynamic gestalt during
individual and group interactions by using a wider
variety of potential shapes. In this paper we provide
background for our approach, and present a new
valence-arousal model for identifying and constructing
affective shapes and the Constructive Sensual
Evaluation Instrument (CSEI).
Author Keywords
Affect; Emotion; Nonverbal; Construction;
Collaboration; Ambiguity; Evaluation; Dynamic Gestalt;
Embodiment; Gesture-based Interaction.
In recent years, HCI researchers and practitioners have
recognized the importance of emotion and pleasure in
user experience [3]. However, traditional affective
measurement tools and models of emotion have found
difficulty in exploring the full complexity and time
Copyright is held by the author/owner(s). Workshop on Tactile User
Experience Evaluation Methods at CHI 2014.
Edward Melcer
New York University
5 MetroTech Center,
Brooklyn, NY 11201
Katherine Isbister
New York University
5 MetroTech Center,
Brooklyn, NY 11201
sensitive nature of affect in an interactive experience.
Such tools are also generally unsuitable for trans-
cultural application, measuring affect among a group,
and users that deviate from the expected norm of
verbal, intellectual, or sensory abilities.
In the past two decades, a variety of affective
measurements have been developed to address these
issues such as the Sensual Evaluation Instrument (SEI)
[4,5,7], Self-Assessment Manikin (SAM) [1], and the
uses of biometric and sensor-based traces of emotion
[12]. While these measurements take distinctly
contrasting approaches tangible objects vs.
anthropomorphic pictorials vs. interpretation of sensor
data they all utilize nonverbal mapping from a
complex emotion space to a delimited set of objects,
images, or emotions. The SEI, in particular, is novel by
virtue of its ambiguous design intended to evoke
unique personal taxonomies of affect for the SEI
objects, rather than forcing all users to obey predefined
affective mappings. These unique taxonomies in turn
lead to a much richer dialogue between designers and
users relating to time specific affective events [7].
While the SEI has proven effective in measuring the
dynamic gestalt of non time-critical interactive stories,
it is not without issues. Some users felt the limited
amount of shapes in SEI was missing affective
dimensions such as ‘droopiness’ [4], while others found
certain objects difficult to interpret and map to affect
[5]. We believe this is where the large shape space of a
constructive approach may prove more effective than
mapping affect to a limited number of shapes.
To aid in uncovering the complex, time sensitive
emotions experienced during interactions, we draw
inspiration from the ambiguous affective objects of SEI,
its mapping of individual taxonomy to shape, and its
focus on generating dialogue. Additionally, we draw
inspiration from participatory design bringing
children into the design process through construction of
physical objects representative of a co-created
language of evaluation [2,10]. We have also been
influenced by art therapy the process of expression
and creation of products through art media that are
perceived predominantly through the tactile-haptic,
visual sensory, and perceptual channels, which are then
processed for their affect, associations, and meaning
through cognitive and verbal channels [9].
Ultimately this has led to our design of the Constructive
Sensual Evaluation Instrument (CSEI). The CSEI is a
multi-touch application that allows multiple users to co-
construct abstract shapes representative of emotional
states. The CSEI works by providing "digital" clay and
distributing the cognition of clay manipulation between
users and device. When users first interact with the
clay they see a neutral sphere. To perform a
manipulation, users draw a circle around the portion, or
whole shape, that they wish to manipulate (see Figure
1A). Users can then employ gestures for quick and
simple manipulations such as pinching fingers to
squash the selection, spreading fingers to stretch the
selection, and dragging four fingers to roll the selection
(see Figure 1B). Additionally, upon selection an edit
menu pops up for complex and precise manipulations
such as pushing the selection out from the shape or
pulling it inwards, smoothing or sharpening, twisting,
increasing size of the selection, and symmetrically
mirroring the manipulations over different axes (see
Figure 1C).
Figure 1. The CSEI tool. A) Users
circle portion of shape to edit. B)
Users can make gestures to edit
selection. C) Edit menu appears
with sliders for more manipulations.
The CSEI utilizes an underlying construct for the
dynamic relation among emotions called the Valence-
Arousal Cube (see Figure 2). We constructed the VA
Cube to address a common issue with many models of
emotion where positive and negative valence are
treated as opposite ends of the same dimensional
spectrum. By instead separating the valence dimension
into distinct positive and negative dimensions, a more
expressive model encapsulating mixed emotions and
contextual/cultural differences is constructed. This type
of model also more accurately represents current
research in neuroscience where highly interconnected
regions in the brain dealing with negative or positive
affect run concurrently and handle complex cognitive-
emotional interactions [11].
The VA Cube model is extremely useful for abstracting
a set of shape manipulations to their related affective
dimensions (see Table 1). By placing these
transformations along their corresponding dimensions
of the VA Cube, we create the Affective Shape Cube
which guides creation and analysis of objects in the
CSEI (see Figure 3). Such a model also captures the
dynamics and persistence of emotions over time since
all manipulations, and corresponding dimensions, can
be represented simultaneously. For instance, in the
game Minecraft one common interaction among players
is to go on a mining expedition. At the start of such an
experience the cube would likely represent excitement
for all players with lots of rounded extrusions (see
Figure 4A). However, over time the players experiences
could diverge. Such as one player maintaining
excitement after finding an area with rich resources,
another feeling bored after finding nothing, and a third
angry after dying and losing supplies. This could lead to
a shape that is squashed to show boredom, spiky to
show anger, and rounded to show excitement in order
to encapsulate the full experience (see Figure 4B).
Later on, the bored player may become excited after
finding a material rich area and remove the squashing
transformation, but the third player may remain angry
from the permanent loss of resources (see Figure 4C).
We envision the CSEI being used for evaluation of non
time-critical individual and group interactions. The CSEI
may be especially useful for highly collaborative and
non time-critical games such as Minecraft. While CSEI
is similar to SEI in the breadth of its use, CSEI offers
several gains with regard to the information it can
obtain. CSEI allows us to collect rich data about how
people vary individually and in group settings when
constructing emotional objects. These objects can also
be analyzed for the persistence and dynamics of
emotion/shape over time by using the Affective Shape
Cube. Ultimately, we could create a database of shape
dynamics from the constructed objects and use mass
Figure 2. Our proposed Valence-
Arousal Cube accounts for the
experience of mixed emotions.
Negative valence, positive valence,
and arousal are placed at right
angles to form a cube.
Calm, Neutral
Smooth Surface [4]
Extrude [4], Stretch [6],
Pull, Increase Size
Squash [6], Fold
Smooth (Bumpy) [4]
Sharpen (Spiky) [4]
Symmetry or Asymmetry
(opposite current shape
valence) [4], Twist
Number of Extrusions, High
Surface Deformation [4]
Table 1. Edges of the Valence-Arousal Cube, corresponding emotions, and related manipulations.
surveying sites, such as MTURK, to obtain cross ratings
of each object's emotional signaling in order to verify
the Affective Shape Cube dimensions.
We also posit CSEI provides key advantages over
physical construction approaches such as art therapy:
Preserves pleasure of constructing objects, but with
significantly more speed and precision due to a
simple gesture and slider based tactile interface.
Provides a detailed, digital trace of dynamic gestalt
by representing and understanding shape changes
over time as a combination of affective
manipulations along the Affective Shape cube.
Effective tool for use with children and adults.
Provides a shared digital space and/or object for
collaborative, real-time construction and
communication among users.
Evokes ambiguous communication of affect among
multiple users and designers through the co-
creation of mixed emotion objects.
The Valence-Arousal Cube
Typical models of emotion are primarily concerned with
mapping an affective response to a single valence
emotion. This approach to affect has mainly resulted in
circumplex models, such as Russell’s valence-arousal
model [13] (see Figure 5), that treat negative and
positive valence as opposite ends of the same
spectrum. While such models are sufficient for many
affective measurement tools, the complex, multi-
layered, and time sensitive nature of an interactive
experience (or story) requires a more expressive
model. Specifically, we need a model that accounts for
the simultaneous experience of opposite-valence
emotions (i.e. mixed emotions [8]) and
cultural/contextual differences. For instance, a user
could enjoy manipulating interface controls of some
system but greatly dislike the sound effects associated
with their use. Traditional models and tools would likely
map the overall experience to only a positive or
negative emotion rather than grasp the concurrent,
highly positive and negative affect occurring throughout
the experience. To address this issue, we altered
Russell’s circumplex valence-arousal model by splitting
the valence dimension into separate positive and
negative dimensions. We then placed arousal, negative
valence, and positive valence at right angles to form
the Valence-Arousal Cube (see Figure 2).
The Valence-Arousal Cube offers three distinct
advantages for construction of the CSEI. First, primary
emotions from Russell’s model can still be reached
along the cube’s edges. Second, mixed emotions fall
towards the center of the cube and can be reached
through combinations of positive and negative valence.
These combinations represent affect that is ambiguous,
eliciting a context and culture based interpretation from
the user. Third, separating valence into its atomic
dimensions allows us to associate distinct affective
manipulations of shape with each dimension, resulting
in an Affective Shape Cube (see Figure 3 and Table 1).
The Affective Shape Cube quantifies ways in which
shapes can be manipulated to evoke affective response.
By transforming a shape along affective dimensions of
the cube, a variety of primary and mixed emotion
objects can be created. It is important to note that the
Affective Shape Cube only maps shape to affective
response. The actual interpretation of emotion evoked
by shape is left to the individual, context, and culture.
Figure 3. Our proposed Affective
Shape Cube couples manipulations of
shape with the affective dimensions
of arousal, negative valence, and
positive valence. Combining differing
affective transformations produces
different affective shapes.
Figure 4. Objects created during a
mining experience in Minecraft. A)
Initial excitement is formed at
beginning of expedition. B)
Frustration from dying, boredom
from lack of events, and excitement
from success are experienced
during play. C) Boredom lessens as
new resources to mine are found.
However, shapes that fall along points closer to the
faces of the cube are far more likely to have a universal
interpretation than ambiguous shapes that lay closer to
the center of the cube. E.g. the anteater and spiky SEI
objects correlate to anger and surprise trans-culturally
because the majority of shape transformations are
along arousal and negative valence dimensions, placing
them near the arousal/negative valence face. On the
other hand, the barba papa SEI object has conflicting
dimensional transformations (asymmetry vs. smooth
extrusion) which places it towards the center of the
cube and leaves interpretation more to context and
culture (see Figure 6). This may explain why the barba
papa was confusing and unused by Swedish test
subjects since there was no cultural equivalent [5].
Current Status and Next Steps
The exact details of how the CSEI tool should function
or be used during tactile evaluation scenarios is still
unclear, and even what scenarios may be best for such
a tool needs further exploration. Ultimately, we want to
examine what people create with the CSEI tool to first
refine the instrument, and then focus on exploring the
proper usage scenarios. As such, we are currently
conducting a study employing a Wizard of Oz mockup
of the CSEI system using clay. Our goal in this study is
to verify and refine the Affective Shape Cube, and
determine manipulations/gestures necessary for
individual and group utilization of the CSEI. We then
plan to prototype the initial CSEI and conduct user
evaluations of the CSEI in tests with individuals,
groups, and finally children. We should have results
from the current study, and the preliminary prototype,
to present at this workshop.
[1] Bradley, Margaret M., and Peter J. Lang.
"Measuring emotion: the self-assessment manikin and
the semantic differential." Journal of behavior therapy
and experimental psychiatry 25.1 (1994): 49-59.
[2] Druin, Allison. "The role of children in the
design of new technology." Behavior and information
technology 21.1 (2002): 1-25.
[3] Höök, Kia, et al. "Evaluation of Affective
Interactive Applications." Emotion-Oriented Systems.
Springer Berlin Heidelberg, 2011. 687-703.
[4] Isbister, Katherine, et al. "The sensual
evaluation instrument: developing an affective
evaluation tool." Proceedings of the SIGCHI conference
on Human Factors in computing systems. ACM, 2006.
[5] Isbister, Katherine, et al. "The sensual
evaluation instrument: Developing a trans-cultural self-
report measure of affect." International journal of
human-computer studies 65.4 (2007): 315-328.
[6] Johnston, Ollie, and Frank Thomas. "The
illusion of life: Disney animation."Disney Editions
[7] Laaksolahti, Jarmo, Katherine Isbister, and
Kristina Höök. "Using the sensual evaluation
instrument." Digital creativity 20.3 (2009): 165-175.
[8] Larsen, Jeff T., Yen M. To, and Gary Fireman.
"Children's understanding and experience of mixed
emotions." Psychological Science 18.2 (2007): 186-
[9] Lusebrink, Vija B. "Art therapy and the brain:
An attempt to understand the underlying processes of
art expression in therapy." Art Therapy 21.3 (2004):
[10] Muller, Michael J. "Participatory design: the
third space in HCI." Human-computer interaction:
Development process (2003): 165-185.
Figure 6. Top - anteater, spiky.
Bottom - barba papa.
Figure 5. Russell’s valence-arousal
[11] Pessoa, Luiz. "On the relationship between
emotion and cognition." Nature Reviews
Neuroscience 9.2 (2008): 148-158.
[12] Picard, Rosalind W., and Shaundra Bryant
Daily. "Evaluating affective interactions: Alternatives to
asking what users feel." CHI Workshop on Evaluating
Affective Interfaces: Innovative Approaches. 2005.
[13] Russell, James A. "A Circumplex Model of
Affect." Journal of Personality and Social Psychology
39.6 (1980): 1161-178.
The image is a basis of any piece of art. This research work envisages the nature of artistic images manifestation, the mechanism of sensory perception and creation of the image‐bearing solution in the designer project activities. The hierarchy of sensory images is represented and the stages of project image formation are distinguished. The historical examples of symbolization of archetypical foretypes and systematization of images arising in the consciousness of the designer in the course of projecting are analyzed. The relationship between the elements of project image structure is introduced, and the process of creation of the new forms such as environment project images, costume style solutions and conceptual ideas of visual data presentation is illustrated. The image concentrates the designer consciousness and forms the inner world of the personality. The language of the images appears in the copies, associations, prints, stamps, and symbols providing the visual information without a word. It is very symbolic, polysemic and contradictory. It is supported by the archetypical symbols and often connected with the ethnic culture and traditions developed in the different art genres.
Conference Paper
Full-text available
We build upon recent research designing a constructive, multi-touch emotional assessment tool and present preliminary qualitative results from a Wizard of Oz study simulating the tool with clay. Our results showed the importance of emotionally contextualized spatial orientations, manipulations, and interactions of real world objects in the constructive process, and led to the identification of two new affective dimensions for the tool.
Full-text available
In this paper, we advocate the use of behavior-based methods for use in evaluating affective interactions. We consider behavior-based measures to include both measures of bodily movements or physiological signals and task-based performance measures.
Full-text available
Factor-analytic evidence has led most psychologists to describe affect as a set of dimensions, such as displeasure, distress, depression, excitement, and so on, with each dimension varying independently of the others. However, there is other evidence that rather than being independent, these affective dimensions are interrelated in a highly systematic fashion. The evidence suggests that these interrelationships can be represented by a spatial model in which affective concepts fall in a circle in the following order: pleasure (0), excitement (45), arousal (90), distress (135), displeasure (180), depression (225), sleepiness (270), and relaxation (315). This model was offered both as a way psychologists can represent the structure of affective experience, as assessed through self-report, and as a representation of the cognitive structure that laymen utilize in conceptualizing affect. Supportive evidence was obtained by scaling 28 emotion-denoting adjectives in 4 different ways: R. T. Ross's (1938) technique for a circular ordering of variables, a multidimensional scaling procedure based on perceived similarity among the terms, a unidimensional scaling on hypothesized pleasure–displeasure and degree-of-arousal dimensions, and a principal-components analysis of 343 Ss' self-reports of their current affective states. (70 ref) (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Full-text available
This chapter surveys methods, techniques, and practices in Participatory Design (PD) that can lead to hybrid experiences - that is, practices that take place neither in the workers' domain, nor in the software professionals' domain, but in an "in-between" region that shares attributes of both the workers' space and the software professionals' space. Recent work in cultural theory claims that this "in-between" region, or "third space," is a fertile environment in which participants can combine diverse knowledges into new insights and plans for action, to inform the needs of their organizations, institutions, products, and services. Important attributes of third space experiences include challenging assumptions, learning reciprocally, and creating new ideas, which emerge through negotiation and co-creation of identities, working languages, understandings, and relationships, and polyvocal (many- voiced) dialogues across and through differences. The chapter focuses on participatory practices that share these attributes, including: site-selection of PD work; workshops; story-collecting and story-telling through text, photography, and drama; games for analysis and design; and the co- creation of descriptive and functional prototypes. Introduction Participatory design (PD) is a set of theories, practices, and studies related to end- users as full participants in activities leading to software and hardware computer products
Conference Paper
Full-text available
In this paper we describe the development and initial testing of a tool for self-assessment of affect while interacting with computer systems: the Sensual Evaluation Instrument. We discuss our research approach within the context of existing affective and HCI theory, and describe stages of evolution of the tool, and initial testing of its effectiveness.
The application of new techniques in brain imaging has expanded the understanding of the different functions and structures of the brain involved in information processing. This paper presents the main areas and functions activated in emotional states, the formation of memories, and the processing of motor, visual, and somatosensory information. The relationship between the processes of art expressions and brain functions is approached from the viewpoint of the different levels of the Expressive Therapies Continuum (Lusebrink, 1990, 1991) with examples from art therapy interventions. The basic level of interventions with art media is through sensory stimulation. Visual feature recognition and spatial placement are processed by the ventral and dorsal branches of the visual information processing system. Mood-state drawings echo the differences in the activation of different brain areas in emotional states. The cognitive and symbolic aspects of memories can be explored through the activation of their sensory components.
Methods are developed for different audiences and purposes. HCI researchers develop methods to shape the future through pure, applied and blue sky research – as is still the case with most affective interactive applications. Unsurprisingly, practitioners will be more concerned that the methods they use not only are tractable but produce better and more innovative results in terms of the systems they ultimately release into the world. Researchers, on the other hand, may have other concerns, such as the novelty of their techniques. Up until recently, most HCI methods (both for researchers and practitioners) were developed for work applications and desktop situations. They focused on efficiency, learnability, transparency, control and other work-related values. They were developed in response to a theoretical orientation which viewed the user as an information processing system not so dissimilar to the computer itself. But now that HCI is concerned with technologies that enter all aspects of life, our methods have begun to change and will need to continue to change. In keeping with our changing conception of what a “user” is and a wider concern with their experience of use of new technologies, a key challenge will be to develop and expand methods for analyzing not just what people do with the technology but how it makes them feel, and not just how people understand technology but how they make sense of it as part of their lives. Methods must be concerned, not only with issues of usefulness and usability, but also with issues of aesthetics, expression, and emotion. In addition we need to focus on evaluating technology not just in the short term under controlled conditions but also in the longer term and in broader social and cultural contexts. In this section, we will therefore provide two strands of evaluation methods. The first concerns what we might see as more traditional usability evaluation: is my system usable for the purpose it was designed for? The second strand tries to get at what we have named “third wave of HCI” in the previous chapters: does my system provide for the kind of (emotional) experience that it aimed to do?
In our research we made use of an instrument previously developed to facilitate nonverbal self-report of emotion, which consists of eight sculpted objects. We describe the use of this instrument in the assessment of three interactive storytelling experiences in a small user study and draw some conclusions about the instrument's effectiveness in supporting design.
In this paper we describe the development and testing of a tool for self-assessment of affect while interacting with computer systems, meant to be used in many cultures. We discuss our research approach within the context of existing cultural, affective and HCI theory, and describe testing of its effectiveness in the US and Sweden.