PillowTalk: can we afford intimacy?
ABSTRACT This paper describes the move.me interaction prototype developed in conjunction with V2_lab in Rotterdam. move.me proposes a scenario for social interaction and the notion of social intimacy. Interaction with sensory-enhanced, soft, pliable, tactile, throw-able cushions afford new approaches to pleasure, movement and play. A somatics approach to touch and kinaesthesia provides an underlying design framework. The technology developed for move.me uses the surface of the cushion as an intelligent tactile interface. Making use of a movement analysis system called Laban Effort-Shape, we have developed a model that provides a high-level interpretation of varying qualities of touch and motion trajectory. We describe the notion of social intimacy, and how we model it through techniques in somatics and performance practice. We describe the underlying concepts of move.me and its motivations. We illustrate the structural layers of interaction and related technical detail. Finally, we discuss the related body of work in the context of evaluating our approach and conclude with plans for future work. Author Keywords social intimacy, tactile interface, somatics, movement analysis, Laban effort-shape, tangible UIs, art/design installation, play, social interaction, user experience, ambient environment, choreography of interaction.
Conference Paper: Sharing the squid: tangible workplace collaboration.[Show abstract] [Hide abstract]
ABSTRACT: Effective communication is central in building trust and negotiating differences in diverse, multidisciplinary working environments. In this paper we discuss a tangible mediated environment designed to facilitate positive social interaction between colleagues in a research workplace. Through our multi-user tangible interface in the form of a plush squid, participants can share media resources and collaborate in a playful and inviting setting. Results from preliminary studies indicate that playful mediated work environments stimulate constructive discourse, strengthen social bonds, and enhance creative output.Extended Abstracts Proceedings of the 2008 Conference on Human Factors in Computing Systems, CHI 2008, Florence, Italy, April 5-10, 2008; 01/2008
- [Show abstract] [Hide abstract]
ABSTRACT: Studies in awareness systems tend to focus on the informational aspects of interactions. This emphasis is warranted in systems that aim to support instrumental activities, such as collaboration and coordination (e.g. Begole and Tang, 2007) or messaging (e.g. Cheverst et al., 2007). Such activities usually involve the use of information, sometimes collected by sensors, about location, status and activity. However, when awareness systems have the core aim to maintain human relationships, the benefits may come not just from the sharing of awareness information per se, but more from the simple act of exchange.12/2008: pages 173-186;
- [Show abstract] [Hide abstract]
ABSTRACT: This paper presents a new approach to make current and future television universally accessible. The proposed approach provides a means of universal accessibility both for remotely operating the TV set and for interacting with online services delivered through the TV. This proposal is based on the ISO/IEC 24752 “Universal Remote Console” (URC) standard. This standard defines an abstract user interface layer called the “user interface socket” and allows the development of pluggable (plug-in) user interfaces for any type of user and any control device. The proposed approach lays the foundation for the development of advanced user interfaces that can be interacted within various modalities. Different prototypes have been developed based on this approach and tested with end users. The user tests have shown this approach to be a viable option for the proposed scenarios. Based on the experience gathered with the prototypes, recommendations and implementation options are suggested for commercial adoption.Universal Access in the Information Society 03/2013; 12(1):73-87. · 0.40 Impact Factor
PillowTalk: Can We Afford Intimacy?
Thecla Schiphorst1, Frank Nack2, Michiel KauwATjoe2, Simon de Bakker2, Stock2, Lora Aroyo3,
Angel Perez Rosillio3, Hielke Schut 3, Norm Jaffe1
1School of Interactive Arts and
Simon Fraser University, Canada
2V2_ Institute for Unstable
Media, Eendrachtsstraat 10,
3012 XL, Rotterdam, NL
3Eindhoven University of
P.O.Box 513, 5600 MD
This paper describes the move.me interaction prototype
developed in conjunction with V2_lab in Rotterdam.
move.me proposes a scenario for social interaction and the
social intimacy. Interaction with
sensory–enhanced, soft, pliable, tactile, throw-able cushions
afford new approaches to pleasure, movement and play. A
somatics approach to touch and kinaesthesia provides an
underlying design framework. The technology developed
for move.me uses the surface of the cushion as an intelligent
tactile interface. Making use of a movement analysis
system called Laban Effort-Shape, we have developed a
model that provides a high-level interpretation of varying
qualities of touch and motion trajectory. We describe the
notion of social intimacy, and how we model it through
techniques in somatics and performance practice. We
describe the underlying concepts of move.me and its
motivations. We illustrate the structural layers of interaction
and related technical detail. Finally, we discuss the related
body of work in the context of evaluating our approach and
conclude with plans for future work.
social intimacy, tactile interface, somatics, movement
analysis, Laban effort-shape, tangible UIs, art/design
installation, play, social interaction, user experience,
ambient environment, choreography of interaction.
ACM Classification Keywords
H5.m. Information interfaces and presentation (e.g., HCI):
A growing trend within tangible and embedded interaction
is a desire to express emotional qualities through
interaction, accompanied by an interest in incorporating
movement and human perceptual-motor abilities .
Within HCI, intimacy is among an emerging set of
experiential values that also include curiosity, enjoyment,
resonance, play and self-awareness . Hummels, et al 
describe this trend as a renewed ‘respect for the human as a
whole’, and cite the shift in contextual focus of HCI from
the work place to ‘quality of experience’ in our everyday
lives. A direct reflection of this contextual refinement is the
development of interactive technologies that mediate
intimacy [14, 20, 28]. The expression of intimacy is vital in
personal and social interaction. It is reflected in the
persistent desire to create technologies that simulate touch,
body contact, and ‘near-space’ interaction , and that
communicate closeness, even at a distance . Gibbs et al
 have coined the term ‘phatic technologies’ to
emphasize the qualitative importance of non-informational
forms of exchange: interactive technologies that are less
concerned with capturing and communicating information
and more involved with establishing and maintaining social
connection. Grivas  argues similarly that assigning
aesthetic and emotional qualities to physical objects or
locations is a key strategy for the achievement of intimate
interactions. Intimacy is connected with physical
togetherness and contingency, and that intimacy can be
heightened by a ‘post-optimal’ approach to technology that
values the evocative and poetic powers of electronic media
over the urge for utility and efficiency. Gaver  reflects
on intimacy and emotional communications systems by
asserting that new forms of aesthetic pleasure can evoke a
deeper and richer experience through increased use of
unusual and sense–based materials and interactions, and
less ‘explicit’ forms of information, encouraging
imagination and expression of value and attitude.
We propose the notion of social intimacy, where the
interplay between people and a set of networked objects in
a social or public space can be used to create awareness
between others, sensitivity and more vital connection
between groups of people in a public space. As a part of the
European ITEA Passepartout project (www.passepartout-
project.org) we explore intimate ambient technologies in
the context of home, and social-intimacy in urban social
spaces, such as a lounge, café, or speak-easy. We have
developed a set of small interactive throw pillows
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. To copy otherwise,
to republish, to post on servers or to redistribute to lists, requires prior
specific permission and/or a fee.
TEI’07, February 15-17, 2007, Baton Rouge, Louisiana, USA.
Copyright 2007 ACM ISBN 978-1-59593-619-6/07/02…$5.00.
Chapter 1 - CONNECTEDNESS TEI'07, 15-17 Feb 2007, Baton Rouge, LA, USA
containing intelligent touch-sensing surfaces, in order to
explore new ways to model the interactions and experiences
of participants and artefacts within the environment, in the
context of expressive non-verbal interaction. Key concepts
investigated by move.me include intimacy, connectivity and
We apply ‘phatic’ approaches to technology, using
networked cushions to explore awareness of connection and
playful interaction. We use aesthetic properties of materials
and their innate sensuality coupled with movement and
pleasure to support socially intimate connectivity, including
interactions of empathy, peripheral awareness, and
engagement, as portrayed in Figure 1.
Figure 1. Interactive Pillow as Intimate Object – Paris ITEA
demonstration Oct 2006
We propose ambient technology that requires the ability to
detect and understand the user's activity, body state and
identity . Additionally, it has to understand the social
signals displayed in user communication, as the this is
always part of a larger social interplay .
This article first describes the motivation for our choice of
the pillow as an everyday intimate object We then outline
the underlying concepts of move.me. We illustrate an in-
depth look at the structural layers that support the
representation of context within move.me and related
technical detail. Finally, this work is placed in the context
of related bodies of work for evaluating our approach. We
conclude with plans for future work.
THE PILLOW AS AN INTIMATE EVERYDAY OBJECT
The pillow is an example of an intimate everyday object. A
pillow can express and extend a large dynamic range of
qualities of affect. Our interaction can range from affection
to ambivalence in a continuous cycle within our daily lives.
Warhol’s “Silver Floating Pillows” and Dunne’s “The
Pillow”  have contextualized the form of the pillow in
both art and design. While Philips “photonic pillow”  is
an extension of display of a ‘soft’ SMS, Dunne and Gaver
describe “the Pillow” as a soft, subtle, gentle emitter of
ambient data, beautiful and evocative, raising its issues and
its content gently, one that has a certain value fiction that
can contextualize information about our environment (in
their case: the presence of electromagnetic radiation)
through immediacy, intimacy and simple pleasure. The
pillow is familiar: it contains our memory, energetically and
physically. Pillows have a rich and evocative metaphor
space: they cushion us, bolster us when we are nervous, can
be cherished, warm, close, and friendly. A pillow keeps
secrets and shares intimate connections . The term
Pillow Talk and Pillow Book both reference this secret
internal world of the body, the sensual or even erotic
connotations that the pillow can suggest. Pillows are used
as forms of urban or folk combat: the pillow fight, a
physicalization of battle, physical play and expression of
affect. They enable both the internalization as well as the
externalization of movement, and ‘afford’ interaction that
can play or slide between these varying scales. But a pillow
is also a safe and humble object, it is held by a child for
safety, for comfort, and to ‘bring a sense of home’ along for
Our exploration of the pillow as intimate technology
embeds both digital technologies along side metaphors of
intimacy to allow us to share, edit and communicate the
evidence of our connection to reflect more subtle – or
poetic – aspects of our identity and connection through
patterns of touch, movement and being. We communicate
embodied intimacy and play through a tactile interface
embedded in the textiles and in the fabric of the cushions.
In this way both the circuit design and the fabric and textile
becomes an aesthetic component of the interactive object.
. This is also an extension of awareness technologies as
discussed in Gaver’s  reference to provocative
MOVE.ME – MOTIVATION AND SCENARIOS
Move.me is an ambient environment in which embedded
technologies act as a “connective tissue” between users and
devices within a contextualised space through domain-
specific interaction strategies.
In move.m we developed a set of small interactive throw
pillows, as portrayed in Figure 2. We utilize these pillows
within two scenarios, a home scenario where a pillow is
used by a single user in the context of digital entertainment,
and in a café or lounge environment where the ambience is
created by the dynamic social activity and interaction of
people coming and going.
Figure 2. Pillow with actuators, touchpad, LED display and
vibrator (photo courtesy of Jan Sprij).
The home scenario explores a child as the interactor with
the pillow. In this scenario the media space is understood as
an experience space, which the child can explore freely.
Thresholds can be set for this space, with respect to levels
TEI'07, 15-17 Feb 2007, Baton Rouge, LA, USAChapter 1 - CONNECTEDNESS
of excitement, as well as temporal aspects such as the
reaction time for adaptations. The aim of this scenario is to
explore a single user single object interaction and its
possibilities to influence the overall environment.
The second move.me scenario is situated in a café/lounge
environment, where participants are invited to re-mix a set
of moving images projected in large scale on the walls
within the café through physical interaction with a set of
small interactive pillows. Figure 3 portrays a setting of the
move.me environment, on the left, and an action performed
with a pillow, on the right.
Figure 3: The move.me scenario setting.
The type of purpose-free social play in this scenario forms a
kind of choreographic experiment in which the result of
interaction with the pillow creates movement on at least
three levels: 1) the movement of the participant as they
interact with the pillows (touching, caressing, throwing,
hitting, holding); 2) the movement of the pillows
themselves as a result of the participants interaction; and 3)
the movement of re-mixed images derived and
rematerialized in direct response to the public intervention.
The focus of this work lies on the representation of
movement in an environment that is aware of users and
objects but not necessarily knows much about them. This
work integrates somatics  and gesture interaction 
with textiles and interactive object design . A detailed
description of this environment can be found here .
SENSORS, ADAPTATION AND FEEDBACK IN MOVE.ME
The main idea behind move.me is to establish an
environment, which constantly collects raw data from
various modality-sensitive objects that is then
communicated to a context engine. The context engine
interprets the derived parameters to manipulate in turn the
presentation of audio-visual material displayed in the
environment as well as the overall ambience of the
environment itself, e.g. by manipulating light and sound
As a result we developed an interaction model that involves
• The user.
• The interface, which in our case is a conceptual unit
containing the interactive pillow as input sensor and other
devices, such as vibrator, fan, light-emissive fibre, light-
emissive diode, earphones, screen, sound system, lamp, et
cetera, as output sources.
• The context engine as a back end server.
With respect to its interaction part the model extends Don
Norman’s traditional execution-evaluation model 
beyond the user’s view of the interaction by including not
only the interface but all the elements necessary to judge
the general usability of the interactive system as a whole.
This allows placing the move.me environment in different
social contexts with an overlap on a particular task.
With respect to the contextual aspects of our interaction
model we adopt strategies from case-based reasoning
(CBR) , in particular those strategies which argue that
tracing the history of actions  provides the means to
improve a systems capability to adaptively interact with a
user. We establish a set of raw data, on which we then
elaborate based on user and environmental data (both
together form our context) to perform contextualized
adaptation. The adaptation as well as the context it was
performed in are then stored and will be used in the
ongoing process of ambient user adaptation for further
In the remaining part of this section we will outline the
various modules of the move.me environment, namely:
• Sensors, which are the input sensors of the pillow.
• Sensor evaluation module, which instantiates the device
drivers for every detected pillow. Its main task is to
perform some statistical analysis (high-pass filtering and
mean value calculations) in order to keep the overhead of
• Context module that consists of data structures describing
the current context with respect to users, devices and the
interactions between them.
• Adaptation engine, which uses data from the Context
module to establish a mapping between detected actions
and the appropriate environmental adaptation. It also
conveys instructions about the source to be adapted and
the means of adaptation to the Communicator.
We are aware that a number of the described action efforts
as well as resulting adaptations could also and probably
should be detected by other devices than the pillow.
However, for the sake of clarity we explain the mechanisms
through the pillow.
Pillow sensors and their evaluation
The objectives of the technical research in move.me are
twofold. First, we want to explore smart fabric textiles in
the context of flexible electronics and displays in order to
build these into a wireless network, capable of making body
data available. Second, we wish to develop heuristics of
interaction based on touch, gesture, and movement to infer
action efforts from users while utilising the device and to
manipulate this raw data to enable a higher-level mapping
of action efforts and presentation manipulation techniques.
Using raw data from smart fabric textiles allow qualitative
recognition for two categories of movement: 1) touch on
Chapter 1 - CONNECTEDNESS TEI'07, 15-17 Feb 2007, Baton Rouge, LA, USA
the surface of the pillow, and 2) movement in three-
dimensional space created by the ‘free-throwing’ of the
pillow. We call our touch selection input “threads of
recognition” because it refers to metaphors of input
recognition in the context of smart textiles research .
Pressure is the essential type of data we process to extract a
caress and its effort. We have identified a set of parameters
that can be extracted or calculated from the information that
the response area provides over time. These parameters are
described in Table 1. At the moment we utilise the values
for pressure, size, speed and direction as input parameters.
A pillow in move.me is the main medium for smart fabric
textiles, and can be equipped with any of the sensors and
actuators as displayed in Figure 4.
Figure 4: Pillow technology
The main sensors we apply are a touch-based interface for
measuring pressure and accelerometers to measure motion.
Fans, vibrators and light sources, either in the form of light
emissive fibers or as a LED display, provide localized
The touch-pad itself is a simple grid (64 sensitive regions)
of carbon-impregnated open cell polyethylene foam. This
material has the characteristic that the electrical resistance
of the foam drops as the density of the foam increases. We
utilize this behaviour to identify a point of contact or
The processing unit is a small, lightweight, single-board
computer, which connects to the accelerometers, and a
sensor board card, which can measure up to 64 pressure
sensors. The computer filters the incoming data and
communicates at a rate of approximately 20 Hz to the
server application, where the interpretation of the data is
performed. When switched on, the pillow propagates its IP-
address and port number on the network, which allows
listening programs to detect and register the pillow and start
receiving or sending messages.
For identifying users in the closer surroundings of the
pillow each pillow contains an RFID reader/writer,
allowing us to write our codes into programmable RFID-
Tags, which are embedded within each pillow or worn by
users of the environment as bracelets.
pressure soft, hard The intensity of the touch.
time short, longThe length of time a gesture takes.
size small, medium, big
The size of the part of the interaction object that
touches the pad.
The distinction between one finger or object and many
speednone, slow, fast
The speed of a touch-effort. This is the overall velocity
of movement. This parameter is not used directly to
distinguish efforts, but is used to determine space.
none, left, right, up,
down, and four
The direction of movement. This parameter is not
directly used to distinguish efforts, but is used to
determine space and path.
space (speed) stationary, travelling
A function of speed. If speed is zero then the gesture is
stationary, otherwise it’s traveling.
If the speed is not zero, and there is only one direction
registered, the gesture is straight.
If the pressure maintains a single value after an initial
acceleration the gesture is constant, otherwise it’s
If a gesture is unique in relation to the gesture
immediately before and after, it is continuous. Any
repeated action or gesture is classified as repetitive.
Table 1: Parameters derived from pressure pad data
Emphasizing the sensual aesthetic of a pillow, covers are
designed to encourage connection through feel in an
associative and intuitive way. Pillow prototypes are
portrayed in Figure 5.
Figure 5: Prototype pillows
Pillowcases are made of silk organza , a conductive
fabric. In move.me, we use this material as cables that send
data signals from the touchpad to the embedded processing
unit. In that way we achieve that users, such as children, not
only interact naturally with the pillow but also whish to do
so, as the surfaces of textiles or light-immersive material
asks for touch. Moreover, all the required hardware is
lightweight and wrapped in soft material to avoid edgy
sharpness that could destruct the intimate character of the
The pillow serves mainly as an affectionate transmitter that
provides a basic analysis of the signals, e.g. calculating time
and space variables, which are then sent out to the central
system along with additional information, such as which
other pillows or users are near to this pillow.
Once the data for pressure, size, speed and direction is
received by the central system, the sensor evaluation
module performs a first abstraction on the data. The way
that incoming sensor data is analyzed depends on the
context and the configuration of a pillow. For example, if
the pillow contains a pad that allows measuring pressure as
well as galvanic skin response (GSR) and an accelerometer
TEI'07, 15-17 Feb 2007, Baton Rouge, LA, USAChapter 1 - CONNECTEDNESS
for computing movement, a different abstraction scheme is
provided as output compared to a pillow that only contains
a pressure pad. This means that the system handles every
pillow individually, as it also does for users. Only at a later
stage are the individual views combined to the global
When analysing streams of data, in this case the input data
for every taxel of the pad, it is desirable to keep a history of
past events especially when we are looking for trends in the
data . However, storing the whole dataset and re-
iterating over the last n samples whenever a new sample
arrives quickly becomes inefficient as n grows. We,
therefore, apply a method that keeps a history of past events
without actually storing them but summarizing instead the
entire set of (or the last n) past events in a few critical
For the detection of the pillow movement we analyse the
data coming from a pillow’s accelerometer. The basis of the
recognition is a distinction between linear movements in
three directions, clockwise and counter-clockwise circular
movements, as well as rectangles and triangles described in
space. The collected raw data is interpreted as vectors, and
the input vector (the “raw” acceleration vector) is filtered
and further processed to subtract the influence of gravity,
and to yield “Position” and “Orientation” vectors. The
“Motion” vector (i.e. total acceleration - gravity =
acceleration caused by movement) is passed on to the
neural network for the analysis of pillow motion (twirl, pan,
tilt). The interpretation of the sensor data depends on the
context in which it was collected.
Context in move.me
Context in move.me describes an area, namely a living
room (home entertainment context) and the lounge (café
context), in which users interact with the pillows.
The Context Module describes the current (present) status
of the environment with respect to resident users, devices
and the interactions between them. Users as well as devices
become part of the Context Model once their RFID is
detected. For each detected user or object a memory
structure is established that reflects only those
characteristics that are relevant for the current context. User
characteristics are, for example, the user identifier, relevant
thresholds, the current biometric status as well as related
presentation devices. Device characteristics are, e.g. the
device's sensor set, its affector setup, the device IP, activity
state and location as well as preferences for particular users.
That we provide devices with a memory structure that is
similar to that of human users is so that they themselves
become proactive towards users. At the moment we only
store very simple data, such as user id, action performed
and its duration, but later on we wish to explore further in
The memory structures are rather static schema with which
it would be difficult to observe the dynamics of the
environment. We introduced, therefore, the concept of a
session, which monitors the interactions between a user and
device or between devices. A session is a structure
containing the ids of the two agents involved, the general
start time of the session, the end time, the actions performed
and resulting status reports (e.g. sensor values).
A session is instantiated by the Context Module once a
device detects the user id and its sensors show some level
of interaction. Sessions between devices are instantiated if a
device in another session acts as a meta-device. Example: a
child might hug a pillow (session A: child1 – pillow23)
while it watches TV but actually operates through the
pressure it performs on it the presentation of the program
(session B: pillow23 – loudspeaker). Sessions are closed,
depending on the device, either once the interaction stops or
if the user leaves the context. In cases where more than one
user is detected by a device the one using the device the
longest is considered the prime user. If the prime user
leaves the context then the next longest user in the list takes
over. Once a session is terminated it will be stored in the
The History Model is our approach towards an
individualized long-term memory of the interaction patterns
for every user and device in a context. It is updated if a user
or device exits the context or if a session has been
terminated. The model contains, at the moment, two
memory sets, namely identification and session. The
identification set is always instantiated once a user enters a
context. This set serves as a crosscheck source for the
Adaptation engine to evaluate user behaviour (it might turn
out that the user attends certain contexts, thus shows
interest, but does not act in them – no sessions with this
user id in the same time span). The session set describes
every interaction the user or device was involved in. The
data set stores the collected biometric data and the
adaptation list contains the adaptations performed by the
Adaptation engine based on the data in the same time
frame. At the moment we keep track of sessions, and thus
make them accessible to the Adaptation engine, in the form
of a relational database. The outlined representation
structures serve as sources for the Adaptation engine to
determine if an adaptation is required and which type of
adaptation needs to be performed.
Adaptation and Feedback
In move.me the Adaptation engine uses a finite state
machine (FSM), where the session structure and
descriptions of the Context module are used as to represent
the states. Changes of theses states are triggered through the
input devices’ touch pad and accelerometer. The transitions
are based on the constraints set for the context as well as in
the user models. Actions finally describe the adaptation that
is to be performed at a given moment, either in the form of
the adaptation of a pillow’s actuators, such as vibrator, fan,
or light-emissive diodes, or the performance of presentation
devices in the environment, such as a change of the noise
Chapter 1 - CONNECTEDNESSTEI'07, 15-17 Feb 2007, Baton Rouge, LA, USA
level. Each context can be understood as a set of possible
actions and moods that then again trigger certain
adaptations. Thus, the organisation of rules in move.me is
based on context scripts.
The Adaptation engine constantly evaluates the Context
module for every identified user and device and reacts on
changes only if they are outside the provided constraint set.
In move.me adaptation focuses on three major processes,
namely stimulation, relaxation and representation.
Stimulation describes the attempt to either engage a non-
active user into an interaction with the environment or to
increase a low-base activity. Relaxation tries to reduce the
amount of activity or excitement. Representation aims to
present the state of the environment and the user in a visible
and audible form.
A typical situation for stimulation in the home scenario is,
for example, if the child is in the living room but either
does not interact with anything, i.e. simply sits on the sofa
(the child is detected by the system but no session is
established); or the child might hold the pillow but does that
for a long time without changing neither effort nor gesture
(there is a session instantiated but the changes of values are
infrequent and generally low).
If the Adaptation engine cannot identify a session it tries to
engage the child. The first step is to investigates which type
of devices are available that are equipped with actuators
that provide means to connect with the child (e.g. all types
of global actuators, such as LEDs, emissive fibers,
loudspeakers, etc.). Comparing the neighbourhood relations
between these objects and the user the adaptation can
activate the closest non-active pillow to start an interaction
with the user. However, if any of the pillows already has a
preference for the child, as represented in the pillow’s user
model, or the child has a preference for any of the pillows,
this particular pillow will be instantiated to become active.
In that way our system tries to utilise already established
relations between objects and users. The start of an
interaction can begin with already established pattern, such
as the pillow shows known visual pattern that invite the
child to hug the pillow. Once the contact is established the
adaptation engine will use other established relations to
stimulate further interest, e.g. switching on a TV or radio
A different type of stimulation is the detection of potential
non-interest. Assuming the adaptation engine discovers
over a period of time (a constraint determined by the
context) a steady decrease of one of the threshold values, it
might determine that the user is bored, and it may then
activate both an icon on the LED matrix in the centre of the
touch-pad, as well as causing a vibrator within the pillow to
generate a shiver-like action. This pattern might also be
used to instantiate a change of context, for example the
change from TV mode into game mode.
The order of rules for the Adaptation engine is established
based on the current state and the outcome of tracking the
performance of the instantiated adaptation. In the example
of the change between the state of watching TV and playing
a game the adaptation engine would not launch the game if
the child had not responded with acceptance of that change
in time. In case the child ignored the suggested game, the
Adaptation engine would try another strategy, e.g.
instantiate a change in the environment, like increasing the
volume to attract attention.
The dynamic interpretation of user actions and its efforts as
well as pillow movement results in a change of visual and
auditory patterns as well as task contexts, which in turn
might stimulate new associations in users, resulting in a
behaviour that might require that new adaptations be
performed by the system. The result is a constant feedback
loop where the data from a pillow triggers the interpretation
mechanism, which directly affect the audio-visual outputs
of the system and vice versa.
We ran a small user study on the initial prototype of the
system, consisting of a qualitative elicitation study, in the
form of a one-day participatory workshop with 10 users,
which was designed to explore user needs and system
requirements. The test users were representatives of the
target user group, 3 females and 7 males between the ages
of 20 and 30. Aiming to gain an insight into interaction
patterns, the workshop covered:
• A hands-on free exploration session with a medium-
fidelity pillow prototype (the hardware as well as
adaptation software for different modalities worked in
real time). The free exploration sessions were conducted
first from an individual starting point, and latter on a
• A “Wizard of Oz” simulation of the complete intended
functionality of the system.
The participants’ experiences with the pillow through the
free exploration of the move.me system, and the discussion
during the ‘Wizard of Oz’ sessions were both video taped
for later analysis. A detailed evaluation of the workshop is
described here . The major findings of the workshop
• The current architecture is stable.
• Even though the adaptation of iconic messages on the
LED display responded too slowly, causing in some users
the impression that the icons were randomly generated,
the initial exploration interaction pattern lead to the users'
full engagement with the system.
• The way the gestures were performed by the users
implied that they do not make a distinction between
gestures based on the size of the area on the touch pad
occupied by the gesture. For example a tap and a slap
meant the same to them. A more important mean of
distinction was, however, the number of repetition of
TEI'07, 15-17 Feb 2007, Baton Rouge, LA, USAChapter 1 - CONNECTEDNESS
movements or the strength (or intensity) in which they
We are not only aware of the fact that the pillow is still a
limited prototype (even if the current system is stable), but
also that the test sample is rather small. That is the reason
why, at the moment, we were only able to perform a
qualitative evaluation of the system. This means that the
findings should be taken as general guidelines, which will
allow us to make educated decisions from a user centric
point of view for further developments.
A presentation of the pillow functionality at the ITEA
symposium in October 2006 showed that the adaptation
based on stimulation, relaxation and representation is
sufficiently sound to let users understand how the
environment reacts to their actions. However, the
performed user sessions are again too short to provide
significant statistical data about the effectiveness of our
suggested context representation. Regarding possible
quantitative tests, we are aiming for a later workshop as the
basis for fine-tuning the system before the public
presentation of the installation in spring 2007 at the Dutch
Electronic Arts Festival.
In our work we apply sensing and biofeedback technologies
to establish a new way of interpreting human movements in
real-time to enable expressive non-verbal interaction in the
context of ambient, public, urban, social spaces. This
section includes a summary of background and informing
There has been a great deal of general research in sensing
and biofeedback in human-computer interaction [2, 8, 19,
20] Although we use these technologies in different
environments, this research has indicated that a number of
well established sensory methods, such as pressure and
GSR, obtain a window into the state of an individual.
Buxton et al  provides early descriptions of the unique
characteristics of touch tablets relative to other input
devices such as mice and trackballs. Chen et al 
describe the use of a touch-sensitive tablet to control a
dynamic particle simulation using finger strokes and whole-
hand gestures, where the gestures are interpreted as a form
of command language for direct manipulation. The fabrics
used in our work differ, though, as they can be multiple-
touch. In our work we go beyond direct manipulation by
language to including the quality effort into the recognized
The Laban notation , which we use in the public, urban,
and social space scenario of move.me, has often been used
to interpret user movements, especially in interactive
artistic settings. Badler  presents a digital representation
of the specific Laban notation. Zhao  has applied Laban
Movement Analysis (LMA) to studies of communication
gestures. Within move.me we interpret gestures, or rather
users’ effort, for establishing communication rather than
precise communication acts. Schiphorst et al  describe
the use of kinematic models to represent movements.
Calvert et al  further describe the development of the
composition tool into the product Life Forms, which uses
Laban notation as the representation language. A computer-
based graphical tool for working with the similar Benesh
Movement Notation is described by Singh . The current
work was influenced by the choreographic approach to
motion description and presentation of these studies.
Haptics and touch have been explored by many researchers.
The University of Tsukuba has also developed a great
number system that makes use of haptics such as
finger/hand manipulation and locomotion . Although
these systems use different technologies, they have
provided us with a motivation for the usage of touch.
With socio-ec(h)o  we share the notion of play. Just like
move.me, socio-ec(h)o explores the design and
implementation of a system for sensing and display.
However, socio-ec(h)o bases its interaction models on
existing serious game structures, where body movements
and positions must be discovered by players in order to
complete a level and in turn represent a learned game skill.
In move.me the emphasis is more on the entertaining,
purpose-free aspect of play than on the learning of skills.
CONCLUSION AND FUTURE WORK
We have described move.me, an ambient environment in
which a set of small interactive throw pillows containing
intelligent touch-sensing surfaces allow the exploration of
new ways to model the environment, participants, artefacts,
and their interactions, in the context of social intimacy
through expressive non-verbal interaction.
The novel aspect of move.me is the approach to map efforts
of actions to higher-level adaptation activities, which opens
the mapping space between biometric data and its potential
meaning. Though the first prototype shows promising
results, we have to provide significant improvements with
respect to adaptation response time as well as the range of
adaptations to facilitate an experience-rich environment that
reflects the broader motions of social interchange. We also
have to fine-tune the relations between context, action and
presentation modalities and the relationships between
intimacy within the context of a one-to-one connection and
within a group.
We consider move.me as a platform for the study of new
forms of ambient-based interaction that integrate networked
connectivity, in the context of social intimacy, and intend to
explore this avenue further.
The presented work is funded by the ITEA Passepartout
project (ITEA 10001895). The authors wish to thank the
Passepartout consortium, in particular Keith Baker, for
providing the space for our investigations. We also wish to
thank Anne Nigten and Siuli Ko-Pullan from V2_, the
Institute for the Unstable Media, for their generous
intellectual, logistic and administrative support during the
Chapter 1 - CONNECTEDNESSTEI'07, 15-17 Feb 2007, Baton Rouge, LA, USA
1. Aamodt, A. and Plaza, E.: “Case-based reasoning:
Foundational issues, methodological variations and system
approaches”, AI Communications, 7, 39-59, 1994.
2. Aarts, E.: “Ambient Intelligence Drives Open Innovation”,
interactions, July-August 2005, pp. 66-68.
3. Aroyo, L. et al: “Personalized Ambient Media Experience:
move.me Case Study”, Submitted to International Conference
on Intelligent User Interfaces (IUI) 2007, Hawaii, January 28th
– 31st. 2006.
4. Badler, N. and Smoliar, S.W.: “Digital Representation of
Human Movement”, ACM Comp Surveys, Process of
Composition, Proceedings of the SIGCHI 1979.
5. Berzowska, J.: “Memory Rich Clothing: second skins that
communicate physical memory”, Proceedings of the 5th
conference on Creativity and Cognition C&C 05, London,
April 12-15, 2005, pp. 32-40.
6. Blythe, M.A., Monk, A.F., Overbeeke, K. and Wright, P.C.
(eds): Funology: From Usability to Enjoyment. Kluwer
Academic Publishers, The Netherlands, 2003.
7. Buxton, W.: Gesture and Marking. Chapter 7 in Baecker,
R.M., Grudin, J., Buxton, W. and Greenberg, S. (Eds.),
Readings in Human Computer Interaction: Toward the Year
2000, Morgan Kaufmann Publishers, San Francisco, 1995.
8. Calvert, T.W. and Mah, S.Y.: Choreographers as Animators:
Systems to support composition of dance, in Interactive
Computer Animation, N. Magnenat-Thalmann and D.
Thalmann (eds), pp: 100-126, Prentice-Hall, 1996.
9. Champin, P.A., Mille, A. and Prié, Y.: “MUSETTE:
Modelling USEs and Tasks for Tracing Experience.” ICBBR
’03 Workshop, “From structured cases to unstructured
problem solving episodes” NTNU, 2003, 279-286
10. Chen, T.T.H., Fels, S.S. and Min, S.S.: “Flowfield and
Beyond: Applying Pressure-Sensitive Multi-Point Touchpad
Interaction”, Proceedings IEEE Intl Conf on Multimedia and
Expo, Baltimore, 2003.
11. Dunne, A. and Gaver, W.W.: “The Pillow: Artist-Designers in
the Digital Age”, CHI Proceedings, March 1997, pp. 361-362.
12. Fayyad, U.M., Piatetsky-Shapiro, G. and Smyth, P.: “From
Data Mining to Knowledge Discovery in Databases”, AI
Magazine 17(3): 37-54 (1996).
13. Gaver, B.: “Provocative Awareness”, Computer Supported
Cooperative Work (CSCW), Volume 11, No. 3-4, September
2002, pp. 475-493. Kluwer Academic Publishers, The
14. Gibbs, M.R., Vetere, F., Bunyan, M. and Howard, S.:
“Sychromate: a phatic technology for mediating intimacy”,
Proceedings of the 2005 conference on Designing for user
experience DUX’05, November 2005, pp 2-6.
15. Goleman, D.: Emotional Intelligence. Bantam Books, New
York, NY, USA, 1995.
16. Grivas, K.: “Digital Selves: Devices for intimate
communications between homes”, January 2006, Personal and
Ubiquitous Computing, Volume 10 Issue 2, Springer Verlag,
17. Hanna, T.: Bodies in Revolt: A Primer in Somatic Thinking.
Freeperson Press, 1985.
18. Hummels, C., Overbeeke, K. and Van der Helm, A.: “The
Interactive Installation ISH: In Search of Resonant Human
Product Interaction”, in Funology: From Usability to
Enjoyment, Blythe, M.A., Monk, A.F., Overbeeke, K. and
Wright, P.C. (eds), 2003, Kluwer Academic Publishers, The
Netherlands, pp 265-274.
19. Iwata, H.: “Art and technology in interface devices”,
Proceedings of the ACM symposium on Virtual reality
software and technology VRST '05, Monterey, CA, USA,
2005, pp. 1-7.
20. Kaye, J.: “I just clicked to say I love you: rich evaluations of
minimal communication”, CHI 06 extended abstracts on
Human factors in computing systems CHI’06, April 2006,
Montreal, Canada, pp 363-368.
21. Laban, R. and Lawrence, F.C.: Effort: economy of human
movement, 2nd ed. MacDonald and Evans, 1973.
22. Norman, D.A.: The Design of Everyday Things. Doubleday,
New York, NY. 1988.
23. Philips Research: “Photonic pillow”,
24. Post, E.R. and Orth, M.: “Smart Fabric, or Washable
Computing”, First IEEE International Symposium on
Wearable Computers, Cambridge, 1997.
25. Schiphorst, T.: “Affectionate Computing: Can We Fall In
Love With a Machine?”, IEEE Multimedia, Vol. 13 Number 1,
January – March 2006, pp. 20-23.
26. Schiphorst, T., Jaffe, N. and Lovell, R.: “Threads of
Recognition: Using Touch as Input with Directionally
Conductive Fabric”, Proceedings of the SIGCHI conference on
Human Factors in computing systems, April 2–7, 2005,
Portland, Oregon, USA.
27. Singh, B., Beatty, J.C., Booth, K.S. and Ryman, R.: “A
Graphics Editor for Benesh Movement Notation”, Computer
Graphics and Interactive Techniques, ACM Computer
Graphics, Detroit, 1983.
28. Vetere, F., Gibbs, M.R. et al.: “Mediating Intimacy: designing
technologies to support strong-tie relationships”, Proceedings
of the SIGCHI conference on Human Factors in computing
systems, Portland, Oregon, 2005, pp. 471-480.
29. Wakkary, R., Hatala, M., Lovell, R. and Droumeva, M.: “An
ambient intelligence platform for physical play”, Proceedings
of the 13th annual ACM international conference on
Multimedia, Singapore, 2005, pp. 764 - 773.
30. Zhao, L.: Synthesis and Acquisition of Laban Movement
Analysis Qualitative Parameters for Communicative Gestures.
PHD Thesis, CIS, University of Pennsylvania, 2001.
TEI'07, 15-17 Feb 2007, Baton Rouge, LA, USAChapter 1 - CONNECTEDNESS