Conference PaperPDF Available

To Challenge Textile with Music

  • Norwegian University of Science and Technology, Gjøvik, Norway


Content may be subject to copyright.
To Challenge Textile with Music
Birgitta Cappelen
Arts and Communication / Design Sciences
Malmoe University / Lund University
Anders-Petter Andersson
Interactive Sound Design / Musicology
Kristianstad University / Gothenburg University
In this paper we present some of the challenges we
approached in order to design and develop interactive
experience environments in textile. Our interactive
environments are created to facilitate communication on
equal terms through music, between children with
severe disabilities and their families. We call these
environments “fields” because of their openness
towards many interpretations, interaction forms and
activity levels. We present some of the problems,
experiments and explorations we made in textile, when
designing these fields. We show how we designed to
motivate musical actions, like playing an instrument,
improvising with the field as a co-musician, and
experiencing it as a “soundscape”.
Interaction design, ubiquitous computing, physical
computing, music, ambiguity, open, smart textile de-
The ground for this paper is a research and development
project called “iFields”1. The project goal is to develop
interactive, audio-tactile experience environments for
children with severe disabilities and their families. We
wanted to make possible collaboration and communi-
cation on equal terms, between all family members in
their own homes.
Today most interactive media, developed for our
target group, has the goal to exercise something, under
supervision of a parent or teacher, usually using screen
and some sort of keyboard. So, in the use situation, there
are always one that knows how to perform a task and
one, the child, who does not. When interacting with the
media this hierarchy maintains and the asymmetric
power relation between them. There are few media that
offers collaboration possibilities between all the
members of a family, where the participants can choose
their role, individual focus and level of participation.
That is our goal when creating our experience
Our experience environments are interactive
multimodal media, also called multi-sensoric
environments [8]. They consist of tactile interfaces
mostly soft and in textile, combined with light (LED,
Fiber optics etc.) and other materials. The output side
always include dynamic video graphics as “living
tapestries”, light and interactively generated music. The
speakers are part of the physical environment and offer
vibrations as a physical user experience, in addition to
music. Since few children in our target group speak,
music is an ideal language for all our users to commu-
nicate in.
The experience environments are designed as
ambiguous [1] hybrids, between toys and furniture, to
attract both children and grown ups. In musical terms;
The interactive environments are hybrids between
musical instruments to play on, musical actors [5] (e.g.
co-musicians) to communicate with, and ambient
soundscapes [9] to be in.
Figure 1. A family playing, communicating and
experiencing the interactive field “Vinings”.
Permission to take digital/hard copies of all or part of this material
for personal or classroom use is granted without fee provided that the
copies are not made or distributed for profit or commercial
advantage, the copyright notice, the title of the publication and its
date appear, and notice is given that copyright is by permission of
the University College of Borås, (CTF, The Swedish School of
Textiles). To copy otherwise, to republish, to post on servers or to
redistribute to lists, requires specific permission and/or fee.
Ambience’08, Borås Sweden.
1 The theories and interactive media we use in iFields [4] are created by
MusicalFieldsForever’s members [7] at Malmö University, K3 and
Interactive Institute from 2000 and onwards. From 2006–08 Lund
University has hosted the project with finance from the Swedish
Inheritance Fund for iFields’ target group and use.
The ambiguous design invites the user to make many
different interpretations, which are socially negotiated,
changing over time. We call our environments open
“fields” [2], because they simultaneously offer high
degree of potentiality by being open for many interpre-
tations, offer many interaction forms and activity levels.
Users can choose level of activity individually, if they
should interact casually or repeatedly, fast or slow. They
can also decide what role to play, if they, for instance,
should be musicians making music, friends
collaborating or playing games. They can also choose
between different forms of interaction, for instance to
speak, or move the whole body, the head, or just a
finger. Or simply sit in the field sleeping or reading. Our
fields do not divide between right or wrong interactions,
or reward game values like frequency, specific sequen-
ces, speed or activity level. It’s about communicating on
equal and individual terms, through music.
We believe that by designing these interactive
environments as open fields, we can tear down
hierarchies between people with and without
disabilities. A full discussion of the results of our user
testing regarding the project goals, and of open design
are too extensive to describe here, but will be presented
in forthcoming papers.
Within music, “musical gestures” is a common term,
referring to how a musician interacts with a musical
instrument, as well as to the form of the sound his
activities cause. We used them as design input, to
encourage musical actions in our fields.
We made use of three categories of musical gestures,
and complex combinations of several gestures [3]:
Iterative gestures, when a user interacts many times,
following a rhythmic pulse. For instance when hitting a
gong that swings back and forth, encouraging repeated
Sustained gestures, like when a Cellist’s continuous
bow movement on a string creates an unbroken tone.
Impulsive gestures, when a user make a short
explosive action, such as hitting a drum.
Complex gestures, combining two or many categories
of gestures. For example when many people play
together in a band with sustained, impulsive, iterative
movements, hitting, shaking, rubbing, rolling, holding,
singing, etc.
When performing traditional music, the sounds are
created mechanically caused by the action. Working
with electronic and interactive music, controlled by
software, other design possibilities is at hand. A weak
clap, for instance, can sound as a full orchestra starting
many seconds after the initial clap. It is not necessary a
relation between interaction and response. The relation
is controlled by software. We use this possibility
because some of the children are very weak, while their
sisters and brothers might be stronger. In the software
we can make a weak movement respond with a strong
answer and the opposite to level out the differences, and
encourage collaboration on equal terms.
The interactive field can simultaneously be experien-
ced as a musical instrument and a musical actor that
takes active part, and drives the collaboration and
improvisation forward. The advanced software program
listens and learns, in order to create musically satisfying
response. Since we design for the home situation the
field have to be experienced by the users as motivating
over very long time. Hence, we use principles of
musical variation and offer the user many roles, forms
of interaction and activity levels. So the user can role,
hit, jump, clap, stroke, throw, hold, bend, all at the same
time in the field. And these actions the user can combine
with traditional musical iterative, sustained, and
impulsive gestures.
The fields we develop consist of mobile wireless
modules, so called sensor networks, with the possibility
to combine modules into larger landscapes and thereby
offer new possibilities of interaction and experiences.
For example a child considered a small module or part
being a "friend" to play with. But when modules were
linked together, the child acted as if the modules were a
big soft landscape in textile with light and sound to dive
into, like a big bed or ocean. Divided, the modules were
distributed as ambient, layers of potentiality through out
the home.
In addition we chose to work with users with very
special needs, such as the need to get help in order to
uphold ones head, and where the head’s movement
could be the only possibility for activating a sensor.
Textile was of many reasons the obvious choice of
material for our purpose, since the interactive fields we
create are hybrids between toys and furniture, to be
interpreted and experienced as musical instruments, co-
actors and soundscapes.
We wanted textile that was stretchable, in order to
give physical response and inspire iterative gestures and
rhythmic interaction. Further we wanted textile with a
surface that invited to many forms of interactions, like
hitting, clapping and hugging. And we needed textile in
a wide range of colours, in order to attract all in our
target group and support different functions. The textile
also had to be robust and light weight to endure the
different forms of interactions and many activity levels,
from iterative hitting, impulsive jumps by a child to
sustained relaxation by a grown up (Figure 1 & 2).
We searched among all kinds of textiles and related
materials, such as neoprene, lighting materials, and
conductive textiles on the marked. The conductive
textiles available didn’t offer a satisfying colour palette,
and the attractive surface qualities we needed. Most
furniture textile was not stretchable enough in order to
stimulate interaction. We ended up using textile for car
interiors and high-tech materials for sportswear (soft-
shell, Lycra, etc.) and stretch velour. We worked in
parallel on the textile investigations and experiments,
designing shapes and developing sensors. All, in order
to encourage the user to perform musical actions.
We searched among all available sensors, and worked
for a long time with bendable digital sensors,
microphones, capacitive sensors and accelerometers
(motion detectors).
We found accelerometers to be interesting because
they could register if a user was present, and also detect
movements in three axes (x, y, z). This was a
combination of features that was important, since we
wanted to detect everything from tiny to large user
movements. An accelerometer is small and can easily be
incorporated in the interface, so that it register if the
user for example shake or hit as if playing an instrument
However, we found it difficult to use for gentle touch in
the fashion of caressing a friend or a child. But in the
software we could control the sensitivity so that it
considered different users’ level of activity and strength,
which was necessary for our target group that varied in
size, strength and ability to move.
Capacitive sensors using metal as detector covered in
textile, also offered very attractive possibilities for our
purposes. The use of metal made it possible to give the
sensor different surfaces, sizes and structures,
encouraging musical actions. The capacitive sensors
offered the multitude of interaction forms we wanted for
the user to experience the environments as musical
instrument, musical actors and soundscapes. The users
could for instance hit impulsive, once or twice, and
iterative, repeatedly, as well as stroke sustained,
continuous, over the surface. We sewed metal on textile,
threaded metal into textile, and in-and-out of textile to
design different user surfaces for capacitive sensors.
Metal surfaces are aesthetically very interesting but
became too hard and unpleasant for our user group. We
created earthed conducting sensors, investigated
different lengths and dimensions of capacitive sensors
and experimented with the interactive potential of short
Figure 2. Interacting with the tail and “wire curl”.
What we found most interesting from an aesthetical,
interaction and a functional point of view was to stretch
Lycra-like textile around a metal wire. The wire
functioned as a capacitive sensor since the textile
became very thin, when it was stretched along the wire.
The stretched textile deformed the wire into a spiral
shaped curl, and offered a physical response to
interaction, caused by its flexing firmness. In addition it
was robust and had an attractive use surface. The wire
sensor functioned satisfying when being hit, both with
impulsive and iterative movements, like a string on a
musical instrument. But, because of its exciting shape,
the user also found it pleasant to follow along the wire
curl in a sustained manner, like following a string on an
instrument or along a “living” pleasant shape. We
experimented with different filling to stuff the wire curl
in order to make it maximally ambiguous, more string
like as an instrument and more body like as an actor.
Our biggest problem was the relation between the
need for earth of capacitive sensors, location and
wireless modules. We could not fill the inside of the
module with metal to make a stable conductive earth,
because it would put the wireless network out of work.
After many rounds of experiments, material
explorations, interaction and user tests we have
collected some experiences. Experiences we will
develop further, when designing interactive media in
textile, that motivate musical actions. Some are related
to form and structure, whole and parts. Some
experiences are related to surfaces and sensors, and
others are more general interaction design [6], related to
“open” design.
Open and Ambiguous
An open and ambiguous [1] environment, an interactive
field, invites the users to create many interpretations
and perform many different actions.
When interacting in the field it was important that the
user experienced direct response to his actions in the
beginning to motivate further explorations and
interpretations. But later the unpredictable and varying
response opened for a more “communication-like”
interaction and collaboration. Even the way the users
spoke of and acted towards the field became gradually
more “human-like”.
The field should to be able to handle simultaneity in
order to offer satisfying openness. This follow on our
definition of openness where the field, ought to offer the
user the choice of many simultaneous roles, interaction
forms, and activity levels. The design consequences are
that many sensors should be active at the same time, and
handle complex interaction forms and different activity
levels related to the sensors. For example that one user
stroke with sustained movements on one module, at the
same time as his friend hit iterative on another module.
Therefore many sensor input forms increased the
possibilities and thereby the degree of openness.
Modular, Transformable and Inscribable
A modular, buildable form motivated to use over time.
Modularity made the experience environment possible
to change, distribute, and restructure. It offered the user
the possibilities to adopt roles, like being in a landscape,
playing and creating together with a “friend”.
The dynamically changeable form motivated
interaction and offered new functions. The curled wire,
for example, functioned as a collar to uphold a weaker
child’s head.
An inscribable [2], physical system gave the
opportunity for personification and individual inter-
pretation. Such as a physical system for tagging that the
user utilised to separate one module from another.
There ought to be consistency between the identity of
the sound and the interaction form. When observing we
found it to be more inspiring for the users when a
sustained action was followed by a soft sustained sound.
So, the sound should be consistent with the character of
the field. The sound could change over time as long as
the user experienced it to be consistent with the
character. This was not necessary when the user acted as
playing on a musical instrument.
Registration of presence gave the system input to
create long use sequences, and relations with the media
as a coplayer. An example is how the system invited the
user to communicate after some time without activity.
This kind and unpredictable invitation to collaboration
strengthened the relation between the environment and
the user, and the actor qualities of the system.
The dynamically changing, surprising and unpredict-
able form of the curled wire in stretched textile
encouraged communication, when varying with every
user movement. When the user treated, it as an
instrument, the tension of the wire and the large
dimensions, inspired impulsive gestures and slow
movements. The exciting shape of the line and attractive
surface of the textile wire, invited the user to make
sustained gestures. Small wire curls and small
dimensions gave small and fast flexing movements that
motivated to iterative gestures.
Similar and Varying
The flexing wire gave very satisfying user experience
and dynamic affordance, both tactile, visual and rhythm-
ic. The continuously flexing wire offered variation with
a dynamically changing shape that encouraged the user
to make rhythmic iterative actions.
The texture of the user surface ought to have mat
finish, be inviting, soft and skin like. We observed that
this motivated the user to touch, skin-to-skin, as the user
compared it with other inviting skin like objects. For
instance playing on a drum skin, or touching other
actors, such as human friends and stuffed animal toys.
The mat, skin like surface helped the user to play like if
it was a musical instrument, giving friction, and better
grip. When experiencing the ambient soundscape the
mat surface blended in with the background, not taking
unwanted focus from other activities. As a contrast, the
user considered a shiny, cold, metal-like surface less
nice to touch repeatedly, since it was hard, cold and
easily could get filthy.
A heterogenic structure in the textile with few but
pregnant elements was more motivating than a
homogeneous structure. It offered the user a graspable
amount of elements that the user could interact with
individually when playing.
When communicating the users preferred
homogeneous textile, interacting with sustained
continuous actions. This because behaviour such as
stroking needed less control. When experiencing being
in a sound environment, it was important that the
pregnant elements could blend in with the background.
Here integrated light had a unique potential of blending
in and out of focus. Seams, as pregnant elements, also
had great interaction potential.
A firm but flexing, stretch structure in the textile
offered qualities that motivated to rhythmic and musical
gestures. When interacting, it bounced back and forth,
encouraged the user who played iterative rhythms on
the textile as if it was an instrument. The user treating
the field as an actor was inspired by the varying
response, as if he was communicating with a living
These are some of the experiences we made, when
designing fields in textile, motivating musical actions.
We thank Fredrik Olofsson for all his work with music,
software and graphics, and Stefan Magnusson for his
work with hardware and sensors. We thank the Swedish
Inheritance Fund, Borgstena Textile and all who have
contributed to MusicalFieldsForever’s work over the
1. Andersson A.P., Cappelen B. Ambiguity – A User
Quality. Collaborative Narrative in a Multimodal
User Interface. Proc. AAAI 2000, Smart Graphics.
2. Cappelen B., Andersson A.P. From Designing
Objects to Designing Fields—From Control to
Freedom. Digital Creativity 14, 2 (2003), 74–90.
3. Godøy, R.I., Haga, E., Jensenius, A. R. Exploring
Music-Related Gestures by Sound-Tracing.
“A Preliminary Study.” Congas 2006, Leeds.
4. Hedvall, P.O., Cappelen, B., Andersson A.P.,
Rydeman, B., Schulerud, A., Magnusson, S. iFields
– ”Design för Alla” ur ett familjeperspektiv,
Sv. Rehabilitering, 2007.
5. Latour B. Pandora’s hope: essays on the reality of
science studies. Harvard U P, Cambridge, Mass.,
6. Löwgren J, Stolterman E. Thoughtful interaction
design: a design perspective on information
technology. MIT Press, Cambridge, Mass., 2005.
7. MusicalFieldsForever.
8. Pagliano, P. Multisensory Environments, David
Fulton Publishers, London, 1999.
9. Schafer, R.M. The Soundscape, 1977.
ResearchGate has not been able to resolve any citations for this publication.
Full-text available
In this paper we want to explore Field as a concept and as a metaphor for understanding interactive systems. By interactive systems we mean both systems and artworks, where the user by interacting changes the course of events. We intend to show why we need new terms and why we consider Field to be a fruitful concept and term. Further we will show how the Field concept changes both our understanding of what we do as designers and composers and how we acknowledge our audience. We will exemplify the design consequences of the Field concept by going through some design considerations we made when designing the audio tactile installation Mufi.
Full-text available
In this paper we describe an application model for a collaborative narrative in a multimodal tangible user interface. We investigate how a story can be developed in negotiation between three users. By collaborating the users create a soundtrack and negotiate the next video sequence. We developed rhetorical principles for editing moving image material for a node based interactive system. We present a new concept of quality in the user experience that we define as ambiguity, user experience caused by the tension between complexity, control and risk.
Full-text available
This is an exploration of listeners association of gestures with musical sounds. The subjects listen to sounds that have been chosen for various salient features, and the tracing movements made by the subjects are recorded and subsequently compared in view of common features in the tracings.
Multisensory Environments
  • P Pagliano
Pagliano, P. Multisensory Environments, David Fulton Publishers, London, 1999.
Design för Alla" ur ett familjeperspektiv, Sv. Rehabilitering
  • P O Hedvall
  • B Cappelen
  • A P Andersson
  • B Rydeman
  • A Schulerud
  • S Magnusson
Hedvall, P.O., Cappelen, B., Andersson A.P., Rydeman, B., Schulerud, A., Magnusson, S. iFields -"Design för Alla" ur ett familjeperspektiv, Sv. Rehabilitering, 2007.