Content uploaded by Jacques Rémus
Author content
All content in this area was uploaded by Jacques Rémus
Content may be subject to copyright.
Non haptic control of music by video analysis of hand
movements : 14 years of experience with the «Caméra
Musicale»
Jacques Rémus
Ipotam Mécamusique
19 rue des Frigos 75014 Paris France
+33 1 45 82 874 40
jr@directmail.org
ABSTRACT
The "Caméra Musicale" is an interface which allows a musical
practice based on the movement of hands and fingers under a
camera. Born from technologies and methods developed in the
early eighties to create music from choreographic movements,
the “Camera Musicale” has evolved during the nineties to
become a user friendly device since 2003. This article describes
its fundamental principles of operation, which have remained
relatively unchanged. It then focuses on the importance of the
choices necessary to be made while analyzing the video image.
Keywords
Musical Camera, camera musicale, non haptic instrument
interface, musical hand's mappings, Jacques Rémus'sound
sculptures and mechanical musical machines.
1. INTRODUCTION
The "Caméra Musicale" is an instrumental interface which
allows musical practice based on the position and movements
of hands in space, in the scope of a video camera. After the first
prototype was elaborated, its development was insured by
constant experimentations. The first version (1992) had an
audience interact by moving their hands in mid-air, controlling
large and mechanical musical devices. The appeal of this
version lied in the contrast between the immaterial nature of the
movements and the materiality of the music produced by the
machines.
The system was immediately embraced by audiences of all
kinds : Festival audiences, school kids, the handicapped, and of
course, musicians. As years went by, thousands of people have
played "Caméra Musicale" in various contexts.
Consequentially, its development was in close relation with the
audience and the "Caméra Musicale" evolved on grounds of
experimentation - a very pragmatic, rather than scientific
approach.
These experiments also led to the development of another
technology: an interface controlling audio or MIDI systems.
2. ORIGINS AND DEVELOPMENTS
The "Caméra Musicale" was born from research and works by
Sylvain Aubin, with the "Manorine.
A first prototype was developed in 1982 " (patent n° 82695,
december 1982). The point of this work was to combine a
logical interface between a video camera and analogical
synthesizers. The main goal of such a machine was to create
music for choreographic performances (Stéphanie Aubin,
performances, Paris, Rennes 1986).
The principle was based on the analysis of a clear surface (the
body, or a part of the body) being detached in hyper-contrast on
a black screen. This analysis brought up many characteristics of
the moving spot:
X and Y, global parameters giving the position of the object
dX and dY, local parameters describing the size of the object
dY
dX
Y
X
Mobile
Reference point
Figure 1. Global and local parameters.
After these first experiments, focusing on dance movements
generating music, it was clear that the prototype was a relevant
tool for artistic creation. However, its potential was restrained
by the fact that it was hard to operate by non-technicians.
In 1992 Jacques Rémus, Sylvain Aubin and Gwek Bure Soh,
artist, associated to undertake the project of the "Caméra
Musicale".
A Manorine with a MIDI interface coupled with developments
on the Max [1] software was then created.
David Rokeby's VNS ("Very Nervous System") [2] initially
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies
are not made or distributed for profit or commercial advantage and that
copies bear this notice and the full citation on the first page. To copy
otherwise, or republish, to post on servers or to redistribute to lists,
requires prior specific permission and/or a fee.
NIME 06, June 4-8, 2006, Paris, France.
Copyright remains with the author(s).
P
roceedings of the 2006 International Conference on New Interfaces for Musical Expression (NIME06), Paris, France
250
developed for dance as well, was connected in parallel to the
Manorine-MIDI, hence completing the first version of the
"Caméra Musicale".
The analysis of the Manorine was the core of the system, while
the VNS was used to enhance the movement/sound reactions.
3. GENERAL PRINCIPLES
The "Caméra Musicale" is assembled in part from mainstream
elements, available in the market. Those elements are combined
to relatively easy developments, considering one is familiar
with live electronics software (Max-MSP)
Here are its main components:
- A black and white camera. A standard IR photo filter,
positioned on the lens, allows exclusive infrared detection. The
camera is aimed towards the floor and stands 1 to 3 meters
above the player's hands.
- An infrared projector - the type used for night surveillance
(invisible in darkness). Next to the camera, it also aims towards
the ground. This system allows the "Caméra Musicale" to
remain unaffected by the various lightings, often problematic in
live performances.
Infra red light
(50 - 80 W
halogen type)
black and white
small camera
with infra red filter
on the lens
2.30 up to
2.50 m
from the
ground to the
camera
screw for
microphone
stand
Very easy
position-
axles
standard
microphone
stand
0.80 to 1 m
from
the
microphone
stand axle
Figure 2. Structure of the “Camera Musicale”.
- An interface or a standard video computer card (e.g. a TV card
for computer can very well do the job)
- A software development on Max-MSP-Jitter [1], allowing to
control and chose, live, various parameters of video analysis
and musical patches. Cyclops [1] or VNS [2] plug-ins or Jitter
[1] developments are used to filter simple information from the
very complex original image. The filtered data then enters in
Max, in the form of a continuous flow (25 or 30 images per
second). It's basically a software "Manorine" with important
improvements from the plug-ins.
- Musical instruments: synthesizers (expanders), musical
machines, sound sculptures or computer generated sound (Max-
MSP)
- One or many screens, showing the hands, allowing visual
interaction for the player and audience.
4. DESCRIPTION OF THE VIDEO/MIDI
GENERATED INFORMATION
The information resulting from the video analysis remain the
center of the development. As described above (§2), there are
four:
- X and Y measure the position of the mobile
- dX measures the widest part of the mobile's image.
- dY measures its highest part.
The dynamic of the mobile's movement (hands or fingers) is
essentially rendered by the speed in which X', Y', dX' and dY'
evolve. Their calculation in Max creates four new data flows.
The calculation of the second derivation of the first four
parameters (accelerations X", Y", dX", dY") creates once again
four new data flows.
The hands and fingers can be individualized and independent
(by definitions of areas or mobile objects' individualization).
On this basis, all the data that can be rendered by pixel
variations in defined areas (principle of the VNS) and the
indications of the grids from Cyclops complete and enrich the
possible video analysis parameters
5. NUMBER OF INDEPENDENT
PARAMETERS A MUSICIAN CAN
CONTROL
The essential choice while operating the "Caméra Musicale" is
the number of independent parameters the musician can control
with the camera.
In theory, the "Caméra Musicale" allows to play with a number
of parameters, dependent or independent, in various predefined
areas in space (drawn on the screen). (cf. §4) However, if the
complexity can produce gratifying results with lots of feedback
and automatisms, they are seldom relevant.
The number of independent parameters a classical
instrumentalist can control is relatively low: pitch, intensity,
articulation, timbre and rhythm (succession of events in time).
It hardly exceeds four or five.
The same goes for the "Caméra Musicale", with three or four
independent parameters. The time parameter, rhythm, is
considered independently from the others.
The "Caméra Musicale" is developed to read a limited number
(from one to four) of data flows from the incoming information.
Only very simple patches with two to four independent
parameters could allow a good control of the outcome. This
applies to the beginner as well as to the accomplished musician.
On the other hand, the chosen parameters can be very different
from one try to another : X, X', X", Y, Y,' Y", dX, dX,' dX",
dY, dY', dY", surfaces, number of pixels and their variations,
etc.
Example 1 (basic) : X specifies the pitch of the note or sound, Y
specifies the intensity and dY" a change in the attack or timbre.
Example 2: dots or small areas defined in X, Y are drawn on the
screen and the parameters in which they accelerate to reach one
another are used (variations of pixels combined with dX" and
dY" ensembles). For example: the faster the movement is, the
P
roceedings of the 2006 International Conference on New Interfaces for Musical Expression (NIME06), Paris, France
251
longer (or shorter, or harmonically transformed...) the note at
the point of release (X, Y) will be. The hands move in space at
different speeds and "touch" predetermined points (X, Y) which
trigger the sounds, modulated by the speed of the movements.
Example 3: a 12 note loop is automatically played: Its speed is
increased in Y, its velocity in X, the length of its notes in dX
and their attack in dX'. The mapping then takes into account
the width of the hands, or the distance between both hands
(dX). However, if we add a change of timbre for dY', the
control of yet another parameter would become too difficult.
Once the choices are made, the possibilities of the mapping
remain very wide, even if in practice (see § 7) the movements
chosen by the players tend to imitate those of a pianist, harpist
or drummer !
6. NON “HAPTIC” BODY LANGUAGE.
These gestures are performed in mid-air and the player has
counterbalance every movement he makes with its opposite, in
order to keep balance. It is hard to master at the beginning, like
all non return (non haptic) interfaces, but this doesn't take away
the fact that the mapping is precise and pleasant. Moreover, the
body being entirely engaged in these gestures, the musician's
body language can become an object of interest for the
audience.
Practice has proved that the changes between playing modes
should be done by a different system that the gestures and the
camera. Simple midi-pedals now allow such changes, with no
risk of errors due to bad control of the gestures.
7. EXPERIENCE WITH PUBLIC AND
MUSICIANS
The "Caméra Musicale" in use since 1992 by large audiences
with midi robotized acoustic machines and since 2003 by
musicians on electro-acoustic sounds
The "Caméra Musicale" was first used by the "Concertomatique
N°2" ensemble [3] [9] made up of several pipe organs, a string
quartet, percussion and machines with ringed pipes [3] [5].
Then, it was coupled with the "Carillons des Zic-phones" [3]
[6]. the "Carillon Concertomatique N°3" (Festival Résonnances,
Ircam, 2002) [3] [7]. and recently, with the "Pic-Verts" (Green
Woodpeckers) [3], the "Ensemble des machines à laver
musicales" (The Musical Washing Machine Ensemble) [3] [7]
[8] and the "Orgabulles" (Bubleorgan) [3].
In concerts, expositions, installations, animations and festivals,
thousands of people played with such installations and the
mappings were progressively developed and improved. All
these experiments show the strength of cultural perceptions
related to musical gestures of the instrumentalists or conductor.
Surprisingly enough, such gestures were particularly present
with musicians.
The "Caméra Musicale" has always proposed innovative
mappings (examples 2 and 3 from §5 only being a sample), but
users will usually seek three primary types of gestures:
1) The movement related to percussion - with the tip of the
finger or pretending to hold a drumstick or mallet (triggering
related to a neutral zone at the edge, below the screen, which
takes into account the entry of the finger, hand or arm in the
image (area drawn in X Y)
2) The movement referring to the organization of pitches on a
keyboard (low pitches at the left, high pitches at the right: X
parameter).
3) Reference to the intensity, or strength they put in their
movements (Y', Y" or more oftenly, dY' and dY" or the
variations of pixels in the VNS).
If some mappings do not respect one of those three primary
approaches, they are considered to be difficult, beyond
understanding or unnatural.
The "Caméra Musicale" is not adapted to the type of
movements an instrumentalist would do, in order to imitate the
basic technique of his instrument: moving the fingers, trying to
find the notes on an invisible instrument.
Research has to take account of such requests, but usually
focuses on exploring other possibilities.
It is actually with pre-written modules in the patches that the
"Caméra Musicale" takes form of an original instrument:
automatic arpeggios, prerecorded loops, prewritten sequence
modulations (as the Max Mathews sticks) erase the note
interpretation difficulty, and opens to modulations related to a
conductor's work.
In addition, the relationship with the Theremin is acquired by
pitch-bends on the midi system and more easily again with
sounds generated by calculation (Max-MSP).
Finally, the last developments ("Signa" duo concerts with
theremin player Rolf Sudman (Berlin)[3]) focused on the live
treatment and samples (scratching) of the sung voice, spoken
word and noise. The deformation of sound was, in all cases,
controlled by movements based on the three primary criteria..
Table 1 indicates the use of the parameters in the plays.
Mappin
gs type X Y X' Y' X" Y" dX dY dX' dY' dX" dY"
INSTR. impuls 5 5
INSTR pitch 5 2 3
INSTR velo 4 5 4 4 2 2
INSTR modul 2 3 2 1 4 2 3
ORCH. impuls
ORCH. pitch 5 1
ORCH. velo 4
ORCH. modu 3 4
THER. impuls
THER. pitch 3 3
THER. velo 3 3
THER. modul 1 2 2 2
D.AUD impuls 5 5
D.AUD pitch 4 1 3 2 1
D.AUD velo 4 2 4 2 3
D.AUD modu 2 1 1
Table 1. Frequency of the parameters in the mappings
(classified from 1 to 5)
Legend :
NSTR. = INSTRUMENTAL, ORCH. = ORCHESTAL
THER. = THEREMIN LIKE, D.AUD = DIRECT AUDIO
impuls = impulsion, velo =velocity, modul=modulations.
This table is not a statistical study, but merely a representation
of the users' tendencies. The numbers (0 to 5) represent about
fifty mappings: 1 = seldom use to 5= always or almost. We
notice that instrumental mappings and mappings using the
audio are related to the instrumentalist's basic movements
whereas the theremin or conductor mappings allow more
fantasy and imagination in the choice of movements.
P
roceedings of the 2006 International Conference on New Interfaces for Musical Expression (NIME06), Paris, France
252
8. CONCLUSION
The "Caméra Musicale" is an interface for musical performance
and also a tool for musical creation. Other research has been
developed in the same direction. In addition to the works of
David Rokeby, already mentionned [2], the works of Tom
Demeyer with the Big Eye software [10] have influenced this
type of work - particularly with the residence at the Steim
foundation.
The "Caméra Musicale" opens new paths for future musical
practices. Its originality and reliability has been tested and
approved by a wide audience. Still far away from its full
potential, it should however always consider the natural
movements of instrumentalists.
9. REFERENCES
[1] Cycling 74 home page : http://cycling74.com
[2] David Rokeby home page :
http://homepage.mac.com/davidrokeby/
[3] Jacques Rémus and Ipotam Mécamusique home pages :
http://jacquesremus.com
http://mecamusique.com
[4] Yamawaki, K., Jacques Rémus: the musical garden with
informatics, robotics and musical instruments In Artec'97,
The 5th International Biennale in Nagoya 1997, official
catalogue,
(June 1997), The council for the international Biennale in
Nagoya, Naganae printing Co, Japan, p.42-49.
[5] Rémus, J. The music of ringed pipes In Proceedings of the
International Symposium on Musical Acoustics
(July 2-6, 1995), SFA Jouve, Paris, France, p.69-73.
[6] Rhobiny, S., Because of the wonderful things he does, In
Kilometer zero N°3,
The Kilometer zero project, 2002, ISSN: 1629-3274, p.50-
56
[7] L'oeil de la nuit 2003 Parcours Paris Rive Gauche, official
catalogue of Nuit blanche 2003,
Paris Musées, Les Musées de la ville de Paris, Paris, ISBN
20042-87900-831-X, p.136-137.
[8] Installation sonore, Lavage de tympans In Nova Magazine,
Connexions pirates, N°101, Paris, Mai 2003, p.25.
[9] Official Catalogue of Musique en Scène Festival, Grame,
Lyon, France, april 1995, p.8-9.
[10] Steim foundation, (Amsterdam) homepage :
http://steim.org
[11] Mécamusique, In La marionnette et les autres arts,
Musiques en mouvement, Puc n°6
1993 Editions Institut International de la Marionnette,
France, ISSN 0993-0701, p.56-57.
[12] Jacques Rémus,Double Quatuor à cordes, De la Motte-
Haber, H., In Klangkunst,Tönende Objekte und klingende,
Räume, Laaber-Verlag, 1999,ISBN 3-89007-432-4, p.160-
161.
[13] Jacques Rémus, Musica ex machina, Nicollet, G., Brunot
V., In Les Chercheurs de sons, Instruments inventés,
machines musicales, sculptures et installations,
Ed Alternatives, Athènes, 2004, ISBN: 2-86277434-8,
p.120-125.
[14] Arts Ménagers, arts sonores , Le Graverend, F., In Area
Revues, Délirantes ou célibataires, les machines dans
l’art, N°11,
Area corp, Paris, march 2006, ISBN: 2-35276-002-X,
p.160.
[15] Les nouveaux gestes de la musique,Genevois, H., de Vivo,
R.,
Parenthèses , collection eupalinos, cultures musicales may
1999, ISBN :2-86364-616-8.
[16] La Sculpture sonore, pratique artistique en recherche de
définition, Rémus, J. In Musiques Arts Technologies, pour
une approche critique, Music arts tecnologies, toward a
critical approach", (Barbanti R., Lynch E, Pardo, C.,
Solomos, M),
L'Harmattan, France, june 2004, ISBN : 2-7475-6691-9 ,
p.61-75.
P
roceedings of the 2006 International Conference on New Interfaces for Musical Expression (NIME06), Paris, France
253
