Family ensemble: a collaborative musical edutainment system for children and parents.

Full text

Page 1

Family Ensemble: A Collaborative Musical Edutainment
System for Children and Parents
Chika Oshima
ATR Media Information Science Labs.
2-2-2, Hikaridai, Keihan’na Gakken
Toshi, Kyoto, 619-0288, Japan
Phone: +81 774 95 1465
chika-o@atr.jp
knishi@jaist.ac.jp
Kazushi Nishimoto
Japan Advanced Institute of Science
and Technology
1-1, Tatsunokuchi, Nomi, Ishikawa,
923-1292, Japan
Phone: +81 761 51 1812
Masami Suzuki
ATR Media Information Science Labs.
2-2-2, Hikaridai, Keihan’na Gakken
Toshi, Kyoto, 619-0288, Japan
Phone: +81 774 95 1465
msuzuki@atr.jp


his/her child to readily enjoy ensembles together at home, in this
case, a piano duo, even if the parent has little or no experience in
playing a musical instrument.
ABSTRACT
In this paper we propose a collaborative musical edutainment
system named “Family Ensemble” (FE). FE allows a parent and
his/her child to readily enjoy ensembles together at home, in this
case, a piano duo, even if the parent has little or no experience in
playing a musical instrument. FE makes it easier for the parent to
correctly perform given melodies along with his/her child’s
performance, using a score-tracking algorithm that can cope with
the particular errors commonly made by beginners, i.e., children.
By supporting the parent, FE prompts the child to practice the
musical instrument more willingly. In experiments, we confirmed
that FE can facilitate the playing of duos by pre-instrumental
performers and beginners. Furthermore, we found, during a joint
practice using FE, that some subjects discussed musical ideas that
they could not have talked about without the system. Thus, not
only does FE encourage children to willingly practice the piano
but it also allows even pre-instrumental performers and beginners
to aim for richer musical expression and a deeper understanding
and appreciation of music.
Categories and Subject Descriptors
H.5.5. [Sound and Music Computing]. K.3.1. [Computer Uses in
Education].
General Terms
Experimentation.
Keywords
music edutainment, support system, score tracking, beginner,
piano duo.
1. INTRODUCTION
In this paper, we propose a collaborative musical edutainment
system named “Family Ensemble” (FE). FE allows a parent and
Some families have enjoyed playing music in the home since the
eighteenth century. After dinner, sisters might play a Mozart
symphony as a piano duo, or a father might play the jazz music on
a violin with his son’s accompaniment on the piano. This custom
has played an important role not only in communication among
family members but also in music education. Today, despite the
ready availability of recorded music, there are still many families
who want to play music at home. Usually, however, we see only a
few families performing ensembles at home.
Recently, in piano playing education, the efficacy of duo play in
the learning process has come to be widely recognized [1][10][12].
Even if a pupil who has just started to practice the piano one week
ago plays a piece that consists only of “Do” notes, it is desired that
a teacher accompanies him/her with various chords on the piano.
It has often been reported that children who have played duos
come to practice more enthusiastically and to perform more
musically [13].
However, most children can play duos only in school, since most
of their parents cannot play the piano. Few marketed scores of
piano duos are arranged for two beginners. This fact also shows
the difficulty in playing a duo by beginners. Recently, there are
various “minus-one” software applications in which all parts
except for the pupil’s part are recorded [4][10]. Professors of
music, however, comment that minus-one software prevents the
pupils from acquiring the skill of cooperative performance that is
essential for ensembles [13].
On the other hand, the parents of children who are learning to play
a musical instrument are often eager to play music with them.
Recently, music schools for adults and software for learning to
play particular musical instruments have become popular, at least
in Japan. These schools and/or software usually focus only on
enabling the learners to perform their favorite pieces. However, if
they want to perform various works with other people, e.g., their
children, they must eventually start from the basics. The need to
undertake difficult training for many years makes most parents
give up the dream of being able to play duos with their children.
Accordingly, in this paper, we propose “Family Ensemble,” which
supports ensemble performances by a parent who has little or no
experience in playing the musical instrument and his/her child
who is a beginner in playing the instrument. Though any type of
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that
copies bear this notice and the full citation on the first page. To copy
otherwise, or republish, to post on servers or to redistribute to lists,
requires prior specific permission and/or a fee.
MM’04, October 10–16, 2004, New York, New York, USA.
Copyright 2004 ACM 1-58113-893-8/04/0010...$5.00.
556

Page 2

performance interface can be applied to FE, we employed a piano
keyboard to conduct our experiments. We designed FE for
performing “reproduction-type music” like classical music, in
which musicians should correctly perform the notes described in a
given score (in contrast to jazz, in which musicians are expected to
improvise freely). FE tracks the child’s performance in real-time.
By referring to the tracking results, FE decides which notes the
parent should perform at each point. Even if the parent strikes the
wrong keys on the keyboard, FE replaces the incorrect notes with
the correct ones [14].
Thus, FE allows any parents to immediately play a duo with
her/his child, following the child’s performance. FE will bring
about a new cooperative learning method in music education at
home for parents and children. We would like to emphasize that
FE directly supports the parent, not the child. We believe that a
child who is studying the piano should use a normal instrument
and that direct support would detract from the learning process.
We also think that the following are important aspects for
constructing a system as music edutainment: 1) Interaction with
not a computer but a person, 2) Knowing incorrect elements, e.g.,
incorrect pitches and no sense of rhythm, through listening to
one’s performance, and 3) Room for growth, i.e., the system
should not intervene in musical expression. A system that includes
these aspects gives even pre-instrumental performers the
enjoyment of musical performance, eagerness to practice, and the
knowledge that they can perform richer music.
Technologies
performances have been studied mainly in terms of automatic
accompaniment systems. Dannenberg [5] developed an algorithm
for tracking a score based on pitch-sequence data by applying a
DP-matching method. Vercoe [19] developed a score-tracking
method by referring not only to the pitch-sequence but also to the
sounded time of each note. In addition, a human cognition model
for estimating musical performance in an ensemble was taken into
account. Studies of an automatic jam session system attempt to
follow the emotive transition of human performer(s) [7]. Although
the score-tracking function of FE is an extension of Dannenberg’s
algorithm, as described later, all of these preceding studies aim at
enabling ensemble play by an advanced human performer and a
machine. FE, in contrast, aims at enabling ensemble play by
human pairs with a little and no experience in playing the musical
instruments.
for automatically tracking human musical
In regard to the study of interactive musical systems, the
Continuator [16] is a system that supports a user in learning
musical styles. When the user plays a musical sequence of any
kind (i.e., a phrase), the Continuator, in turn, generates a musical
phrase in response to the user’s phrase based on a Markov model
of the musical phrase. The experimental results showed that they
played with the system in a self-motivated way. The Jum-O-Drum
[2] is a system that allows up to six players to practice
collaborative musical improvisation. pianoFORTE [17] is a
system that records and displays several aspects of a user’s
performance. This system aims to have the teacher discuss with a
student about the distinction between the art of playing piano and
the technique of playing the correct notes. DrumSteps [11] is a
dynamic graphical interface that enables children to create,
manipulate, edit and save original pieces of percussion music. Toy
Symphony [9] is an international music performance and
education project that allows children to play alongside some of
the world's most accomplished musicians, and to learn by doing.
Several interesting tools or instruments have been developed in
this project. For example, Music Toys are specially developed
musical instruments that allow everybody to easily enjoy musical
performances. Hyperscore [6] is a tool that enables children to
'compose-by-drawing'. It is designed to introduce children to
musical composition and creativity in an intuitive and dynamic
way. FE provides no special tools to the learners while these
systems provide special tools to them. FE indirectly prompts the
children’s learning with self-motivation as well as communication
with his/her parents.
In the next section, we will investigate features of the beginners’
mistakes in performances and describe the FE system design based
on the investigation results. Then, we will illustrate our
experiments and discuss the effects of FE. Additionally, we
discuss a system for music edutainment. In the final section, we
will conclude this paper and outline future work.
2. FEATURES OF BEGINNERS’
PERFORMANCES
FE has to be adapted to even beginners’ performances. We thus
examine the kinds of errors that beginners are apt to make.
2.1 Outline of Experiments
We conducted experiments with two children who have been
studying the piano. Subject-A had studied the piano for four years.
She is a fourth year student of an elementary school. Subject-B
had studied the piano for one year. She is a nursery school toddler
and a sister of Subject-A. Subject-A was asked to perform
seventeen pieces that she had never performed before in four
thirty-minute experiments. Subject-B was asked to perform
thirteen pieces that she had never performed before in four thirty-
minute experiments. Their pieces include piano pieces for four
hands. They were asked to perform pieces from one to five times
according to the level of difficulty. We recorded the sounds and
utterances of these experiments on a DVD-RAM disk.
2.2 Results
Ordinary score tracking systems (e.g., [5][8][15]) deal with three
types of performance errors as defined by Bloch & Dannenberg
[3], assuming that the target users are experienced performers and
that their performances closely match the scores.
1. extra: An extra note that does not match any notes in the
score is performed.
wrong: Though a performed note matches a note in the score,
its pitch is incorrect.
2.
Table 1. Frequency of errors
Pl aceAB Sum%
ext r a
wr ong
mi ssi ng
1
0
3
671. 2
6. 1
0. 5
75. 4
3. 1
1. 0
0. 8
0. 7
8. 4
0. 8
1. 7
0. 3
100. 0
37
0
141
37
3
456
19
1st beat
2nd downbeat
Last beat
St ar t of phr ase
Al t er nat i on
1st best ( pr e-bar )
Last beat ( pr e-bar )
St ar t of pi ece
St ar t of col umn
Sum
315
172
6
0
0
0
5
4
6
5
4
26
2
7
1
381
25
3
3
1
224
51
5
10
2
605
r epl ayi ng

557

Page 3

3. missing: A note in the score is not performed.
While beginners also make these three errors, it is necessary to
add a fourth type:
4. replaying: An extra note sequence that matches a part of the
score is performed1. In other words, an undirected refrain is
performed.
Table 1 shows the frequency of errors, i.e., extra, wrong, missing,
and replaying. As for replaying, in addition, the places where the
subject restarted and the frequency of each restarted place are
shown. “1st beat” means that the subject restarted at the 1st beat of
the same bar as where the subject stopped her performance. “2nd
downbeat” means the 2nd downbeat, i.e., the 3rd beat of a
quadruple measure or the 2nd beat of a binary measure. “Last beat”
means the last beat, i.e., the 4th beat of a quadruple measure. “Start
of phrase” means the start of the slur mark. “Alternation” means
where a performer needs to alternate right hand with left hand in
one phrase. “column” means a group of bars forming a line on a
printed score. “pre-bar” means that the subject restarted at the bar
preceding where the subject stopped performing. The three
categories of “1st beat,” “2nd downbeat” and “Last beat” do not
include “Start of phrase,” “Start of piece,” “Start of column” and
“Alternation.”
Subject-A made 381 mistakes and Subject-B made 224 mistakes
in total. Surprisingly, the cases of replaying were 377 times in 381
errors in Subject-A performances, and 181 times in 224 errors in
Subject-B. Moreover, “1st beat” accounted for about 75% of all
errors. Thus, we need to consider all four types of error in order to
construct a supporting system for beginners’ ensemble playing.
3. SYSTEM DESIGN OF Family Ensemble
3.1 Setup of Family Ensemble
We developed FE as a system to support the playing of duos on
the piano. The system was implemented by Visual C++ and runs on
Windows XP or 2000. We call the part of the child “Primo,” that
is, the part that performs the upper register of a duo, and the part
of the parent “Secondo,” that is, the part that performs the lower
register. FE supports Secondo by detecting where Primo is playing
in the score in real time.

1 The extra note sequence may include errors of the other three
types.
Figure 1 shows the setup of FE. Primo’s performance is output as
MIDI (Musical Instrument Digital Interface) data and directly
input to a sound generator as the notes are played. At the same
time, Primo’s performance data are input to the score-tracking
module that determines Primo’s current position in the score. The
obtained current position data are input to the note-number-
acquisition module. This module compares Primo’s current
position with Secondo’s score. As a result, the MIDI note
number(s)2 that Secondo should play at that moment is (are)
acquired. The acquired MIDI note numbers are input to the note-
number-replacement module. This module replaces the MIDI note
number(s) that Secondo played with the acquired correct MIDI
note number(s), while the other characteristics of the performed
note(s), i.e., loudness and length of sound, are preserved as they
were performed. Consequently, by using any keys, Secondo can
perform the correct notes, with his/her own expression [11],
alongside Primo’s performance.
If Secondo stops playing for a moment while Primo keeps on
performing, Secondo can immediately resume the performance
from the correct place in the score. On the other hand, if Primo
suddenly stops playing while Secondo keeps on performing,
Secondo cannot proceed beyond the point where Primo stopped.
When Primo resumes the performance, Secondo can resume the
performance, too. Even if Primo returned several notes back when
Primo resumes to play, Secondo can quickly catch up with it by
the score-tracking function that is described in 3.2.
3.2 Score-tracking Module for Beginners’
Performances
We extended Dannenberg’s score-tracking algorithm [5], which is
based on the DP-matching method, to cope with the fourth type of
error, i.e., replaying. Similar to Dannenberg’s algorithm, our
algorithm deals only with pitch data. Let the number of notes in
Primo’s score be N, where a chord is regarded as a note and is
represented by its highest note, and let the “performance time” be
tj where j is the number of notes performed by Primo. Now,
assuming that the current position at the performance time tj-1 was
decided at the note
) 1 (NiSi
≤≤
note Pj at performance time tj. In this case, our score-tracking
module determines the correspondence between the note in the
score and the performed note Pj based on the following algorithm:
1. First, Pitch(Pj), i.e., the pitch of the performed note Pj, is
compared with Pitch(Sk) of each note in the score
) 1 (NkSk
≤≤
. Then, the weight of each note Sk at the
performance time tj, i.e., W(Sk, tj), is calculated as follows:
in the score, Primo performed a
(a) if Pitch(Sk) = Pitch(Pj)
then W(Sk, tj) := W(Sk-1, tj-1)+ 1.
(b) else W(Sk, tj) := W(Sk-1, tj-1) - 1.
where if W(Sk-1, tj-1) – 1 < 0 then W(Sk, tj) := 0

If W(Si+1, tj) ≠ W(Si, tj-1) + 1, some error is detected. In this
case the following correction algorithm is applied.


2.
(a) To correct a replaying error, let W(Sh, tj) := W(Si+1, tj) – m,
where Sh is the point in the score to which the beginner is likely

2 The MIDI note number represents the pitch of the note.
Modified
performance
data
Primo’s performance
interface interface
Secondo’s
performance interface performance interface
Score Database Score Database
Performance data i MIDI j
Performance data i MIDI j
Note-number-
replacement
module module
Note-number-
acquisition
module module
Performance data i MIDI j
Performance data i MIDI j
MIDI Sound
GeneratorGenerator
SpeakerSpeaker
Primo’s
score score
Secondo’s
score score
Score-tracking
module module
Modified
performance
data
Primo’s performance
Secondo’s
Note-number-
replacement
Note-number-
acquisition
MIDI Sound
Primo’s
Secondo’s
Score-tracking

Figure 1. Setup of Family Ensemble
558

Page 4

to return when he/she stops playing, as enumerated in the
previous subsection. At present, m is empirically set to 2.
(b) To correct extra, wrong and missing errors, let W(Sg, tj) :=
W(Si+1, tj), where Sg is all notes from Si-r to Si+r. At present, r is
empirically set to 2.

3. Finally, the score-tracking module decides a note Sk whose
weight W(Sk, tj) is the largest as the current position at
performance time tj. If there are multiple notes whose weights
are the largest, the module chooses one note as follows:
(a) If W(Si+1, tj) is the largest then Si+1 is decided upon as the
current position.
(b) If W(Si+1, tj) is not the largest then
i. a note Sk whose weight is the largest and that is the nearest
to the note Si is decided upon as the current position.
ii. If two notes Si-d and Si+d have the same largest weight and
are both nearest to the note Si, Si-d is decided upon as the
current position.
Figure 2 illustrates our score-tracking algorithm.
This algorithm achieves robust score-tracking even for the
performances by the children with a little musical experience in
virtue of the function to cope with the “replaying” type error as
well as the three ordinary types of errors. Figure 3 shows
situations of score-tracking. (a) shows the result in the case in
which the system corrects a replaying error. (b) shows the result in
the case in which the system does not correct a replaying error.
Column corresponds to notes described in the score. Row
corresponds to notes performed by Primo, Pj. The hatched part of
the performed notes shows where Primo replayed. Consequently,
in this case, Primo replayed “sol, la, sol, mi…” although she
should have played “mi, sol, sol, sol….” Furthermore, cells that
include outline characters show the estimated current position that
the score-tracking module decided upon. Cells surrounded by a
black border show the actual correct notes that Primo is playing at
each point.
In figure (a), we can see that the notes “sol” and “do” have weight
“11” at the start of replaying. We had appointed these two notes to
correspond with the start of bars beforehand. Accordingly, the
system could follow Primo’s performance after mistakes of only
three notes. On the other hand, in figure (b), the system mistook
thirteen notes before following Primo’s performance correctly.
Thus, we can see that our score-tracking algorithm, which adds to
correct replaying errors, is practical.
As for usual musical pieces, this algorithm works enough robustly.
In some extreme cases, however, it might not be able to be
restored from an error. For example, if one makes a mistake
during the first eight bars of “One note samba” that consist of only
notes of an identical pitch (F), it is impossible to find the correct
position. Time values of notes should be taken into account to
track such a piece. It is also difficult to track a piece that consists
of only an identical motif, e.g., a simple repeat of “do, re, mi, re”
over and over. As for the processing time of this algorithm,
although it scans over the entire piece each time, pieces that
beginners perform are usually short and simple. Practically,
therefore, the total processing time can be ignored.
sol l a sol mi sol l a
000
000
mi 000
mi 000
r e000
sol 1101
l a 10 120
sol 2913
mi 018
sol 102
l a020
sol 103
mi 000
do000
r e 000
mi 000
mi 000
r e000
mi 0 00
sol 101
sol 201
sol 111
sol mi do
00
00
11
02
00
10
00
30
04
10
00
17
0
4
18
03
00
11
02
00
01
10
10
20
r e
0
11
0
0
1
0
10
9
0
0
2
0
0
0
20
1
0
1
0
0
0
0
mi mi r e
00
00
12
1
1
13
00
13 13 13 13
012 12
90
10 10
09
00
10
12
00
00
21
1
2
22
01
21
01
00
00
mi sol sol sol l a
000
000
200
110
000
131
12 12 12
11 13 13
12 10 12
0 13 11
80 12
791
168
005
000
200
110
000
24
00
0
25
1
01
26
012
do do
0
0
0
0
0
0
0
1
do sol l a
00
00
00
00
00
01
00
01
00
01
80
99
88
117
610
95
08
30
02
01
01
01
mi r e do
00
01
10
10
02
00
00
10
10
00
00
10
10
70
68
75
106
311
82
07
00
00
do
r e
0
0
1
1
0
0
0
0
14
7
1
0
4
0
0
1
1
0
1
0
0
0
3
0
0
0
0
1
0
1
0
15
6
2
0
3
0
0
0
0
0
2
1
1
0
2
0
0
0
0
2
0
0
0
16
5
1
0
2
0
0
0
0
0
1
0
10
0
0
0
1
11
10
0
0
3
0
0
0
19
2
0
0
1
0
0
0
0
1
1
0
0
14
0
0
0
0
0
1
12
13 11
12 12 10
13 11 11
10 14 10 10
139
0 12
70
46
03
00
00
00
10
20
27
1
0
0
0
0
0
0
2
0
0
0
0
0
0
0
0
0
9
0
0
0
0
0
0
2
0
0
0
2
0
8
7
6
9
4
7
0
1
0
0
1
0
0
0
0
1
0
0
0
0
0
0
0
1
0
7
4
5
10
1
6
0
11
0
9
8
0
0
1
1
0
0
23
0
0
0
0
13
8
11
0
5
2
0
0
0
0
0
9
12
7
10
0
4
1
0
0
0
0
sol l a sol mi sol l a
000
000
mi 000
mi 000
r e000
sol 101
l a020
sol 203
mi 010
sol 102
l a020
sol 103
mi 000
do000
r e000
mi 000
mi 000
r e000
mi 000
sol 1 01
sol 201
sol 111
sol mi do
00
00
11
02
00
10
00
30
04
10
00
70
08
00
00
11
02
00
01
10
10
20
r e
0
11
0
0
1
0
0
0
0
0
2
0
0
0
10
0
0
1
0
0
0
0
mi mi r e
00
00
12
1
1
13
00
13 13 13 13
012 12
00
11
00
00
10
12
00
00
111
112
00
21
01
00
00
mi sol sol sol l a
000
000
200
110
000
13 1
12 12 12
11 13 13
12 12 12 12
013 13 13
00 12
01 13
100
000
000
200
110
000
14
00
0
15
1
01
16
012
do do
0
0
0
0
0
0
0
1
do sol l a
00
00
00
00
00
01
00
01
00
01
100
911
12 12 10
11 11 11
10 10
99
88
07
00
01
01
01
mi r e do
00
01
10
10
02
00
00
10
10
00
00
10
10
0
10 10
11 11
8
12
77
68
05
00
00
do
r e
0
0
1
1
0
0
0
0
4
0
1
0
4
0
0
1
1
0
1
0
0
0
3
0
0
0
0
1
0
1
0
5
0
2
0
3
0
0
0
0
0
2
1
1
0
2
0
0
0
0
2
0
0
0
6
0
1
0
2
0
0
0
0
0
1
0
10
0
0
0
1
0
0
0
0
3
0
0
0
9
0
0
0
1
0
0
0
0
1
1
0
0
14
0
0
0
0
0
1
12
13 11
0
0
0
0
0
0
2
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
2
0
0
0
2
0
1
0
0
0
0
1
0
0
0
0
0
0
0
1
0
11
0
0
0
0
0
1
1
0
0
13
0
0
0
0
12 12
11 11 11
12 14
13 13 13 13
12 12 12 12
011 11 11 11
00 10 10
000
000
000
000
100
200
17
10
10 10
12
11 11
10 10
13
6
7
4
0
9
0
0
0
0
0
0
9
12
5
6
3
Per f or med not es ( Pj )
( a) Cor r ect r epl ayi ng er r or
( b) Not cor r ect r epl ayi ng er r or
Descr i bed not es i n t he scor e
Descr i bed not es i n t he scor e
Per f or med not es ( Pj )

Figure 3. Result of score-tracking
StartStart
Primo performed a note Pjat performance time tj
Primo performed a note Pjat performance time tj
Pitch(Sk) = Pitch(Pi) ?
Skis the kthnote in the score
Skis the kthnote in the score
The weight of each note Sk
at performance time tj
W(Sk, tj)
© W(Sk-1, tj-1) + 1W(Sk, tj)
© W(Sk-1, tj-1) + 1
The weight of each note Sk
at performance time tj
W(Sk, tj)
© W(Sk-1, tj-1) - 1W(Sk, tj)
© W(Sk-1, tj-1) - 1
W(Si+1, tj) = W(Si, tj-1) + 1 ?
where Siis the estimated performed note
at performance time tj-1
at performance time tj-1
W(Sk-1, tj-1) – 1 > 0 ? W(Sk-1, tj-1) – 1 > 0 ?
The weight of each note Sk
at performance time tj
W(Sk, tj) W(Sk, tj)
© 0
© 0
W(Sh, tj) W(Sh, tj)
© W(Si+1, tj) – m @© W(Si+1, tj) – m @
for each note Shto which the beginners
are likely to return. i m is empirically set to 2 j
are likely to return. i m is empirically set to 2 j
W(Sg, tj) W(Sg, tj)
© W(Si+1, tj)
© W(Si+1, tj)
@ for each note Sgthat is included between
Si+rand Si+ri r is empirically set to 2 j
Si+rand Si+ri r is empirically set to 2 j
Are there multiple notes
whose weights W(Sk, tj) @ (k=1~N)
are the largest?are the largest?
Skwhose weight W(Sk, tj) is the largest
is determined as the performed note
at performance time tj
at performance time tj
Are there multiple
notes Skwhose weights W(Sk, tj) are
the largest and whose distances from
the note Siare the nearest?
the note Siare the nearest?
Sk(k<i) is determined as
the performed note
at performance time tj
at performance time tj
EndEnd
YesYesNoNo
NoNoYesYes
NoNo
NoNo
YesYes
k > N?k > N?
N is the number of notes in Primo’s scoreN is the number of notes in Primo’s score
k k
© 1
© 1
NoNo
YesYes
k k
© k+1
© k+1
Si+1is determined as the
performed note
at performance time tj
at performance time tj
YesYes
NoNoYesYes
Skis determined as
the performed note
at performance time tj
at performance time tj
Pitch(Sk) = Pitch(Pi) ?
The weight of each note Sk
at performance time tj
The weight of each note Sk
at performance time tj
W(Si+1, tj) = W(Si, tj-1) + 1 ?
where Siis the estimated performed note
The weight of each note Sk
at performance time tj
for each note Shto which the beginners
@ for each note Sgthat is included between
Are there multiple notes
whose weights W(Sk, tj) @ (k=1~N)
Skwhose weight W(Sk, tj) is the largest
is determined as the performed note
Are there multiple
notes Skwhose weights W(Sk, tj) are
the largest and whose distances from
Sk(k<i) is determined as
the performed note
Si+1is determined as the
performed note
Skis determined as
the performed note

Figure 2. Score-tracking algorithm
559

Page 5

4. TRIAL EXPERIMENTS OF FAMILY
ENSEMBLES
4.1 Methods
4.1.1 Subjects
We employed five pairs of subjects as follows:
[Pair-A] Primo was a female junior high school student who had
studied piano for eight years (intermediate). Secondo was her
father who had never played the piano and it was hard for him
to read the scores (pre-instrumental performer).
[Pair-B] Primo was a female elementary school student who had
played piano for four years (beginner). Secondo was her mother,
who had studied the electronic organ for two years (extreme
beginner).
[Pair-C] Primo was a nursery school child, the other daughter of
the Secondo of Pair-B. She had been studying piano for one
year (extreme beginner).
[Pair-D] Primo was a male student at our institute who had studied
the electronic organ for eight years (intermediate). Secondo was
a male student at the same institute who had never played the
piano and it was hard for him to read the scores (pre-
instrumental performer).
[Pair-E] Primo was the same as Pair-D. Secondo was another male
student at the same institute who had never played the piano and
it was hard for him to read the scores (pre-instrumental
performer).
4.1.2 Set pieces
We prepared three set pieces for duo play that the subjects all
knew well: “Prelude Op.28-7” and “Waltz Op.18” by F. Chopin,
and “Takibi” (“Bonfire”), a popular Japanese children’s song.
“Prelude” consists of 16 bars arranged for beginners. “Waltz”
consists of a Primo part arranged for beginners and a Secondo part
for intermediate. “Takibi” consists of a Primo part arranged for
extreme beginners and a Secondo part arranged for intermediate.
They are all arranged in C major.
The ranking of technical ease for Primo is “Takibi,” “Prelude” and
“Waltz.” On the other hand, the ranking for Secondo is “Prelude,”
“Takibi” and “Waltz” without FE, as well as “Waltz,” “Takibi”
and “Prelude” with FE. The ranking of technical ease with FE is
decided by the rhythmic difficulty of the score.
Hence, we asked Pair-A, the Primo who was the most experienced
among the five pairs, to perform all three pieces with FE, and to
perform “Prelude” without FE. According to each Primo’s ability
and preparation period, Pair-B and E tried “Prelude” and “Takibi”
with FE, and tried “Prelude” without FE. Pair-C and D tried
“Takibi” with and without FE.
Additional to using FE, we asked the subject who played Primo
for Pair-D and E to try to use minus-one software that cuts out
Primo’s part from a duo. We prepared minus-one software for two
set pieces, “Takibi” and “Prelude.” The set piece, “Takibi” was
prepared in three tempo patterns, i.e., 44 (slow), 60 (medium), and
72 (fast) quarter notes are beat per minute. Another set piece
“Prelude” was also prepared in three tempo patterns, i.e., 72
(slow), 88 (medium), and 104 (fast) quarter notes are beat per
minute.
4.1.3 Procedure of Experiments
We asked each Primo to practice the given set pieces beforehand,
until they could perform them with few mistakes. Then, we asked
each pair to jointly practice the duos, both with and without the
use of FE. Each joint session was fifteen minutes long.
[Record-1] The parent sat by his/her child who was practicing
alone for five minutes in Pair-A, B and C.
[Record-2] Pair-A, B and E had a joint practice to perform
“Prelude,” and Pair-C and D had a joint practice to perform
“Takibi” without FE for fifteen minutes.
[Explanation] We explained how to use the system and let the
Secondo try it, using a short trial piece.
[Record-3] Pair-A, B and E had a joint practice to perform
“Prelude,” and Pair-C and D had a joint practice to perform
“Takibi” with FE for fifteen minutes.
[Record-4] Pair-A, B and E had a joint practice to perform
“Takibi” with FE for ten minutes.
[Record-5] Pair-A had a joint practice to perform “Waltz” with FE
for ten minutes.
[Record-6] The Primo of Pair-D and E tried to perform “Takibi”
and “Prelude” with minus-one software in the three kinds of
tempo for each piece.
Our experiments were conducted on an acoustic grand piano that
can output MIDI data in a soundproof room in the second author’s
institute. The piano was connected to a personal computer on
which the FE system ran and to a sound generator.
In the joint practices with FE, the Secondo was allowed to use any
keys below C3, i.e., “Do” on the observer’s left, which Primo did
not need to use for playing the set pieces. Moreover, Secondo
used a special score. Since FE replaces the pitch of the performed
notes with the correct pitch, regardless of which keys Secondo
plays, Secondo does not need to read the actual notes on the score.
Therefore, we gave Secondo a so-called “rhythm score” in which
pitch data was omitted. In this score, a chord is represented by
piled note symbols, in numbers which correspond to the number
of notes in the chord. The Secondo can play a chord by hitting
either only one key or multiple keys. By hitting multiple keys to
perform a chord, the Secondo can separately control the volume
and length of each constituent note in the chord.
4.1.4 Questionnaires
After recording the performances with and without FE, the
subjects were asked to answer the following questions.
[To both Primo and Secondo]
Table 2. Results of the number of performance in the fifteen-
minute joint practices without and with FE
D uoSol oD uoSol o
P ai r-AP rel ude00131 ( P ri m o)
P ai r-BP rel ude
5*
2**
0140 ( P ri m o)
P ai r-CTaki bi
02 4 ( P ri m o)
P ai r-D Taki bi0057 ( P ri m o)
P ai r-E P rel ude0091 ( P ri m o)
W i thout FEW i th FE
P i ece

*The Secondo played a part of right hand.
**The Secondo played only five notes.
560

Page 6

Question-1: Is it difficult for the Secondo to perform a set piece
without FE? (“1” is very easy to “5” is very difficult)
Question-2: Why do you think so? (Optional)
Question-3: Is it difficult for the Secondo to perform set pieces
with FE? (“1” is very difficult to “5” is very easy)
Question-4: Why do you think so? (Optional)
Question-5: How do you feel about Secondo being able to
perform with only one finger? (Optional)
Question-6: Do you want to play a duo with FE again? (“Yes,”
“Marginal,” and “No”)
Question-7: Why do you think so? (Optional)
Question-8: Who do you want to play a duo with?
Question-9: Please comment on some expected improvements, etc.
(Optional)
[About minus-one software to the Primo of Pair-D and E]
Question-10: Could you perform the set pieces with the correct
melody and rhythm?
Question-11: Could you express agogics and fermata, i.e., subtle
changes in tempo?
Question-12: Could you express dynamics?
(In the case of the Slow, Middle, and Fast tempo, “1” is “could do
it” to “3” is “could not do it.”)
4.2 Results
Table 2 shows the number of subjects playing duo and solo in
fifteen minutes in the joint practice without and with FE. We
counted the frequency of a pair or one of the performers playing
from start to end even if there were many errors. Consequently,
even if the Secondo could perform only five notes in a duo, we
considered it one duo as shown in Pair-B and C. Table 3 shows
the results of Question-1, 3, 6, 8 as well as Table 4 shows the
results of Question-11, 12, 13, i.e., concerning to using minus-one
software.
We can see that none of the pairs could complete the duo in the
joint practice without FE. On the other hand, with FE they
immediately performed the duo more than ten times within the
fifteen-minute session (see Table 2). In the joint practices of Pair-
A, B, D, E without FE, the Primo was busily occupied telling the
Secondo which key(s) to hit. According to answers of Question-2,
especially the Secondos of Pair-A, D, and E did not know how to
read notes or the correct position corresponding to each note on
the score. Hence, the results of Question-1 on Table 3 show that
most Primos and Secondos thought it was very difficult to play a
duo without FE, i.e., answer of “5.”
Moreover, we should notice the increase in the number of solo
practices by the Primos in joint practice with FE (see Table 2).
According to the answers of Question-9, the Secondo of Pair-D
wrote that FE prompted him to practice playing the musical
instrument. Primos’ performances became poorer than those of
Secondos supported by FE, and so Primos wanted to recover their
domination by practice. This result suggests that FE can prompt
children who are studying the piano to practice more eagerly.
According to the answer of Question-5, The Primos of all pairs
did not feel envious of the FE support for their partner. However,
in the case of joint practice with FE, the performance became
imperfect due to the Primo’s mistakes in reproducing correct
pitches. Therefore, the Primo of Pair-A felt frustrated that she
could not blame her father when the performance was not perfect
(answer for Question-5).
According to the answers of Question-4 and 7, the Secondos of
Pair-D and E answered it was very wonderful that they could
perform from the start to the end of the pieces. According to the
answers of Question-6, all subjects want to use FE again. The
subjects of Pair-D and E wanted to use FE with their friends as a
form of entertainment (see the answer of Question-8).
On the other hand, the Secondo of Pair-B and C, who has a little
experience in playing the organ, answered that she hopes to be
able to play the piano more naturally. In the session of pair-B
without FE, she tried to perform only the right-hand-part. The rate
of correct notes that she played became 76.7% of all right-part
notes on the score in fifteen minutes. According to the answer of
Question-9, she proposed a way of support such that the Secondo
can perform the right-hand-part practically with the left-hand-part
supported by FE.
The subject who played the Primo of Pair-D and E tried to use
minus-one software in “Takibi” and “Prelude.” Table 4 shows that
he could almost perform it in the medium tempo. However, he
answered that he could not perform with subtle changes in tempo
and dynamics, i.e., musicality in any tempo (see answers for
Question-11 and 12).
4.3 Discussion
The results of the experiments clearly show that FE allows even
extreme beginner to immediately play duo performances. There
are differences in parent-child relations between the cases of joint
practicing with FE and without FE. These changes prompt the
children to practice more enthusiastically. Moreover, it is
suggested that FE has a desirable effect on their musicality.
4.3.1 Motivate children to practice more
In the joint practice without FE, the Primo taught the Secondo
how to read notes on the score, what position of keys the Secondo
should touch, and so on, in Pair-A, B, D, and E. We can see that
the Primos are very active in teaching their partners. However,
they could not perform a set piece once. In Pair-C, the Primo is a
nursery school child as well as an extreme beginner. Therefore,
successfully performing her part is all she can do. As a result, she
is displeased that her mother, who has a little experience,
generated wrong pitches.
In the joint practice with FE, the following dialogue took place
between the child and her father of Pair-A.

(After the first duo)
Child: machigaeta=
(I made mistakes.)
Father: =damedane::=
(That’s too bad.)
Child: =mukatuku
(Damn it!)
Father: ja, mou 1 kai ikimasyone
(Well, shall we play the piano one more time?)
561

Page 7

Father & Child: Play duo
(Made a bad start, and stopped immediately)
Father: omae mazu rensyu sei!
(Do your practice firstly!)
Child: un rensyu suruwa
(I see. I am going to practice.)
Child: Play solo
Child: yosya ikou
(OK! Let’s go!)
Father: eeka?
(Are you ready?)
Child: otosan to hikuto yarinikuiyona
(It is tricky to play with dad.)
[
Father: nandeyanen
(Why?)
Father & Child: Play duo
By using FE, a pre-instrumental parent and child with a little
experience are on almost even ground in reproducing correct
pitches. Consequently, if the child cannot perform the piece
perfectly through lack of practice, the parent can achieve the rank
of the child. In this dialogue, an utterance of the child, “otosan to
hikuto yarinikuiyona (It is tricky to play with dad.)” shows that
she is impatient with her incomplete performance. After all
experiments, her comment in this utterance relates that the
performance not being perfect is due to the Primo part, in
reproducing correct pitches (section 4.2.2). Therefore, she is
frustrated that she could not blame her father when the
performance is not perfect. However, the children’s impatience
may promote more enthusiastic practice. Actually, the results of
the experiments show that in the joint practice using FE the
Primos practiced by themselves (Table 2).
4.3.2 Construct their performance more musically
The fact that the parents could easily play their part correctly with
the help of FE also had a desirable effect on another aspect of
musicality. The following dialogue took place between Pair-A in
the joint practice with FE.

Child: ano ko tan tan tan tte hikuto oto togireru wakeyo=
(Well, your key-touch is like “tan tan tan.” This way of
touch indicates disconnected tapping sounds.)
Father: =un un un
(yeah yeah yeah)
Child: dakara motyotto sa
(So, a bit more like this…)
She played the Secondo part
Father: un motto ko nobasette iuno? =
(You mean I should sustain the sounds, right?)
Child: =so
(Right!)
In other words, the child wanted her father to play his part more
legato. This kind of conversation on musical expression was never
found in the joint practice of any of the pairs without FE; the
Secondo could consider nothing other than performing his/her part.
Moreover, though the father of Pair-A was an extreme beginner,
he was able to point out that their timing was off at dotted notes;

Child: o dekitayan=
(Oh! We completed it!)
Father: =a koko aitaine
(Well, I want the timing in this part to be right.)
Circling dotted notes with his finger
Child: ta-n ta tan
(ta-n ta tan)
Playing the dotted notes
[
Father: so so ta-n ta tan un kokodake rensyusuru
(Yeah yeah, ta-n ta tan. O.K. Shall we practice from this
part?)
Primarily, ensemble practice should be a time when the players
construct their own performance by conveying their musical ideas
and performance plans to each other. Without FE, such
discussions of musicality could not be achieved because of the
knowledge gap between the Primo and Secondo (e.g., the Secondo
cannot even read a normal score) and because of the Secondo’s
lack of technical ability. However, FE can fill this gap, that is, the
parent and the child become almost technically equal with the
support of FE. Thereby, they become able to practice jointly
toward a richer musical performance.
4.3.3 Edutainment
Tollinger [18] criticizes music education software based on the
following four aspects: a) educational value, b) pedagogical
validity, c) user-friendliness, and d) developmental consideration
of the audience/user. In the case of FE, the results of experiments
show the “user-friendliness” of FE (section 4.2.5). The subjects
answer that they want to use FE more even after experiments
using each way. The fact that the subjects started to practice more
eagerly and practice toward a richer musical performance shows
the “educational value” and “pedagogical validity” of FE.
Moreover, the children and their parents can learn the children’s
development level objectively by listening to their performance
while playing. Concerning the development of reproducing correct
pitches, for example, if the child mistakes some notes, he/she can
listen to poor harmonies. On the other hand, concerning
musicality, the expression in playing can be conveyed to both
partners. Consequently, the children and the parents can learn the
development of each other’s musicality. We think these good
effectiveness are brought by three important aspects for
constructing system as music edutainment that were enumerated in
Introduction: 1) Interaction with not a computer but a person, 2)
Knowing incorrect elements through listening to the performance,
and 3) Room for growth.
Please note that the parent's part of FE is actually not the piano but
another "new" musical instrument: the way of performance of this
part is very different from that of the conventional piano. This is
based on a practical decision that the parent need not master to
perform the piano: one who should master it is the child. It is also
important, however, FE does not intervene with the parent’s musical
expression at all. He/she would get to play music with good musical
expression as he/she wants through practice. This feature of the
parent's part of FE indirectly prompted the child to practice playing
the musical instrument. We think that the child also gets to perform
more musically through cooperatively constructing musical
performance by continuous use of FE with his/her parent.
562

Page 8

5. CONCLUSION
We have proposed and implemented a “Family Ensemble” that
supports ensemble performances by a pre-instrumental parent and
his/her child who has a little experience in playing a musical
instrument, specifically, piano. FE will help foster a new
cooperative learning method for use at home by parents and
children. FE does not directly support the Primo part so it does not
detract from the learning process with teachers. On the other hand,
FE always tracks where Primo is performing on the score. Using
the tracking results, FE supports the Secondo to perform the
correct notes. Our score-tracking module is based on
Dannenberg’s score-tracking algorithm, but we extended it to cope
with a peculiar error of beginners, replaying, in addition to the
extra, wrong, and missing types of error that are dealt with by
ordinary score-tracking systems. As a result, even if the Primo
makes many errors and even if the Secondo has little or no
experience in playing the piano, with FE they can immediately
enjoy performing duos together. In our experiments, we confirmed
that FE facilitates duo play by even the child with a little
experience in playing the musical instrument and even the pre-
instrumental parent. Furthermore, we found that in the joint
practices using FE some subjects discussed musical ideas that
helped them achieve richer musical expression. Such a situation
was never found in the joint practices without FE because the
Secondo, in particular, could consider nothing other than
performing his/her part. Thus, FE allows even beginners to aim for
richer musical expression. Moreover, the experiments showed that
FE prompts children to practice more eagerly.
We would like to conduct more experiments, with more subjects
and more musical pieces, at homes and at piano schools as well as
at our institute. This will enable us to determine the effect of other
environments on the process and assess the long-term usefulness
of the system.
6. ACKNOWLEDGMENTS
We would like to give an address of thanks to the subjects.
Oshima’s and Suzuki’s parts were supported by the National
Institute of Information and Communications Technology (NICT).
Nishimoto’s parts were partially supported by the Ministry of
Education, Science, Sports and Culture, Grant-in-Aid for
Scientific Research (C), 16500580, 2004 and were partially
supported by Hayao Nakayama Foundation for Science &
Technology and Culture.

7. REFERENCES
[1] Agay, D. Learning to Play Piano: Book 1, 1987.
[2] Blain, T. and Perkis, T. The Jum-O-Drum Interactive Music
System: A Study in Interaction Design, Proceedings of the
conference on Designing interactive systems: processes,
practices, methods, and techniques, pp.165-173, 2000.
[3] Bloch, J. and Dannenberg, R. B. Real-time computer
accompaniment of keyboard performance. Proc. ICMC 1985,
ICMA (1985), 279-289.
[4] Casio, LK-55 Full Size Keyboard with Key Lighting System,
http://www.casio.com/index.cfm?fuseaction=products.detail
&product=LK%2D55.
[5] Dannenberg, R. B. An on-line algorithm for real-time
accompaniment. Proc. ICMC 1984, ICMA (1984), 193-198.
[6] Farbood, M., Pazstor, E., Jennings, K. Hyperscore, A
graphical Approach to Composing Music. IEEE Computer
Graphics and Applications, Special Issue on Emerging
Technologies (2004), 50-54.
[7] Goto, M., Hidaka, I., Matsumoto, H., Kuroda, Y., and
Muraoka, Y. A Jazz session system for interplay among all
players --- VirJa Session (Virtual Jazz Session System) ---,
Proc. ICMC 1996, ICMA (1996), 346-349.
[8] Grubb, L. and Dannenberg, R. B. A stochastic method of
tracking a vocal performer. Proc. ICMC 1997, ICMA (1997),
301-308.
[9] Jennings, K. Toy Symphony: An International Music
Technology Project for Children. Music Education
International, 2 (2003), 3-21.
[10] Kihara R. Piano Land, Vol.1-5, with music data, 1991-92 (an
educational material in Japan).
[11] McCarthy, C., Bligh, J., Jennings, K., Tangney, B. Virtual
Collaborative Learning Environments for Music: Networked
Drumsteps. Proceedings of the Computer Assisted Learning
Conference (2003)
[12] Miyoshi, A. Miyoshi Piano Method, Vol.1-8 (an educational
material in Japan).
[13] Miyoshi, A. Piano Duo Sympathetic World Outside language,
MUSICA NOVA, Vol. 31 No. 9, pp.30-32, Ongaku no tomo
sya, 2000 (It is a Japanese magazine).
[14] Oshima, C., Nishimoto, K., Miyagawa, Y., and Shirosaki, T.
A Concept to Facilitate Musical Expression, Proc. Creativity
& Cognition 2002, ACM Press(2002) pp.111-117.
[15] Orio, N., Lemouton, S., and Schwarz, D. Score following:
State of the art and new developments, Proc. NIME 2003,
36-41.
[16] Pachet, F. and Addessi, A. When Children Reflect on Their
Playing Style: Experiments with the Continuator and
Children, ACM Computers in Entertainment, Vol.2, No.1,
Article 05, 2003.
[17] Smoliar, S. W., Waterworth, J. A., and Kellock, P. R.
pianoFORTE: A System for Piano Educatio Beyond Notation
Literacy, ACM Multimedia 95, 1995.
[18] Tollinger, V. L. Multimedia for the Music Educator: A
Review and Critique of Programs to be Used in the School
and Home, Technological Directions in Music Learning
(ejournal), 1999.
[19] Vercoe, B. The synthetic performer in the context of live
performance. Proc. ICMC 1984, ICMA (1984), 199-200.

563