Content uploaded by Bradley W Vines
Author content
All content in this area was uploaded by Bradley W Vines
Content may be subject to copyright.
BRADLEY W. VINES, REGINA L. NUZZO,
AND DANIEL J. LEVITIN
Department of Psychology, McGill University
THIS ARTICLE INTRODUCES THEORETICAL and
analytical tools for research involving musical
emotion or musical change. We describe techniques
for visualizing and analyzing data drawn from time-
varying processes, such as continuous tension judg-
ments, movement tracking, and performance tempo
curves. Functional Data Analysis tools are demon-
strated with real-time judgments of musical tension (a
proxy for musical affect) to reveal patterns of tension
and resolution in a listener’s experience. The derivatives
of tension judgment curves are shown to change with
cycles of expectation and release in music, indexing the
dynamics of musical tension. We explore notions of
potential energy and kinetic energy in music and
propose that affective energy is stored or released in the
listener as musical tension increases and decreases.
Differential calculus (and related concepts) are intro-
duced as tools for the analysis of temporal dynamics in
musical performances, and phase-plane plots are
described as a means to quantify and to visualize
musical change.
Received September 5, 2003, accepted March 28, 2005
MUSIC IS SAID TO REPRESENT the dynamics
of human emotion. Conveying the changing
nature of those emotions is fundamental to
musical performance (Scherer & Zentner, 2001), and
understanding this process is among the most impor-
tant problems in music cognition (Jones, 1993; Meyer,
1956). Indeed, Aristoxenus (364-304 B.C.E.)—perhaps
the first music cognition theorist—argued that music
can only be understood by studying both the musician
and the mind of the listener. Music listening also
involves lower-level perceptual processes. The human
mind perceives relations between simultaneous
elements of sound (e.g., notes in a chord) as well as
relations between sequential elements of sound
(Bregman, 1990; Dowling & Harwood, 1986; Lerdahl &
Jackendoff, 1983). Melodic perception, which develops
from infancy (Dowling, 1999; Plantinga & Trainor, in
press; Saffran, 2003; Trehub, 2003), relies on a sensitiv-
ity to change over time, as does the perception of har-
monic chord progressions and rhythmic relations.
In this article, we introduce new analytic techniques
drawn from Newtonian physics that are useful for
quantifying the dynamic nature of stimuli that change
over time, such as music, and human participants’
dynamic reactions to these stimuli. Shepard (1984) has
persuasively argued for the plausibility of analogies
between physics and human cognition with his ideas of
resonant kinematics for perceiving and imagining sen-
sory stimuli. The techniques we introduce will allow
researchers to solve several problems inherent to
dynamic data sets, not the least of which are issues asso-
ciated with multiple comparisons and repeated meas-
ures when the data set (sampled responses to music
over real time) may contain thousands of observations.
(A subset of these new techniques was first applied
to music in McAdams, Vines, Vieillard, Smith, &
Reynolds, 2004.)
In what follows, we will refer to time-dependent
processes in music as “musical dynamics,” based on the
terminology introduced by Jones (1993). The effect of
temporal context in music—what has played before and
what is about to play—continuously influences a lis-
tener’s experience. An identical physical stimulus may
be perceived differently, depending on the context
(Jones, 1993; Shepard, 1984); thus music perception is a
dynamic, time-dependent process. Changes in loudness
in a musical performance are but one example of tem-
poral dynamics in music in the larger Jonesian sense.
Other examples include fluctuations in tempo, changes
in pitch, and adjustments to timbre.
The study of emotions has recently become a central
focus in music cognition research (Juslin & Sloboda,
2001; Scherer, Zentner, & Schacht, 2002). The continu-
ous tracking technique, introduced by Nielsen (cited in
Madsen & Fredrickson, 1993) and further explored
ANALYZING TEMPORAL DYNAMICS IN MUSIC:
Differential Calculus, Physics, and Functional Data Analysis Techniques
Music Perception VOLUME 23, ISSUE 2, PP. 137-152, ISSN 0730-7829, ELECTRONIC ISSN 1533-8312 © 2005 BY THE REGENTS OF THE
UNIVERSITY OF CALIFORNIA. ALL RIGHTS RESERVED. PLEASE DIRECT ALL REQUESTS FOR PERMISSION TO PHOTOCOPY OR REPRODUCE ARTICLE CONTENT
THROUGH THE UNIVERSITY OF CALIFORNIA PRESS’S RIGHTS AND PERMISSIONS WEBSITE AT WWW.UCPRESS.EDU/JOURNALS/RIGHTS.HTM
Analyzing Temporal Dynamics in Music 137
04.MUSIC.23_137-152.qxd 01/12/2005 14:53 Page 137
by Madsen, Fredrickson, Krumhansl, and other
researchers (Fredrickson, 2000; Krumhansl, 1996, 1997;
Krumhansl & Schenck, 1997; Madsen & Fredrickson,
1993; Schubert, 2001; Sloboda & Lehmann, 2001; Vines,
Krumhansl, Wanderley, & Levitin, in press) has become
an important tool for measuring musical emotion. In an
experiment that employs this form of measurement,
participants continuously adjust a computer input
device to report their ongoing, real-time judgments of
the emotional, structural, or aesthetic content in a piece
of music. Functional Data Analysis1tools (FDA; Ramsay
& Silverman, 1997, 2002) are ideal for studying contin-
uous measures to reveal underlying structures in the
data and relations between the data and other continu-
ous processes, such as tempo change, loudness, rate of
movement, and note density (Levitin, Nuzzo, Vines, &
Ramsay, 2005). FDA was developed primarily as an
alternative to general linear model-based statistics that
assume the dependent variables come from independ-
ent, discrete observations; FDA treats a curve represent-
ing multiple observations as the fundamental unit of
analysis. Although continuous data may be obtained in
a variety of ways, we will use continuous tracking data
in this article as an illustrative and typical example.
This article introduces the interpretation of judgment
curves obtained in continuous tracking paradigms as well
as the analysis of the derivatives of those curves. We
demonstrate a way to visualize the data with phase-plane
plots to reveal the dynamics of change in musical per-
formance. FDA enables researchers to pinpoint in time—
with reference to the musical score—when participants
perceive important musical and emotional events. We
hope to provide new insight into musical processes that
involve change, such as tension and release in music, and
to show how FDA can be applied to address a broad range
of research questions in music perception and cognition.
Tension in Music
Musicians and composers speak metaphorically of ten-
sion and release in music as dynamic processes. Musical
tension gives rise to an emotional experience in listen-
ers that references real-world, nonmusical counter-
parts, such as tension in physical objects, in the body,
and in social situations. These common life experiences
cause a convergence of meaning for “tension” across
individuals (Fredrickson, 1997), thus making tension
a practical index of emotional experience in scientific
research.
The experience of “release” complements the experi-
ence of “tension,” and the ebb and flow of these oppo-
sites elicits emotional responses to music (Krumhansl,
2002; Patel, 2003). Generally, music theorists view ten-
sion in a piece of music as being related to phrasing
structure (A. Vishio, personal communication, June,
2003), and empirical investigations have found that ten-
sion tends to build up during a phrase, to peak toward
the phrase ending, and then to subside, often rapidly
(Krumhansl, 1996; Krumhansl & Schenck, 1997).
Musicians and composers employ tension in a variety of
ways by utilizing listeners’ expectations and by exploit-
ing basic psychophysical perceptual principles (Balkwill
& Thompson, 1999; Meyer, 1956). Fundamental to all
uses of tension is its relation to “release”—the inevitable
decrease in tension (Storr, 1972).
Measuring Tension in Music: The Continuous Judgment
The continuous tension judgment, introduced by Frede
Nielsen (see Madsen & Fredrickson, 1993; Fredrickson,
1995), is a way to assess a participant’s real-time experi-
ence of a musical piece. Participants indicate the ongoing
tension they feel while listening to a musical piece by
adjusting a dial-shaped Continuous Response Digital
Interface (Fredrickson, 1995, 1997, 1999, 2000; Madsen
& Fredrickson, 1993), a foot pedal with a range of motion
(Krumhansl & Schenck, 1997), or a moveable slider
(McAdams et al., 2004; Vines et al., in press; Vines,
Wanderley, Krumhansl, Nuzzo, & Levitin, 2003). The
apparatus type does not appear to materially affect the
judgment itself or its contour through time. These ten-
sion judgments are influenced by a wide variety of struc-
tural, harmonic, and rhythmic features in music,
including pitch height, note density, loudness, and
harmonic dissonance (Fredrickson, 1995; Madsen &
Fredrickson, 1993; Krumhansl, 1996; Krumhansl &
Schenck, 1997); they are informative about a person’s
affective state (Fredrickson, 1995) and are robust across
participant populations differing in age, level of musical
skill, and degree of familiarity with the musical selection
being judged (Fredrickson, 1997, 1999, 2000). Tension
judgments also correlate with emotional states and with
physiological measures (Krumhansl, 1997; Krumhansl &
Schenck, 1997). Furthermore, music theoretical analyses
of tension (Lerdahl, 1996, 2001) predict judgments of
tension in music (Bigand, 2003; Bigand & Parncutt, 1999;
Lerdahl & Krumhansl, 2003; Krumhansl, 1996; Patel,
2003; Smith & Cuddy, 2003; Vega, 2003).
138 B. W. Vines, R. L. Nuzzo and D. J. Levitin
1Functional Data Analysis software tools are available to all inter-
ested researchers at http://ego.psych.mcgill.ca/pub/ramsay. The data
analysis functions can be downloaded in Matlab code from that Web
site. Sample applications of those functions, which provide a tem-
plate for doing analyses, are also obtainable from the site.
04.MUSIC.23_137-152.qxd 01/12/2005 14:53 Page 138
Taken together, there is thus a growing body of evi-
dence for the validity and robustness of the continuous
tension judgment as a measure that captures listeners’
real-time affective experience of a musical piece.
Interpreting Continuous Judgment Responses:
The Meaning of Derivatives
Participants in continuous tension experiments are
asked to move some device in one direction as their
experience of tension increases and in the other direc-
tion as their experience of tension decreases (e.g., up
and down on a slider, in and out on spring-loaded
tongs). In the remainder of this section, we introduce
an interpretation of the first two derivatives of tension
curves thus obtained in a musical context. We concep-
tualize the first derivative as the affective velocity of
(a person’s reactions to) a selection of music, and
the second derivative as the affective acceleration. The
descriptor “affective” is used instead of “tension” for the
derivatives because (as discussed above) judgments of
tension can serve as a proxy for affective experience or
emotion. More specifically, it is likely that tension
corresponds to the “arousal” dimension of emotion, as
opposed to “valence” (McAdams et al., 2004; Russell,
1979; Schubert, 2001); this claim is supported by
Krumhansl and Schenck’s (1997) finding that continu-
ous judgments of tension were consistently correlated
with continuous judgments of fear, an emotion closely
related to arousal. Additionally, the processes we are
describing pertain to general emotional experience;
thus the term “affect,” which is more general than “ten-
sion,” is suitable.
Schubert (2002) introduced the notion of comparing
continuous judgment derivatives in order to determine
if two curves are indeed similar. He showed, for example,
that Pearson correlations are more reliable when the
derivatives of the measures are compared, versus the
observed curves, noting that “differenced data refers to
the change in emotional response which takes place
from moment to moment” (p. 230). Our goal is to
extend this notion by analyzing two derivatives in an
effort to quantify affective aspects of musical experience.
The concepts of affective velocity and acceleration
can be illustrated with an example using idealized ten-
sion curves. Music contains periodicities at hierarchical
levels (Jones, 1993; Krumhansl, 1996; Lerdahl &
Jackendoff, 1983; Lerdahl, 2001; Temperley, 2001), and
for simplicity we will illustrate this with a sinusoidal
wave. Figure 1.1 shows a hypothetical tension curve
made in response to a musical piece that induces a
cosine wave pattern of experienced tension for listeners.
The tension level begins at awith an absolute minimum
and then increases to a theoretical absolute maximum
tension level at c, passing through an intermediate level,
b. A release in tension occurs as the curve passes
through point d, and the pattern begins again at a.
This idealized sinusoidal model exemplifies a simple,
straightforward case in order to illustrate the interpre-
tation of tension curves and their derivatives. The ten-
sion and release cycle can, of course, occur at a slow
(Figure 1.2) or fast (Figure 1.3) rate.
Derivatives and Differential Calculus
Music is dynamic, and the calculus (Leibnitz, 1686;
Newton, 1726) was invented to study changes across
time. If Yrepresents our original function (the tension
curve associated with a musical piece), then the first
derivative (Y) represents the instantaneous rate of
Analyzing Temporal Dynamics in Music 139
FIG. 1. (.1) A sinusoidal wave representation, with shifted phase, as an idealized model for musical tension judgments. (.2) A cosine wave
with half the frequency of the wave in (.1), modeling tension judgments for a musical piece with slower emotional changes. (.3) A cosine wave with
twice the frequency of the wave in (.1), modeling tension judgments for a musical piece with rapid emotional changes.
04.MUSIC.23_137-152.qxd 01/12/2005 14:53 Page 139
change of our primary measure, and the second deriva-
tive (Y) represents the instantaneous rate of change of
the first derivative. It is certainly mathematically possi-
ble to take the first and second derivatives of tension
curves, but do these derivatives have musical (or music-
cognitive) meaning? We explore this question in the
data analysis described below.
Affective velocity, the rate of change of a tension curve
(Y), reveals how quickly the experience of tension is
increasing or decreasing for a listener. The first deriva-
tive of the idealized tension curve of Figure 1.1 is shown
in Figure 2.1.
The derivative curve reveals a combination of the
composer’s and performer’s manipulation of tension as
well as the emotional dynamics of a listener’s experience
by showing how fast tension changes. From Y, we can
learn not only that tension is increasing, for example,
but also whether the tension is increasing quickly or
slowly. Note for example the difference between
Figures 1.1, 1.2, and 1.3 and the corresponding deriva-
tives in Figures 2.1, 2.2, and 2.3. In Figure 1.2, the cosine
wave has a period that is twice as long as the curve in
1.1; the tension and release are spread out over a longer
period of time, yielding a first derivative of smaller
magnitude (Figure 2.2). Such a pattern would be likely
for a musical piece with a relatively slow progression
(e.g., “My Funny Valentine” played by Miles Davis, or
Ravel’s Bolero). The power of this analytical approach
lies in the ability to quantify musical tension over time.
In Figure1.3, the cosine wave has a period that is half as
long as the curve in Figure 1.1; the corresponding first
derivative, indexing rate of change, has a greater magni-
tude (Figure2.3). Such a pattern, characterized by rapid
change, might be found for a musical piece with rapid
fluctuations in tension (e.g., “Cherokee” played by
Charlie Parker). As described here, the first derivative
does have musical meaning: the rate of change of a
person’s emotional response to music (Schubert, 2002).
One could imagine categorizing different musical
pieces by the magnitude of their first derivative in emo-
tional response.
Whether or not the derivatives of affective experi-
ence, velocity and acceleration, have a distinct percep-
tual reality (i.e., that the experience of acceleration is
different from the experience of a steady change in ten-
sion) is an empirical question that must be approached
through experimental investigation. In this article,
we hypothesize that affective acceleration and affective
velocity are related to unique dimensions of the musical
experience.
Energy Transfer in Musical Dynamics
Given that tension increases and decreases throughout
a musical piece, we can introduce an analogy in which
fluctuations in tension correlate with an ebb and flow
of affective (or musical) energy that is stored up and
released within a listener. Increasing tension in the
piece generates a buildup of affective energy in the lis-
tener; a subsequent decrease in tension results in a
release of the pent-up energy. (In this way, the listener
functions like a capacitor by storing up a charge until
a threshold is reached, at which point the charge is
released.) The storage and release of affective energy
causes changes to a listener’s experience of emotional
content in a piece and is presumably under the control,
to a large extent, of the composer and performers. That
is, stored musical energy has the potential to alter
the emotional state of a listener when it is ultimately
released.
140 B. W. Vines, R. L. Nuzzo and D. J. Levitin
FIG. 2. The first derivatives of the idealized tension curves shown in Figures 1.1 through 1.3.
04.MUSIC.23_137-152.qxd 01/12/2005 14:53 Page 140
Musical experience (and performance art in general,
such as dance and theater) usually entails a sequence of
energy transfers over time; the emotive experience of a
performance increases from a baseline level, builds up
to a heightened level, drops to a state of release, and
then builds up again (see Krumhansl, 2002; Meyer,
1956). Krumhansl and Schenck (1997) found that emo-
tional responses to both dance and music built up over
the duration of phrases and ultimately tended to
decrease toward phrase endings. McAdams and col-
leagues (2004) found similar patterns of peaks and
decays in their participants’ emotional responses to a
contemporary electroacoustic musical piece. In this
sense, musical performance can be thought of as involv-
ing the transfer of energy from the composer and per-
formers to the listener, and a cyclical process of storage
and release of that energy within the listener.
The rate at which affective velocity (the first deriva-
tive) changes over time is expressed by the second
derivative, which we refer to as affective acceleration.
Having established that the composer is leading the lis-
tener toward increasing tension (for example), the sec-
ond derivative, affective acceleration, reveals whether
the listener is being brought to that point of tension at a
constant rate, at an increasing rate, or at a decreasing
rate. When the rate of change is increasing, it may feel as
though a goal is being reached more and more quickly,
as in the following thought example (B. Thompson,
personal communication, August 20, 2004). Imagine a
horror movie in which a character is slowly descending
a staircase into a dark basement—the tension has a pos-
itive velocity. Suddenly, the villain, lurking in wait in the
darkness below, becomes visible to the audience—the
rate of change in tension increases, and an increasing
slope of the tension curve is evidence of positive affec-
tive acceleration.
POTENTIAL AND KINETIC ENERGY
We are borrowing the concepts of potential and kinetic
energy from Newtonian physics to describe the dynam-
ics of affective change in music and to better under-
stand how derivatives of tension curves relate to
musical meaning. Past research has explored analogies
between musical phenomena and the laws of physics by
comparing dynamic temporal processes in music, such
as ritardando, with processes in the physical world, such
as the movement of objects through a gravitational field
or a runner’s rate of deceleration (Clarke, 1999;
Davidson & Correia, 2002; Friberg & Sundberg, 1999;
Todd, 1999). These studies have revealed that aspects of
change in music are structured in a similar way to
movement and change in the physical world. Larson
(2004) drew parallels between melodic expectation and
the forces of inertia, gravity, and magnetism, arguing
“we experience musical motions metaphorically in
terms of our experience of physical motions” (Larson,
2004, p. 462). Shepard (1987, 1995) posits that the
human mind evolved to incorporate external physical
laws into mental representation, explicitly mentioning
the principles of physical dynamics and movement
(analogous to our temporal derivatives) among those
universal constraints that shaped human cognition
(Shepard, 1984). The analytical tools that have emerged
in physics to describe flux in the physical world are
also useful for understanding musical fluctuation and
experience.
In our idealized example, we can say that affective
experience has a tendency to gravitate toward a stable
state of complete release. This is the baseline (y1)
seen in Figure 1. Musical pieces may not always resolve
to a point of low tension, especially in twentieth-
century art music, as the composer flouts various max-
ims of musical expectation or seeks different aesthetic
aims than a sense of completion or release. (This is
perhaps the musical equivalent of flouting Grice’s con-
versational maxims in language; Grice, 1975.) However,
this does not negate the value of derivatives in under-
standing the ongoing nature of changes in tension
throughout a piece—in fact, the derivative analysis we
are proposing does not depend on how a piece ends, but
rather is a tool for analyzing the changes in musical
tension throughout; it can, furthermore, permit us to
quantify the state of tension/resolution that exists at the
end of a piece.
In physics (and in our analogy to music), potential
energy is synonymous with stored energy; here, stored
musical energy is the potential to alter the emotional
state of a listener. The potential for change is at its great-
est at a point of maximum tension in music, such as
point cin Figure1. The music has brought listeners to a
peak of tension; hence, a resolution is likely to follow to
satisfy the listeners’ expectations and to adhere to musi-
cal convention. (There may also be physiological con-
straints that prevent a person from maintaining a
high-tension experience for very long durations of
time.) The higher the tension becomes—or the longer
that it is held at a high level—the stronger the tendency
or “pull” exerted by the music to return the listener to a
more moderate state; in our idealized tension curve,
this pull is indicated by an acceleration in the direction
of the pull—it is negative at point cof Figures 1 and 2,
for instance. (The slope of affective velocity is negative
at point c; see Figure 2.) Note that in our simplified
model, which represents the pattern of musical tension
Analyzing Temporal Dynamics in Music 141
04.MUSIC.23_137-152.qxd 01/12/2005 14:53 Page 141
as a sinusoid, nonzero values for velocity and accelera-
tion always co-occur, except at the maxima and min-
ima; thus we can use the existence of acceleration as an
indicator that potential energy is being created. But real
musical pieces could certainly have stretches of positive
(or negative) velocity while the acceleration is zero.
A strong potential for change in emotional states
also occurs when the tension level is very low—small
manipulations by performers and composers can easily
increase tension, and listeners may expect such an
increase as well. At such points (a, and ain Figures 1
and 2), there will be high potential energy for increasing
tension levels, as indicated by a strong positive acceler-
ation in the idealized tension curves. (The slope of
affective velocity, shown in Figure 2, is positive at points
a, and a.) Therefore, the maxima in potential energy
occur at both points aand cof Figures 1 and 2.
In addition, we hypothesize that kinetic energy, the
energy of movement, is related to the rate of change in
emotion—affective velocity. If musical energy is con-
served over time, as in our idealized tension model,
then as a listener’s kinetic energy increases, potential
energy decreases—the transfer of musical energy facili-
tates a change in emotional state. This exchange of
energy continues until potential energy and the associ-
ated affective acceleration decrease to zero, at which
point kinetic energy reaches a maximum (points band
din Figure 1). The potential energy for change in the
opposite direction then increases, while kinetic energy
and the associated affective velocity decrease toward
zero at points aand c. This trade-off between kinetic
energy and potential energy, velocity and acceleration,
continues throughout the hypothetical tension judg-
ment. In real music, the composer or performers
contribute energy to the music to create contours of
experience that are not strictly periodic.
Phase-Plane Plots
A Functional Data Analysis technique for visually rep-
resenting the dynamics of continuous processes is the
phase-plane plot, in which the second derivative is plot-
ted against the first derivative (Ramsay & Silverman,
2002). In the musical context we have introduced, the
phase-plane plot graphs affective acceleration against
affective velocity. Figure 3.1 shows the phase-plane plot
for the idealized cosine curve of Figure 1.1 (it is thus a
plot of Figure 2.1 versus its derivative). Purely oscilla-
tory behavior yields a perfectly circular phase-plane
plot. A larger radius in the plot corresponds to a greater
amount of musical energy transfer, as would occur for a
piece of music with rapid changes in tension or in the
velocity of tension. A piece of music with no changes
in tension would yield a single point at the center.
Figure 3.2 shows a plot in which there are large acceler-
ations and relatively small velocities, and Figure 3.3
shows a plot in which there are small accelerations and
large velocities. Note that the phase-plane plot does not
explicitly plot time, though time markers along the path
can be annotated, as we will demonstrate with Figure 6.
We can relate the poles of the phase-plane plot to
the point markers in Figures 1.1 and 2.1. Points of
142 B. W. Vines, R. L. Nuzzo and D. J. Levitin
FIG. 3. (.1) Phase-plane plot for the cosine wave shown in Figure 1.1, with affective acceleration plotted against affective velocity.
(.2) A hypothetical phase-plane plot for a judgment with large affective accelerations and small affective velocities. (.3) A hypothetical
phase-plane plot for a judgment with small affective accelerations and large affective velocities.
04.MUSIC.23_137-152.qxd 01/12/2005 14:53 Page 142
maximum kinetic energy and affective velocity occur at
band d, as indicated by their distance from the y-axis
in Figure 3. The maximum positive velocity occurs at
point b, the maximum negative velocity occurs at point
d, and the velocity is zero at the center of the plot (the
origin). Points aand care points of maximum potential
energy, as indicated by their distance from the x-axis in
Figure 3. Point ahas maximum positive acceleration
and point chas maximum negative acceleration. The
acceleration is 0 at the center of the plot. At aand c, the
first derivative (affective velocity) is 0.
An Example With Real Data
To illustrate the new application of techniques based on
differential calculus and phase-plane plots, we will dis-
cuss a segment of the data collected in a recent study by
Vines et al. (in press).
Data Collection
Thirty musically trained participants were randomly
divided into three treatment groups and were presented
with a performance of Stravinsky’s second piece for
solo clarinet (1920/1993). (Final N28 after discarding
two outliers.) One group (auditory only, n9) heard
the performance without seeing it, a second group
(visual only, n9) saw the performance without hear-
ing it, and the third (auditoryvisual, n10) both
saw and heard the performance. This piece was chosen
because it is an unaccompanied work that lacks a metric
pulse; hence the performer was free to move expres-
sively and idiosyncratically, and the experimental
participants who saw the performance could not rely
on metrically based body movements to make their
judgments.
Each participant performed the continuous tension
judgment using a slider with a span of 7 cm, sampled
at 10 Hz. This rate is relatively high compared to
past studies involving continuous musical ratings.
Krumhansl and Schenck (1997) used 4 Hz, as did
Krumhansl (1996), and Madsen and Fredrickson
(1993) used 2 Hz, for example. Madsen and
Fredrickson (1993) found that sampling rates above
2 Hz did not add information to the tension curves
that they obtained. If 2Hz is close to the maximum
frequency of relevant information, then a 10 Hz sam-
pling rate more than adequately satisfies the Nyquist
theorem, which states that the sampling rate must be at
least twice the frequency of the signal being measured;
thus no distortions due to sampling rate were intro-
duced. In our data, the average change per sample was
approximately 1% of the total range of the slider. The
following instructions were given:
Use the full range of the slider to express the TENSION
you experience in the performance. Move the slider
upward as the tension increases and downward as
the tension decreases. Begin with the slider all the
way down.
Each participant used his or her best intuition about
the meaning of “tension,” following previous studies
(Fredrickson, 1995, 1997, 1999, 2000; Krumhansl,
1996, 1997; Krumhansl & Schenck, 1997; Madsen &
Fredrickson, 1993).
Analysis
DATA PREPARATION, SCALING, CENTERING
The raw data consisted of 28 records (one for each
participant) of 800 samples each (80 seconds of music
at 10 Hz sampling rate). The values obtained from the
MIDI slider ranged from 0 to 127. To begin the analysis,
each participant’s judgment was scaled to an interval of
0 to 1. Due to the scaling, each sample over time repre-
sented the proportion of the maximum tension experi-
enced by a participant during the entirety of the piece.
(Other transformation schemes and interpretations are
discussed in Vines et al., in press). Each vector then was
centered by subtracting the median of all values from
each element in the vector (a process also referred to as
“zero-meaning”). This, in essence, equated the central
tendency in each participant’s judgment. The scaled
and centered data for a single representative participant
in the auditory-only condition are shown in the top
panel of Figure 4.
CREATING A FUNCTIONAL OBJECT
In order to calculate the derivatives of the responses, we
converted the data from discrete points (at 10 Hz) into
smoothed representations known as functional objects
(Levitin et al., 2005; Ramsay, 2000; Ramsay & Silverman,
1997). Ideally, such functional curves eliminate noise
and collection artifacts in the discrete, raw data, and
they approach the true, underlying process from which
the data were collected with greater accuracy.
Each observed record of discrete data was modeled
by a basis expansion of 150 sixth-order B-splines
(Deboor, 1978; Unser, 1999) and a lambda smoothing
parameter of 0.1, using custom software (Ramsay, 2003)
written in Matlab (The Mathworks Inc., 2003). See
Levitin and colleagues (2005) for a description of
Functional Data Analysis modeling and guidelines
for choosing the number and order of B-splines for
Analyzing Temporal Dynamics in Music 143
04.MUSIC.23_137-152.qxd 01/12/2005 14:53 Page 143
smoothing. The bottom panel of Figure 4 shows the
FDA-modeled data for the auditory-only participant
whose raw data appear in the top panel. The visual sim-
ilarity between the two panels of Figure 4 demonstrates
the accuracy of the B-spline approximation.
INTERPRETATION OF THE DATA
In the empirical data we are referring to, we assume that
the position of the slider corresponds to each partici-
pant’s real-time experience of the tension in the musical
piece, perhaps with some response delay or anticipa-
tion. Past research with continuous measures found
response delays from 1 second (Kuwano & Namba,
1985) to over 3.25 seconds (see Schubert, 2001; Smith &
Cuddy, 2003). Here, we are not concerned with the
response delays, which are relatively minor compared
to the time spans of interest, and methods exist for
normalizing delays among participants (Levitin et al.,
2005; Ramsay & Silverman, 1997). The individual
participants vary around a mean response lag.
Mathematically, this delay has the effect of adding a
small constant, which does not materially affect inter-
pretations at the level of musical phrases.
PHASE-PLANE ANALYSIS WITH DATA
To illustrate the preceding concepts, we consider Vines
et al. (in press) study introduced earlier, using the
methods presented in this article. Figure 5.1 shows the
mean functional tension judgment for participants in
the auditory-only condition. All 80 seconds of the
judgment mean are shown. There are clear periodicities
in the mean curve that correspond to the musical
structure of the piece. The composition has three main
sections (Friedland, n.d.; A. Vishio, personal communi-
cation, April, 4, 2003) that differ in musical content.
The first and third sections (0 to ~33.5 seconds and
~66 seconds to the end, respectively) mirror each other
musically in that they both consist of fast-moving lines
with a wide pitch range that elaborate upon similar
thematic material. The second section (~33.5 to ~66
seconds) consists of slower-moving lines at a low pitch
height that elaborate upon different thematic material.
The mean tension curve in Figure 5.1 is relatively high
in section one, with some small-scale fluctuations. The
second section is marked by a low tension rating with
two major subsection peaks. The tension level returns
to a high level in the third section, which shows another
subdivision. Overall, the global shape follows a sinu-
soidal path with two full cycles (low to begin, high in
section one, low in section two, high in section three,
and low to end) with small-scale cycles of tension and
release adding fluctuation within the global sinusoid.
First and second derivatives of the tension mean are
depicted in Figures 5.2 and 5.3, respectively.
144 B. W. Vines, R. L. Nuzzo and D. J. Levitin
FIG. 4. (Top panel) Scaled raw data obtained from a single representative participant in the auditory-only condition of the experiment described in
text. The measurement was a continuous judgment of musical tension. (Bottom panel) The same judgment after functional
modeling. Note the similarity between the two panels, indicating an accurate B-spline estimation.
04.MUSIC.23_137-152.qxd 01/12/2005 14:53 Page 144
The phase-plane plot in Figure 6.1 plots a segment of
the curve in 5.3 against the corresponding segment
of the curve in 5.2, to illustrate three things: (a) the
dynamics of tension experienced in a performance of
Stravinsky’s piece, (b) the cycles of energy transfer
between potential and kinetic, and (c) the ongoing
changing relations between affective velocity and affec-
tive acceleration in the piece. For brevity, we have
plotted only the judgments made during an especially
interesting 15-second section of the piece: 25 seconds
from the beginning up to 40 seconds from the begin-
ning. (Arabic numerals indicate the time markers in
clock seconds.) This segment corresponds to an impor-
tant transition in the musical piece when the first sec-
tion ends with a high peak note and the second section
begins after a rest; the musician’s body movements con-
tinued through the silence. The time markers corre-
spond to specific events in the performance.
Figure 7 (please see color plate) shows an image of the
clarinetist at each time marker as well as a pointer to the
corresponding location in the musical score. Graphs of
the mean tension curves, the first derivative curves cor-
responding to affective velocity, and the second deriva-
tive curves corresponding to affective acceleration are
shown below the score for all three presentation groups
(auditory only, visual only, and auditoryvisual) over a
duration that includes all of the important time points.
The phrase that builds up to a high peak note begins in
second 27. In second 28 the high peak note is reached
and held. In second 31, the high note ends and a silent
rest ensues, with the performer’s movements continu-
ing into the silence. The performer begins to move in
preparation for the next section in second 32; he takes a
breath and moves the clarinet bell downward in antici-
pation of the new phrase, which starts in second 33. The
second phrase in the new section begins in second 36.
Analyzing Temporal Dynamics in Music 145
FIG. 5. (.1) The smoothed mean curve obtained from nine functionalized auditory-only judgments. (.2) Representing affective velocity with
the first derivative of the mean curve. (.3) Representing affective acceleration with the second derivative of the mean curve.
FIG. 6. (.1) Phase-plane plot for the auditory-only tension mean, showing
Y
vs.
Y
, or
affective velocity
vs.
affective acceleration
.
Arabic numerals in the graph mark clock time in seconds from the beginning of the piece. (.2) Phase-plane plot for the visual-only mean judgment.
(.3) Phase-plane plot for the auditoryvisual mean judgment.
04.MUSIC.23_137-152.qxd 01/12/2005 14:53 Page 145
The curve in Figure 6.1 begins with moderate levels
of energy change, as indicated by its proximity to the
origin at second 25. The energy change reaches greater
levels from second 31 through second 33, before return-
ing to a low energy state at second 36. The rounded
path that circles the origin indicates that the listeners
reported a varied and dynamic (i.e., rapidly changing)
musical experience during this section of the piece;
both positive and negative affective velocities were felt
as well as positive and negative affective accelerations.
An analysis of the relation between acceleration, veloc-
ity, and the performance stimuli yields insight into
listeners’ emotions and demonstrates the utility of
phase-plane plotting.
The phase-plane plot shown in 6.1 reveals emotional
dynamics that were driven by events in the performance.
Affective velocity becomes positive in second 27 (accel-
eration was already positive) just as the phrase that built
to the peak note began. When the high peak note is
reached in second 28, the velocity achieves a positive
maximum and the acceleration passes through 0,
becoming negative. The change in acceleration signals
the end of an emotional buildup. Once the target peak
note is reached, acceleration becomes negative, signaling
a pull in potential energy toward release. This moment
in the energy transfer is analogous to point bin our ide-
alized model (compare point bin Figure 3.1 to second 28
in Figure 6.1). When the high note ends in second 31,
the velocity becomes negative. The auditory-only group
could not see the musician; hence his movements in
second 32 could not have influenced the emotional
dynamics experienced by the group. Just as the new
musical phrase begins in second 33, the affective accel-
eration becomes positive. This moment in energy trans-
fer is analogous to point din the idealized tension curve.
In second 33, an event in the music (the onset of a new
phrase) appears to have directly influenced affective
acceleration, which changed from negative to positive.
Though the judgment of tension continues to change
with negative velocity, the acceleration grows more and
more positive in response to the musical change. This
example demonstrates the utility of phase-plane plot-
ting: Hidden dynamics in the dependent variable were
revealed. We posit that changes in music can act on a lis-
tener’s experience at the level of affective acceleration
and that there may be very little temporal delay for such
an effect. As the new section continues into second 36,
the overall amount of energy transfer dies down, as
shown by a phase-plane trajectory that moves closer and
closer to the origin.
The phase-plane plot makes it possible to compare the
dynamics of two or more judgment curves (e.g., from
different participants, different musical pieces, or differ-
ent conditions of an experiment). In the experiment
under consideration here, recall that there were three
participant conditions that differed in sensory modality
of presentation: auditory only, visual only, and
auditoryvisual. Figure 6.2 depicts the phase-plane plot
for the visual-only condition. We can deduce that the
magnitude of energy transfer was relatively weak for the
visual-only condition (as indicated by a path that is gen-
erally closer to the origin). Note that the difference
in phase-plane magnitude involves the derivatives of the
mean tension judgment and not the undifferentiated
place data, which were scaled to eliminate varying uses
of the slider. The visual-only phase-plane plot follows the
same general trajectory as the auditory-only phase-plane
plot, but with a phase that is shifted forward in time.
The graph in 6.2 shows a maximum negative affective
acceleration in second 31, whereas the corresponding
maximum in 6.1 occurs at least 1 second later in second
32. Additionally, the visual-only phase-plane plot
achieves a maximum negative velocity before second 32,
which is well before the auditory-only plot reaches its
corresponding maximum in second 33. A comparison of
the auditory and visual phase-plane plots suggests that
during this segment similar emotional dynamics were
conveyed by the sound and by the sight of the perform-
ance, though the overall magnitude of energy transfer
was less in the visual-only condition and emotional
information was conveyed earlier by the visual modality.
The first portion of the auditoryvisual phase-plane
plot shown in Figure 6.3 is nearly identical to the audi-
tory phase-plane plot. The velocity becomes positive in
second 27, as the phrase leading to the peak high note
begins. When the peak high note arrives in second 28,
the velocity reaches a positive maximum and the accel-
eration becomes negative. The high note ends at second
31, just before acceleration reaches a negative maximum.
At this point, the timing of the auditoryvisual phase-
plane plot deviates from that of the auditory-only plot.
The visual content can account for this divergence.
As mentioned above, the clarinetist begins his move-
ments in anticipation of the new section. In second 32,
he takes a breath and prepares his clarinet by moving
it forward before initiating any sound. Based on the
phase-plane plot depicted in Figure 6.3, we posit that
the performer’s expressive movements influenced par-
ticipants in the auditoryvisual group. During second
32, the acceleration becomes positive; the velocity
remains negative, but the rate of change begins to
increase once the player’s anticipatory movements
begin. This is another example for which changes in
the performance are reflected as changes in affective
146 B. W. Vines, R. L. Nuzzo and D. J. Levitin
04.MUSIC.23_137-152.qxd 01/12/2005 14:53 Page 146
Analyzing Temporal Dynamics in Music 147
acceleration. The auditory-only phase-plane plot does
not achieve a positive acceleration until second 33,
when the sound begins; those participants could not see
the performer and, therefore, could not be influenced
by his movements. The fact that the change in accelera-
tion begins earlier for the auditoryvisual group sug-
gests that the performer’s anticipatory movements (his
breathing, body lunging, and swooping down of the
clarinet bell) influenced the judgments by providing
affective information in advance of the sound.
Summary
The phase-plane plot analysis just described generated
insight into the dynamics of emotional experience in
music. Visualizing the derivatives of tension judgments
revealed patterns of energy exchange that were not
obvious in the raw position data. The phase-plane plot
enabled comparisons of energy transfer across experi-
mental conditions and brought to light different aspects
of musical influence on emotion. Based on this analysis
we posit that changes in a musical stimulus can have a
direct effect on affective acceleration and that this effect
is revealed with a shorter time lag than position data
(i.e., a peak in slider position lags well behind the causal
music event that influences affective acceleration more
quickly). Using a comparison of phase-plane plots
across experimental conditions, we also found that a
musician’s movements influence perception at key
points in the music for those who can see the perform-
ance. This influence itself is dynamic and evolving over
time, making techniques such as the phase-plane plot
ideal for analyses.
A striking conclusion of Vines et al. (in press)
was that the body movements of musicians do carry
meaningful information about the music and that visual
information can indeed contribute to the overall experi-
ence of emotion and the perception of phrasing struc-
ture in a musical performance. The independent
contribution of the present article is the finding that
emotional dynamics conveyed visually were similar to,
but out of phase with, emotional dynamics conveyed
aurally during a performance segment; participants
who could both hear and see the performances were
influenced by the phase-advanced affective information
in the visual channel.
Discussion
This article introduces new techniques for theoretical
and quantitative analysis of musical dynamics (change
over time in music). We show that concepts from
differential calculus are well suited for use in the
domain of music cognition and that physics analogies
can deepen interpretations of emotional response.
Functional Data Analysis techniques for modeling,
smoothing, and taking derivatives are described, and
we present phase-plane plots as a practical tool for visu-
alizing musical dynamics. We demonstrate the value of
the analytical tools that we introduce by means of an
idealized model for musical tension and an analysis of
continuous tension ratings made in response to a musi-
cal performance. Here, tension is treated as a proxy for
emotion, based on experimental evidence; hence the
ideas discussed may be generalized to affective experi-
ence in general.
A primary contribution of this article is to introduce
differential calculus as a means for analyzing the dynam-
ics of musical performances. Calculus has refined tech-
niques for exploring rates of change and movement
over time. A person’s emotional experience is changing
continuously while listening to music, and a piece of
music that creates strong affective changes can be called
“moving.” With respect to one’s continuous magnitude
of emotional experience, we label the first derivative
affective velocity and the second derivative affective
acceleration and we show how these derivatives relate to
expectation and release, a primary catalyst for affective
energy in music.
Terms from physics such as energy have an intuitive
sense in the domain of music. We suggest here that
analogies between physical mechanics and musical
dynamics may lead to insights related to emotional
change in a listener’s experience. In Newtonian mechan-
ics, acceleration (the second derivative of position—
affective acceleration in the sinusoidal model of musical
tension) is related to potential energy while the velocity
(the first derivative of position—affective velocity in the
sinusoidal model of musical tension) is related to
kinetic energy. We explicate the links between these
concepts and musical dynamics, by describing the
transfer of energy that occurs during a musical piece.
Specifically, affective energy is stored or released in the
listener as tension increases and decreases, and the
changing values of the first and second derivatives of
tension ratings give evidence of this energy transfer.
This article also introduces phase-plane plots as a
practical tool for visualizing relations between a mea-
sure’s first and second derivatives, two latent variables
that quantify musical change and affective dynamics in
a piece. With this plotting technique it is possible to
compare the underlying dynamics of different data sets
and to attribute meaning to the velocity and accelera-
tion of a musical measurement. A phase-plane plot
04.MUSIC.23_137-152.qxd 01/12/2005 14:53 Page 147
analysis of collected tension judgments revealed that
the timing of peaks in emotional dynamics differed
across presentation conditions. The experience for
those who could see the performer was phase shifted in
advance compared to those who could only hear, thus
demonstrating the influence of the musician’s move-
ments through a direct comparison of emotional
dynamics across presentation conditions.
The techniques described herein will allow emotion
and music cognition researchers to refine their analysis
of musical dynamics. Music listening is the experience
created by mentally organizing sound over time; there-
fore precise methods are necessary for characterizing
not just musical events with respect to time but also the
ways in which those events change—and the ways in
which those changes change over time. Methods from
calculus, physics, and FDA are well suited for applica-
tions with musical data, and they have the potential to
reveal latent relations in data sets and to enhance our
understanding of temporal dynamics in music.
Extensions
Having already introduced the concepts of energy,
velocity, and acceleration to a musical context, it is use-
ful to contemplate additional analogies between musi-
cal and physical dynamics. Concepts from physics that
have meaning in the physical world may well have anal-
ogous forms in music (see Davidson & Correia, 2002).
Some such concepts already have common currency
in music criticism—people speak intuitively about the
musical momentum of a piece, for example. Whether
these can be described by mathematical equations
drawn from the physics paradigm is a question for fur-
ther inquiry. Less intuitive, perhaps, is the concept of
external force. In physics, we know that an external force
must be applied to alter the direction or speed of a phys-
ical object in motion. In music, we might hypothesize
that the composers and performers apply the external
force that influences the direction or speed of affective
experience in a listener. Additional physical concepts
may be useful for quantifying musical meaning: friction,
mass,gravity,power,work, and oscillating systems.
Of course, the limitations of analogies between musi-
cal phenomena and physics must be kept in mind.
Concepts in physics describe relations and changes in
matter and its alternate form, energy. Science has cre-
ated methods by which to quantify and, to some extent,
to control matter and energy. However, no such method
is yet available for quantifying musical experience itself,
though introspective feedback and physiological meas-
urements can be used as indicators. Tools developed in
physics comprise an analytical technology that is suited
for processes involving change; musical experience is
undoubtedly characterized by change and, therefore,
its analysis benefits from the methods developed in
physics.
Other Data Sets
We illustrated the methods introduced here with rat-
ings of musical affect. The methods are quite general,
however, and they can be applied to any data for which
the underlying process is continuous—even if the actual
data are “discrete.” The data need not be derived from
participant judgments either. As an example, Repp
(1992, 1996, 1997) has extensively investigated the tim-
ing profiles of expressive piano performances. With the
methods described here, which were not available when
Repp did his seminal work, it would be possible to
examine the functional dynamics of tempo changes
(such analyses have not yet been performed). For
instance, a first derivative taken from the tempo profile
of a Chopin Étude would represent the tempo velocity,
or the speed at which tempo was changing, and the sec-
ond derivative would reveal the acceleration of tempo.
The derivatives could be used to ask questions such as:
How smoothly or monotonically does the performer
slow down or speed up? Are there particular points in
time when the tempo acceleration is greatest? Does the
performer replicate similar tempo profiles, at the level
of velocity and acceleration, across performances of the
same piece?
Repp extensively characterized interperformer differ-
ences by factor analyzing the raw timing (using Inter
Onset Intervals) and loudness performance curves.
Repp (1992), for example, extracted four principal com-
ponents to distinguish the interperformer differences
among 28 pianists. Functional principal component
analysis techniques (fPCA) now exist which differ
from the methods used by Repp in that they enable a
researcher to explore major modes of variation over
time. For example, fPCA might reveal that, compared to
the average, one group of performers tends to decrease
in tempo at the beginning of a phrase while increasing
in tempo toward the end. Functional techniques also
yield greater control over the smoothness and visual
presentation of the principal component curves. But
perhaps the biggest advantage of applying these meth-
ods to a data set like Repp’s is the ability to analyze
derivatives of the curves. Functional factor analyses on
the derivatives—tempo velocity and tempo accelera-
tion, in this case—provide a new way to describe inter-
performer variability. These methods may well uncover
148 B. W. Vines, R. L. Nuzzo and D. J. Levitin
04.MUSIC.23_137-152.qxd 01/12/2005 14:53 Page 148
Analyzing Temporal Dynamics in Music 149
latent aspects of pianists’ styles that will help to further
quantify and characterize the performers’ aesthetics.
Furthermore, one could correlate the tempo velocity
and tempo acceleration with participants’ judgments of
emotion in the piece to better understand how tempo
dynamics contribute to affective experience (Sloboda &
Lehmann, 2001). Such an analysis would reveal the
functional relations between the two dependent vari-
ables, tempo and tension, and their derivatives.
Alternate measurements of emotion, such as those
obtained by Clynes’ sentographic recordings (see
Clynes & Nettheim, 1982), are analyzable with the tech-
niques presented here. It would also be possible to apply
these methods to the analysis of human movement
data, and in particular to the movements of musicians
(Wanderley, 2002).
Inferential Statistics
It is possible to examine the sampling variability of
functional curves and their derivatives using methods
that are still under development, including functional
tests of significance (Ramsay & Silverman, 1997), func-
tional bootstrapping (Efron, 1979; Efron & Tibshirani,
1993; Ramsay & Silverman, 2002), and random fields
(Shafie, Sigal, Siegmund, & Worsley, 2003). These
methods allow researchers to determine whether a
derivative differs significantly from zero or whether two
sets of curves differ significantly from one another.
Functional Data Analysis and Time-Series Analysis
FDA and Time-Series Analysis (TSA) are complemen-
tary areas of statistics, each of which is tailored to a dif-
ferent aspect of data sampled over time. TSA focuses on
short-term stochastic behavior in a system that satisfies
the assumptions of stationarity; that is, TSA assumes
that the patterns in data will remain constant over time
with a relatively unchanging mean and an unchanging
variance. FDA is designed to explore global changes in
data over time, including changes in mean and changes
in variance within and between curves. FDA does not
maintain assumptions about stationarity. Additionally,
Functional Data Analysis offers analogous tools to
those used in traditional statistics (e.g., correlations,
general linear modeling, principal components analy-
sis, analysis of covariance, and significance testing) but
with functions of time as the units for analysis as
opposed to data points. FDA and TSA techniques may
be used in coordination, for example, to analyze the sto-
chastic properties in a time series, using TSA, after the
global patterns of change and variation have been
identified and removed, using FDA (J. Ramsay, per-
sonal communication, September 13, 2004).
Authors’ Note
A version of this article was submitted in partial
fulfilment of the requirements for the PhD degree by
the first author at McGill University. The authors are
grateful to Jim Ramsay, Bruno Repp, Associate Editor
Bill Thompson, and the anonymous reviewers for help-
ful comments on earlier drafts of this manuscript. This
work was supported by grants from NSERC, CFI,
SSHRC, and VRQ to Daniel J. Levitin, and a J. W.
McConnell McGill Majors Fellowship and a CIRMMT
Doctoral Fellowship to Bradley W. Vines.
Address correspondence to: Daniel J. Levitin,
Department of Psychology, McGill University, 1205
Avenue Penfield, Montreal, QC H3A 1B1 Canada.
E-MAIL dlevitin@psych.mcgill.ca
References
BALKWILL, L. L., & THOMPSON, W. F. (1999). A cross-cultural
investigation of the perception of emotion in music:
Psychophysical and cultural cues. Music Perception, 17,
43-64.
BIGAND, E. (2003). Traveling through Lerdahl’s tonal pitch space
theory: A psychological perspective. Musicae Scientiae, 7,
121-140.
BIGAND, E., & PARNCUTT, R. (1999). Perceiving musical tension
in long chord sequences. Psychological Research, 62, 237-254.
BREGMAN, A. S. (1990). Auditory scene analysis: The perceptual
organization of sound. Cambridge, MA: MIT Press.
CLARKE, E. F. (1999). Rhythm and timing in music. In D.
Deutsch (Ed.), The psychology of music (2nd ed., pp. 473-500).
San Diego: Academic Press.
CLYNES, M., & NETTHEIM, N. (1982). The living quality of music:
Neurobiologic patterns of communicating feeling. In M.
Clynes (Ed.), Music, mind, and brain: The neuropsychology of
music (pp. 47-82). New York: Plenum Press.
DAVIDSON, J. W., & CORREIA, J. S. (2002). Body movement. In
R. Parncutt & G. E. McPherson (Eds.), The science and
psychology of music performance: Creative strategies for teaching
and learning (pp. 237-250). Oxford: Oxford University Press.
04.MUSIC.23_137-152.qxd 01/12/2005 14:53 Page 149
DEBOOR, C. (1978). A practical guide to splines. Berlin: Springer-
Verlag .
DOWLING, W. J. (1999). The development of music perception
and cognition. In D. Deutsch (Ed.), The psychology of music
(2nd ed., pp. 603-625). San Diego: Academic Press.
DOWLING, W. J., & HARWOOD, D. L. (1986). Music cognition.
Orlando, FL: Academic Press.
EFRON, B. (1979). Bootstrap methods: Another look at the
jackknife. Annals of Statistics, 7, 1-26.
EFRON, B., & TIBSHIRANI, R. J. (1993). An introduction to the
bootstrap. New York: Chapman & Hall.
FREDRICKSON, W. E. (1995). A comparison of perceived musical
tension and aesthetic response. Psychology of Music, 23, 81-87.
FREDRICKSON, W. E. (1997). Elementary, middle, and high school
student perceptions of tension in music. Journal of Research
in Music Education, 45, 626-635.
FREDRICKSON, W. E. (1999). Effect of musical performance on
perception of tension in Gustav Holst’s First Suite in E-flat.
Journal of Research in Music Education, 47, 44-52.
FREDRICKSON, W. E. (2000). Perception of tension in music:
Musicians versus nonmusicians. Journal of Music Therapy,
37, 40-50.
FRIBERG, A., & SUNDBERG, J. (1999). Does music performance
allude to locomotion? A model of final ritardandi derived
from measurements of stopping runners. Journal of the
Acoustical Society of America, 105, 1469-1484.
FRIEDLAND, S. (n.d.). Stravinsky’s three pieces for clarinet solo.
Retrieved June 20, 2002, from
http://clarinet.cc/Repertoire003.html
GRICE, H. P. (1975). Logic and conversation. In P. Cole (Ed.),
Syntax and semantics (Vol. 3, pp. 41-58). New York: Academic
Press.
JONES, M. R. (1993). Dynamics of musical patterns: How do
melody and rhythm fit together? In T. J. Tighe & W. J.
Dowling (Eds.), Psychology and music: The understanding of
melody and rhythm (pp. 67-92). Hillsdale, NJ: L. Erlbaum.
JUSLIN, P. N., & SLOBODA, J. A. (2001). Music and emotion:
Theory and research. New York: Oxford University Press.
KRUMHANSL, C. L. (1996). A perceptual analysis of Mozart’s
Piano Sonata K. 282: Segmentation, tension, and musical
ideas. Music Perception, 13, 401-432.
KRUMHANSL, C. L. (1997). An exploratory study of musical
emotions and psychophysiology. Canadian Journal of
Experimental Psychology, 51, 336-352.
KRUMHANSL, C. L. (2002). Music: A link between cognition and
emotion. Current Directions in Psychological Science, 11,
45-50.
KRUMHANSL, C. L., & SCHENCK, D. L. (1997). Can dance
reflect the structural and expressive qualities of music?
A perceptual experiment on Balanchine’s choreography of
Mozart’s Divertimento No. 15. Musicae Scientiae, 1 (Spring),
63-85.
KUWANO, S., & NAMBA, S. (1985). Continuous judgment of
level-fluctuating sounds and the relationship between overall
loudness and instantaneous loudness. Psychological Research,
47, 27-37.
LARSON, S. (2004). Musical forces and melodic expectations:
Comparing computer models and experimental results. Music
Perception, 21, 457-498.
LEIBNITZ, G. W. (1686). Nova methodus pro maximis et
minimis, itemque tangentibus, qua nec irrationals quantitates
moratur [A new method for maxima and minima, and also
for tangents, which is not obstructed by irrational qualities].
Acta Eruditorum.
LERDAHL, F. (1996). Calculating tonal tension. Music Perception,
13, 319-363.
LERDAHL, F. (2001). Tonal pitch space. Oxford: Oxford University
Press.
LERDAHL, F., & JACKENDOFF, R. (1983). A generative theory of
tonal music. Cambridge, MA: MIT Press.
LERDAHL, F., & KRUMHANSL, C. L. (2003). La teoría de la tensión
tonal y sus consecuencias para la investigación musical [The
theory of tonal tension and its implications for musical
research]. In J. Martín Galán & C. Villar-Taboada (Eds.), Los
últimos diez años en la investigación musical (pp. 63-89).
Valladolid: Servicio de Publicaciones de la Universidad de
Valladolid.
LEVITIN, D. J., NUZZO, R., VINES, B. W., & RAMSAY, J. O. (2005).
Introduction to functional data analysis. Manuscript
submitted for publication.
MADSEN, C. K., & FREDRICKSON, W. E. (1993). The experience of
musical tension: A replication of Nielsen’s research using the
Continuous Response Digital Interface. Journal of Music
Therapy, 30, 46-63.
MEYER, L. B. (1956). Emotion and meaning in music. Chicago:
University of Chicago Press.
MCADAMS, S., VINES, B. W., VIEILLARD, S., SMITH, B. K., &
REYNOLDS, R. (2004). Influences of large-scale form on
continuous ratings in response to a contemporary piece in a
live concert setting. Music Perception, 22, 297-350.
NEWTON, S. I. (1726/1972). Philosophial Naturalis Principia
Mathematica. Cambridge, MA: Harvard University Press.
PATEL, A. D. (2003). Language, music, syntax and the brain.
Nature Neuroscience, 6, 674-681.
PLANTINGA, J., & TRAINOR, L. J. (in press). Memory for melody:
Infants use a relative pitch code. Cognition.
RAMSAY, J. O. (2000). Functional components of variation in
handwriting. Journal of the American Statistical Association,
95, 9-15.
RAMSAY, J. O. (2003). FDA Analysis Toolkit. Unpublished
software application for Matlab, available for download from
ftp site ego.psych.mcgill.ca/pub/ramsay/
RAMSAY, J. O., & SILVERMAN, B. W. (1997). Functional data
analysis. New York: Springer-Verlag.
150 B. W. Vines, R. L. Nuzzo and D. J. Levitin
04.MUSIC.23_137-152.qxd 01/12/2005 14:53 Page 150
Analyzing Temporal Dynamics in Music 151
RAMSAY, J. O., & SILVERMAN, B. W. (2002). Applied functional data
analysis: Methods and case studies. New York: Springer-Verlag.
REPP, B. H. (1992). Diversity and commonality in music
performance: An analysis of timing microstructure in
Schumann’s “Traumerei.” Journal of the Acoustical Society of
America, 92, 2546-2568.
REPP, B. H. (1996). Patterns of note onset asynchronies in
expressive piano performance. Journal of the Acoustical
Society of America, 100, 3917-3932.
REPP, B. H. (1997). Expressive timing in a Debussy prelude: A
comparison of student and expert pianists. Musicae Scientiae,
1, 257-268.
RUSSELL, J. A. (1979). Affective space is bipolar. Journal of
Personality and Social Psychology, 37, 345-356.
SAFFRAN, J. R. (2003). Mechanisms of musical memory in infants.
In I. Peretz & R. J. Zatorre (Eds.), The cognitive neuroscience of
music (pp. 32-41). New York: Oxford University Press.
SCHERER, K. R., & ZENTNER, M. R. (2001). Emotional effects of
music: Production rules. In P. N. Juslin & J. A. Sloboda (Eds.),
Music and emotion: Theory and research. Oxford, New York:
Oxford University Press.
SCHERER, K. R., ZENTNER, M. R., & SCHACHT, A. (2002). Emotional
states generated by music: An exploratory study of music
experts. Musicae Scientiae, Special Issue 2001-2002, 149-171.
SCHUBERT, E. (2001). Continuous measurement of self-report
emotional response to music. In P. N. Juslin & J. A. Sloboda
(Eds.), Music and emotion: Theory and research (pp. 393-414).
New York: Oxford University Press.
SCHUBERT, E. (2002). Correlation analysis of continuous
emotional response to music: Correcting for the effects of
serial correlation. Musicae Scientiae, Special Issue 2001-2002,
213-236.
SHAFIE, K., SIGAL, B., SIEGMUND, D., & WORSLEY, K. J. (2003).
Rotation space random fields with an application to fMRI
data. Annals of Statistics, 31, 1732-1771.
SHEPARD, R. N. (1984). Ecological constraints on internal
representation: Resonant kinematics of perceiving, imagining,
thinking, and dreaming. Psychological Review, 91, 417-447.
SHEPARD, R. N. (1987). Toward a universal law of generalization
for psychological science. Science, 237, 1317-1323.
SHEPARD, R. N. (1995). Mental universals: Toward a twenty-first
century science of mind. In R. L. Solso, & D. W. Massaro
(Eds.), The science of the mind: 2001 and beyond (pp. 50-62).
New York: Oxford University Press.
SLOBODA, J. A., & LEHMANN, A. C. (2001). Tracking performance
correlates of changes in perceived intensity of emotion during
different interpretations of a Chopin piano prelude. Music
Perception, 19, 87-120.
SMITH, N. A., & CUDDY, L. L. (2003). Perceptions of
musical dimensions in Beethoven’s Waldstein Sonata: An
application of Tonal Pitch Space theory. Musicae Scientiae,
7, 7-34.
STORR, A. (1972). The dynamics of creation. New York:
Atheneum.
STRAVINSKY, I. (1920/1993). Second piece for solo clarinet. In
Three pieces. London: Chester Music Ltd.
TEMPERLEY, D. (2001). The cognition of basic musical structures.
Cambridge, MA: MIT Press.
TODD, N. P. M. (1999). Motion in music: A neurobiological
perspective. Music Perception, 17, 115-126.
TREHUB, S. E. (2003). Musical predispositions in infancy.
In I. Peretz & R. J. Zatorre (Eds.), The cognitive neuroscience
of music (pp. 3-20). New York: Oxford University Press.
UNSER, M. (1999). Splines: A perfect fit for signal and image
processing. IEEE Signal Processing Magazine, 16,
22-38.
VEGA, D. A. (2003). A perceptual experiment on harmonic
tension and melodic attraction in Lerdahls’ Tonal Pitch Space.
Musicae Scientiae, 7, 35-55.
VINES, B. W., KRUMHANSL, C. L., WANDERLEY, M. M., &
LEVITIN, D. J. (in press). Cross-modal interactions in the
perception of musical performance. Cognition.
VINES, B. W., WANDERLEY, M. M., KRUMHANSL, C. L., NUZZO, R. L.,
& LEVITIN, D. J. (2003). Performance gestures of musicians:
What structural and emotional information do they
convey? In A. Camurri & G. Volpe (Eds.), Gesture-based
communication in human-computer interaction (pp. 468-478),
the proceedings of the 5th International Gesture Workshop,
GW 2003, Genova, Italy, April 15-17, 2003. Berlin: Springer
Verlag .
WANDERLEY, M. M. (2002). Quantitative analysis of non-obvious
performer gestures. In I. Wachsmuth & T. Sowa (Eds.),
Gesture and sign language in human-computer interaction
(pp. 241-253). Berlin: Springer Verlag.
04.MUSIC.23_137-152.qxd 01/12/2005 14:53 Page 151
Analyzing Temporal Dynamics in Music
152
FIG. 7.
Analyzing Temporal Dynamics in Music
153
FIG. 7. Images corresponding to important clock time locations in the phase-
plane plots with lines pointing to corresponding locations in the score. Graphs of
the tension judgment, the affective velocity (Y’), and the affective acceleration
(Y’’) are shown below the score for the duration of interest, with lines of
corresponding colors to mark clock time locations. The solid curve corresponds
to auditory only, the dotted curve represents the visual only group mean, and the
dashed and dotted curve signifies the auditory+visual mean.
