Conference PaperPDF Available

The Effects of Pitch and Dynamics on the Emotional Characteristics of Piano Sounds

Authors:

Abstract

The piano is an instrument extensively used in classical, jazz, and pop music, since its broad pitch range and ample dynamic levels allow the instrument to become self-contained and versatile for various kinds of music. Previous work has found the piano to be emotionally neutral among eight tested non-sustaining instruments. This paper further explores the emotional characteristics of piano sounds with a listening test comparing isolated one-second sounds of different pitches and dynamics on the piano over ten emotional categories: Happy, Sad, Heroic, Scary, Comic, Shy, Romantic, Mysterious, Angry, and Calm. In the experiment, the loud bass was ranked the most Angry and Heroic, and the effect dropped with increasing pitch. The soft treble was ranked the most Calm and Shy, and the effect dropped with decreasing pitch. The trend was clear across the octaves. Both loud and soft sounds have distinguishing emotional characteristics. In contrast, the emotional categories Mysterious and Happy were not much affected by dynamics.
TheEffectsofPitchandDynamicsontheEmotionalCharacteristics
ofPianoSounds
Chuck-jeeChau AndrewHorner
DepartmentofComputerScienceandEngineering
TheHongKongUniversityofScienceandTechnology
ClearWaterBay,Kowloon,HongKong
chuckjee@cse.ust.hk,horner@cs.ust.hk
ABSTRACT
The piano is an instrument extensively used in classical, jazz,
and pop music, since its broad pitch range and ample dy-
namic levels allow the instrument to become self-contained
and versatile for various kinds of music. Previous work has
found the piano to be emotionally neutral among eight tested
non-sustaining instruments. This paper further explores the
emotional characteristics of piano sounds with a listening test
comparing isolated one-second sounds of different pitches
and dynamics on the piano over ten emotional categories:
Happy, Sad, Heroic, Scary, Comic, Shy, Romantic, Mysteri-
ous, Angry, and Calm. In the experiment, the loud bass was
ranked the most Angry and Heroic, and the effect dropped
with increasing pitch. The soft treble was ranked the most
Calm and Shy, and the effect dropped with decreasing pitch.
The trend was clear across the octaves. Both loud and soft
sounds have distinguishing emotional characteristics. In con-
trast, the emotional categories Mysterious and Happy were
not much affected by dynamics.
1. INTRODUCTION
Previous research has shown that both sustained [1, 2, 3] and
non-sustained [4, 5] instrument sounds have strong emotional
characteristics. For example, it has found that the trumpet,
clarinet, and violin are relatively joyful compared to other
sustained instruments, even in isolated sounds apart from mu-
sical context, while the horn is relatively sad. The marimba
and xylophone are relatively happier compared to other sus-
tained instruments, while the harp and guitar are relatively
depressed.
In our recent study of non-sustained instrument sounds [4,
5], several musical instruments were tested: plucked violin,
guitar, harp, marimba, xylophone, vibraphone, piano, and
harpsichord. Among the eight instruments, the piano was
found to be ranked neutral for eight emotional categories:
Happy, Sad, Heroic, Scary, Comic, Shy, Joyful, and De-
pressed. It was often ranked in the middle, indicating neutral
emotional characteristics relative to the other tested instru-
ments. Perhaps one might find this surprising since the piano
has the widest repertoire among all instruments in classical,
Copyright: c
2015 Chuck-jee Chau et al. This is an open-access article
distributed under the terms of the Creative Commons Attribution License 3.0
Unported, which permits unrestricted use, distribution, and reproduction in
any medium, provided the original author and source are credited.
jazz, and pop music, and there are also abundant transcrip-
tions written for the piano. What is the range of emotional
characteristics for the piano?
Previous work has only studied single mid-range pitches of
the instrument, and the loudness was also equalized to allow
consistent comparison. We were curious about the effects of
pitch and dynamics in isolated sounds on the emotional char-
acteristics of the piano.
We then formulated the current study to carefully compare
the emotional characteristics of piano pitches at different dy-
namic levels. The study includes representative pitches rang-
ing from the lowest to the highest octave. The dynamic levels
included loud, medium, and soft (forte,mezzo, and piano).
All sounds were isolated with a duration of one second. They
were compared pairwise over ten emotional categories.
This work provides a systematic overview of the emotional
characteristics of the piano across the different octaves at dif-
ferent dynamics. This research will help recording and au-
dio engineers, composers, and pianists manipulate the emo-
tional characteristics of the instrument in live performances
and recordings.
2. BACKGROUND
Much work has been done on emotion recognition in music,
especially addressing melody [6], harmony [7], rhythm [8],
etc.
Researchers have gradually established connections be-
tween music emotion and timbre. Scherer and Oshinsky [9]
found that timbre is a salient factor in the rating of synthetic
sounds. Peretz et al. [10] showed that timbre speeds up dis-
crimination of emotion categories. It was also found that tim-
bre is essential to musical genre recognition and discrimina-
tion [11, 12].
Eerola et al. [1, 13] showed a direct connection between
music emotion and timbre. Eerola carried out listening tests
to investigate the correlation of emotion with temporal and
spectral sound features. The study confirmed strong correla-
tions between features such as attack time and brightness and
the emotion dimensions valence and arousal for one-second
isolated instrument sounds.
We followed up Eerola’s work with our own studies of mu-
sic emotion and timbre [2, 3, 4, 14, 15] to find out if some
sounds were consistently perceived as being happier or sadder
in pairwise comparisons. We designed listening tests to com-
pare sounds from various string, wind, and percussion instru-
ments. The results showed strong emotional characteristics
for each instrument. The horn and flute were highly ranked
ICMC 2015 – Sept. 25 - Oct. 1, 2015 – CEMI, University of North Texas
– 372 –
for Sad, while the violin, trumpet, and clarinet were highly
ranked for Happy. The oboe was ranked in the middle. In
another experiment, the harp, guitar and plucked violin were
highly ranked for Sad, while the marimba, xylophone, and
vibraphone were highly ranked for Happy. And piano was
ranked in the middle.
3. EXPERIMENT
Our experiment was a listening test, where subjects com-
pared pairs of instrument sounds over different emotional cat-
egories.
3.1 Test Materials
3.1.1 Stimuli
The stimuli used in the listening tests were sounds of a grand
piano with different combinations of dynamics (loudness)
and pitch.
All sounds were from the RWC [16] sample library. The
sounds were played by the same pianist on a Steinway grand
piano. Three different level of dynamics were used: forte,
mezzo, and piano, with forte being the loudest, piano the soft-
est, and mezzo in between. To avoid the effect of intervals
of pitches interfering the experiment results, we chose only
the C pitches of the piano (C1–C8), with C1 the lowest and
C8 the highest (as shown in Figure 1). All sounds used a
44,100 Hz sampling rate.
Any silence before the onset of each sound was removed.
The sound durations were then truncated to 1.0 second using
a 30 ms linear fade-out before the end of each sound. In all
cases, the fade-outs sounded like a natural damping or release
of the sound.
3.1.2 Emotional Categories
The subjects compared the stimuli in terms of ten emotional
categories: Happy, Sad, Heroic, Scary, Comic, Shy, Roman-
tic, Mysterious, Angry, and Calm. When picking these ten
emotional categories, we particularly had dramatic musical
genres such as opera and musicals in mind, where there are
typically heroes, villains, and comic-relief characters with
music specifically representing each. The emotional charac-
teristics in these genres are generally more obvious and less
abstract than in pure orchestral music.
We chose to use simple English emotional categories so
that they would be familiar and self-apparent to non-native
English speakers, which are similar to Italian music expres-
sion marks traditionally used by classical composers to spec-
ify the character of the music. These emotional categories
also provide easy comparison with the results of our previous
work [2, 3, 4, 14, 15].
3.2 Test Panel
There were 26 subjects hired for the listening test, with an
average age of 20.8 (ranging from 19 to 24). All subjects
were undergraduate students at the Hong Kong University of
Science and Technology. None of them reported any hearing
problems.
3.3 Test Procedure
The subjects were seated in a “quiet room” with less than
40 dB SPL background noise level. Residual noise was
mostly due to computers and air conditioning. The noise level
was further reduced with headphones. The Sound Blaster
sound card uses 24-bits with a maximum sampling rate of
96 kHz and a 108 dB S/N ratio.
The subjects were provided with an instruction sheet con-
taining definitions of the ten emotional categories from the
Cambridge Academic Content Dictionary [17].
Every subject made pairwise comparisons on a computer
among all the 24 combinations of pitch and dynamics for each
emotional category. During each trial, subjects heard a pair
of sounds from different instruments and were prompted to
choose the sound that represented the given emotional cate-
gory more strongly. Each combination of two different in-
struments was presented once for each emotional category.
For each emotional category, the overall trial presentation or-
der was randomized (i.e., all the Happy comparisons were
first in a random order, then all the Sad comparisons were
second, and so on). However, the emotional categories were
presented in order to avoid confusing and fatiguing the sub-
jects.
The listening test took about 3 hours, with a short break of
5 minutes after every 30 minutes to help minimize listener
fatigue and maintain consistency.
3.4 Analysis Procedures
The voting results from the subjects were used for the correla-
tion figures. The Bradley-Terry-Luce (BTL) model [18] was
then used to derive rankings based on the number of posi-
tive votes each sound received for each emotional category.
For each emotional category, the BTL scale values for all
the combinations of dynamic and pitch sum up to 1. The
BTL value for each sound is the probability that listeners will
choose that sound when considering a certain emotional cat-
egory. The 95% confidence intervals of the BTL values were
obtained to test the significance of the instrument ranks.
4. EXPERIMENT RESULTS
The raw results were votes for each sound pair and each emo-
tional category. Figure 2 displays the BTL scale values of the
sounds, with the corresponding 95% confidence intervals.
From the charts, it can be observed that the low notes (C1–
C3) were significantly ranked more Angry than the high notes
(C5–C8). For the low notes, the forte dynamics were signifi-
cantly more Angry than piano. The difference is not as much
for the high notes. The rankings were generally in order by
pitch. The rankings were much more compressed for piano,
and in-between for mezzo.
On the other hand, Shy was the opposite with wide-spread
rankings for piano.The high notes at piano were significantly
more Shy than at forte, and the dynamics did not make a dif-
ference for low notes. The rankings also followed the pitch
order.
Heroic showed similar behavior as Angry. Romantic and
Romantic were similar to Shy.
It is interesting that the extremes (C1 and C8) were not al-
ways highest or lowest. For example, for Comic at forte, C1
and C7 were ranked similarly and the mid-range pitches were
ICMC 2015 – Sept. 25 - Oct. 1, 2015 – CEMI, University of North Texas
– 373 –
C1 C2 C3 C4 C5 C6 C7 C8
f = 32.7 Hz 65.4 Hz 130.8 Hz 261.6 Hz 523.3 Hz 1046.5 Hz 2093.0 Hz 4186.0 Hz
Figure 1: Selected piano pitches and the corresponding frequencies.
0.00 0.05 0.10 0.15 0.20
Happy
BTL Scale Value
f
m
c1 p
f
m
c2 p
f
m
c3 p
f
m
c4 p
f
m
c5 p
f
m
c6 p
f
m
c7 p
f
m
c8 p
0.00 0.05 0.10 0.15 0.20
Sad
BTL Scale Value
f
m
c1 p
f
m
c2 p
f
m
c3 p
f
m
c4 p
f
m
c5 p
f
m
c6 p
f
m
c7 p
f
m
c8 p
0.00 0.05 0.10 0.15 0.20
Heroic
BTL Scale Value
f
m
c1 p
f
m
c2 p
f
m
c3 p
f
m
c4 p
f
m
c5 p
f
m
c6 p
f
m
c7 p
f
m
c8 p
0.00 0.05 0.10 0.15 0.20
Scary
BTL Scale Value
f
m
c1 p
f
m
c2 p
f
m
c3 p
f
m
c4 p
f
m
c5 p
f
m
c6 p
f
m
c7 p
f
m
c8 p
0.00 0.05 0.10 0.15 0.20
Comic
BTL Scale Value
f
m
c1 p
f
m
c2 p
f
m
c3 p
f
m
c4 p
f
m
c5 p
f
m
c6 p
f
m
c7 p
f
m
c8 p
0.00 0.05 0.10 0.15 0.20
Shy
BTL Scale Value
f
m
c1 p
f
m
c2 p
f
m
c3 p
f
m
c4 p
f
m
c5 p
f
m
c6 p
f
m
c7 p
f
m
c8 p
0.00 0.05 0.10 0.15 0.20
Romantic
BTL Scale Value
f
m
c1 p
f
m
c2 p
f
m
c3 p
f
m
c4 p
f
m
c5 p
f
m
c6 p
f
m
c7 p
f
m
c8 p
0.00 0.05 0.10 0.15 0.20
Mysterious
BTL Scale Value
f
m
c1 p
f
m
c2 p
f
m
c3 p
f
m
c4 p
f
m
c5 p
f
m
c6 p
f
m
c7 p
f
m
c8 p
0.00 0.05 0.10 0.15 0.20
Angry
BTL Scale Value
f
m
c1 p
f
m
c2 p
f
m
c3 p
f
m
c4 p
f
m
c5 p
f
m
c6 p
f
m
c7 p
f
m
c8 p
0.00 0.05 0.10 0.15 0.20
Calm
BTL Scale Value
f
m
c1 p
f
m
c2 p
f
m
c3 p
f
m
c4 p
f
m
c5 p
f
m
c6 p
f
m
c7 p
f
m
c8 p
Figure 2: BTL scale values and the corresponding 95% confidence intervals. The dotted line represents no preference.
p=piano,m=mezzo,f=forte
ICMC 2015 – Sept. 25 - Oct. 1, 2015 – CEMI, University of North Texas
– 374 –
ranked highest. C1 and C8 were together ranked highest for
Scary.
Sad is not as sensitive to the pitch difference. A difference in
ranking is seen among the dynamic levels instead. In contrast,
Happy and Mysterious are less sensitive to dynamics.
5. DISCUSSION
Based on the results, our main observations are the following:
1. The emotional characteristics for different pitch at dif-
ferent dynamics on a piano is distinctly distinguishable.
2. Many of the emotional characteristics show a trend
across the octaves. Some of the emotional character-
istics are more significant in the middle octaves.
3. Low notes at forte often shows opposite emotional
characteristics compared to high notes at piano.
Although this is only the preliminary analysis of the emo-
tional effect of dynamics and pitch in the piano, our results
have shown discernible differences. The results of the tested
pitches were generally consistent, with a slight variation in a
consistent trend.
Dynamic level and pitch showed the strongest effect on the
emotional categories Angry and Shy. The more powerful and
lower the sound on the piano, the more Angry and Heroic it
sounds. The softer and higher the sound on the piano, the
more Calm and Shy it sounds. We also found that categories
like Happy or Mysterious were relatively unaffected by dy-
namics.
Further timbral analysis of the experiment results will give
more insights about the emotional characteristics of piano
sounds. This will help recording and audio engineers, com-
posers, and pianists better understand the emotional dynam-
ics of the piano and use them to engineer even more expres-
sive recordings and performances, and to present and alter the
mood and ambiance of their musical work.
Acknowledgments
This work has been supported by Hong Kong Research
Grants Council grant HKUST613112.
6. REFERENCES
[1] T. Eerola, R. Ferrer, and V. Alluri, “Timbre and affect di-
mensions: evidence from affect and similarity ratings and
acoustic correlates of isolated instrument sounds,” Music
Perception, vol. 30, no. 1, pp. 49–70, 2012.
[2] B. Wu, A. Horner, and C. Lee, “Musical Timbre and Emo-
tion: The Identification of Salient Timbral Features in
Sustained Musical Instrument Tones Equalized in Attack
Time and Spectral Centroid.” in Proc. 40th Int. Comp.
Music Conf. (ICMC), 2014, pp. 928–934.
[3] ——, “The Correspondence of Music Emotion and Tim-
bre in Sustained Musical Instrument Sounds,” J. Audio
Eng. Soc., vol. 62, no. 10, pp. 663–675, 2014.
[4] C.-j. Chau, B. Wu, and A. Horner, “Timbre Features and
Music Emotion in Plucked String, Mallet Percussion, and
Keyboard Tones.” in Proc. 40th Int. Comp. Music Conf.
(ICMC), 2014, pp. 982–989.
[5] ——, “The Emotional Characteristics and Timbre of
Nonsustaining Instrument Sounds,” J. Audio Eng. Soc,
vol. 63, no. 4, 2015.
[6] L.-L. Balkwill and W. F. Thompson, “A cross-cultural in-
vestigation of the perception of emotion in music: Psy-
chophysical and cultural cues,Music Perception, vol. 17,
no. 1, pp. 43–64, 1999.
[7] J. Liebetrau, S. Schneider, and R. Jezierski, “Application
of Free Choice Profiling for the Evaluation of Emotions
Elicited by Music,” in Proc. 9th Int. Symp. Comput. Mu-
sic Modeling and Retrieval (CMMR 2012): Music and
Emotions, 2012, pp. 78–93.
[8] M. Plewa and B. Kostek, “A Study on Correlation be-
tween Tempo and Mood of Music,” in Audio Eng. Soc.
Conv. 133. Audio Eng. Soc., 2012.
[9] K. R. Scherer and J. S. Oshinsky, “Cue utilization in emo-
tion attribution from auditory stimuli,Motivation and
Emotion, vol. 1, no. 4, pp. 331–346, 1977.
[10] I. Peretz, L. Gagnon, and B. Bouchard, “Music and emo-
tion: perceptual determinants, immediacy, and isolation
after brain damage,” Cognition, vol. 68, no. 2, pp. 111–
141, 1998.
[11] J.-J. Aucouturier, F. Pachet, and M. Sandler, ““The way
it sounds”: timbre models for analysis and retrieval of
music signals,” IEEE Trans. Multimedia, vol. 7, no. 6, pp.
1028–1035, 2005.
[12] G. Tzanetakis and P. Cook, “Musical genre classification
of audio signals,” IEEE Trans. Speech Audio Process.,
vol. 10, no. 5, pp. 293–302, 2002.
[13] T. Eerola and J. K. Vuoskoski, “A comparison of the dis-
crete and dimensional models of emotion in music,” Psy-
chology of Music, vol. 39, no. 1, pp. 18–49, 2011.
[14] B. Wu, S. Wun, C. Lee, and A. Horner, “Spectral Cor-
relates in Emotion Labeling of Sustained Musical Instru-
ment Tones,” in Proc. 14th Int. Soc. Music Inform. Re-
trieval Conf. (ISMIR), November 4–8 2013.
[15] B. Wu, A. Horner, and C. Lee, “Emotional Predisposi-
tion of Musical Instrument Timbres with Static Spectra,
in Proc. 15th Int. Soc. Music Inform. Retrieval Conf. (IS-
MIR), 11 2014, pp. 253–258.
[16] M. Goto, H. Hashiguchi, T. Nishimura, and R. Oka,
“RWC Music Database: Music genre database and mu-
sical instrument sound database.” in Proc. Int. Soc. Music
Inform. Retrieval Conf. (ISMIR), vol. 3, 2003, pp. 229–
230.
[17] Cambridge University Press. Cambridge Academic
Content Dictionary. [Online]. Available: http://dictionary.
cambridge.org/dictionary/american-english
[18] R. A. Bradley, “Paired comparisons: Some basic proce-
dures and examples,Nonparametric Methods, vol. 4, pp.
299–326, 1984.
ICMC 2015 – Sept. 25 - Oct. 1, 2015 – CEMI, University of North Texas
– 375 –

Supplementary resource (1)

... Music emotion research has flourished in recent years. In particular, various recent studies have found that different musical instruments have strong and different emotional characteristics [2][3][4][5][6][7][8][9][10][11][12][13]20,21]. The vast majority of these studies have focused on a single common pitch, usually a note just above middle C so that as many treble and bass clef instruments can be compared against one another as possible. ...
... We also investigated piano's emotional characteristics changed with pitch and dynamics from C1 to C8 over piano, mezzo, and forte dynamic levels [10,13]. Especially relevant to the current paper is our experimental finding on how the emotional characteristics of the solo bowed strings changed with pitch and dynamics [11,20]. ...
... Each note also had two dynamic levels, corresponding to forte (f) and piano (p)loud and soft. Only two dynamic levels were tested, as a previous study with 3 dynamic levels, namely forte (f), mezzo (m), and piano (p)-loud, medium, and soft, showed that the result for mezzo were consistently inbetween the results of forte and piano [10,13]. The total number of sounds was 28 (14 notes × 2 dynamic levels). ...
Conference Paper
Full-text available
Recent research has shown that different musical instrument sounds have strong emotional characteristics. It has also shown how these emotional characteristics change with different pitch and dynamics for the piano and solo bowed strings. This paper investigates how the emotional characteristics of the section bowed strings vary with pitch and dynamics, comparing and contrasting the solo and section string results. We conducted listening tests where listeners compared the section string sounds pairwise over ten emotional categories. The section and solo string results were somewhat similar overall, but with some notable differences as well. The emotional characteristics Happy, Heroic, Romantic, Comic, Calm, and Shy generally increased with pitch in an arching shape that peaked at C5 and decreased at the highest pitches. The characteristics Angry and Sad generally decreased with pitch. Scary was somewhat U-shaped and especially strong in the extreme high register. In terms of dynamics, the results showed that Happy, Heroic, Comic, Angry, and Scary were stronger for loud notes, while Romantic, Calm, Shy, and Sad were stronger for soft notes. These results provide audio engineers and musicians with possible suggestions for emphasizing various emotional characteristics of the section strings in orchestral recordings and performances.
... Previous research has shown that musical instruments have distinctive emotional characteristics [1][2][3][4][5][6][7][8][9] and that these characteristics can be significantly changed with reverberation [10][11][12][13]. This paper considers whether these changes in character are relatively uniform or instrument-dependent. ...
... Researchers have considered various relationships between timbre and music emotion [14][15][16][17][18][19][20][21][22][23][24][25][26][27][28][29] and, in particular, have found that dfferent instruments have different timbral and emotional characteristics [1][2][3][4][5][6][7][8]. By changing the pitch, dynamics, and other aspects of the performance, the timbre and emotional characteristics also change [9,[30][31][32][33][34][35][36]. These characteristics are further modified by the performance environment-by the amount and length of reverberation in the space [10,11,37,38], which smears the temporal and spectral envelopes and changes the emotional character of the sound. ...
... Some choices of emotional characteristics are fairly universal and occur in many previous studies (e.g., Happy, Sad, Scary/Fear/Angry, Tender/Calm/Romantic) roughly corresponding to the four quadrants of the Valence-Arousal plane, but there are lots of variations beyond that [52]. For this study we used the same categories we have used in our previous research on musical instruments [3][4][5][6][7][8][9]12]. The ratings of the emotional categories according to the Affective Norms for English Words [53] are shown in Fig. 5 using the Valence-Arousal model. ...
Article
Full-text available
Previous research has shown that musical instruments have distinctive emotional characteristics and that these characteristics can be significantly changed with reverberation. This paper considers whether these changes in character are relatively uniform or instrument-dependent. We compared eight sustained instrument tones with different amounts and lengths of simple parametric reverberation over eight emotional characteristics. The results show a remarkable consistency in listener rankings of the instruments for each of the different types of reverberation, with strong correlations ranging from 90 to 95%. These results indicate that the underlying instrument space does not change much with reverberation in terms of emotional characteristics, and that each instrument has a particular footprint of emotional characteristics. Among the tones we tested, the instruments cluster into two fairly distinctive groups: those where the positive energetic emotional characteristics are strong (e.g., oboe, trumpet, violin) and those where the low-arousal characteristics are strong (e.g., bassoon, clarinet, flute, horn). The saxophone is an outlier and is somewhat strong for most emotional characteristics. In terms of applications, the relatively consistent rankings of emotional characteristics between the instruments certainly helps each instrument retain its identity in different reverberation environments and suggests possible future work in instrument identification.
... Music emotion research has flourished in recent years. In particular, various recent studies have found that different musical instruments have strong and different emotional characteristics [1][2][3][4][5][6][7][8][9][10][11][12][13][14][15]. The vast majority of these studies have focused on a single common pitch, usually a note just above middle C so that as many treble and bass clef instruments can be compared against one another as possible. ...
... We also investigated piano's emotional characteristics changed with pitch and dynamics from C1 to C8 over piano, mezzo, and forte dynamic levels [9,12]. Especially relevant to the current paper is our experimental finding on how the emotional characteristics of the bowed strings changed with pitch and dynamics [10,13,15]. ...
Conference Paper
Full-text available
Recent research has shown that different musical instrument sounds have strong emotional characteristics. It has also shown how these emotional characteristics change with different pitch and dynamics for the piano and bowed strings. This work differentiates the distinctive emotional characters of the brass instruments, and investigate how pitch and dynamics influence their characters. We conducted listening tests where listeners compared the brass instrument sounds pairwise over ten emotional categories. The emotional characteristics Happy increased with pitch. Heroic, Romantic, and Comic generally increased with pitch in an arching shape that peaked at C5 and decreased at the highest pitches. Calm, Mysterious , and Shy are also in arching shape but peaked at C3. Angry and Scary was somewhat U-shaped and especially strong in the extreme high register. Sad decreased with pitch. In terms of dynamics, the results showed that Heroic, Comic, Angry, and Scary were stronger for loud notes, while Romantic, Calm, Mysterious, Shy, and Sad were stronger for soft notes. These results help orchestra-tors and composers make the jump from knowing a particular pitch is technically possible on an instrument, to understanding how its pitch register shapes its emotional character.
... A number of recent studies have found that different musical instruments have strong and different emotional characteristics [1 -15]. In particular, our previous work has shown that the strength of emotional characteristics varies significantly with pitch and dynamics in instruments such as the piano and bowed strings [9,10,12,13,15]. The woodwind instruments remain unexplored in this respect. ...
Conference Paper
Full-text available
Recent research has shown that emotional characteristics change with different pitch and dynamics for the piano and bowed strings. This work investigates how pitch and dynamics influence their characters in the woodwind instruments. We conducted listening tests where listeners gave absolute judgements on the woodwind sounds over ten emotional categories, and analyzed the data with logistic regression. For pitch, Happy, Heroic, Romantic, Calm, Mysterious, and Shy had an arched shape that peaked at C5. Mysterious and Shy increased until C4 and decreased afterwards. Comic and Sad decreased with Pitch. Angry had a V-shape that stronger in the high register. Scary had an asymmetric U-shape that was especially strong at the highest pitch. For dynamics, Happy, Heroic, Comic, Angry were stronger for loud notes. Romantic , Calm, Mysterious, Shy, and Sad were stronger for soft notes. For Scary, loud and soft notes were about the same. Overall, pitch and dynamics had about an equally important effect on woodwind emotional characteristics ; some characteristics such as Happy were affected more by pitch and others such as Shy were affected more by dynamics. Instrument was more of a secondary factor though still important. Also, the bassoon was the most different from the other woodwind instruments.
... Some choices of emotional characteristics are fairly universal and occur in many previous studies roughly corresponding to the four quadrants of the Valence-Arousal plane [6]. For this study, we used the same categories that we used in our previous research on musical instruments and parametric reverberation [1,2,3,7,8,9,10,11,12,13,14,15,16,17,18,19]. Before the listening test, subjects read online definitions of the emotional categories used in our experiment, which were taken from the Cambridge Academic Content Dictionary [20]. ...
Conference Paper
Full-text available
Previous work has shown that the emotional characteristics of musical instrument sounds are significantly changed with parametric reverberation. This paper considers the effects of reverberation time on the emotional characteristics of instrument sounds with convolution reverberation. We compared eight musical instruments and ten emotional characteristics over five hall impulse responses ranging from the 1-second Royal National Theatre to the 5-second King's College Chapel. The results showed that convolution reverberation had more pronounced effects on the emotional characteristics compared to parametric reverberation. While the results were more pronounced for convolution reverberation compared to parametric reverberation, there was also a strikingly strong agreement in their results, and correlation coefficient between them was 0.74 over all emotional characteristics. This strong correlation indicates that reverberation time has a remarkably consistent effect on the emotional characteristics regardless of whether using convolution or parametric reverberation
... Some choices of emotional characteristics are fairly universal and occur in many previous studies roughly corresponding to the four quadrants of the Valence-Arousal plane [10]. For this study, we used the same categories we have used in our previous research on musical instruments [1,6,11,12,13,14,15,16,17,18,19,20,21]. ...
Conference Paper
Full-text available
Previous research has shown that the distinctive emotional characteristics in musical instruments can be significantly changed with reverberation. This paper considers whether these changes in character are relatively uniform or instrumentdependent. We compared eight sustained instrument tones with different amounts and lengths of simple parametric reverberation over eight emotional characteristics. The results show a remarkable consistency in listener rankings of the instruments for each of the different types of reverberation, with strong correlations ranging from 90 to 95%. These results indicate that the underlying instrument space does not change much with reverberation in terms of emotional characteristics, and that each instrument has a particular footprint of emotional characteristics. Among the tones we tested, the instruments cluster into two fairly distinctive groups: those where the positive energetic emotional characteristics are strong, and those where the low-arousal characteristics are strong. The saxophone is an outlier, and is somewhat strong for most emotional characteristics. In terms of applications, the relatively consistent rankings of emotional characteristics between the instruments certainly helps each instrument retain its identity in different reverberation environments, and suggests possible future work in instrument identification.
... Most relevant to the current study, we recently studied how the piano's emotional characteristics changed with pitch and dynamics from C1 to C8 over piano, mezzo, and forte dynamic levels [36,37]. In that study we found that the emotional characteristics Happy, Romantic, Comic, Calm, Mysterious, and Shy generally increased in pitch in an arching shape that decreased at the highest pitches. ...
Article
Full-text available
Previous research has shown that different musical instrument sounds have strong emotional characteristics. This paper investigates how emotional characteristics vary with pitch and dynamics within the bowed string instrument family. We conducted listening tests to compare the effects of pitch and dynamics on the violin, viola, cello, and double bass. Listeners compared the sounds pairwise over 10 emotional categories. Results showed that the emotional characteristics Happy, Heroic, Romantic, Comic, and Calm generally increased with pitch but decreased at the highest pitches. Angry and Sad generally decreased with pitch. Scary was strong in the extreme low and high registers, while Shy and Mysterious were unaffected by pitch. For dynamics, the results showed that Heroic, Comic, and Angry were stronger for loud notes, while Romantic, Calm, Shy, Sad, and the high register for Happy were stronger for soft notes. Scary and Mysterious were unaffected by dynamics. The results also showed significant differences between different bowed string instruments on notes of the same pitch and dynamic level. These results help quantify our understanding of the relative emotional characteristics of the strings. They provide audio engineers and musicians with suggestions for emphasizing emotional characteristics of the bowed strings in sound recordings and performances.
... Each of us tried to describe the excerpt in a single one-word adjective, and the 14 words we used most frequently were selected as the 14 categories. It turns out that about half of the 14 words were used in our previous related studies [79][80][81][82][83][84][85][86][87][88][89][90][91][92][93][94][95][96] and most of the others were used in studies by other researchers [9,20,21,35,47]. All the categories included in the 4-quadrant model in Figure 1 appear in Figure 4 except Angry. ...
Article
Full-text available
Music emotion recognition and recommendation systems often use a simplified 4-quadrant model with categories such as Happy, Sad, Angry, and Calm. Previous research has shown that both listeners and automated systems often have difficulty distinguishing low-arousal categories such as Calm and Sad. This paper seeks to explore what makes the categories Calm and Sad so difficult to distinguish. We used 300 low-arousal excerpts from the classical piano repertoire to determine the coverage of the categories Calm and Sad in the low-arousal space, their overlap, and their balance to one another. Our results show that Calm was 40% bigger in terms of coverage than Sad, but that on average Sad excerpts were significantly more negative in mood than Calm excerpts were positive. Calm and Sad overlapped in nearly 20% of the excerpts, meaning 20% of the excerpts were about equally Calm and Sad. Calm and Sad covered about 92% of the low-arousal space, where 8% of the space were holes that were not-at-all Calm or Sad. The largest holes were for excerpts considered Mysterious and Doubtful, but there were smaller holes among positive excerpts as well. Due to the holes in the coverage, the overlaps, and imbalances the Calm-Sad model adds about 6% more errors when compared to asking users directly whether the mood of the music is positive or negative. Nevertheless, the Calm-Sad model is still useful and appropriate for applications in music emotion recognition and recommendation such as when a simple and intuitive interface is preferred or when categorization is more important than precise differentiation.
Article
Full-text available
Previous literature suggests that structural and expressive cues affect the emotion expressed in music. However, only a few systematic explorations of cues have been done, usually focussing on a few cues or a limited amount of predetermined arbitrary cue values. This paper presents three experiments investigating the effect of six cues and their combinations on the music's perceived emotional expression. Twenty-eight musical pieces were created with the aim of providing flexible, ecologically valid, unfamiliar, new stimuli. In Experiment 1, 96 participants assessed which emotions were expressed in the pieces using Likert scale ratings. In Experiment 2, a subset of the stimuli was modified by participants (N = 42) via six available cues (tempo, mode, articulation, pitch, dynamics, and brightness) to convey seven emotions (anger, sadness, fear, joy, surprise, calmness, and power), addressing the main aim of exploring the impact of cue levels to expressions. Experiment 3 investigated how well the variations of the original stimuli created by participants in Experiment 2 expressed their intended emotion. Participants (N = 91) rated them alongside the seven original pieces, allowing the exploration of similarities and differences between the two sets of related pieces. An overall pattern of cue combinations was identified for each emotion. Some findings corroborate previous studies: mode and tempo were the most impactful cues in shaping emotions, and sadness and joy were amongst the most accurately recognised emotions. Novel findings include soft dynamics being used to convey anger, and dynamics and brightness being the least informative cues. These findings provide further motivation to investigate the effect of cues on emotions in music as combinations of multiple cues rather than as individual cues, as one cue might not give enough information to portray a specific emotion. The new findings and discrepancies are discussed in relation to current theories of music and emotions.
Article
Full-text available
In this paper, a study is carried out to identity a relationship between the mood description and combinations of various tempos and rhythms. First, a short review of music recommendation systems along with music mood recognition studies is presented. In addition, some details on tempo and rhythm perception and detection are included. Then, the experiment layout is explained in which a song is first recorded and then its rhythm and tempo are changed. This constitutes the basis for a mood tagging test. Six labels are chosen for mood description. The results show a significant dependence between tempo and mood of the music.
Article
Full-text available
Music is one of the strongest triggers of emotions. Melody, rhythm, and harmony are important triggers, but what about timbre? Do musical instruments have an emotional predisposition? For example, is the melancholy sound of the English horn due to its timbre or how it is used? Though music emotion recognition has received a lot of attention; researchers have only recently begun considering the relationship between emotion and timbre. Therefore, we designed listening tests to compare sounds from eight wind and bowed string instruments. We wanted to know if some sounds were consistently perceived as being happier or sadder in pairwise comparisons. Eight emotions were tested. The results showed strong emotional predispositions for each instrument. The emotions Happy, Joyful, Heroic, and Comic were strongly correlated with one another, and the violin, trumpet, and clarinet best evoked these emotions. Sad and Depressed were also strongly correlated and were best evoked by the horn and flute. Scary was an emotional outlier, and the oboe had an emotionally neutral disposition. We also correlated emotion and several spectral features and found that emotion correlated significantly with average spectral centroid and spectral centroid deviation for nearly all emotions. To determine which spectral features were most important aside from spectral centroid, we conducted follow-up listening tests of centroid-equalized sounds as well as static sounds. The results showed that the even/odd harmonic ratio significantly correlated with most emotions. This suggests that the even/odd harmonic ratio is perhaps the most salient timbral feature after attack time and brightness. These results provide new insights for orchestration.
Article
Full-text available
Recent research has shown the connection between timbre and music emotion and has focused on sustaining instruments. This paper undertakes listening tests to investigate the connection for nonsustaining instruments including plucked string, pitched percussion, and keyboard sounds compared pairwise by listening test subjects over eight emotional categories. Different durations were studied to investigate the effect of early release on the emotional characteristics. The experiments showed that the harpsichord, marimba, vibraphone, and xylophone were highly rated for positive emotional characteristics; the guitar, harp, and plucked violin were highly rated for negative emotional characteristics; and the piano was rated emotionally neutral. There were differences in emotional characteristics due to early release, such as the harpsichord decreasing in ranking for positive emotional characteristics on sounds with early releases. By correlating timbral features with the listener rankings, we found that decay slope and density of significant harmonics were significant timbral features for many emotional characteristics.
Conference Paper
Full-text available
Music conveys emotions by means of pitch, rhythm, loudness, and many other musical qualities. It was recently confirmed that timbre also has direct association with emotion, for example, that a horn is perceived as sad and a trumpet heroic in even isolated instrument tones. As previous work has mainly focused on sustaining instruments such as bowed strings and winds, this paper presents an experiment with non-sustaining instruments, using a similar approach with pairwise comparisons of tones for emotion categories. Plucked string, mallet percussion, and keyboard instrument tones were investigated for eight emotions: Happy, Sad, Heroic, Scary, Comic, Shy, Joyful, and Depressed. We found that plucked string tones tended to be Sad and Depressed, while harpsichord and mallet percussion tones induced positive emotions such as Happy and Heroic. The piano was emotionally neutral. Beyond spectral centroid and its deviation, which are important features in sustaining tones, decay slope was also significantly correlated with emotion in non-sustaining tones.
Conference Paper
Full-text available
Music evokes and carries emotions. Despite many studies having investigated the relation between music and emotion, current research lacks a systematic and empirically derived taxonomy of musically induced emotions [1]. This work contributes to the question which musical features in particular are able to induce emotions while listening. Problems of defining and measuring emotions are explained. A method to measure affective states induced by music with the help of Free Choice Profiling (FCP) is outlined. Two FCP experiments, assessing the usefulness of the method for emotional research and the selection of test stimuli are described. The shown results are in line with psychological theories of emotions, i.e., the valence/arousal model.
Article
Full-text available
CONSIDERABLE EFFORT HAS BEEN MADE TOWARDS understanding how acoustic and structural features contribute to emotional expression in music, but relatively little attention has been paid to the role of timbre in this process. Our aim was to investigate the role of timbre in the perception of affect dimensions in isolated musical sounds, by way of three behavioral experiments. In Experiment 1, participants evaluated perceived affects of 110 instrument sounds that were equal in duration, pitch, and dynamics using a three-dimensional affect model (valence, energy arousal, and tension arousal) and preference and emotional intensity. In Experiment 2, an emotional dissimilarity task was applied to a subset of the instrument sounds used in Experiment 1 to better reveal the underlying affect structure. In Experiment 3, the perceived affect dimensions as well as preference and intensity of a new set of 105 instrument sounds were rated by participants. These sounds were also uniform in pitch, duration, and playback dynamics but contained systematic manipulations in the dynamics of sound production, articulation, and ratio of high-frequency to low-frequency energy. The affect dimensions for all the experiments were then explained in terms of the three kinds of acoustic features extracted: spectral (e.g., ratio of high-frequency to low-frequency energy), temporal (e.g., attack slope), and spectrotemporal (e.g., spectral flux). High agreement among the participants' ratings across the experiments suggested that even isolated instrument sounds contain cues that indicate affective expression, and these are recognized as such by the listeners. A dominant portion (50-57%) of the two dimensions of affect (valence and energy arousal) could be predicted by linear combinations of few acoustic features such as ratio of high-frequency to low-frequency energy, attack slope, and spectral regularity. Links between these features and those observed in the vocal expression of affects and other sound phenomena are discussed. © 2012 by the regents of the University of California all rights reserved.
Article
Full-text available
The primary aim of the present study was to systematically compare perceived emotions in music using two different theoretical frameworks: the discrete emotion model, and the dimensional model of affect. A secondary aim was to introduce a new, improved set of stimuli for the study of music-mediated emotions. A large pilot study established a set of 110 film music excerpts, half were moderately and highly representative examples of five discrete emotions (anger, fear, sadness, happiness and tenderness), and the other half moderate and high examples of the six extremes of three bipolar dimensions (valence, energy arousal and tension arousal). These excerpts were rated in a listening experiment by 116 non-musicians. All target emotions of highly representative examples in both conceptual sets were discriminated by self-ratings. Linear mapping techniques between the discrete and dimensional models revealed a high correspondence along two central dimensions that can be labelled as valence and arousal, and the three dimensions could be reduced to two without significantly reducing the goodness of fit. The major difference between the discrete and categorical models concerned the poorer resolution of the discrete model in characterizing emotionally ambiguous examples. The study offers systematically structured and rich stimulus material for exploring emotional processing.
Article
This chapter describes some basic procedures and examples related to paired comparisons. The chapter presents the methods for the examination of specified treatment contrasts and the analysis of factorial paired comparison experiments together with examples. These methods provide much new flexibility. Several applications are arisen in many areas, but most notably in food technology, marketing research, and sports competition. Probabilistic models for paired comparisons may be devised to represent the experimental situation and permit appropriate data analysis. The models provide probabilities of possible choices of items or treatments from pairs of items and hence depend on orderings. Then extensions of the method are developed for factorial treatment combinations and for multivariate responses, responses on several attributes for each paired comparison. Multivariate responses to paired comparisons are obtained. For example, this happens in consumer testing where, on paired samples, preferences on a number of characteristics are solicited. The chapter concludes with comments on additional methods of analysis.
Article
Electronically synthesized tone sequences with systematic manipulation of amplitude and pitch variation, pitch level and contour, tempo, envelope, and filtration were rated on emotional expressiveness. The results show that two-thirds to three-quarters of the variance in the emotion attributions can be explained by the manipulation of the acoustic cues, and that a linear model of the judges'' cue utilization seems to be a good approximation to their response system. Implications for phylogenetic and ontogenetic aspects of the vocal expression of emotion and for the psychology of music are discussed.