ArticlePDF Available

Crossmodal transfer of emotion by music

Authors:

Abstract and Figures

Music is one of the most powerful elicitors of subjective emotion, yet it is not clear whether emotions elicited by music are similar to emotions elicited by visual stimuli. This leads to an open question: can music-elicited emotion be transferred to and/or influence subsequent vision-elicited emotional processing? Here we addressed this question by investigating processing of emotional faces (neutral, happy and sad) primed by short excerpts of musical stimuli (happy and sad). Our behavioural experiment showed a significant effect of musical priming: prior listening to a happy (sad) music enhanced the perceived happiness (sadness) of a face irrespective of facial emotion. Further, this musical priming-induced effect was largest for neutral face. Our electrophysiological experiment showed that such crossmodal priming effects were manifested by event related brain potential components at a very early (within 100 ms post-stimulus) stages of neuronal information processing. Altogether, these results offer new insight into the crossmodal nature of music and its ability to transfer emotion to visual modality.
Content may be subject to copyright.
GOLDSMITHS Research Online
Article (refereed)
Logeswaran, Nidhya and Bhattacharya, Joydeep
Crossmodal transfer of emotion by music
Originally published in Neuroscience LettersThe publisher's version is
available at: http://dx.doi.org/10.1016/j.neulet.2009.03.044
You may cite this version as: Logeswaran, Nidhya and Bhattacharya,
Joydeep, 2009. Crossmodal transfer of emotion by music. Neuroscience
Letters, 455 (2). pp. 129-133. ISSN 03043940 [Article]: Goldsmiths Research
Online.
Available at: http://eprints.gold.ac.uk/4213/
This document is the author’s final manuscript version of the journal article,
incorporating any revisions agreed during peer review. Some differences
between this version and the publisher’s version remain. You are advised to
consult the publisher’s version if you wish to cite from it.
Copyright © and Moral Rights for the papers on this site are retained by the
individual authors and/or other copyright owners.
http://eprints-gro.goldsmiths.ac.uk
Contact Goldsmiths Research Online at: lib-eprints@gold.ac.uk
This article appeared in a journal published by Elsevier. The attached
copy is furnished to the author for internal non-commercial research
and education use, including for instruction at the authors institution
and sharing with colleagues.
Other uses, including reproduction and distribution, or selling or
licensing copies, or posting to personal, institutional or third party
websites are prohibited.
In most cases authors are permitted to post their version of the
article (e.g. in Word or Tex form) to their personal website or
institutional repository. Authors requiring further information
regarding Elsevier’s archiving and manuscript policies are
encouraged to visit:
http://www.elsevier.com/copyright
Author's personal copy
Neuroscience Letters 455 (2009) 129–133
Contents lists available at ScienceDirect
Neuroscience Letters
journal homepage: www.elsevier.com/locate/neulet
Crossmodal transfer of emotion by music
Nidhya Logeswarana, Joydeep Bhattacharyaa,b,
aDepartment of Psychology, Goldsmiths College, University of London, London SE14 6NW, United Kingdom
bCommission for Scientific Visualization, Austrian Academy of Sciences, Vienna A1220, Austria
article info
Article history:
Received 31 October 2008
Received in revised form 3 March 2009
Accepted 11 March 2009
Keywords:
Music
Face
Emotion
Priming
ERP
abstract
Music is one of the most powerful elicitors of subjective emotion, yet it is not clear whether emotions
elicited by music are similar to emotions elicited by visual stimuli. This leads to an open question: can
music-elicited emotion be transferred to and/or influence subsequent vision-elicited emotional process-
ing? Here we addressed this question by investigating processing of emotional faces (neutral, happy and
sad) primed by short excerpts of musical stimuli (happy and sad). Our behavioural experiment showed
a significant effect of musical priming: prior listening to a happy (sad) music enhanced the perceived
happiness (sadness) of a face irrespective of facial emotion. Further, this musical priming-induced effect
was largest for neutral face. Our electrophysiological experiment showed that such crossmodal priming
effects were manifested by event related brain potential components at a very early (within 100 ms post-
stimulus) stages of neuronal information processing. Altogether, these results offer new insight into the
crossmodal nature of music and its ability to transfer emotion to visual modality.
© 2009 Elsevier Ireland Ltd. All rights reserved.
Music is often considered as the language of emotion and one of
the oldest held views is that music arises principally from human
communication—a performer delivers some message to a receptive
listener. This message is supposed to be an emotional one and this
emotional communication is postulated to be the principal pur-
pose of music [20]. In an extensive review of music performance
[9], the analysis of communication accuracy showed that profes-
sional music performers are able to communicate basic emotions
(e.g., happy, sad, anger) to listeners with an accuracy almost as
high as in facial and vocal expression of emotions. Further, there
is considerable empirical evidence supporting the statement that
emotion is an integral part of a musical experience (see Ref. [10] for
a review).
But are musically induced emotions similar to other emotional
experiences [23]? An early EEG study [4] demonstrated a character-
istic difference in cortical brain activationpatterns: positive musical
excerpts produced a more pronounced lateralisation towards the
left fronto-temporal cortices, whereas negative musical excerpts
produced a right fronto-temporal activation pattern. This early
result is supported by recent studies showing that left frontal areas
are involved with the processing of positive music and right frontal
areas with the negative music [1,8,16]. Similar frontal asymmetry is
well reported for the processing of affective visual stimuli [2,3].
Corresponding author at: Department of Psychology, Goldsmiths College, Uni-
versity of London, New Cross, London SE14 6NW, United Kingdom.
Tel.: +44 2079197334; fax: +44 2079197873.
E-mail address: j.bhattacharya@gold.ac.uk (J. Bhattacharya).
Therefore, it is reasonable to infer that there are some overlaps
between musical emotions and visual emotions.
But can these musically induced emotions arising through the
auditory channel influence our interpretation of emotions aris-
ing through other sensory channels (i.e. visual)? Research on
crossmodal integration of auditory and visual emotions [5] shows
that rating of affective information in one sensory modality can
be biased towards the direction of the emotional valence of
information in another sensory modality. Event-related-potential
(ERP) studies presenting emotionally congruent and incongru-
ent face–voice pairs reveal early ERP effects (N1, P2 components)
for congruent face–voice pairs, suggesting an early interaction
between auditory and visual emotional stimuli [15,22]. Therefore,
musical emotion can interact with visual emotion for simultaneous
music and visual processing.
But can musical emotion interact with or even influence the
visual emotion for non-simultaneous processes? In other words,
can music be used as an affective priming stimulus which could
systematically influence the emotional processes of target visual
stimuli? Music was earlier used as a priming stimulus in semantic
context [12,21]. To the best of our knowledge, the current study is
the first to address this issue in a crossmodal context by using both
behavioural and ERP experiments.
We performed two separate experiments – (i) behavioural and
(ii) electrophysiological (EEG) – on a total of 46 adult human par-
ticipants. Thirty participants (15 males and 15 females, mean age
26.1±4.31 years) took part in (i) without any cash incentive, and
sixteen participants (8 males and 8 females, mean age 27.5±5.88
years) took part in (ii) against a small cash incentive. All partici-
0304-3940/$ – see front matter © 2009 Elsevier Ireland Ltd. All rights reserved.
doi:10.1016/j.neulet.2009.03.044
Author's personal copy
130 N. Logeswaran, J. Bhattacharya / Neuroscience Letters 455 (2009) 129–133
Fig. 1. (a) Emotion ratings of happy, sad and neutral faces, regardless of musical primes. (b) Ratings for six individual conditions: happy, sad and neutral faces primed by
happy or sad musical excerpts. (c) Difference (happy sad) in ratings for three facial emotions. Note that the largest effect was found for neutral facial emotion.
pants were healthy right-handed university students, had normal
hearing, normal or corrected-to-normal vision, and had no special
musical expertise or musical education. The study was conducted
in accordance with the Declaration of Helsinki, and was approved
by the Internal Ethics Committee at Goldsmiths College, University
of London. All participants gave informed written consent before
both experiments.
All musical stimuli were taken from a previous study [1]. Briefly,
there were 120 instrumental musical excerpts belonging to two
emotional categories: happy and sad. Each piece was played for
15s with both beginning and end faded in and out, respectively, to
minimize surprise. The visual stimuli were faces of 40 different indi-
viduals with each individual showing threetypes of facial emotions:
happy, sad and neutral (http://www.macbrain.org/stim/faces.htm).
There were 90 trials equally divided into six possible conditions
(2 musical emotions ×3 facial emotions). Each trial lasted for 16s,
where a 15-s musical excerpt was followed by a facial stimulus pre-
sented for 1 s. At the end of each trial, participants were required
to rate the facial emotion on a 7-point scale: 1= extremely sad,
2 = moderately sad, 3 = slightly sad, 4 = neutral, 5 = slightly happy,
6 = moderately happy, and 7 = extremely happy. Participants were
told to try and to feel the emotion of the musical stimuli and
to rely mainly on their feelings while they rated the facial stim-
uli.
The EEG study followed a similar procedure to the behavioural
study but with the following exceptions. There were 120 trials with
20 trials for each condition. Further, instead of an explicit emo-
tional evaluation, the participants were asked to press a button
whenever a female face was shown. This minimized the explicit
components of emotional processing, and the remaining differ-
ences, if any,would reflect the implicitness of emotional processing.
Trials were randomized within each block and across participants.
EEG signals were recorded from 28 Ag/AgCl scalp electrodes (Fp1,
Fp2, F3, F4, F7, F8, Fz, FC3, FC4, FCz, C5, C6, C3, C4, Cz, CP5, CP6, CP3,
CP4, CPz, P7, P8, P3, P4, Pz, O1, O2, Oz) according to the Interna-
tional 10/20-system. Horizontal and vertical eye movements were
recorded from electrodes placed around the eyes. Impedances were
kept below 5 k. All scalp electrodes were algebraically referenced
to the average of two earlobes. The sampling frequency was 500 Hz.
Perceived emotional ratings were assessed using factorial
repeated-measures analysis of variance (ANOVA) with the factors,
musical emotion (two levels: happy and sad) and facial emotion
(three levels: happy, sad and neutral). Further post hoc compar-
isons between pairs of specific conditions were carried out by using
Fig. 2. Grand average ERP profiles at 13 selected electrodes (see the electrode locations on the top) during processing of neutral facial stimuli primed by either happy music
(thick line) or sad music (thin line).
Author's personal copy
N. Logeswaran, J. Bhattacharya / Neuroscience Letters 455 (2009) 129–133 131
paired t-tests. Greenhouse–Geisser correction wasapplie d toadjust
for degrees of freedom.
The data pre-processing was carried out by EEGLAB [6]. Stim-
ulus epochs of 1500ms starting 500 ms before the face onset
were generated offline from the continuous data. Extracted epochs
were subsequently checked for artefacts by visual inspection. In
order to correct for ocular artefacts including eye blinks, Inde-
pendent Component Analysis was performed on the remaining
epochs. ERP was obtained by averaging artefact free epochs for
each of the six conditions. A series of 2 ×2 factorial repeated-
measures ANOVAs with factors, priming (happy vs. sad) and region
(as selected after scalp maps), were conducted on mean ERP
amplitudes at specific regions of interest (ROI) between any two
conditions with identical facial emotion but with different musical
primes. Temporal regions of interest were selected on the basis of
global field power (GFP) [13] which quantifies the instantaneous
global activity across the spatial potential fields. Spatial regions
of interest were selected on the basis of scalp maps of mean ERP
amplitudes at the pre-selected temporal region of interest. Across
statistical comparison, we found that spatial regions of interest
consisted of two levels—anterior and posterior. Instead of a data-
blind procedure of selecting region of interests on an ad-hoc basis,
this data-driven method selected only a few regions of interests,
thereby minimizing the error variance and maximizing the effect
size.
Fig. 1(a) shows the mean ratings of happy, neutral and sad
faces regardless of the type of musical primes. It was clear that
the facial emotions were rightly rated and classified by our par-
ticipants. Mean ratings for six conditions, separately, are shown
in Fig. 1(b). The happy faces when primed by happy music were
rated (6.13 ±0.36) higher (i.e. more happy) than when primed
by sad music (5.56 ±0.35), and the sad faces when primed by
sad music were rated (1.87±0.30) lower (more sad) than when
primed by happy music (2.44 ±0.30). Further, when the neutral
faces were primed by happy music, the rating (4.68±0.33) was
much higher than when it was primed by sad music (3.37±0.54).
Therefore, the differential effects of priming (happysad) were
similar for happy and sad faces (mean difference rating of 0.57)
but was almost doubled (mean difference rating of 1.31) for
neutral face (Fig. 1(c)). Repeated-measures ANOVA showed that
there were highly significant main effects for musical emotion
(F(1,29)= 103.29, p< 0.001) and facial emotion (F(2,28) = 1358.89,
p< 0.001). The music ×faces interaction effect was also found to
be highly significant (F(2,58) = 34.37, p< 0.001). These results show
that the effect of musical priming was largest for emotionally neu-
tral target stimuli.
ERPs were always compared between two conditions with same
facial emotion but with different priming, and same analysis pro-
cedure was applied for all three types of facial emotions. Since the
behaviour study showed largest effect for neutral faces, we strate-
gically emphasized the results for neutral faces in details as follows,
and the results for other two types of facial emotions will be briefly
mentioned.
Visual inspections revealed that ERP profiles for neutral faces
primed by happy music were markedlydif ferent from those primed
by sad music (Fig. 2). Enhanced N1 component was seen across all
frontal and central electrodes for happy, as opposed to sad, musi-
cal primes. The classical N170 face component was exhibited in
occipital and parietal (P7, P8, not shown) regions bilaterally for
both priming conditions. Between 180 and 250 ms, an increased
positivity (or reduced negativity) for happy primes as compared to
sad primes was noticed in frontal, central. At a later stage (300 ms
onward), posterior positivity was observed for both conditions. GFP
values were plotted in Fig. 3 (top panel) and the two profiles were
separated as early as from 50ms till 150ms, and then again for the
time period 190–210ms. Scalp maps at these time windows were
Fig. 3. Global field power values for three emotional facial stimuli: neutral face
(upper panel), happy face (middle panel), and sad face (lower panel). For each emo-
tion type, two types of musical priming, happy and sad, were shown in thick and
in thin lines, respectively. The high GFP values correspond to pronounced potential
fields with high peaks (both positive and negative)and steep gradients, whereas low
GFP values correspond to flat potential fields. Foreach facial emotion, time windows
where the two profiles showed maximal separation between two musical priming
were used for successive statistical analysis.
displayed in Fig. 4 for both conditions and their differences. As com-
pared to negative musical prime, positive musical prime showed
an enhanced negativity in frontal and central brain regions during
50–150ms and enhanced positivity during 190–210 ms in similar
anterior brain regions. For the N1 time window (50–150 ms), statis-
tical analysis showed a significant effect of priming (F(1,15)= 5.35,
p= 0.03), and a priming ×region interaction effect (F(1,15)= 8.62,
p= 0.01). For the later time window (190–210 ms), a significant
priming effect (F(1,15) = 4.66, p= 0.047) was found.
GFP plots for other two types of facial emotions were shown in
Fig. 3 (middle and lower panels). For happy facial stimuli, differ-
ences between happy and sad musical primes were found between
0–50 and 160–210ms. Statistical analysis of mean ERP ampli-
tudes showed a near significant priming effect during 160–210 ms
(F(1,15) = 4.33, p= 0.06) and an almost significant priming×region
interaction effect between 160 and 210 ms (F(1,15) = 4.16, p= 0.06).
For sad facial stimuli, the early (0–50ms) difference between
the two priming conditions was also found, but the later differ-
ences were found during 430–450 ms. Statistical analysis of mean
ERP amplitudes showed significant region effects, F(1,15)= 7.54,
p= 0.015 and F(1,15) = 32.29, p<0.001, for both time windows,
Author's personal copy
132 N. Logeswaran, J. Bhattacharya / Neuroscience Letters 455 (2009) 129–133
Fig. 4. The scalp distribution of mean ERP amplitudes at two different time windows (50–150 and 190–210ms) for target neutral faces primed by happy (left column) and
sad (middle column) musical stimuli. The right column shows the same but for the difference potentials (sadhappy). As compared to sad musical prime, happy musical
prime produced enhanced negativity at the first time window and enhanced positivity at the second time window, both over anterior brain regions.
0–50 ms and 430–450 ms, respectively, and a near significant effect
of priming (F(1,15) = 4.30, p= 0.06) was found at the later time win-
dow.
Our behavioural experiment confirmed that emotional rating
of the target facial stimuli could be biased towards the direction
of the emotional valence of the musical primes. Earlier research
[5,22] reported that emotions in auditory stimuli interactwith emo-
tions in visual stimuli for simultaneously presented auditory–visual
stimuli. But our result extends it further by showing that such
interaction could also occur for non-simultaneous processing, i.e.
when the emotional auditory stimuli precede the emotional visual
stimuli. In other words, priming musical stimuli can systematically
influence the perceived emotional contents in target visual stimuli.
Music was earlier used in semantic priming paradigms. Using chord
sequences, either consonant or dissonant, as priming stimuli, it was
shown [21] that target words are faster recognized for emotionally
congruent chord–word pairs than for incongruent ones. Further,
when target words are preceded by semantically unrelated musical
primes, N400, an ERP component reflecting contextual integration,
effect is reported [12]. Altogether, this suggests that music has an
excellent potential to be used as an emotional priming stimulus.
Our behaviour data also shows that the largest effect of musi-
cal priming was found for neutral faces with an effect size almost
twice those for happy or sad faces. It was shown earlier [14] that
as compared to emotionally distinct (i.e. happy, angry, sad) facial
stimuli, emotionally ambiguous (i.e. neutral) facial stimuli are more
likely to be influenced by auditory cues in a facial emotion detec-
tion task. The information-content of neutral faces are supposedly
lower than those of happy or sad faces, and since the brain relies on
cues from multiple sensory stimuli to create an optimal represen-
tation of the external environment, emotionally neutral stimuli is
being influenced by emotionally conspicuous stimuli, even though
being generated by different senses. Although in our paradigm,
there is no such explicit requirement of integration of informa-
tion across musical and visual stimuli, our findings suggest that
a generic mechanism of multimodal affective interaction might
exist also in a priming paradigm. Alternatively, this could also be
explained by the affect-as-information hypothesis [19] which relies
on the assumption that affective processes mainly occur outside
of awareness. In contrast to traditional assumption of judgement
and decision making which emphasizes the role of target related
features, this hypothesis states that when making evaluative judge-
ments, participant often ask themselves, “How do I feel about it?”
[18], and therefore, “they may mistake feelings due to a pre-existing
state as a reaction to the target” [17]. Since the emotionally neutral
facial stimuli contain less information than emotionally conspic-
uous (happy or sad) facial stimuli, the transient affect from the
priming musical stimuli has the maximal impact in determining
the evaluation of the neutral facial stimuli.
Our ERP data showed that for neutral facial emotion, happy
music, as compared to sad music, showed a significant effect
during the N1 time period (50–150ms). Earlier Pourtois et al.
[15] have found that simultaneous presentation of emotionally
congruent face–voice pairs produce an enhancement of auditory
N1 component as compared to incongruent pairs, suggesting an
early crossmodal binding. In contrast to this study which reported
enhancement over auditory cortex, our N1 effect was predomi-
nant over fronto-central and midfrontal regions (FC3/4, FCz, Fz, Cz).
Taken together, this suggests that happy or positive auditory emo-
tion is more likely to influence neutral visual emotion by engaging
brain regions responsible for top-down projections.
ERP results also showed priming related enhancement of P2
(190–210ms) component for happy and neutral target faces but
not for sad target faces. Similar modulation of P2 has also recently
been reported [22] for happy picture–voice pairs presented simul-
taneously but not for sad pairs. Enhanced positivity at similar time
window has also been found for processing isolated emotional faces
as compared to neutral faces [7]. However, the functional role of
P2 is not yet clear (but see Ref. [22] for some possible explana-
tions) in mediating interaction between priming musical stimuli
and emotionally selective (happy and neutral, but not sad) target
faces.
Finally, let us offer a few practical remarks. First, the current
paradigm of music-induced emotional priming is quite different
from other mood-induction procedures which are associated with a
longer lasting change of emotional states, whereas our study inves-
tigated the effect of emotional changes on a much shorter time
scale [11]. Secondly, unlike previous ERP studies of facial emotion
Author's personal copy
N. Logeswaran, J. Bhattacharya / Neuroscience Letters 455 (2009) 129–133 133
processing, we alwayscompared the same facial emotional type but
differed only in priming. Therefore, our results indicate a more sub-
tle component in early neural responses which can potentially bias
subsequent emotional evaluation occurring at later stages. This was
also manifested by our robust behavioural findings which called
for an explicit evaluation of facial emotions. Thirdly, our ERP data
primarily reflects an implicit emotional processing since the par-
ticipants were naïve with respect to the specific aims of the study.
Further, as the task of gender detection did not require the par-
ticipants to focus on the presented emotions, the results are less
likely to be attributed to differences in directed attention as a func-
tion of presented emotions. Therefore, the ERP results suggest an
early processing of emotional facial expression primed by musical
stimuli.
In summary, the results of our behavioural and ERP study
revealed some patterns of crossmodal influences by musical emo-
tion. Behavioural data clearly showed that listening to musical
excerpts, albeit short, could significantly influence the subsequent
explicit evaluation of visual emotional stimuli. ERP data showed
that such musical priming could also influence implicit visual emo-
tional processes.
Acknowledgements
The study was supported by JST.ERATO Shimojo project (JB). We
are thankful to Prof. Eckart Altenmüller for the music stimuli, to
Rob Davis for technical support, to Job Lindsen for help in data
pre-processing, and to Prof. Rolf Reber for his helpful comments
as a reviewer. Author contributions: J.B. conceived research; N.L.
collected data; J.B. and N.L. analyzed data and wrote the paper.
References
[1] E. Altenmuller, K. Schurmann, V.K. Lim, D. Parlitz, Hits to the left, flops to the
right: different emotions during listening to music are reflected in cortical
lateralisation patterns, Neuropsychologia 40 (2002) 2242–2256.
[2] T.Canli, J.E. Desmond, Z. Zhao, G. Glover, J.D.E. Gabrieli, Hemispheric asymmetry
for emotional stimuli detected with fMRI, Neuroreport 9 (1998) 3233–3239.
[3] R.J. Davidson, Anterior cerebral asymmetry and the nature of emotion, Brain
and Cognition 20 (1992) 125–151.
[4] R.J. Davidson, G.E. Schwartz, C. Saron, J. Bennett, D.J. Goleman, Frontal versus
parietal EEG asymmetry during positive and negative affect, Psychophysiology
16 (1979) 202–203.
[5] B. de Gelder, J. Vroomen, The perception of emotions by ear and by eye, Cogni-
tion & Emotion 14 (2000) 289–311.
[6] A. Delorme, S. Makeig, EEGLAB: an open source toolbox for analysis of single-
trial EEG dynamics including independent component analysis, Journal of
Neuroscience Methods 134 (2004) 9–21.
[7] M. Eimer, A. Holmes, Event-related brain potential correlates of emotional face
processing, Neuropsychologia 45 (2007) 15–31.
[8] E.O. Flores-Gutierrez, J.L. Diaz, F.A. Barrios, R. Favila-Humara, M.A. Guevara, Y.
del Rio-Portilla, M. Corsi-Cabrera, Metabolic and electric brain patterns during
pleasant and unpleasant emotions induced by music masterpieces, Interna-
tional Journal of Psychophysiology 65 (2007) 69–84.
[9] P.N.Juslin, P. Laukka, Communication of emotions in vocalexpression and music
performance: different channels, same code? Psychological Bulletin 129(2003)
770–814.
[10] P.N. Juslin, J.A. Sloboda (Eds.), Music and Emotion, Oxford University Press,
Oxford, 2001, p. 487.
[11] P.N. Juslin, D. Vastfjall, Emotional responses to music: the need to consider
underlying mechanisms, The Behavioral and Brain Sciences 31 (2008) 559–575
(discussion 575–621).
[12] S. Koelsch, E. Kasper, D. Sammler, K. Schulze, T. Gunter, A.D. Friederici, Music,
language and meaning: brain signatures of semantic processing, Nature Neu-
roscience 7 (2004) 302–307.
[13] D. Lehmann, W. Skrandies, Reference-free identification of components of
checkerboard-evoked multichannel potential fields, Electroencephalography
and Clinical Neurophysiology 48 (1980) 609–621.
[14] D.W. Massaro, P.B. Egan, Perceiving affect from the voice and the face, Psycho-
nomic Bulletin & Review 3 (1996) 215–221.
[15] G. Pourtois, B. de Gelder, J. Vroomen, B. Rossion, M. Crommelinck, The time-
course of intermodal binding between seeing and hearing affectiveinformation,
Neuroreport 11 (2000) 1329–1333.
[16] D.L.Santesso, L.A. Schmidt, L.J. Trainor, Frontal brain electrical activity(EEG) and
heart rate in response to affective infant-directed (ID) speech in 9-month-old
infants, Brain and Cognition 65 (2007) 14–21.
[17] N. Schwarz, Feelings as information: informational and motivational functions
of affective states, in: E.T. Higgins, R. Sorrentino (Eds.), Handbook of Motivation
and Cognition: Foundations of Social Behaviour, vol. 2, Guildford Press, New
York, 1990, pp. 527–561.
[18] N.Schwarz, G.L. Clore, How do I feel about it? The informative function of mood,
in: K. Fiedler,J. Forgas (Eds.), Affect, Cognition and Social Behaviour, C.J. Hogrefe,
Toronto, 1988, pp. 44–62.
[19] N. Schwarz, G.L. Clore, Mood, misattribution, and judgments of well-
being—informative and directive functions of affective states, Journal of
Personality and Social Psychology 45 (1983) 513–523.
[20] M. Serafine, Music as Cognition: The Development of Thought in Sound,
Columbia University Press, New York, 1988.
[21] B. Sollberger, R. Reber, D. Eckstein, Musical chords as affective priming context
in a word-evaluation task, Music Perception 20 (2003) 263–282.
[22] K.N. Spreckelmeyer, M. Kutas, T.P. Urbach,E. Altenmuller, T.F. Munte, Combined
perception of emotion in pictures and musical sounds, Brain Research 1070
(2006) 160–170.
[23] L.J. Trainor, L.A. Schmidt, Processing emotionsinduce d bymusic, in: I. Peretz, R.
Zatorre (Eds.), The Cognitive Neuroscience of Music, OxfordUniversity of Press,
Oxford, 2007, pp. 310–324.
... Meanwhile, several studies have investigated the effect of emotional auditory primes on face encoding. In contrast to studies utilizing visual primes, these studies did not find an effect of auditory emotional primes on N170 responses, implying that the emotional content of auditory primes does not influence structural face encoding (Lense et al., 2014;Logeswaran and Bhattacharya, 2009;Paulmann and Pell, 2010;Wang et al., 2017). Regarding encoding-related LPPs, Paulmann and Pell (2010) showed that the encoding-related LPP responses were smaller for fearful and angry faces preceded by fearful and angry vocal expressions than those preceded by happy vocal expressions when the duration of the vocal stimuli was short (i.e., 200 ms), whereas the effect was reversed when the duration of the vocal stimuli was longer (i.e., 400 ms). ...
... The findings suggest that the effect of negative vocal primes on cognitive face encoding depends on the duration of the vocal primes. With a long presentation duration (500 ms -15 s), however, the reversed effect could not be replicated in other studies (Lense et al., 2014;Logeswaran and Bhattacharya, 2009;Wang et al., 2017). ...
... The nonsignificant effect of emotional auditory primes on N170 responses and the inconsistent findings regarding encoding-related LPP responses might be due to differences in the auditory primes. Specifically, in several studies, the auditory primes were used by music clips (Lense et al., 2014;Logeswaran and Bhattacharya, 2009) or sentences spoken with different emotional prosodies (Paulmann and Pell, 2010). In addition to conveying emotion information, these stimuli conveyed other information (e.g., melody and rhythm in music clips and semantic information in sentences) that might distract individuals from emotion processing and thus alter the effect of auditory emotional priming on face encoding. ...
Article
Previous studies have suggested that emotional primes, presented as visual stimuli, influence face memory (e.g., encoding and recognition). However, due to stimulus-associated issues, whether emotional primes affect face encoding when the priming stimuli are presented in an auditory modality remains controversial. Moreover, no studies have investigated whether the effects of emotional auditory primes are maintained in later stages of face memory, such as face recognition. To address these issues, participants in the present study were asked to memorize angry and neutral faces. The faces were presented after a simple nonlinguistic interjection expressed with angry or neutral prosodies. Subsequently, participants completed an old/new recognition task in which only faces were presented. Event-related potential (ERP) results showed that during the encoding phase, all faces preceded by an angry vocal expression elicited larger N170 responses than faces preceded by a neutral vocal expression. Angry vocal expression also enhanced the late positive potential (LPP) responses specifically to angry faces. In the subsequent recognition phase, preceding angry vocal primes reduced early LPP responses to both angry and neutral faces and late LPP responses specifically to neutral faces. These findings suggest that the negative emotion of auditory primes influenced face encoding and recognition.
... Interestingly, there is some evidence from experiments that used an affective priming paradigm indicating that human faces are a stronger source of stimulus control than musical stimuli (e.g., Bakker & Martin, 2015;Huziwara et al., 2023;Logeswaran & Bhattacharya, 2009;Zhou et al., 2019). In a typical affective priming task, participants are asked to classify target stimuli as positive (e.g., a happy face) or negative (e.g., a sad face) by pressing designated keys on a computer keyboard. ...
... Considering that combined properties really increase the evocative effect of music excerpts, higher levels of transfer of function should be found by participants of original group than by those in the tempo and mode groups. Through a slightly modified task but still based on the priming paradigm, a behavioral process similar to transfer of function was described by Logeswaran and Bhattacharya (2009). The authors evaluated the extent to which music could be used as an affective priming to influence the emotional processes of visual stimuli. ...
Article
Empirical evidence has supported that musical excerpts written in major and minor modes are responsible for evoking happiness and sadness, respectively. In this study, we evaluated whether the emotional content evoked by musical stimuli would transfer to abstract figures when they became members of the same equivalence class. Participants assigned to the experimental group were submitted to a training procedure to form equivalence classes comprising musical excerpts (A) and meaningless abstract figures (B, C, and D). Afterward, transfer of function was evaluated using a semantic differential. Participants in the control group showed positive semantic differential scores for major mode musical excerpts, negative scores for minor mode musical excerpts, and neutral scores for the B, C, and D stimuli. Participants in the experimental groups showed positive semantic differential scores for visual stimuli equivalent to the major modes and negative semantic differential scores for visual stimuli equivalent to the minor modes. These results indicate transfer of function of emotional content present in musical stimuli through equivalence class formation. These findings could provide a more comprehensive understanding of the effects of using emotional stimuli in equivalence class formation experiments and in transfer of function itself.
... The priming effect is defined as the influence of a preceding stimulus on the processing of a subsequent stimulus [49]. For example, Logeswaran et al. asked subjects to give impressions of three different facial images after listening to music and found that the faces looked happier when they listened to happy music beforehand and sadder when they listened to sad music beforehand [50,51]. In the present study, subjects looked at a picture of a pop idol (a visual stimulus) and/or listened to the idol's music under TVA, TV, and TA conditions before grasping the handshake device. ...
Article
Full-text available
“Handshaking parties,” where pop idols shake hands with fans, can be exciting. The multimodal stimulation of tactile, visual, and auditory sensations can be captivating. In this study, we presented subjects with stimuli eliciting three sensory responses: tactile, visual, and auditory sensations. We found that the attraction scores of subjects increased because they felt the smoothness and obtained a human-like sensory experience grasping a grip handle covered with artificial skin, faux fur, and abrasive cloth with their dominant hand as they looked at a picture of a pop idol or listened to a song. When no pictures or songs were presented, a simple feeling of slight warmth was correlated with the attraction score. Results suggest that multimodal stimuli alter tactile sensations and the feelings evoked. This finding may be useful for designing materials that activate the human mind through tactile sensation and for developing humanoid robots and virtual reality systems.
... That is why knowing the history of the person is crucial to know musical effects. Our participant's musical training feasibly influenced her ability to transfer auditory emotional experiences to other sensory modalities experiences (e.g., visual or tactile), as it is observed in trained musicians (Logeswaran and Bhattacharya, 2009). Moreover, it may have influenced her intuition to use music to relieve opioid withdrawal, to identify pieces with certain harmonies, pitches, tones, and rhythms that relieve pain, to appreciate music in a more emotionally and cognitively sophisticated way, and to avoid discordant pieces provoking her pain and evoking stressful situations (e.g., on social injustice). ...
Article
Full-text available
Listening to music has progressively been proposed as a complementary alternative for chronic pain; understanding its properties and its neurobiological bases is urgent. We show a phenomenological investigation of a woman who has lived 20 years with chronic pain. The inquiry involved her experience of the context in which she listens to music, the intensity and quality of pain, body mapping, memories, emotions, and cognition. The participant listens to music for different reasons, such as pain and anxiety relief, motivation to exercise, and quality of sleep, but all seem to revolve around different strategies for pain management. Experiences in physiological and cognitive aspects included perceived restorative sleep that may have improved the participant's general wellbeing and improved cognitive and motor performance as well as communication skills. The music enabled the participant not only to relieve pain but also withdrawal effects after discontinuing her opioid-based treatment. These effects may encompass endogenous opioid and dopamine mechanisms involving natural analgesia associated with pleasurable experiences. Future studies could consider phenomenological case studies and therapeutic accompaniment to reorient subjective properties of pain and expand quantitative and qualitative knowledge for more comprehensive reports on music and analgesia.
... Prior literature suggests that observers may be influenced by a simultaneously presented auditory stimulus when asked to make judgments of a visual stimulus (Gerdes et al., 2014). For example, short excerpts of music have been shown to modulate judgments of the emotion depicted in facial expressions or other images (Logeswaran & Bhattacharya, 2009;Marin et al., 2012). Additionally, Braun Janzen et al. (2022) demonstrated that museum visitors rated a Kandinsky painting as depicting more positive emotional valence if they listened to music they liked rather than disliked while viewing the painting. ...
Article
Observers can make independent aesthetic judgments of at least two images presented briefly and simultaneously. However, it is unknown whether this is the case for two stimuli of different sensory modalities. Here, we investigated whether individuals can judge auditory and visual stimuli independently, and whether stimulus duration influences such judgments. Participants (N = 120, across two experiments and a replication) saw images of paintings and heard excerpts of music, presented simultaneously for 2 s (Experiment 1) or 5 s (Experiment 2). After the stimuli were presented, participants rated how much pleasure they felt from the stimulus (music, image, or combined pleasure of both, depending on which was cued) on a 9-point scale. Finally, participants completed a baseline rating block where they rated each stimulus in isolation. We used the baseline ratings to predict ratings of audiovisual presentations. Across both experiments, the root mean square errors (RMSEs) obtained from leave-one-out-cross-validation analyses showed that people's ratings of music and images were unbiased by the simultaneously presented other stimulus, and ratings of both were best described as the arithmetic mean of the ratings from the individual presentations at the end of the experiment. This pattern of results replicates previous findings on simultaneously presented images, indicating that participants can ignore the pleasure of an irrelevant stimulus regardless of the sensory modality and duration of stimulus presentation. (PsycInfo Database Record (c) 2023 APA, all rights reserved).
Article
Full-text available
Introduction and aim: Music has a profound relationship with human emotions. However, the characteristics and correlates of this relationship have not yet been determined. The aim of this review is to describe the results of the research on the capacity of music in the generation and modification of the emotional state in the listener, the respective methodological designs and the evaluation tests used. Methods: A systematic review of articles written between 2009 and 2021 was carried out. The criteria for inclusion, exclusion, analysis, and data recording are based on the structure proposed in the Preferred Items of Reports for Systematic and Meta-analyses Reviews (PRISMA). Results: The results show that music can generate and modify changes in the emotional state of listeners, in addition to modifying cognitive performance in recognition tasks and executive performance. However, the neuropsychological foundations and characteristics that would determine these modifications are diverse and inconclusive. Conclusions: It is necessary to carry out studies that, through a rigorous methodology, allow to establish consistent conclusions on the human characteristics that sustain the capacity of music in the generation or modification of the emotional state.
Book
This book presents advances in speech and music in the domain of audio signal processing. The book begins with introductory chapters on the basics of speech and music, and then proceeds to computational aspects of speech and music, including music information retrieval and spoken language processing. The authors discuss the intersection in the field of computer science, musicology and speech analysis, and how the multifaceted nature of speech and music information processing requires unique algorithms, systems using sophisticated signal processing, and machine learning techniques that better extract useful information. The authors discuss how a deep understanding of both speech and music in terms of perception, emotion, mood, gesture and cognition is essential for successful application. Also discussed is the overwhelming amount of data that has been generated across the world that requires efficient processing for better maintenance, retrieval, indexing and querying and how machine learning and artificial intelligence are most suited for these computational tasks. The book provides both technological knowledge and a comprehensive treatment of essential topics in speech and music processing.
Article
RESUMEN Introducción y objetivo: Existe una profunda relación entre las emociones humanas y la música. Sin embargo, las características y correlatos neurobiológicos de esta relación aún no se han determinado. La presente revisión tiene como objetivo describir los resultados de la investigación sobre la capacidad de la música en la generación y modificación del estado emocional del oyente, los respectivos diseños metodológicos y las pruebas de evaluación utilizadas. Métodos: Se realizó una revisión sistemática de artículos escritos entre 2009 y 2021. Los criterios de inclusión, exclusión, análisis y registro de datos se basan en la estructura propuesta en los Ítems Preferidos de Reportes para Revisiones Sistemáticas y Metaanálisis (PRISMA). Resultados: Los resultados muestran que la música tiene la capacidad de generar cambios en el estado emocional de los oyentes, además de modificar el desempeño cognitivo en tareas de reconocimiento y desempeño cognitivo. Sin embargo, los fundamentos neuropsicológicos y las características que determinarían estas modificaciones son diversas y no concluyentes. Conclusiones: Es necesario realizar estudios que, mediante una metodología rigurosa, permitan establecer conclusiones consistentes sobre las características humanas que sustentan la capacidad de la música en la generación o modificación del estado emocional. ABSTRACT Introduction and aim: Music has a profound relationship with human emotions. However, the characteristics and correlates of this relationship have not yet been determined. The aim of this review is to describe the results of the research on the capacity of music in the generation and modification of the emotional state in the listener, the respective methodological designs and the evaluation tests used. Methods: A systematic review of articles written between 2009 and 2021 was carried out. The criteria for inclusion, exclusion, analysis, and data recording are based on the structure proposed in the Preferred Items of Reports for
Chapter
In this paper, we try to classify the emotional cues of sound and visual stimuli solely from their source characteristics, i.e., from the 1D time series generated from the audio signals and the two-dimensional matrix of pixels generated from the affective picture stimulus. The sample data consists of six audio signals of 15 s each and six affective pictures, of which three belonged to positive and negative valence, respectively. Detrended Fluctuation Analysis (DFA) has been used to calculate the long-range temporal correlations or the Hurst exponent corresponding to the audio signals. The 2D analogue of the DFA technique has been applied on the array of pixels corresponding to affective pictures of contrast emotions. We obtain a single unique scaling exponent corresponding to each audio signal and three scaling exponents corresponding to red/green/blue (RGB) component in each of the visual images. Detrended Cross-correlation (DCCA) technique (both 1D and 2D) has been used to calculate the degree of nonlinear correlation present between the sample audio and visual clips. To assess the proportion of cross-modal correlation in the emotional appraisal, Pearson correlation coefficient was calculated using the DFA exponents of the two modalities. The results and findings have been corroborated with a human response study based on the emotional Likert scale ratings. To conclude, we propose a novel algorithm with which emotional arousal can be classified in cross-modal scenario using only the source audio and visual signals while also attempting to assess the degree of correlation between them.
Chapter
Music is not just a complex sound encompassing multiple tones but an amalgamation between the harmonicity of the spectral components and its temporal relationships. Music perception involves complex auditory processing, starting at cochlea wherein fundamental pitch, duration, and loudness are encoded in the tiny hair cells of the inner ear. Derailing of the hair cells causes hearing loss, which in turn affects the spectrotemporal resolution. The offshoot of the hearing loss at the cochlear level translates physiologically into deficits in the extraction and coding of the pitch at the brainstem level, which in turn impairs pitch and temporal perception at the cortical level. Thus, hearing loss adversely affects perception of pitch, temporal, and loudness, all of which compound as difficulty appreciating the normal aspects of music perception. The contemporary solution FOR improving audibility and perception of speech sounds in the hearing-impaired population are the use of hearing aids and cochlear implants. Although these devices are primarily meant to amplify speech sounds, their utility is also advocated for music perception. Digital hearing aids employ sophisticated signal processing techniques to improve the perception of speech sounds. However these techniques do not, in music processing, and are not an alternative to human cochlea. The multichannel amplitude compression, used in hearing aids to improve audible range of loudness levels, can cause distortions in the temporal envelope of sounds resulting in poor quality for music perception. Additionally, fast-acting compression circuitry used in the modern digital hearing aids causes more temporal smearing (compared to slow-acting compression), adversely affecting music perception. The limited input dynamic range and higher crest ratio in AD converters of hearing aids fall short of processing live music. Unlike hearing aids, cochlear implants work on the principle of electrical stimulation. This auditory prosthesis processes the incoming sound and delivers its electrical output directly into the auditory nerve bypassing the cochlea. Though it is more sophisticated and advanced than hearing aids, it was developed to improve speech perception rather than music perception. A cochlear implant uses “N” number of electrode channels situated along the human cochlea to deliver its output. Since the partition along the human cochlea codes various frequencies in sounds (place coding) which cannot be matched by a surgically implanted electrode array, the cochlear implant users experience difficulties with pitch perception. The rate of stimulation used by the cochlear implant may not be higher enough to deliver the higher harmonics of musical sounds. The above discussed physiological limitations of hearing loss and technological limitations of hearing amplification devices on music perception are elaborated in this chapter. The book chapter also features comprehensive discussion on the advancements in signal processing techniques available in hearing amplification devices (hearing aids and cochlear implants) that can address these shortcomings.
Chapter
Full-text available
Can music induce emotions directly and, if so, are these emotions experienced similarly to emotions arising in other contexts? This chapter analyzes these questions from the perspective of neuroscience. Despite the fact that music does not appear to have an obvious survival value for modern adults, research indicates that listening to music does activate autonomic, subcortical, and cortical systems in a manner similar to other emotional stimuli. It is proposed that music may be so intimately connected with emotional systems because caregivers use music to communicate emotionally with their infants before they are able to understand language. In particular, it examines whether music engages the autonomic nervous system, sub-cortical emotion networks, and cortical areas involved in the emotional processing of other types of stimuli. It also investigates whether emotional reactions to music are simply cultural conventions by asking whether and how infants process musical emotions.
Article
Full-text available
This experiment examines how emotion is perceived by using facial and vocal cues of a speaker. Three levels of facial affect were presented using a computer-generated face. Three levels of vocal affect were obtained by recording the voice of a male amateur actor who spoke a semantically neutral word in different simulated emotional states. These two independent variables were presented to subjects in all possible permutations-visual cues alone, vocal cues alone, and visual and vocal cues together-which gave a total set of 15 stimuli. The subjects were asked to judge the emotion of the stimuli in a two-alternative forced choice task (either HAPPY or ANGRY). The results indicate that subjects evaluate and integrate information from both modalities to perceive emotion. The influence of one modality was greater to the extent that the other was ambiguous (neutral). The fuzzy logical model of perception (FLMP) fit the judgments significantly better than an additive model, which weakens theories based on an additive combination of modalities, categorical perception, and influence from only a single modality.
Article
Full-text available
Emotions are expressed in the voice as well as on the face. As a first step to explore the question of their integration, we used a bimodal perception situation modelled after the McGurk paradigm, in which varying degrees of discordance can be created between the affects expressed in a face and in a tone of voice. Experiment 1 showed that subjects can effectively combine information from the two sources, in that identification of the emotion in the face is biased in the direction of the simultaneously presented tone of voice. Experiment 2 showed that this effect occurs also under instructions to base the judgement exclusively on the face. Experiment 3 showed the reverse effect, a bias from the emotion in the face on judgement of the emotion in the voice. These results strongly suggest the existence of mandatory bidirectional links between affect detection structures in vision and audition.
Chapter
Full-text available
reviews research on the impact of affective states on evaluative judgments, presenting evidence that is difficult to reconcile with the assumption that emotional influences on social judgment are mediated by selective recall from memory / rather, the presented research suggests that individuals frequently use their affective state at the time of judgment as a piece of information that may bear on the judgmental task, according to a "how do I feel about it" heuristic extends the informative-functions assumption to research on affective influences on decision making and problem solving, suggesting that affective states may influence the choice of processing strategies / specifically it is argued that negative affective states, which inform the organism that its current situation is problematic, foster the use of effortful, detail oriented, analytical processing strategies, whereas positive affective states foster the use of less effortful heuristic strategies (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Article
Full-text available
Investigated, in 2 experiments, whether judgments of happiness and satisfaction with one's life are influenced by mood at the time of judgment. In Exp I, moods were induced by asking 61 undergraduates for vivid descriptions of a recent happy or sad event in their lives. In Exp II, moods were induced by interviewing 84 participants on sunny or rainy days. In both experiments, Ss reported more happiness and satisfaction with their life as a whole when in a good mood than when in a bad mood. However, the negative impact of bad moods was eliminated when Ss were induced to attribute their present feelings to transient external sources irrelevant to the evaluation of their lives; but Ss who were in a good mood were not affected by misattribution manipulations. The data suggest that (a) people use their momentary affective states in making judgments of how happy and satisfied they are with their lives in general and (b) people in unpleasant affective states are more likely to search for and use information to explain their state than are people in pleasant affective states. (18 ref) (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Article
Using an affective priming paradigm, we demonstrated that the affective tone of musical chords influences the evaluation of target words. In Experiment 1, participants heard either consonant chords with three tones or dissonant chords with four tones as primes and then saw a positive or a negative word as target. Even participants who were unaware of the hypothesis of the experiment evaluated target words faster if the words were preceded by a similarly valenced chord (e.g., consonant-holiday) as compared to affectively incongruent chord-word pairs (e.g. dissonant-humor). In Experiment 2, results of Experiment 1 were replicated even when chord density was held constant at three tones per chord. Results suggest that the affective tone of single musical elements is automatically extracted and might therefore be viewed as A basic process contributing to the strong connection between music and affect.