Content uploaded by Kun Qian
Author content
All content in this area was uploaded by Kun Qian on May 10, 2023
Content may be subject to copyright.
How does Music Affect Your Brain? A Pilot Study on
EEG and Music Features for Automatic Analysis
Gang Luo1,2, Shuting Sun∗1,2, Kun Qian∗1,2,Senior Member, IEEE,
Bin Hu∗1,2,Fellow, IEEE, Bj¨
orn W. Schuller3,4,Fellow, IEEE, and Yoshiharu Yamamoto5,Member, IEEE
Abstract— Music can effectively induce specific emotion
and usually be used in clinical treatment or intervention. The
electroencephalogram can help reflect the impact of music.
Previous studies showed that the existing methods achieved
relatively good performance in predicting emotion response
to music. However, these methods tend to be time consuming
and expensive due to their complexity. To this end, this
study proposes a grey wolf optimiser-based method to predict
the induced emotion through fusing electroencephalogram
features and music features. Experimental results show that,
the proposed method can reach a promising performance for
predicting emotional response to music and outperform the
alternative method. In addition, we analyse the relationship
between the music features and electroencephalogram features
and the results demonstrate that, musical timbre features are
significantly related to the electroencephalogram features.
Clinical relevance— This study targets the automatic predic-
tion of the human response to music. It further explores the
correlation between EEG features and music features aiming to
provide the basis for the extension to the application of music.
The grey wolf optimiser-based method proposed in this study
could supply a promising avenue for the emotion prediction as
induced by music.
I. INTRODUCTION
Music is ubiquitous and sophisticated due to it being widely
used and multiple structures, which include rhythm, melody,
mode, and tonality. Music can express emotion or mood
through these structures [
1
], and music has the capacity of
communicating emotion or mood between the musician and
the listener [
2
]. Generally speaking, these structures bear
numerous potential to affect an individual’s emotion and
This work was partially supported by the Ministry of Science and Tech-
nology of the People’s Republic of China with the STI2030-
Major Projects (No. 2021ZD0201900, No. 2021ZD0202000, and
No. 2021ZD0200601), the National Natural Science Founda-
tion of China (No. 62227807 and No. 62272044), the Teli Young Fellow Pro-
gram from the Beijing Institute of Technology, China, and the Grants-in-
Aid for Scientific Research (No. 20H00569) from the Ministry of Educa-
tion, Culture, Sports, Science and Technology (MEXT), Japan. Correspond-
ing authors: S. Sun, K. Qian and B. Hu.
1,2
Gang Luo, Shuting Sun, Kun Qian, and Bin Hu are with the
Key Laboratory of Brain Health Intelligent Evaluation and Intervention,
Ministry of Education (Beijing Institute of Technology), Beijing 100081,
China, and also with School of Medical Technology, Beijing Institute
of Technology, Beijing 100081, China.
{gang, sunsht, qian,
bh}@bit.edu.cn
3,4
Bj
¨
orn W. Schuller is with GLAM – the Group on Language, Audio, &
Music, Imperial College London, 180 Queen’s Gate, Huxley Bldg., London
SW7 2AZ, UK, and also with the Chair of Embedded Intelligence for Health
Care and Wellbeing, University of Augsburg, Eichleitnerstr.30, Augsburg
86159, Germany. schuller@ieee.org
5
Yoshiharu Yamamoto is with the Educational Physiology Laboratory,
Graduate School of Education, The University of Tokyo, 7-3-1 Hongo,
Bunkyo-ku, Tokyo 113-0033, Japan.
yamamoto@p.u-tokyo.ac.jp
mental state when one listens to music. For example, Kim
and Kang reported that music timbre, pitch, and mode were
considered to influence the individual mood [
3
]. Meanwhile,
a set of theories and applications of music are being created
and utilised, e. g., music intervention [
4
]. In the past decade,
researchers attempted to utilise music to induce specific
emotions in an individual for the purpose of improving
individual mental health. Wang et al. studied the effect of
group music on college students, and they indicated that
the measured score of depression was significantly decreased
subsequent to the music intervention [
5
]. Hsu et al. researched
on music to treat psychiatric inpatients with major depression
and they selected various kinds of music, such as western
music and Chinese music. They observed that depression
symptoms were decreased after music intervention [6].
With the development of wearable devices, the electroen-
cephalogram (EEG) is often used to record an individual’s
brain activity when receiving musical stimulus [
7
]. Further,
a wealth of studies focuses on predicting the emotional
response to music based on EEG data. For example, Qian
et al. proposed an end-to-end branch LSTM-CNN to extract
emotion features for emotion recognition during music
listening [
8
]. Dutta et al. utilised reinforcement learning
to continuesly recognising emotion when listeners received
musical stimuli, and they applied the method to the publicly
available DEAP EEG dataset [
9
]. Although the experimental
results of these studies are encouraging, the above mentioned
methods might be time consuming and expensive in the
practical usage because of the complex deep neural network
models employed and the need of sufficient training data.
Therefore, it is vital to find a simple and cheap method to
predict the emotional response of an individual to music.
The grey wolf optimisation algorithm (GWO) inspired by
the hunting activities of grey wolves is one of the recent
advanced metaheuristics swarm intelligence methods [
10
].
Compared to other swarm intelligence methods, GWO comes
with the advantage of less parameters and no derivation
information being required in the initial search [11].
We propose an emotion prediction framework, and we apply
the framework to an EEG database recorded during listening
affective music [
12
]. Firstly, we analyse the relationship
between EEG features and music features. Then, we utilise
GWO to build a model to predict the emotional response.
The main contributions are as follows: To the best of our
knowledge, it is the first time to apply GWO to the mentioned
EEG database for predicting the emotion response, and
the experimental results demonstrate that GWO appears
promising. We further analyse the correlation between EEG
features and music features, which can provide help in
applications in the future.
The rest of this paper will be organised as follows: Firstly,
the database and methods used will be introduced in Section II.
Next, experimental results and a discussion will be given in
Section III and Section IV, respectively. Finally, Section V
concludes the work and the findings in this paper.
II. MATERIALS AND METHO DS
A. Database and EEG Preprocessing
An EEG database recorded during listening to affective
music is used in this study [
12
]. The affective music was
measured by self-evaluation from the individuals, and it was
divided into different genres including happy, sad and so
on. The database contains the EEG data of 31 participants
aging from 18 to 66 years (median 35 years, 18 subjects
being female). All participants are right-handed and healthy
without any mental disorder or affective disorder. Moreover,
the participants did not have any problem in hearing. Each
of the participants needed to accomplish 6 tasks. During the
6 tasks, the first and the last task were resting and the other
tasks were listening to music. Every participant listened to
40 pieces of affective music, and every music was played for
about 12 s under 4 tasks (task 2 to task 5). The EEG data was
recorded by a pair of devices with 19 channels electrodes that
locate in the cerebral cortex according to the international
10/20 standard. More details can be found in [
13
], [
14
], [
15
].
Note that, the EEG dataset of happy music listening served
for this study because we would like to induce happy emotion
in improving mental health in the future.
EEG is a kind of weak physiological signal and sensitive
to the environment. Apart from the signal we wanted, various
artefacts are collected during the EEG recording. For instance,
the electrooculogram (EOG) resulting from eye movement
of the individual and the signals of current interference are
collected at the same time. Therefore, EEG preprocessing is
the first step.
The EEG signal contains different frequency signals, and
only a part of the frequency signals are suited for analysis.
We firstly filter the 50 Hz current interference and reserve the
signals between 0.5 Hz and 45 Hz by a high pass filter and
a low pass filter. Meanwhile, we remove the EOG artefacts
by independent component analysis (ICA) which is a power
signal processing algorithm for separating different signals
[
16
]. Moreover, the alpha band is confirmed to be related with
emotion, e. g., happy emotion [
17
] as of major interest, here.
As a result, the alpha band is taken as interesting data and
selected. Extracting features from a long EEG signal is not
beneficial to analyse profound results. Thus, it is important
to apply framing to the EEG signal prior to feature extraction
when the EEG signal is too long. In general, 2 s and 50 %
overlap are a common choice for framing an EEG signal.
B. Feature Extraction
The past research has demonstrated that frontal EEG
asymmetry has high correlation with emotion regulation and
TABLE I
SUMMARY OF DIFFERENT FEATURES
Feature Genre Feature
EEG Features Asymmetry Frontal Asymmetry
Band Power Alpha Power of Frontal
Alpha Power of Prefrontal
Music Features
Pitch Features Fundamental Frequency
Timbre Features
Zero Crossing Rate
Brightness
Centroid
Skewness
Tonality Features Key
Mode
emotion state [
18
]. In addition, the power of the frontal lobe
and the power of the prefrontal cortex are also related to
emotion [
19
], [
20
]. We extract the above mentioned three EEG
features and use
FEasym
,
FEf ront
, and
FEpref
to represent
the frontal EEG asymmetry, the power of the frontal lobe and
the power of the prefrontal cortex, respectively. In the EEG
database, the frontal electrodes include Fp1, Fp2, F7, F3, F4,
and F8, and Fp1 and Fp2 are the prefrontal electrodes at the
same time.
Music, on the other hand, can be represented by a series of
musical features. In this study, we extract pitch features, tim-
bre features, and tonality features. We also apply windowing
to the music signal and the length is equal to the one used for
the EEG signal. For pitch features, the fundamental frequency,
being set as
FMf 0
, carries significant emotional information
of the music. In terms of timbre features, mean zero crossing
rate, mean brightness, mean centroid and mean skewness
features are extracted, and we use
FMz cr
,
FMbr i
,
FMcent
,
and
FMsk ew
to name these. Accordingly, for tonality features,
key and mode, named
FMk ey
and
FMmo
, including major and
minor are also important to express music emotion and are
taken as additional features. From music theory, major music
usually shows positive impact and minor music expresses
sad emotion. In total, we extract 10 dimensions of features.
Table I gives a summary of the extracted features.
C. Feature Selection and Dimension Reduction
Too many features might lead to high computational
complexity because many coefficients need to be optimised
when build the predictive model. Hence, we select some
features from the original feature set as feature subset. For
EEG features, we select the one feature which has the
strongest correlation from the three features according to
the average correlation between every EEG feature and music
features. Similarly, for music features, we select the top 2
features through the average correlation between every music
feature and EEG features. Therefore, we obtain a subset
including
FEasym
,
FMz cr
, and
FMbr i
. We choose to fuse
the features into one feature:
X= (a1x2
1+a2x1+a3) + ... + (a7x2
3+a8x3+a9) (1)
where
X
represents the fused feature,
x
represents the
feature in the feature subset,
a
represents the coefficient
needed to be found.
Emotion Prediction
EEG
Music Music Features
EEG Features
Emotion Response
Grey Wolf
Optimization Algorithm
Easym
F
Mzcr
F
Mbri
F
Feature ExtractionData Acquisition
.
.
.
Correlation Analysis
Feature Selection
Easym
F
Mzcr
F
Mbri
F
Selected Features
Feature Fusion
Easym
F
Mzcr
F
Mbri
F
2
1 1 2 1 3
X ( )a x a x a
2
7 3 8 3 9
+( )a x a x a
2
4 2 5 2 6
( )a x a x a
1
( 1) ( )S t S t A D
2
( 1) ( )S t S t A D
3
( 1) ( )S t S t A D
Fig. 1. The proposed framework to predict emotion response to music
TABLE II
CORRELATION BETWEEN EEG FEATUR ES A ND M US IC F EATU RE S
FMf 0FM zcr FM br i FMcent FM skew FM key FM mo
FEasym 0.094 0.148 0.113 0.082 -0.082 0.073 0.064
FEf ront -0.074 -0.182 -0.223* -0.147 0.128 0.043 0.012
FEpref -0.072 -0.178 -0.232* -0.160 0.151 0.050 0.036
* represents the significant value p<0.05.
D. GWO Algorithm Set
Evolutionary algorithms (EAs) imitate the regulation by
survival of the fittest in the wild and are considered to
solve our optimisation problems. With the development of
intelligent computation theory, the past few years have seen
the increasing power of EAs in solving optimisation problems
[21]. The fitness of our GWO is set as follows:
min PN
i=1|y−y|
N(2)
y=b1X2+b2X+b3(3)
where
y
is the actual value of the emotion response and
y
is the predicted value.
N
represents the grey wolf population
size in the GWO and
b
is the coefficient needed to be found.
In GWO, the position of the three leading wolves plays
an important role in finding the optimal solution. Hence, the
grey wolf position updates in time to reach an objective and
we set the position updating strategy using the equations as
follows:
Sα(t+ 1) = Sα(t)−A1·Dα(4)
Sβ(t+ 1) = Sβ(t)−A2·Dβ(5)
Sδ(t+ 1) = Sδ(t)−A3·Dδ(6)
where
Sα(t+ 1)
,
Sβ(t+ 1)
, and
Sδ(t+ 1)
represent the
position of three different grey wolves including the alpha,
beta, and delta next iteration, respectively.
t
indicates the
current iteration.
A
is a coefficient matrix and
D
is a vector.
The framework is shown in Fig. 1.
III. EXP ER IME NTA L RES ULTS
A. Experimental Setup
For GWO, the experimental parameters as follows: the
maximum number of iterations is 3 500 and the grey wolf
population size
N
is set to 100. We split the data into 70 %
training data and 30 % test data.
B. Results
The correlation value between the EEG features and music
features is listed in Table II. As can be seen in Table II, the
results show that
FMbr i
is significantly related to
FEf ront
and
FEpref
with the correlation values being -0.223 and -0.232
(
p <
0.05 by
t
-test) while others are not having significant
relation, and one can see that as
FMbr i
increase,
FEf ront
and FEpref decrease.
Meanwhile, Table III shows the results of the comparison of
different methods between predicted values and actual values.
The absolute value of the presented results are better when
bigger. Specifically, the proposed method is compared with
the method in [
15
], and the results indicate that the proposed
method is able to predict the response emotion value with
r
being 0.381 with significant correlations (
p <
0.05), which
exceeds the value of 0.207 in the literature [
15
] when using
FMbr i
and
FEasym
. Similarly, in another case,
r
is 0.520
for the proposed method compared with the absolute value
of [
15
] being 0.408 (the value of -0.408), which demonstrates
the proposed method shows better performance in predicting
the response to music. Besides, the reached correlation of
0.520 is the best result among the presented results when
using all selected features.
IV. DISCUSSION
For the above presented experimental results, we can firstly
find that there is correlation between some of music features
and EEG features. In particular, the brightness feature is
TABLE III
COMPARISON FOR THE CORRELATION VALUE rOF DIFF ER EN T ME TH OD S BE TW EE N PR ED IC TE D AN D ACT UA L VALUE S
FMz cr +FEasym FM bri +FE asym FM zcr +FM bri +FE asym
Proposed Method -0.169 0.381* 0.520*
Method in [15] -0.408** 0.207* -0.408**
* represents the significant value p<0.05; ** represents the significant value p<0.001.
significantly related to both of the frontal lobe power and
the prefrontal power. These findings not only can help us
explore the complex relationship between EEG features and
music features, but also provide the basis when considering
how to modulate emotion during music therapy. For practical
usage in the future, as a result, they might assist in music
therapy for brain functional disorders. For example, we can
develop an intelligent music therapy by leveraging artificial
intelligence (AI) technology, e.g., computer audition [
22
],
wearable devices, and music-based technology in terms of
treating depression and more general mental disorder. In
another case, as the ageing population has become more and
more prevalent [
23
], designing human-centred AI combining
music and AI is an interesting research filed. Secondly,
the experimental results have indicated the validity of the
proposed method. It is noteworthy that the first result is the
same as the third result in [
15
] of Table III, which indicates
that the method of [
15
] is unable to effectively utilise all
selected features to predict emotion response.
V. CONCLUSION
We proposed a method for predicting a listener’s emotion
via EEG and music features. The experimental results show
promising performance of the proposed method. Apart from
prediction, the discovery of a relationship between EEG data
and music is encouraging. In future work, we attempt to
analyse EEG data of other emotions and apply functional
brain network feature extracting approaches for a higher and
more complex representation from EEG data and explore
the deeper relationship between them and the music. Last
but not least, we will also attempt to design a simpler and
cheaper method to predict the emotion induced by audio,
which includes affective music and environmental sound,
e. g., bird sound.
REFERENCES
[1]
R. Panda, R. M. Malheiro, and R. P. Paiva, “Audio features for
music emotion recognition: A survey,” IEEE Transactions on Affective
Computing, vol. 14, no. 1, pp. 68–88, 2023.
[2]
T. Nakamura, “The communication of dynamics between musicians and
listeners through musical performance,” Perception & Psychophysics,
vol. 41, no. 6, pp. 525–533, 1987.
[3]
J. Kim and M. Y. Kang, “Sustainable success in the music industry:
Empirical analysis of music preferences,” Journal of Business Research,
vol. 142, pp. 1068–1076, 2022.
[4]
K. Qian, W. S. Bj
¨
orn, X. H. Guan, and H. Bin, “Intelligent music
intervention for mental disorders: Insights and perspectives,” IEEE
Transactions on Computational Social Systems, vol. 10, no. 1, pp. 2–9,
2023.
[5]
J. Wang, H. Wang, D. Zhang et al., “Impact of group music therapy
on the depression mood of college students,” Health, vol. 3, no. 3, pp.
151–155, 2011.
[6]
W.-C. Hsu and H.-L. Lai, “Effects of music on major depression in
psychiatric inpatients,” Archives of Psychiatric Nursing, vol. 18, no. 5,
pp. 193–199, 2004.
[7]
S. Sun, X. Li, J. Zhu, Y. Wang, R. La, X. Zhang, L. Wei, and B. Hu,
“Graph theory analysis of functional connectivity in major depression
disorder with high-density resting state eeg data,” IEEE Transactions
on Neural Systems and Rehabilitation Engineering, vol. 27, no. 3, pp.
429–439, 2019.
[8]
W. Qian, J. Tan, Y. Jiang, and Y. Tian, “Deep learning with con-
volutional neural networks for eeg-based music emotion decoding
and visualization,” Brain-Apparatus Communication: A Journal of
Bacomics, vol. 1, no. 1, pp. 38–49, 2022.
[9]
E. Dutta, A. Bothra, T. Chaspari, T. Ioerger, and B. J. Mortazavi,
“Reinforcement learning using eeg signals for therapeutic use of music
in emotion management,” in Processdings of IEEE Engineering in
Medicine and Biology Society (EMBC). IEEE, 2020, pp. 5553–5556.
[10]
S. Mirjalili, S. M. Mirjalili, and A. Lewis, “Grey wolf optimizer,”
Advances in Engineering Software, vol. 69, pp. 46–61, 2014.
[11]
H. Faris, I. Aljarah, M. A. Al-Betar, and S. Mirjalili, “Grey wolf
optimizer: A review of recent variants and applications,” Neural
Computing and Applications, vol. 30, no. 2, pp. 413–435, 2018.
[12]
I. Daly, N. Nicolaou, D. Williams, F. Hwang, A. Kirke, E. Miranda,
and S. J. Nasuto, “Neural and physiological data from participants
listening to affective music,” Scientific Data, vol. 7, no. 1, pp. 1–7,
2020.
[13]
I. Daly, A. Malik, F. Hwang, E. Roesch, J. Weaver, A. Kirke,
D. Williams, E. Miranda, and S. J. Nasuto, “Neural correlates of
emotional responses to music: An eeg study,” Neuroscience Letters,
vol. 573, pp. 52–57, 2014.
[14]
I. Daly, J. Hallowell, F. Hwang, A. Kirke, A. Malik, E. Roesch,
J. Weaver, D. Williams, E. Miranda, and S. J. Nasuto, “Changes in
music tempo entrain movement related brain activity,” in Processdings
of IEEE Engineering in Medicine and Biology Society (EMBC). IEEE,
2014, pp. 4595–4598.
[15]
I. Daly, D. Williams, J. Hallowell, F. Hwang, A. Kirke, A. Malik,
J. Weaver, E. Miranda, and S. J. Nasuto, “Music-induced emotions
can be predicted from a combination of brain activity and acoustic
features,” Brain and Cognition, vol. 101, pp. 1–11, 2015.
[16]
P. Comon, “Independent component analysis, a new concept?” Signal
Processing, vol. 36, no. 3, pp. 287–314, 1994.
[17]
V. Bajaj and R. B. Pachori, “Detection of human emotions using
features based on the multiwavelet transform of eeg signals,” in Brain-
Computer Interfaces. Springer, 2015, pp. 215–240.
[18]
J. A. Coan and J. J. Allen, “Frontal eeg asymmetry as a moderator
and mediator of emotion,” Biological Psychology, vol. 67, no. 1-2, pp.
7–50, 2004.
[19]
E. H. Houssein, A. Hammad, and A. A. Ali, “Human emotion recogni-
tion from eeg-based brain–computer interface using machine learning:
A comprehensive review,” Neural Computing and Applications, vol. 34,
no. 15, pp. 12 527–12 557, 2022.
[20]
R. Ramirez-Melendez and X. Reija, “The creative drummer: An eeg-
based pilot study on the correlates of emotions and creative drum
playing,” Brain Sciences, vol. 13, no. 1, pp. 88–99, 2023.
[21]
C. He, M. Li, C. Zhang, H. Chen, X. Li, and J. Li, “A competitive swarm
optimizer with probabilistic criteria for many-objective optimization
problems,” Complex & Intelligent Systems, vol. 8, no. 6, pp. 4697–4725,
2022.
[22]
K. Qian, X. Li, H. Li, S. Li, W. Li, Z. Ning, S. Yu, L. Hou, G. Tang,
J. Lu et al., “Computer audition for healthcare: Opportunities and
challenges,” Frontiers in Digital Health, vol. 2, 2020.
[23]
K. Qian, Z. Zhang, Y. Yamamoto, and B. W. Schuller, “Artificial
intelligence internet of things for the elderly: From assisted living to
health-care monitoring,” IEEE Signal Processing Magazine, vol. 38,
no. 4, pp. 78–88, 2021.
