Available via license: CC BY 4.0
Content may be subject to copyright.
Page 1/13
Eyes Compensate Smile When Wearing Mask
Shuntaro Okazaki ( shuntaro.okazaki@shiseido.com )
MIRAI Technology Institute, Shiseido co. Ltd
Haruna Yamanami
MIRAI Technology Institute, Shiseido co. Ltd
Fumika Nakagawa
MIRAI Technology Institute, Shiseido co. Ltd
Nozomi Takuwa
MIRAI Technology Institute, Shiseido co. Ltd
Keith James Duncan Kawabata
MIRAI Technology Institute, Shiseido co. Ltd
Research Article
Keywords: Facial muscle, Electromyography, Facial expression, Covid-19, Social communication
DOI: https://doi.org/10.21203/rs.3.rs-576144/v1
License: This work is licensed under a Creative Commons Attribution 4.0 International License.
Read Full License
Page 2/13
Abstract
The use of face masks has become ubiquitous. Although mask wearing is a convenient way to reduce
the spread of disease, it is important to know how the mask affects our communication via facial
expression. For example, when we are wearing the mask and meet a friend, are our facial expressions
different compared to when we are not? We investigated the effect of face mask wearing on facial
expression, including the area around the eyes. We measured surface electromyography from
zygomaticus major, orbicularis oculi, and depressor anguli oris, when people smiled and talked with or
without the mask. We found that only orbicularis oculi were facilitated by wearing the mask. We thus
concluded that mask wearing increases the use of eye smiling as a form of communication. In other
words, we can express joy and happiness even when wearing the mask using eye smiling.
Introduction
How does the use of a face mask inuence our daily communication? Although a part of Japanese
culture since the Meiji Era (1868–1912) 1, face mask usage has only recently become ubiquitous across
the world during the outbreak of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), the virus
that causes coronavirus disease 2019 (COVID-19). Unsurprisingly, interest has grown in the physiological
and psychological effects of mask wearing 2, especially as concern regarding interference of
communication may contribute to resistance to mask wearing.
The psychological impacts of the face mask on social communication are highly relevant. Facial
expression is the one of the most ecient channels to communicate our feelings to others 3. Among
these expressions, smiling is a fundamental facial expression to communicate positive affect and
strongly inuences the attention, brain activity, and various behaviors of others 4. Smiling is achieved by
mouth smiling and eye smiling, where the eye smiling was believed to be relevant most when people feel
enjoyment 5,6. This type of smile has been called a Duchenne smile 5,7. A Duchenne smile has a stronger
effect than a non-Duchenne smile (i.e. without eye smiling) on the evaluation of happiness 8 and
desirable impression 9,10. The importance of eye smiling is shown by the nding that botulinum toxin
application on the eye muscle diminishes eye smiling and decreases the perceived happiness of the
person who received the treatment 11. Recent studies have argued that facial mask wearing interferes
with the recognition of the wearer’s emotion and impression 12,13. To our knowledge, however, there is no
investigations into the effect of the mask wearer expressing the emotion. Understanding how the mask
wearer expresses their emotion despite the mask is important for as it may provide insights into how
mask interference of emotion communication can be overcome.
As the upper face in emotional expression is increased in signicance when the lower face is hidden by
the mask 3,14, we hypothesized that wearing a mask will enhance eye smiling as the mask wearer
attempts to communicate their smile to others. It was widely believed that the Duchenne smile provides a
reliable and spontaneous sign of enjoyment because most people are not able to contract orbicularis
Page 3/13
oculi voluntarily 15. However, there is evidence that Duchenne smiles can be voluntarily reproduced 15–17,
regardless of whether joy is felt. These ndings suggest that when a person wearing a mask wants to
communicate a smile to others, they may try to compensate for the loss of mouth smile information by
exaggerating their eye smiling.
To investigate the effect of wearing a mask on facial expression, we used a smiling and speaking task
with concurrent facial EMG recording of orbicularis oculi, zygomaticus major activity, and depressor
anguli oris. These muscles were chosen as orbicularis oculi is used for eye smilling, zygomaticus major
for mouth smiling and depressor anguli oris is used during speech 18. There are 4 possibilities. First, if our
hypothesis is correct and the participants try to overcome the mask intereference using eye smiling, only
orbicularis oculi activity will be enhanced when smiling with the mask. Second, if the mask disturbs facial
expression visually, but the individual control of the smile related muscles is dicult, both orbicularis
oculi and zygomaticus major will be enhanced when smiling with the mask. Third, if the mask disturbs
general facial motion physically, the mask wearer may universally exaggerate their facial motions,
enhancing both smiling (orbicularis oculi, zygomaticus major) and speaking (depressor anguli oris) to
resist the diculty in the motion. Finally, if the mask does not inuence our facial motion, there will be no
change in any muscles.
Materials And Methods
2.1 Participants
Twenty females have received remuneration for participation in this experiment (Age: 39.8 ± 11.9 years
old), after providing written informed consent. They reported normal or corrected-to-normal vision and no
history of psychiatric and neurological disorders. This study was performed in accordance with the
principles in the Declaration of Helsinki and approved by the Ethical committee at the Shiseido Global
Innovation Center (Approval number: C02121).
2.2 Stimuli and procedure
The participants comfortably sat on a chair and were asked to perform 4 behavioural tasks according to
the instructions that appeared on the PC screen in front of them (Fig.1). The rst task was the "photo"
task where the participants watched a movie. An experimenter appeared in the movie and pretended to
take a picture of the participant, who was asked to smile. The second was the "smiling" task. The
participants made their biggest smile that they can immediately after a beep was heard from the PC.
Participants were instructed how to make the biggest smile (pull the mouth corner back maximally) and
the participants decided whether to open their mouth or not themselves before the experiment. During the
experiment, they replicated this type of smile. The third was the "speech” task. The participants spoke the
designated 4 sentences (from an ATR 503 phonetically balanced sentences) on the screen. The fourth
task was the "free talk" task. The participants talked about 2 themes (“What I'm addicted to these days”
Page 4/13
and “What I have enjoyed recently”) as long as they could up to a maximum of 30 seconds. They were
also asked to pretend to talk with a person familiar to them in this task.
Photo and smiling tasks were both conducted twice (once at the beginning and once at the end of the
experiment) for each mask condition. All tasks were implemented with or without a surgical mask on their
face (Fig.1). The order of mask condition (with or without mask) was counterbalanced for the repetition
(once at the beginning and once at the end of the experiment) and across participants. After the
experiment was started, the experimenter left the participants alone in the experimental room as an anti-
coronavirus infection risk countermeasure and let them initiate the start of the experiment by themselves
which they did by pushing the spacebar of the PC. During this time, the participant was recorded on
video. Detailed procedures of the behavioural tasks and timing of wearing and taking off the mask were
instructed by the in-house developed Python 3.7.3 (Python software foundation,
https://www.python.org/) program made with the Psychopy library 19.
2.3 Data acquisition and analysis
Three active bipolar Ag electrodes (AP-C140-020, Miyuki Giken Co., Ltd., Tokyo, Japan) were attached on
the face to acquire the electromyography (EMG) from orbicularis oculi, zygomaticus major of the left
facial sites, and depressor anguli oris of the right facial sites. The detailed positioning was decided in
accordance with published guidelines 20. Reference and Grand electrodes (AP-C151-015 and MA-C004-
015 with general disposal electrode, respectively) were both attached on the forehead and covered with a
hairband. To lower the electrode conductance, we applied Ten20 conductive paste (Weaver and Company,
Aurora, CO, USA) on the electrode tips and attached a pierced double-sided tape between the body of the
electrode and skin. They were also covered with elastic tape.
EMG signals were recorded by a 32-ch physiological signal recording device (Polymate V AP5148, Miyuki
Giken Co., Ltd., Tokyo, Japan) with a sampling rate of 1000 Hz. 50-Hz noise in the EMG was excluded by
a notch lter (48–52 Hz). Then the spiky outlier data over 200 mV was removed. The denoised signal was
rectied and integrated with each 20-ms time window 20 to monitor its time series (Fig.2–5). The
preprocessed EMG signals were divided into epochs of each behavioural task based on the audio signals
(beep for the smiling task and voice onset for the photo, speech, and free talk tasks). Due to the self-
proceeding experimental procedure, the epochs where the participants failed the procedures were
manually checked on the recorded video and excluded from further analyses. This process was done by a
rater who took no further role in the data analysis. These failures included failure to comply with the task
(e.g. due to failure to hear instructions), misspeaking or misreading. In the photo and smiling tasks, if
both trials were rated as successful, the signals from the two trials were averaged. If only one trial was
rated as a success and one as a fail, only the successful trial was used in further analysis. The EMG
signal was averaged during the tasks and compared between conditions with or without the mask with
Wilcoxon signed-rank test with a Bonferroni correction (N = 12 for tasks and measured muscles).
Results
Page 5/13
Figure 2 shows the group averaged EMG signals recorded on the 3 measured muscles (orbicularis oculi,
Zygomaticus major, and Depressor anguli oris) during the photo task. The data in 2 out of 20 participants
were excluded from statistical analysis (see Materials & methods). EMG of the orbicularis oculi was
signicantly enhanced by wearing the mask during the photo task (V = 3, n = 18, corrected p < .001, r =
.847). The effects of wearing the mask on the other muscles were not statistically signicant (all
corrected p > .250). Figure 3-5 shows the averaged EMGs for the smiling (the data in 2 out of 20
participants were excluded), speech, and free talk tasks (the data in 1 out of 20 participants were
excluded) respectively. The effects of wearing the mask on these tasks were not statistically signicant
(all corrected p > .482).
Discussion
During the coronavirus pandemic, wearing a mask has become common and is obligatory in some
places. As such, there has been increased interest in the various effects of mask-wearing, including the
possible interference effect the mask has on the communication of emotion via facial expression.
Concerns regarding possible interference may have exacerbated the resistance to mask-wearing seen in a
number of countries. Overcoming the interference during wearing the mask is thus an important goal.
Although mask-wearing blocks the transmission of emotional information from the lower face, we thus
hypothesized that wearing a face mask increases eye smiling, compensating for the hidden mouth
smiling, allowing communication of smiling.
We tested this hypothesis by measuring face muscle activity using EMG. Specically, we looked at the
pattern of change in the activity of orbicularis oculi, zygomaticus major, and depressor anguli oris while
smiling during mask-wearing. We found that eye smiling (as measured by orbicularis oculi activity) was
enhanced when smiling for a photograph during mask-wearing compared to no mask (Fig.2). In contrast,
the muscles related to the mouth smiling (zygomaticus major) and speaking (depressor anguli oris)
showed no change when the participants smile (Fig.2, 3) and speak with wearing the mask (Fig.4, 5). In
other words, when smiling for a photo while wearing a mask, participants increased the use of eye
smiling without increasing mouth smiling, consistent with our hypothesis and highlighting the
adaptability of human communication.
Recent studies have demonstrated that the emotion of the person wearing a facial mask is less
accurately recognized 12,13. These ndings focus on how is the mask wearer is perceived, while our study
focuses on how the mask wearer modies their behaviour during communication while wearing a mask.
The eye smiling was previously thought to be engaged when a person really feels enjoyment 5,6 and the
related muscles (orbicularis oculi) thought to be dicult to voluntarily control 7. However, recent studies
have argued that eye smiling can be intentionally controlled even in the absence of enjoyment. As
wearing a mask is unlikely to increase enjoyment, the increase in eye smiling in the current study also
support the existence of voluntary control of eye smiling. Note that we could nd a signicant difference
in the EMG activity around the eye during the photo task (Fig.2) but not during the smiling task (Fig.3).
Krumhuber and colleagues have argued that the occurrence of Duchenne smile (smile with eye smiling) is
Page 6/13
associated with the spontaneity of the expression 15. The different results of eye smiling in the photo and
smiling tasks may be explained by a difference in spontaneity between the tasks. Smiling as a task in
response to a beep stimulus with no person watching is less spontaneous than the photo task.
There are two limitations of this study. The rst is that the participants in this study were all Japanese. As
noted before, mask culture in Japan has survived for a very long time 1. It may be the case that through
long exposure to the existing mask culture, our Japanese participants have developed more control over
eye smiling that may not be the situation in countries where there is no mask culture. In addition,
increased control over eye smiling may also result from the cultural tendency in Japan (and other East
Asian countries) to derive emotional information more from the eyes than the mouth, which is the
opposite of Western cultures 21–23. This interesting point can be addressed in a future cross-cultural
study.
The second limitation is that we did not measure the effect of the enhanced eye smiling during the mask
condition of a receiver of the emotional information. It thus remains unclear to what extent the
enhancement of eye smiling seen in this study compensates for the hidden mouth smile. Previous
studies suggest that masks do impair recognition of emotion 12,13 but this interference effect may be
overestimated because these previous studies used photographs where the facial expressions were
masked using a digitally added mask. Therefore, there would have been no eye smiling compensation in
the mask stimuli. It may be the case that the enhanced eye smiling seen in the current study is enough to
compensate, especially if occurring together with the enhancement in gestures and prosody that might be
expected in a natural setting.
Conclusion
Smiling motion around the eyes can be enhanced when people communicate their smile while wearing a
face mask compared to without a mask. Together with the other modes of communication, such as
gestures and prosody 14 that occur in natural settings, we suggest that it possible for humans to maintain
communication of emotion even while wearing a mask and that the concern about mask interference of
emotion communication may be unnecessary.
Declarations
Conict of Interest
The authors report no conicts of interest.
Author Contributions
Page 7/13
S.O., F.N., N.T., and H.Y. conceived and conducted the experiments. S.O. analysed the results. S.O., H.Y.,
and K.J.D.K. wrote the paper. All authors reviewed the manuscript.
Funding
Details of all funding sources should be provided, including grant numbers if applicable. Please ensure to
add all necessary funding information, as after publication this is no longer possible.
Data availability
Due to the condentiality agreements with the participants, the data in this study are available only at
Shiseido co. Ltd.
Acknowledgements
We thank Keiko Tagai for supporting and coordinating this experiment.
References
1. Martin, A. K. T. The history behind Japan’s love of face masks.
The Japan times
https://www.japantimes.co.jp/news/2020/07/04/national/science-health/japans-history-wearing-
masks-coronavirus/ (2020).
2. Scheid, J. L., Lupien, S. P., Ford, G. S. & West, S. L. Commentary: Physiological and psychological
impact of face mask usage during the covid-19 pandemic.
Int. J. Environ. Res. Public Health
17, 1–
12 (2020).
3. Mheidly, N., Fares, M. Y., Zalzale, H. & Fares, J. Effect of Face Masks on Interpersonal
Communication During the COVID-19 Pandemic.
Front. Public Heal.
8, (2020).
4. Martin, J., Rychlowska, M., Wood, A. & Niedenthal, P. Smiles as Multipurpose Social Signals.
Trends
in Cognitive Sciences
vol.21 864–877 (2017).
5. Ekman, P., Davidson, R. J. & Friesen, W. V. The Duchenne Smile: Emotional Expression and Brain
Physiology II.
J. Pers. Soc. Psychol.
58, 342–353 (1990).
. Ekman, P., Friesen, W. V. & O’Sullivan, M. Smiles when lying.
J. Pers. Soc. Psychol.
54, 414–420
(1988).
7. Duchenne, G. B. & de Boulogne, G. B. D.
The mechanism of human facial expression.
(Cambridge
university press, 1990).
. Gosselin, P., Perron, M., Legault, M. & Campanella, P. Children’s and adults’ knowledge of the
distinction between enjoyment and non enjoyment smiles.
J. Nonverbal Behav.
26, 83–108 (2002).
Page 8/13
9. Frank, M. G. & Ekman, P. (1993). Not all smiles are created equal: the differences between enjoyment
and non enjoyment smiles.
Humor Int. J. Humor Res.
6, 9–26 (1993).
10. Woodzicka, J. A. Sex differences in self-awareness of smiling during a mock job interview.
J.
Nonverbal Behav.
32, 109–121 (2008).
11. Etcoff, N., Stock, S., Krumhuber, E. G. & Reed, L. I. A Novel Test of the Duchenne Marker: Smiles After
Botulinum Toxin Treatment for Crow’s Feet Wrinkles.
Front. Psychol.
11, 612654 (2021).
12. Marini, M., Ansani, A., Paglieri, F., Caruana, F. & Viola, M. The impact of facemasks on emotion
recognition, trust attribution and re-identication.
Sci. Rep.
11, 1–14 (2021).
13. Grundmann, F., Epstude, K. & Scheibe, S. Face masks reduce emotion-recognition accuracy and
perceived closeness.
PLoS One
16, e0249792 (2020).
14. Hombach, S. M. Emotion researcher Ursula Hess explains why a facial expression can be detected
when obscured by a face covering.
Scientic American
https://www.scienticamerican.com/article/from-behind-the-coronavirus-mask-an-unseen-smile-can-
still-be-heard/ (2020).
15. Krumhuber, E. G. & Manstead, A. S. R. Can Duchenne Smiles Be Feigned? New Evidence on Felt and
False Smiles.
Emotion
9, 807–820 (2009).
1. Gunnery, S. D., Hall, J. A. & Ruben, M. A. The Deliberate Duchenne Smile: Individual Differences in
Expressive Control.
J. Nonverbal Behav.
37, 29–41 (2013).
17. Schmidt, K. L., Ambadar, Z., Cohn, J. F., Reed, L. I. & Schmidt, K. Movement differences between
deliberate and spontaneous facial expressions: zygomaticus major action in smiling.
J. Nonverbal
Behav.
30, 37–52 (2006).
1. Kurogi, N., Nagai, H. & Nakamura, T. Continuous inaudible recognition of Japanese vowels using
features detected at lips shape peaks based on surface electromyography. in
3rd International
Conference on Systems and Informatics (ICSAI)
959–964 (2016).
19. Peirce, J.
et al.
PsychoPy2: Experiments in behavior made easy.
Behav. Res. Methods
51, 195–203
(2019).
20. Fridlund, A. J. & Cacioppo, J. T. Guidelines for human electromyographic research.
Psychophysiology
23, 567–589 (1986).
21. Yuki, M., Maddux, W. W. & Masuda, T. Are the windows to the soul the same in the East and West?
Cultural differences in using the eyes and mouth as cues to recognize emotions in Japan and the
United States.
J. Exp. Soc. Psychol.
43, 303–311 (2007).
22. Yamamoto, H., Kawahara, M., Kret, M. & Tanaka, A. Cultural Differences in Emoticon Perception:
Japanese See the Eyes and Dutch the Mouth of Emoticons.
Lett. Evol. Behav. Sci.
11, 40–45 (2020).
23. Jack, R. E., Garrod, O. G. B., Yu, H., Caldara, R. & Schyns, P. G. Facial expressions of emotion are not
culturally universal.
Proc. Natl. Acad. Sci. U. S. A.
109, 7241–7244 (2012).
Figures
Page 9/13
Figure 1
Schematic representation of the experimental design.
Page 10/13
Figure 2
EMG signal measured over the orbicularis oculi, zygomaticus major, and depressor anguli oris, during the
photo task. The graphs in the left column indicate the time series of EMG signal of these muscles that
were integrated with each 20-ms time window (sampling rate was 50 Hz). The solid line is averaged EMG
and the coloured area indicates a standard error of the mean (SEM) across participants. Orange and
green lines indicate the EMG signal when the participants wore the mask and when they did not,
Page 11/13
respectively. Right columns indicate comparisons of averaged EMG during a whole task (15 s) when the
participants wore the mask (M) and when they did not (NM). Asterisk indicates a signicant difference in
the averaged EMG between NM and M conditions by Wilcoxon signed-rank test with a Bonferroni
correction.
Figure 3
Page 12/13
EMG signal measured over the orbicularis oculi, zygomaticus major, and depressor anguli oris, during the
smiling task (3 s). Plots and colours are the same as in gure 2.
Figure 4
EMG signal measured over the orbicularis oculi, zygomaticus major, and depressor anguli oris, during the
speech task (5 s). Plots and colours are the same as in gure 2.
Page 13/13
Figure 5
EMG signal measured over the orbicularis oculi, zygomaticus major, and depressor anguli oris, during the
free talk task (30 s). Plots and colours are the same as in gure 2.
