Content uploaded by Connor Tom Keating
All content in this area was uploaded by Connor Tom Keating on Sep 07, 2022
Content may be subject to copyright.
It’s all in the mind: linking internal representations of emotion with facial
Connor T. Keating1 & Jennifer L Cook1
1University of Birmingham
University/Lab Homepage: https://jencooklab.com
Twitter Handle: @ConnorTKeating
Keywords: Emotion recognition; Emotion Representations
Background and aims of the research
Over the past six decades, researchers have extensively studied emotion recognition
(e.g., Ekman & Friesen, 1975; Bassili, 1979; Young et al., 1997; Goldman & Sripada, 2005;
Zheng et al., 2017; Sowden, Schuster, Keating, Fraser & Cook, 2021). Despite being highly
related to (and potentially important for) emotion recognition, it was only recently that
researchers began investigating internal representations of emotion (e.g., Jack, Garrod, Yu,
Caldara & Schyns, 2012; Jack, Garrod & Schyns, 2014; Jack, Sun, Delis, Garrod & Schyns,
2016; Chen, Garrod, Ince, Schyns & Jack, 2021). Such studies have typically adopted
psychophysical approaches to index the way in which facial expressions appear in the
“mind’s eye” (i.e., internal representations) and compared these across emotions (e.g., Jack,
Garrod & Schyns, 2014; Chen et al., 2018), cultures (e.g., Jack, Caldara & Schuns, 2012;
Jack, Sun, Delis, Garrod & Schyns, 2016), and participant groups (e.g., Pichon et al., 2020).
Despite great progress in these areas, research has not yet investigated the extent to which
internal representations influence emotion recognition. For example, studies have not
explored whether the precision/clarity of internal representations contributes to emotion
recognition abilities and/or difficulties. In our recent study, we tested the hypothesis that
individuals with less clear internal representations of emotion would have low scores on an
emotion recognition task.
To test this hypothesis, participants completed two tasks which employed dynamic
point light displays (a series of dots that convey biological motion) of angry, happy and sad
facial expressions (PLFs). In the first task (taken from Sowden, Schuster, Keating, Fraser &
Cook, 2021; Keating, Fraser Sowden & Cook, 2021), participants viewed emotional PLFs
and rated how angry, happy and sad the facial expressions appeared. We calculated emotion
recognition accuracy scores by subtracting the mean of the two incorrect ratings from the
correct rating. For example, for a trial that displayed an angry expression, the mean rating of
the two incorrect emotions (happy and sad) was subtracted from the rating for the correct
The second task was an adapted version of a task we had employed previously
(Keating, Sowden & Cook, under revision). In this task, on each trial, participants moved a
dial to manipulate the speed of a PLF until it moved at the speed of a typical angry, happy or
sad expression. This task operates on the premise that, compared to participants with clear
internal representations, those with less clear representations of emotion would attribute more
variable speeds to the expressions. For instance, someone with a clear internal representation
anger would be consistent in their attributions (e.g., by attributing 120% speed, 121% speed
and 119% speed to the angry expression). In contrast, someone with a less clear internal
representation would be more variable (e.g., by attributing 120% speed, 60% speed and 180%
speed to an angry expression). Therefore, to index the clarity (or lack thereof) of participants’
internal representations, we calculated variability by taking the standard deviation of the
speeds attributed to the angry, happy and sad expressions respectively. Mean variability was
calculated by taking a mean of the variability scores for the angry, happy and sad PLFs.
Our preliminary results
Our preliminary results suggest that people that have less clear internal
representations of emotion find it more difficult to recognise emotional facial expressions.
However, further work needs to be done to replicate these findings and, to determine the
direction of causality. In the case of the latter- it could be that those with less clear internal
representations of facial expressions do not have consistent “templates” to compare observed
expressions to, thus resulting in poorer emotion recognition. Alternatively, it could be that
those who struggle to read emotional expressions do not build up clear internal
representations as they do not know the correct “label” or give an incorrect “label” to
expressions they observe. In addition, further work needs to be done to a) identify how other
emotional processes are implicated in emotion recognition (e.g., the interoceptive experience
of emotion) and b) identify how different traits (e.g., autistic and alexithymic) are implicated
in these different emotional sub-abilities.
In our next experiment, we aim to test how features of internal emotional experiences,
such as the consistency and overlap between emotions, contribute to internal representations
and facial emotion recognition. By doing so, we hope to construct a mechanistic model of
emotion recognition that elucidates how different emotional sub-abilities and traits are
associated with one another. We hope that such work will illuminate potential pathways for
supporting emotion recognition in clinical and sub-clinical groups (see Keating & Cook,
We thank the British Psychological Society Cognitive Section for awarding us the
grant. The funds were used to offset the costs of online testing (e.g., recruiting participants
Bassili, J. N. (1979). Emotion recognition: The role of facial movement and the relative
importance of upper and lower areas of the face. Journal of Personality and Social
Psychology, 37(11), 2049–2058. https://doi.org/10.1037/0022-35188.8.131.529
Chen, C., Crivelli, C., Garrod, O. G., Schyns, P. G., Fernández-Dols, J. M., & Jack, R. E.
(2018). Distinct facial expressions represent pain and pleasure across
cultures. Proceedings of the National Academy of Sciences, 115(43), E10013-E10021.
Chen, C., Garrod, O. G., Ince, R. A., Schyns, P. G., & Jack, R. E. (2021). Facial Expressions
Reveal Cross-Cultural Variance in Emotion Signaling. Journal of Vision, 21(9), 2500-
Ekman, P., & Friesen, W. V. (1975). Unmasking the face: A guide to recognizing emotions
from facial clues. Prentice Hall.
Goldman, A. I., & Sripada, C. S. (2005). Simulationist models of face-based emotion
recognition. Cognition, 94(3), 193-213.
Jack, R. E., Caldara, R., & Schyns, P. G. (2012). Internal representations reveal cultural
diversity in expectations of facial expressions of emotion. Journal of Experimental
Psychology: General, 141(1), 19-25. https://doi.org/10.1037/a0023463
Jack, R. E., Garrod, O. G., & Schyns, P. G. (2014). Dynamic facial expressions of emotion
transmit an evolving hierarchy of signals over time. Current biology, 24(2), 187-192.
Jack, R. E., Garrod, O. G., Yu, H., Caldara, R., & Schyns, P. G. (2012). Facial expressions of
emotion are not culturally universal. Proceedings of the National Academy of
Sciences, 109(19), 7241-7244. https://doi.org/10.1073/pnas.1200155109
Jack, R. E., Sun, W., Delis, I., Garrod, O. G., & Schyns, P. G. (2016). Four not six: Revealing
culturally common facial expressions of emotion. Journal of Experimental Psychology:
General, 145(6), 708. https://doi.org/10.1037/xge0000162
Keating, C. T., & Cook, J. L. (2020). Facial expression production and recognition in autism
spectrum disorders: A shifting landscape. Child and Adolescent Psychiatric
Clinics, 29(3), 557-571. https://doi.org/10.1016/j.chc.2020.02.006
Keating, C. T., Fraser, D. S., Sowden, S., & Cook, J. L. Differences Between Autistic and
Non-Autistic Adults in the Recognition of Anger from Facial Motion Remain after
Controlling for Alexithymia. Journal of Autism and Developmental Disabilities, 1-17.
Advance online publication. https://doi.org/10.1007/s10803-021-05083-9
Keating, C. T., Sowden, S., & Cook, J. L. (2022). Comparing internal representations of
facial expression kinematics between autistic and non‐autistic adults. Autism
Research, 15(3), 493-506. https://doi.org/10.1002/aur.2642
Pichon, S., Bediou, B., Antico, L., Jack, R., Garrod, O., Sims, C., Green, C. S., Schyns, P., &
Bavelier, D. (2020). Emotion perception in habitual players of action video
games. Emotion. https://doi.org/10.1037/emo0000740
Sowden, S., Schuster, B. A., Keating, C. T., Fraser, D. S., & Cook, J. L. (2021). The role of
movement kinematics in facial emotion expression production and
recognition. Emotion. http://dx.doi.org/10.1037/emo0000835
Young, A. W., Rowland, D., Calder, A. J., Etcoff, N. L., Seth, A., & Perrett, D. I. (1997).
Facial expression megamix: Tests of dimensional and category accounts of emotion
recognition. Cognition, 63(3), 271-313.
Zheng, W. L., Zhu, J. Y., & Lu, B. L. (2017). Identifying stable patterns over time for
emotion recognition from EEG. IEEE Transactions on Affective Computing, 10(3),