Association of Impaired Facial Affect Recognition with
Basic Facial and Visual Processing Deficits in
Daniel Norton, Ryan McBain, Daphne J. Holt, Dost Ongur, and Yue Chen
Background: Impaired emotion recognition has been reported in schizophrenia, yet the nature of this impairment is not completely
and visual contrast detection tasks, where the emotionality, distinctiveness of identity, or visual contrast was systematically manipulated.
Subjects determined which of two presentations in a trial contained the target: the emotional face for emotion discrimination, a specific
individual for identity discrimination, and a sinusoidal grating for contrast detection.
Results: Patients had significantly higher thresholds (worse performance) than control subjects for discriminating both fearful and happy
faces. Furthermore, patients’ poor performance in fear discrimination was predicted by performance in visual detection and face identity
Conclusions: Schizophrenia patients require greater emotional signal strength to discriminate fearful or happy face images from neutral
ones. Deficient emotion recognition in schizophrenia does not appear to be determined solely by affective processing but is also linked to
the processing of basic visual and facial information.
roscience, perception, schizophrenic
such as those mediated in the amygdala (3–5). It is important,
however, to note that recognition of facial emotion also relies on
perceptual signals and nonaffective face information (6,7). Pa-
tients with schizophrenia are deficient at processing basic visual
information (8,9) and at detecting and recognizing facial images
that are without emotional content (10–13). It has been sug-
gested that impaired emotion recognition in schizophrenia is
secondary to deficient visual processing (14). Also, some re-
searchers have posited that the deficit in emotion recognition is
secondary to a generalized deficit associated with schizophrenia
(15). Yet, performance in emotion recognition and in basic
perception tasks has not been directly compared in schizophre-
nia patients. This comparison would provide a direct test of the
hypothesis that impaired emotion recognition is associated with
basic perceptual processing in schizophrenia.
In this study, emotion discrimination performance was as-
sessed in schizophrenia patients and then compared with basic
visual processing (contrast detection), as well as nonaffective
identity discrimination, using a psychophysical approach. Face
identity discrimination was used as a nonaffective comparison of
acial affect recognition is impaired in schizophrenia (1,2).
This behavioral impairment has been linked to specific
brain mechanisms responsible for affective processing,
face recognition ability, since it depends on similar information
processes to facial emotion recognition. Contrast detection was
used as a measure of basic visual processing, since contrast
provides basic perceptual inputs for almost all vision-related
tasks. Each of these assessments used psychophysical methods
to index the level of signal strength required to perform the task.
The employment of these three tasks in the same patients
allowed direct comparison of behavioral capacities across visual
and emotional domains and permitted inference regarding the
roles of affective, nonaffective, and basic visual processing in
emotion recognition in schizophrenia. We hypothesized that
performance on face identity discrimination and contrast detec-
tion tasks would account for a sizeable proportion of the
variance in facial emotion discrimination in patients.
Methods and Materials
Thirty-two schizophrenia patients and 29 normal control
subjects participated. Inclusion criteria were age 18 to 65 years,
no recent history of drug or alcohol abuse (past 6 months), no
neurological problems, and IQ ? 70. Patients were diagnosed
with schizophrenia (n ? 18) or schizoaffective disorder (n ? 14)
using the Structured Clinical Interview for DSM-IV (SCID-IV).
Demographic and clinical characteristics of the sample are listed
in Table 1. All subjects gave written informed consent, and the
protocol was approved by the Institutional Review Board of
Emotion Discrimination. The targets were face images gen-
erated from the NimStim Face Stimulus Set (http://www.
macbrain.org) (16). The images contained happy, fearful, or
neutral expressions. The task was to indicate which of two
sequentially presented face images, one with a specific emo-
tional intensity and the other with a neutral expression, was more
From McLean Hospital (DN, RM, DO, YC), Belmont; Massachusetts General
Hospital (DJH), Charlestown; and Department of Psychiatry (DJH, DO,
YC), Harvard Medical School, Belmont, Massachusetts.
Address reprint requests to Yue Chen, Ph.D., Harvard University Medical
School, Department of Psychiatry, 115 Mill Street, Room G06, Centre
Building, Belmont MA 02478; E-mail: email@example.com.
Received October 8, 2008; revised January 20, 2009; accepted January 21,
BIOL PSYCHIATRY 2009;xx:xxx
© 2009 Society of Biological Psychiatry
ARTICLE IN PRESS
fearful or happier (Figure 1A). Additional details about this and
the other task procedures are provided in Supplement 1.
Identity Discrimination. The targets were face images with
neutral expressions, either the original images of two individuals
or images morphed between them (Figure 1B). The task was to
determine which of two face images was identical to a face image
presented immediately prior.
Contrast Detection. The target was a sinusoidal grating (2
cycles/degree) displayed at varying contrast levels. The task was
to indicate during which interval, the first or second, the target
was present; a blank screen was present during the other
The critical measure for the tasks was the just
noticeable difference (JND) threshold between the target and
comparison, at which subjects performed at a criterion level. The
thresholds were compared using a two-way repeated measures
analysis of variance (ANOVA) (emotion type and diagnosis).
Thresholds for contrast detection and identity discrimination
were compared using Student t tests. To examine how nonaffec-
tive perceptual and clinical variables contribute to emotion
discrimination performance in patients, multiple regression anal-
yses were performed.
Patients had significantly higher overall thresholds than con-
trol subjects for discriminating between emotional and neutral
Figure 1. (A) Emotion discrimination. Emotional in-
tensity of the face images was manipulated by mor-
phing a neutral face image and an emotional face
image of the same individual. Six levels of emotion
intensity were used: 0, 6, 12, 24, 48, and 100%. The
greater an emotion intensity level, the easier the
emotion discrimination task. Photographs of two in-
dividuals, a male individual and a female individual,
were used. Each face image was presented for 400
msec, with a 500 msec interstimulus interval. (B)
Identity discrimination. Morphed images were cre-
ated by morphing together two separate faces (i.e.,
two different individuals) such that the resultant im-
comparison face images in the second presentation
could be 5, 12.5, 25, 50, or 100%. A 100% difference
would entail a comparison of the original (nonmor-
phed) face photographs of two separate individuals.
phed images: 47.5% of individual #1 and 52.5% of
individual #2, or 52.5% individual of #1 and 47.5% of
ison face images varied across trials according to the
method of constant stimuli. Presentation One lasted
for 600 msec, and Presentation Two lasted for 1200
msec. There was a 500-msec interval between the
2 BIOL PSYCHIATRY 2009;xx:xxx
D. Norton et al.
ARTICLE IN PRESS
facial expressions (F ? 10.55, df ? 1, 59, p ? .002) (Table 2). For
both groups, thresholds were lower for discriminating happiness
than fear (F ? 19.8, df ? 1, 59, p ? .001) (Figure 2). The
interaction between the type of emotion and group was also
significant (F ? 7.58, df ? 1, 59, p ? .008), indicating that patients
were differentially impaired in their ability to discriminate fear.
Post hoc comparisons between groups were significant for
fear discrimination (t ? 3.94, df ? 59, p ? .001, Bonferroni
corrected) but only approached significance for happiness dis-
crimination (t ? 2.21, df ? 59, p ? .07, Bonferroni corrected).
Consistent with this was a larger effect size (ES) for group in fear
discrimination (ES ? .99) than happiness discrimination, which
also had a considerable group effect size (ES ? .55). Summary
and analysis of the performance accuracies, from which the
above perceptual thresholds are derived, are included in Sup-
Identity Discrimination and Visual Detection
Face identity discrimination thresholds were higher for pa-
tients than control subjects, but the group difference was not
significant (t ? 1.76, df ? 52, p ? .09). Contrast detection
thresholds were significantly higher for patients than in control
subjects (t ? 4.33, df ? 52, p ? .001).
Correlations between emotion recognition and perceptual/
clinical variables are included in Figure 3 and Supplement 1.
Multiple Regression Analyses
Multiple linear regression analysis included the five factors
that were significantly correlated with fear discrimination perfor-
mance in patients: identity discrimination thresholds, contrast
detection thresholds, Positive and Negative Syndrome Scale
(PANSS) negative scores, PANSS general scores, and IQ. With all
these variables included, the model accounted for a large portion
of the variance in fear discrimination performance in the patients
(R2? .60). Among these independent variables, only identity
discrimination (p ? .02) and contrast detection thresholds (p ?
.01) significantly predicted fear discrimination performance. Fur-
ther analysis using backward stepwise regression included only
the identity discrimination and contrast detection factors and
accounted for a similar amount of variance (R2? .54) to the
original regression analysis.
For happiness discrimination, the same analysis showed that
the five variables in consideration predicted a smaller amount of
the performance variance than they did for fear discrimination
(R2? .38). Only IQ was a significant predictor for happiness
discrimination performance (p ? .01), accounting for a moderate
amount of performance variance (R2? .20).
This study found that schizophrenia patients had significantly
elevated thresholds for emotion discrimination. Interestingly,
impaired fear discrimination was predicted by basic perceptual
performance, namely, contrast detection and face identity dis-
Table 1. Demographic and Clinical Information of the Sample
(n ? 29)
(n ? 32)
40.7 (14.6) 41.8 (9.8)
12 males, 17 females
15.6 years (2.2)
14.1 years (3.1)
17 males, 15 females
13.6 years (2.1)
13.5 years (3.3)
18.9 years (10.9)
565.9 mg (349.7)
Numbers in parentheses indicate standard deviation.
BDI, Beck Depression Inventory; PANSS, Positive and Negative Syn-
aGroups differ significantly at p ? .05.
bGroups differ significantly at p ? .001.
Table 2. Summary of Results
Group Fear Discrimination Happiness DiscriminationContrast Detection Identity Discrimination
For emotion discrimination (happiness and fear), the threshold represents the level of emotional intensity needed for reliable detection. For contrast
detection, it is the level of contrast (in log units) needed for detecting the target. For identity discrimination, it is the proportion of a given individual in the
image that is morphed between two individuals.
Figure 2. (A) Example of performance scores on fear discrimination by a
patient (right panel) and a control subject (left panel). The horizontal lines
denote 80% accuracy level, and the vertical lines mark the stimulus level
needed for each subject to reach 80% accuracy (the threshold). (B) Individ-
ual subjects’ thresholds for fear discrimination and happiness discrimina-
tion. The ordinate, in a log scale, denotes the values of the threshold. The
abscissa denotes the subject groups and type of emotion discrimination
evaluated. The lower the threshold, the better the performance.
D. Norton et al.
BIOL PSYCHIATRY 2009;xx:xxx 3
ARTICLE IN PRESS
crimination, whereas impaired happiness discrimination was not
predicted by these two perceptual variables, but rather by IQ.
Most previous studies of facial emotion recognition in schizo-
phrenia have found that patients make poor judgments classify-
ing and labeling different emotional expressions (1,2). Extending
this work, the present study found that patients performed
poorly on a task where only one emotional expression was
presented at a time. This result suggests that patients have
difficulty detecting the presence of a given emotion. Classifica-
tion and detection of facial emotion are two distinct but closely
related processes (17). How the deficits in these two emotion
processes are related in schizophrenia remains to be explored.
The connection between affective processing systems (e.g.,
amygdala) and visual processing systems (e.g., striate and extra-
striate cortex) has been a topic of intense research. Damage to
the amygdala results in lower activity in the visual cortex during
emotion recognition (18). Presumably, changes in activity levels
within early visual areas also influence activity within the amyg-
dala. The current study found that schizophrenia patients’ per-
formance in contrast detection and nonaffective face recognition
predicted their fear discrimination performance, suggesting that
the known deficits in the visual and affective processing domains
are associated in schizophrenia. It is not clear whether deficient
visual processing leads to deficient affective processing in
schizophrenia or vice versa. Examining the effects of modulating
signals in one domain (e.g., physical attributes of the visual
stimulus) on performance in the other (e.g., affect discrimina-
tion) may help clarify this issue. Also, other aspects of social
cognitive dysfunction in schizophrenia, particularly those that
rely on abstract representations of other people, may not be
associated with basic perceptual processing abnormalities. This
possibility should be examined in future studies.
Impaired performance on both fear and happiness discrimi-
nation seems to suggest an emotion recognition problem that is
not confined to specific affect categories. However, further
analysis revealed that the impairment in fear discrimination was
predicted by visual and nonaffective face discrimination but not
other variables such as IQ or PANSS scores. In contrast, the
impairment in happiness discrimination was predicted by IQ but
not other variables, suggesting that dysfunctions of distinct brain
systems are implicated in these two domains of emotion discrim-
ination. The result of differential associations in patients is
consistent with previous findings that happiness discrimination
and fear discrimination rely on different facial features (mouth vs.
eyes) (19) and involve different brain mechanisms (20). The
selective association between performance in fear discrimination
and contrast detection in patients is also consistent with the
known connections between the magnocellular pathway (which
processes contrast information) and the amygdala (which pro-
cesses fear information) (6).
In sum, the results of this study, along with previous studies,
suggest that impairment in emotion recognition in schizophrenia
can no longer be viewed solely as a problem of affective
processing: basic perceptual processing deficits appear to play
an integral role. Therefore, novel therapeutic interventions that
aim to improve social cognition in schizophrenia patients should
account for the role of perceptual factors.
This work was supported in part by grants from the National
Institutes of Health (NIH) and Harvard University.
We thank Dr. Ken Nakayama for help during the initial phase
of the study, Dr. Jejoong Kim for his comments on an earlier
version of the article, and Dr. Caitlin Ravichandran for her help
with statistical analysis.
The authors report no biomedical financial interests or other
conflicts of interest.
Supplementary material cited in this article is available
discrimination: III. Behavioral findings in schizophrenia. Psychiatry Res
(2003): Facial emotion recognition in schizophrenia: Intensity effects
and error pattern. Am J Psychiatry 160:1768–1774.
3. Holt DJ, Kunkel L, Weiss AP, Goff DC, Wright CI, Shin LM, et al. (2006):
Increased medial temporal lobe activation during the passive viewing
of emotional and neutral facial expressions in schizophrenia. Schizophr
4. Gur RE, McGrath C, Chan RM, Schroeder L, Turner T, Turetsky BI, et al.
(2002): An fMRI study of facial emotion processing in patients with
5. Holt DJ, Phillips ML (2009): The human amygdala in schizophrenia. In:
6. Pourtois G, Dan ES, Grandjean D, Sander D, Vuilleumier P (2005): En-
hanced extrastriate visual response to bandpass spatial frequency fil-
tered fearful faces: Time course and topographic evoked-potentials
7. Gallagher S (2008): Direct perception in the intersubjective context.
8. Butler PD, Silverstein SM, Dakin SC (2008): Visual perception and its
impairment in schizophrenia. Biol Psychiatry 64:40–47.
9. Chen Y, Bidwell LC, Norton D (2006): Trait vs. state markers for schizo-
phrenia: Identification and characterization through visual processes.
10. Chen Y, Norton D, Ongur D, Heckers S (2008): Inefficient face detection
in schizophrenia. Schizophr Bull 34:367–374.
11. Lee J, Kwon JS, Shin YW, Lee KJ, Park S (2007): Visual self-recognition in
patients with schizophrenia. Schizophr Res 94:215–220.
12. Chen Y, Norton D, McBain R, Ongur D, Heckers S (2009): Visual and
cognitive processing of face information in schizophrenia: Detection,
discrimination and working memory. Schizophr Res 107:92–98.
Figure 3. Summary of correlations between emotion recognition and per-
ceptual and clinical variables. Abscissa shows the nonemotion variables
with which the emotion performance was correlated. Ordinate represents
the correlation coefficient: Pearson R for continuous variables (identity dis-
crimination and contrast detection) and Spearman Rho for the clinical
denote a significant correlation at p ? .001. Gray bars are for correlations
with fear discrimination; white bars are for correlations with happiness
discrimination. BDI, Beck Depression Inventory; PANSS, Positive and Nega-
tive Syndrome Scale.
4 BIOL PSYCHIATRY 2009;xx:xxx
D. Norton et al.
ARTICLE IN PRESS
13. Butler PD, Tambini A, Yovel G, Jalbrzikowski M, Ziwich R, Silipo G, et al. Download full-text
(2008): What’s in a face? Effects of stimulus duration and inversion on
face processing in schizophrenia. Schizophr Res 103:283–292.
(2007): Facial emotion recognition in schizophrenia: When and why
does it go awry? Schizophr Res 94:253–263.
15. Salem JE, Kring AM, Kerr SL (1996): More evidence for generalized poor
performance in facial emotion perception in schizophrenia. J Abnorm
16. Tottenham N, Tanaka J, Leon AC, McCarry T, Nurse M, Hare TA, et al. (in
research participants. Psychiatry Res.
flexibly modifies the perception of faces in rapid visual presentations.
18. Amaral DG, Behniea H, Kelly JL (2003): Topographic organization of
projections from the amygdala to the visual cortex in the macaque
monkey. Neuroscience 118:1099–1120.
19. Vuilleumier P, Richardson MP, Armony JL, Driver J, Dolan RJ (2004):
Distant influences of amygdala lesion on visual cortical activation dur-
ing emotional face processing. Nat Neurosci 7:1271–1278.
20. Adolphs R, Gosselin F, Buchanan TW, Tranel D, Schyns P, Damasio AR
age. Nature 433:68–72.
D. Norton et al.
BIOL PSYCHIATRY 2009;xx:xxx 5
ARTICLE IN PRESS