Content uploaded by Carla Anne Wall
Author content
All content in this area was uploaded by Carla Anne Wall on Apr 20, 2016
Content may be subject to copyright.
COMPUTER-ASSISTED IMPROVEMENTS IN ASD 1
Computer-Assisted Face Processing Instruction Improves Emotion Recognition, Mentalizing,
and Social Skills in Students with ASD
Linda Marie Rice
Moorpark Unified School District
Carla Anne Wall
Yale Child Study Center
Adam Fogel
California Department of State Hospitals Headquarters
Frederick Shic
Yale Child Study Center
Citation:
Rice, L. M., Wall, C. A., Fogel, A., & Shic, F. (2015). Computer-assisted face processing
instruction improves emotion recognition, mentalizing, and social skills in students with
asd. Journal of autism and developmental disorders, 45(7), 2176-2186.
COMPUTER-ASSISTED IMPROVEMENTS IN ASD 2
Abstract
This study examined the extent to which a computer based social skills intervention
called FaceSay® could increase the affect recognition, mentalizing, and social skills of school
aged children with Autism Spectrum Disorder (ASD). FaceSay® offers students simulated
practice with eye gaze, joint attention, and facial recognition skills. This randomized control trial
included school-aged children meeting the educational criteria for autism (N = 31). Results
demonstrated that participants who received the intervention were able to significantly increase
their affect recognition and mentalizing skills, and these improvements were also reflected in
certain generalized social-skills abilities. These findings suggest by targeting face processing
skills, computer-based interventions can efficiently produce changes in broader cognitive and
social-skills domains.
Introduction
Difficulties with social interaction are a hallmark characteristic of autistic spectrum
disorder (ASD); these include challenges with social-emotional reciprocity and impairments in
emotion recognition and expression (American Psychiatric Association, 2014; Baron-Cohen,
1995). Frequently, children with ASD exhibit delays and deviations in the ability to recognize
emotions in themselves and others (Harms, Martin, & Wallace, 2010). Even as adults, many
individuals with ASD struggle to recognize complex emotions, have trouble expressing and
regulating their own emotions, and show evidence of eye contact and facial processing
difficulties (APA, 2013; Baron-Cohen, 1995, 2003; Klin, Jones, Schultz, Volkmar, & Cohen,
2002; McPartland, Webb, Keehn, & Dawson, 2011; Pelphrey et al., 2002; Samson, Huber, &
Gross, 2012).
COMPUTER-ASSISTED IMPROVEMENTS IN ASD 3
In addition to emotion processing deficits, children and adults with ASD struggle with
mentalizing, or the ability to attribute mental states such as beliefs, thoughts, feelings, plans, and
intentions, to themselves and others (Baron-Cohen, 2009). Studies have demonstrated that
emotion recognition and mentalizing are related in children with and without autism (Buitelaar et
al., 1999; Muris et al.,1999). Additionally, neuroanatomical observations have also shown that
the areas of the brain that are critical for engaging in social cognition (i.e., thinking about others’
thoughts, feelings, and intentions) are also implicated in perceiving and interpreting nonverbal
social signals such as facial expressions, social gestures, and eye gaze (Hadjikhani, Joseph,
Snyder, & Tager-Flusberg, 2006; Schultz et al., 2003; Zilbovicus et al., 2013)., Abnormalities in
these brain regions are thought to related to many of the behavioral symptoms observed in ASD
(Chevallier, Kohls, Troiani, Brodkin, & Schultz, 2012).).
The failure of those with ASD to accurately interpret the social dynamics of interactions
reflects reduced predisposition for seeking out emotional information from another’s face,
otherwise known as social referencing (Moore & Corkum, 1994). These deficits, in turn, may be
linked to atypical face processing, a trait also considered by some to be fundamental
characteristic of individuals with autism (Rutishauser et al., 2013). Indeed, children with autism
have difficulty “reading” facial expressions, matching facial expressions with verbal messages,
and comprehending emotion-laden words (Hobson, 1993). Furthermore, studies utilizing
multiple paradigms in addition to faces, including voice intonation and pitch, pictures of eyes,
social scenes, and animated objects, indicate that individuals with ASD are weaker at
interpreting more complex emotions and mental states than their typically developing peers
(Amenta, Ferrari, & Balconi, 2014; Daou, Vener, & Poulson, 2014; Shic, Bradshaw, Klin,
Scassellati, & Chawarska, 2011; Xu & Tanaka, 2014). It is possible that many of the broader
COMPUTER-ASSISTED IMPROVEMENTS IN ASD 4
social impairments observed in individuals with ASD may be derived from these localized face-
processing difficulties.
Social cognitive theories propose that core emotion and social processing deficits
observed in individuals with ASD, including difficulties with face processing, result in
subsequent behavioral symptomatology (Rogers & DiLalla, 1990). For example, problems
recognizing, labeling and understanding the emotional and mental states of others, coupled with
an inability to discern the appropriate empathetic and congruent response, can obstruct
communication and precipitate social misunderstandings. By offering interventions that provide
children with the face processing skills requisite for mentalizing and emotion recognition, we can
remedy a potential causal factor of many of the pervasive social skills deficits exhibited by
children on the autism spectrum.
Computer Assisted Instruction in ASD
For those with ASD, computers and computer-assisted instruction (CAI) provide a
method for receiving instruction and/or interaction that has a number of positive and supportive
features (Golan, LaCava, & Baron-Cohen, 2007; Moore, McGrath, & Thorpe 2000; Smith &
Sung, 2014). Computers are a multimodal, repetitive, predictable, and consistent system; require
fewer social demands; can be used at one’s own pace and difficulty level (Golan et al., 2007).
CAI provides multisensory interactions, controlled and structured environments, multilevel
interactive functions, and the ability to individualize instruction, all of which have been found to
be successful in interventions for children with ASD (Bernard-Opitz, 1989; Chen & Bernard-
Opitz, 1993; Panyan, 1984; Yamamoto & Mira, 1999). In addition, CAI are often purposefully
designed to create an intrinsically motivating environment, a feature that may especially appeal
COMPUTER-ASSISTED IMPROVEMENTS IN ASD 5
to children with ASD (Chen and Bernard-Opitz, 1993; Heiman, Nelson, Tjus, and Gillberg;
1995; Moore & Calvert, 2000). Creating interesting learning environments involves the use of
perceptually salient production features, such as sound effects and action, which are likely to
elicit children’s attention to information, and their subsequent processing of that information
(Calvert, 1999). For example, CAI featuring actions or animations increases poor readers’
memory of nouns by providing a visual, iconic mode that children can use to represent content
(Calvert, Watson, Brinkley, & Penny, 1989).
Although CAI has been shown to be a relevant method to train and develop vocabulary
knowledge and language learning for individuals with ASD (Bosseler & Massaro, 2003;
Yamamoto & Mira, 1999; Whalen et al. 2006; Whalen et al. 2010), research on the use of CAI to
teach complex social skills such as emotion recognition (ER) or affect recognition to individuals
with disabilities is still emerging. Generally, studies have shown that a time-limited use of
computer interventions with individuals with various disabilities was sufficient to teach basic ER
of emotions such as happiness, sadness, anger and fear. (Blocher & Picard, 2002; Bolte, Hubl,
Feineis-Matthews, Prvulovic, Dierks & Poustka, 2006; Moore, Cheng, McGrath, & Powell,
2005; Silver & Oakes, 2001). More recently, results of several studies have suggested that basic
and complex ER can improve with computer intervention (Golan & Baron-Cohen, 2006a, 2006b;
Golan et al., 2010; LaCava, Golan, Baron-Cohen, & Myles, 2007; Young & Posselt, 2012).
Notably, a randomized clinical trial found that 20 hours of training with program called Let’s
Face It! led to improvements in facial recognition and processing skills in children with ASD
(Wolf et al., 2008; Tanaka et al. 2010). Another randomized control trial using an alternate
computerized intervention to improve emotion understanding in children with ASD found
improvements in mental state identification, suggesting that it is indeed feasible to target
COMPUTER-ASSISTED IMPROVEMENTS IN ASD 6
mentalizing skills through CAI (Silver & Oaks, 2001). Although CAI has been instrumental in
teaching specific emotion recognition and processing skills to children with ASD, most existing
programs demonstrate limited generalizability of acquired skills to social behaviors and
environments (Golan et al., 2010; Smith & Sung, 2014; Young & Posselt, 2012).
However, a particular CAI program FaceSay, has had some success teaching
generalizable social skills to children with ASD in a generalizable fashion. Hopkins et al. (2011)
showed in an earlier randomized-control trial that children with low-functioning autism (LFA)
who received a 6-week period of intervention with FaceSay showed improvements in emotion
recognition and social interactions. This same study showed that children with high-functioning
autism (HFA) who received the FaceSay intervention demonstrated improvement in facial
recognition, emotion recognition and social interactions compared to the control group. In
blinded observations of peer interactions on the playground, children with HFA and low-
functioning autism (LFA) who received the FaceSay intervention initiated more social
interactions with their peers, made more eye contact, and exhibited fewer negative behaviors
than their peers in a control group. Nevertheless, further work must be done to both evaluate the
utility of the FaceSay software as a therapeutic tool for children with ASD and to understand the
mechanisms by which FaceSay training impacts social information processing more broadly.
Aims and Predictions
The present study aims to expand upon initial results concerning the efficacy of FaceSay
as an intervention tool (Hopkins et al., 2011). It has already been demonstrated that FaceSay
training improves facial recognition in children with autism. We aim to replicate and improve
COMPUTER-ASSISTED IMPROVEMENTS IN ASD 7
upon these findings by examining what additional skills and behaviors may be impacted by this
intervention.
To this end, we evaluated five different hypotheses in a group of children receiving the
FaceSay intervention (experimental group) and a control group that was administered a CAI
intervention focused on more standard educational constructs such as mathematics and reading:
Hypothesis 1. Participants in the experimental group will have a significantly higher mean pre-to
post-intervention score on affect recognition as compared to participants in the control condition.
Hypothesis 2. Participants in the experimental group will have a significantly higher mean pre-
to post-intervention score on mentalizing assessments as compared to participants in the control
condition.
Hypothesis 3. Participants in the experimental group will have significantly lower post-
intervention scores on teacher report measures assessing the participant’s social impairment as
compared to participants in the control condition.
Hypothesis 4. Participants in the experimental group will have increased positive interactions
with peers’ post-intervention based upon social skills observation ratings as compared to
participants in the control condition.
COMPUTER-ASSISTED IMPROVEMENTS IN ASD 8
Hypothesis 5. Participants in the experimental group will have decreased negative interactions
with peers’ post-intervention based upon social skills observation ratings as compared to
participants in the control condition.
Methods
Participants
This study was contacted with Institutional Review Board approval from the California
Graduate Institute of the Chicago School of Professional Psychology. The administration of the
participating school district has provided written consent to conduct the research/collect data at
their facility. Parents of qualifying students were contacted by mail with a description of the
study, parental consent, and child assent forms for the students, along with stamped, self-
addressed return envelopes and contact information in the event they had questions about the
study or required further information. After all participants had been recruited, they were
randomly assigned to a study group.
Participants included 31 elementary school students in Ventura County, California,
ranging in age from 5 years to 11 years (M=7.77), who were eligible for special education
services under the educationally based handicapping condition of autism in California. Of these
students, 28 were male. Demographics: Caucasian: 71.9%, African America: 9.4%, Hispanic:
9.4%, Asian: 6.3%. 15 control, 16 experimental. Subjects received either the WISC-III or WISC-
IV as a measure of cognitive functioning. All participants were considered high functioning, with
FSIQ > 70 (M = 101, SD = 14.45). For additional information regarding participants, see Table 1.
Design and Instruments
COMPUTER-ASSISTED IMPROVEMENTS IN ASD 9
We designed a randomized, controlled experiment to determine the effects of the
FaceSay computer program on the ability of children with an ASD to recognize emotions,
understand another’s perspective, and improve their social skills in comparison to other ASD
children not receiving the intervention. This study thus involved a 2 (Training) x 2 (Time) mixed
factorial design. The within factor, time, had two levels, pre- and post-intervention. The
between factor, training, also included two levels, experimental (FaceSay program) and control
(SuccessMaker® program, see Procedures).
Materials/Dependent Variables
There were five dependent variables for this study: (1) affect recognition performance,
(2) mentalizing ability, (3) social skills as assessed via teacher questionnaire, and (4) positive and
(5) negative social behavior as assessed by direct observation.
Emotion/affect recognition (AR).
Affect recognition was assessed using standard scores on the NEPSY-II Affect Recognition
subtest (Korkman, Kemp, and Kirk, 2007). This subtest is designed to assess the ability to
identify affect from photographs of children’s faces. The tasks progress from affect identification
to recognition and memory for affect. Low scores in this task suggest difficulties with
recognition and discrimination of facial affect. Participants were given a raw score equivalent to
the number of correct responses on the subtest. The raw score was then converted to a scaled
score based on age norms, and this was defined as the participant’s score.
Mentalizing/theory of mind (ToM)
COMPUTER-ASSISTED IMPROVEMENTS IN ASD 10
Raw score on the NEPSY-II Theory of Mind subtest was used to measure each
participant’s mentalizing skills, which includes the ability to understand mental functions such as
belief, intention, deception, emotion, imagination, and pretending and to understand that others
have their own thoughts, ideas, and feelings that may be different from one’s own (Korkman,
Kemp, and Kirk, 2007). The subtest is comprised of two tasks designed to assess the ability to
understand mental functions and another’s point of view. Participants’ scores were defined as a
raw score equivalent to the number of correct responses on the subtest.
Social skills ratings
We utilized the Social Responsiveness Scale, Second edition (SRS-2; Constantino &
Gruber, 2002). The SRS-2 is a parent/teacher questionnaire consisting of 65 items that measures
the type and severity of ASD-specific social deficits in children and adolescents, such that high
scores indicate greater impairment. It yields a total raw score based upon teachers’ numerical
ratings regarding statements about each participant, which was converted to a standard score
based upon age and gender norms. Participants received a total score and scores for each of five
subcales: Social Awareness, Social Cognition, Social Communication, Social Motivation, and
Restricted Interests and Repetitive Behaviors.
Positive interactions
Positive interactions were rated by counting the total number of times a participant
spontaneously initiated and engaged in positive interactions with a peer. These positive
interactions included when the participant exhibited verbal and nonverbal social behaviors that
lead to effective social processes with peers and could serve to start or maintain social
COMPUTER-ASSISTED IMPROVEMENTS IN ASD 11
interactions. Specifically, these interactions were defined as direct eye contact, direct eye
contact combined with a smile; a smile with no eye contact, an expression of affection delivered
verbally or non-verbally, the sharing of an object or objects, the spontaneous verbal sharing of
experiences or request for such, physical approach with social communication and intention, a
greeting such as “hello” or another appropriate response to a greeting, or the giving of help.
Negative interactions
Negative interactions were rated by counting the total number of times instances the
participant engaged in negative interactions with a peer. These interactions included instances
where the participant exhibited unpleasant social behaviors that function to stop or decrease the
likelihood of the development of an adequate social interaction. Specifically, these interactions
were defined as physical or verbal aggressiveness, controlling behaviors in which the participant
dominates peers without respecting their needs, physical but non-violent actions such as stealing
another child’s toy, teasing/taunting initiated by the participant intended to invoke a negative
reaction, or avoidance where the participant moves three feet or more away from another.
Procedures
To measure participant’s emotion recognition and mentalizing skills, pre- and post-test
measures were administered in the school psychologist’s office at each school site by research
staff. The participant was assessed in a one-on-one format for approximately 20 minutes.
Social skills information was collected from the teachers with the SRS-2. The forms
were given to the teachers directly by the examiner and provided along with a return envelope to
ensure confidentiality and research staff contact information in the event that they required
COMPUTER-ASSISTED IMPROVEMENTS IN ASD 12
additional instruction or support in completing the scale. Teachers were blinded to the training
group membership (FaceSay treatment or control) of the participants.
Social skills observations were also conducted at baseline and post-intervention by two
observers on the playground during recess. These observers were also blinded to training group
membership. Both observers were employed by the school district, held master’s degrees in
mental health and/or education, and had experience and training in gathering observational data.
The observations took place for approximately 10 minutes during regularly scheduled recess
and lunch times. The participants were observed independently by each rater at separate time
points, for a total of 20 minutes.. Inter-rater reliability for the social skills observations was
determined adequate once 90% agreement was reached consistently for the coding of data during
training sessions between the two observers and the primary investigator. For each 10-minute
session, the observer recorded the behaviors of a single participant. The observers maintained
fairly close proximity to the participants; however, they did not interact with the participants and
politely declined any overtures made towards them. The participants were informed that the
observer was simply interested in watching them play if the participants questioned the observer
or other adult.
Intervention Procedures (FaceSay)
Once all of the pre-intervention measures were completed and just prior to beginning the
computer sessions, the participants accessed the FaceSay program and underwent a brief
training session with this examiner and the paraeducator or specialty teacher to ensure their
ability to access the program and navigate through the games.
COMPUTER-ASSISTED IMPROVEMENTS IN ASD 13
In FaceSay, various games are designed to teach specific social skills. The “Amazing
Gazing” game was designed to teach children to attend to eye gaze and respond to joint attention,
given that children with ASD have shown deficits in these areas, and these skills can be taught
through interventions (Mundy, Gwaltney, & Henderson, 2010; Leekman, Lopez, & Moore,
2000). Because research studies have indicated orienting difficulties to both social and nonsocial
stimuli, with even greater problems in response to social stimuli (Dawson, Meltzoff, Osterling,
Rinaldi, & Brown, 1998), “Amazing Gazing” includes both social and nonsocial stimuli. In the
game, an avatar is surrounded by an array of objects, numbers, or faces (see Fig. 1a). The
participant is asked to look at the avatar’s eyes and indicate which object, number, or face the
avatar is attending to. If the participant is correct, the item will light up and a verbal
reinforcement will be given (e.g., “Good job, Johnny!”); if the participant is incorrect, a verbal
and/or visual prompt is given to indicate the correct answer.
Joseph & Tanaka (2003) demonstrated that children with ASD do not adapt a configural
strategy when recognizing faces, but rely on a more object-based featural approach. Another
game, “Band Aid Clinic,” was thus developed to teach facial recognition by building on the local
processing cognitive operations that children with ASD use when viewing faces. In the “Band
Aid Clinic,” the participants were asked to select the appropriate face “band aid” that would fit
over the avatar’s face (see Fig. 1b). The possible matches increase in number and similarity as
the game progresses. The face is reconstructed by identifying the correct band aid. The goal of
the “Band Aid Clinic” is to encourage processing of facial expressions in terms of their features
and configuration.
Third, given that children with ASD have difficulty recognizing and identifying facial
expressions from pictures of people’s eyes (Baron-Cohen, Wheelwright, Hill, Raste, & Plumb,
COMPUTER-ASSISTED IMPROVEMENTS IN ASD 14
2001), and given emotional expression difficulties observed in ASD (Hobson, 1986), the
“Follow the Leader” game in FaceSay was designed to improve the ability of its users to
distinguish and create facial expressions of emotions in avatars. This game component
specifically emphasizes how subtle changes in eye information can alter the perception of facial
expression and is designed to teach participants to look to the eyes for information and to
improve their ability to read facial expression based on the eyes. The ultimate goal of “Follow
the Leader” game is to teach aspects of emotional cognition and facial recognition.
In the first level of “Follow the Leader”, the participant is asked to identify identical
facial expressions and emotions by selecting “Yes” for same and “No” for different expressions
(see Fig. 1c). The similarity of the two faces increases as the game progresses. As the game
advances in levels, the participant is asked to make the avatar’s twin match the avatar’s
expression by selecting appropriate eyes from a selection of eyes. Similarly, as the game
continues, the facial expressions change and become increasingly subtle. The game thus provides
practice both in more passive comparisons between facial expressions as well as more active
online adjustment of an avatar’s facial expressions.
Control Procedures (SuccessMaker®)
The participants underwent training sessions with specialty teachers prior to the current
study. The control group participants received SuccessMaker® a set of computer-based courses
used to supplement regular classroom reading instruction in grades K-8. Using adaptive lessons
tailored to the participant’s reading level, SuccessMaker® aims to improve understanding in
areas such as phonological awareness, phonics, fluency, vocabulary, comprehension, and
concepts of print. The courses aim to help the participants develop and maintain reading skills as
COMPUTER-ASSISTED IMPROVEMENTS IN ASD 15
well as provide opportunities for exploration, open-ended instruction and development of
analytical skills. As the student interacted with the program the computer analyzed the
participant’s skill development and assigned specific segments of the program, introducing new
skills as they became appropriate. Individualization allowed the participant to progress on
his/her own schedule (What Works Clearinghouse, 2009). The control participants utilized
SuccessMaker® for the same time and duration as the intervention group utilized FaceSay.
Results
Primary outcome variables for analyses were pre-post difference scores in dependent
variables as described previously. Correlations between these difference scores are presented in
Table 2. Affect Recognition and Theory of Mind scores were positively correlated with each
other and negatively correlated with SRS-2 scores. No other correlations were found to be
significant. Means and standard deviations of pre- and post- measures of all dependent variables
are presented in Table 3.
Hypothesis 1
We hypothesized that the children who received the FaceSay intervention would
demonstrate improved affect recognition, measured by their responses on the NEPSY Affect
Recognition subtest. To assess this, we used an analysis-of-covariance (ANCOVA) approach in
which the independent variable was Group (intervention or control), and the dependent variable
was post-test score on the NEPSY Affect Recognition subtest. Pre-test NEPSY scores were
entered as covariates in order to allow for individual differences prior to the intervention. No
other covariates were entered.
COMPUTER-ASSISTED IMPROVEMENTS IN ASD 16
There was a significant difference in post-test affect recognition score between the
experimental and control groups after controlling for pre-test score, F(1,28) = 20.45, p <.001,
partial η2 = .42. The adjusted Ms for the experimental and control groups were 12.59 and 8.50,
respectively).
Hypothesis 2
We also hypothesized that participants in the experimental group would have a
significantly greater improvement in their mentalizing scores than participants in the control
condition, as measured by the NEPSY Theory of Mind subtest. As with Hypothesis 1, we used
an ANCOVA approach with post-test NEPSY Theory of Mind score as a dependent variable,
Group as an independent variable, and pre-test NEPSY Theory of Mind score as a covariate.
There was a significant difference in post-test Theory of Mind score between the
experimental and control groups after controlling for pre-test score, F(1,28) = 37.35, p <.001,
partial η2 = .57 (adjusted Ms: 12.39 and 16.85, respectively).
Hypothesis 3
Our third hypothesis was that participants in the experimental group would have
significantly lower post-intervention scores on teacher report measures assessing the
participant’s social skills as compared to participants in the control condition (i.e. exhibit fewer
social difficulties). We conducted another ANCOVA with post-intervention SRS-2 score as the
dependent measure, Group (intervention or control) as the independent measure, and the pre-test
SRS-2 scores as a covariate.
COMPUTER-ASSISTED IMPROVEMENTS IN ASD 17
There was a significant difference in post-test SRS-2 score between the experimental and
control groups after controlling for pre-test score, F(1,28) = 4.523, p <.05, partial η2 = .14
(adjusted Ms: 67.7 and 62.3, respectively).
Hypothesis 4
It was hypothesized that participants in the experimental group would show increased
positive interactions with peers’ post-intervention as compared to participants in the control
condition based upon social skills observation ratings. An ANCOVA approach analogous to
previous analyses showed no significant differences in the number of positive social skills
observations between the experimental and control groups following the intervention after
controlling for pre-test numbers, F(1,28) = 0.61, p >.05 (adjusted Ms: 6.71 and 7.61,
respectively). The covariate, pre-test score was the only significant predictor of post-test positive
observations, F(1,28) = 17.24, p < .01.
Hypothesis 5
Finally, we hypothesized that participants in the experimental group would show
decreased negative interactions with their peers’ post-intervention as compared to participants in
the control condition based upon social skills observation ratings. An ANCOVA approach like
those listed above found no significant difference in the number of post-test negative social skills
observations between the experimental and control groups following the intervention, after
controlling for pre-test numbers, F(1,28) = 0.61, p >.05 (adjusted Ms: 0.18 and 0.55,
respectively). The covariate, pre-test score, did not significantly predict post-test negative
COMPUTER-ASSISTED IMPROVEMENTS IN ASD 18
observations either, F(1,28) = 0.627, p >.05. Results of this and the preceding analyses are
shown in Fig. 2)
Discussion
The purpose of this study was to investigate the extent to which FaceSay improves affect
recognition, mentalizing, and social skills in school-aged children with ASD. The results of the
present study suggest that by practicing simulated activities addressing eye gaze, joint attention,
emotional cognition, and facial recognition skills on a computer, participants were able to
improve their ability to recognize basic emotions such as happiness, sadness, neutrality, anger,
disgust and fear. Furthermore, participants increased in their comprehension of beliefs,
intentions, deception, emotion, imagination, and pretending, and improved their understanding
that others have thoughts, ideas and feelings that may be different from their own. Finally, this
study showed suggests that training through this software program is related to a pattern of fewer
autism symptoms as assessed via teacher ratings. Thus, the results of the present study both
support previous work that has demonstrated that this software improves teacher-observed social
function and emotion-processing skills (Hopkins et al., 2011), as well as present new evidence
that additional domains such as mentalization capability and theory of mind skills may similarly
benefit.
Changes in theory of mind scores were strongly correlated with changes in affect
recognition in the current study, suggesting that FaceSay targets these two skills in similar ways.
Furthermore, changes in both of these domains were negatively correlated with changes in SRS
score; that is, greater improvements in each of these domains were associated with decreased
symptoms of autism as measured by the SRS-2. These results are consistent with between group
COMPUTER-ASSISTED IMPROVEMENTS IN ASD 19
findings suggesting that FaceSay is associated with improvements in examined outcome
variables, and strengthen our understanding of FaceSay treatment effects at an individual level.
These results were particularly encouraging, because they demonstrate the ways that
FaceSay can improve general elements of social functioning over and above those directly
targeted by the intervention. FaceSay does not directly teach mentalizing, nor does it explicitly
label any emotions. Rather, all of the constituent games address attention to eye gaze, joint
attention bids, facial recognition, and the ability to distinguish and create emotional facial
expressions in avatars. Nevertheless, the program does address mentalizing in very subtle ways.
For example, joint attention bids are accompanied with questions like, “What does Rebecca want
next?” that imply that others’ mental states can be deduced through an understanding of facial
expressions and eye gaze. The results of the current study suggest that we can simultaneously
improve emotion recognition, mentalizing skills and autism symptomatology through an
intervention that primarily addresses facial processing. Future work should attempt to
disentangle this relationship, using more advanced statistical models to explore possible causal
relationships and underlying neural mechanisms.
Despite these encouraging results, this study found no improvements in observed positive
and negative interactions on the playground. Although the FaceSay intervention did produce
observable change in social skills, as evidence by improvements in SRS-2 scores, these changes
did not translate into changes in broader prosocial and antisocial behaviors in this setting. It is
possible that the observations made here were simply not an effective measure of generalizability
for this intervention. Prosocial behavior requires an array of complex social and communicative
skills that go beyond face processing, emotion recognition, or mentalizing. Furthermore, the
number of negative observations in both groups was initially quite low, especially in the context
COMPUTER-ASSISTED IMPROVEMENTS IN ASD 20
of the high variability of those outcome measures. Perhaps a more telling post-intervention
measure would be to observe children’s ability to understand and respond to another person’s
perspective in real-life situations.
Nevertheless, the current findings suggest that the FaceSay program is a highly
promising, efficient, and cost effective strategy for teaching affect recognition and mentalizing
constructs to high functioning elementary school-aged children with ASD. It further suggests
that by addressing these particular skills, we can effect real change in other behaviors outside the
scope of this intervention.
Limitations and Future Directions
There are a few noteworthy limitations to the present study. Although the NEPSY is a
standardized and relatively sophisticated measure of affect recognition, this study specifically
assessed the recognition of six basic emotions from static two-dimensional representations of
children’s faces. In some cases, high functioning individuals with ASD can recognize basic
emotions relatively well; however, their emotion deficits becomes apparent when the recognition
of more complex emotions and mental states is required (Adolphs et al., 2001). Generally,
complex emotions involve attributing a cognitive state as well as an emotion, and are more
situationally dependent. Barriers to social referencing can impede the interpretation of social
dynamics; thus, further work should address the effectiveness of FaceSay on the ability to
recognize and identify more complex emotions and mental states from static as well as dynamic
(e.g., video) facial expressions.
Although the study involved a number of social skills assessments, only the teacher-
report measure suggested some generalizability of social skills. The intervention was
COMPUTER-ASSISTED IMPROVEMENTS IN ASD 21
implemented in the computer lab rather than the classroom setting; introducing an element to
improve the generalization effects, such as having an instructional assistant review what the child
learned or provide self-monitoring techniques, or instruction in a more natural setting could
augment the social skills benefits observed here. Additionally, the use of a self-report measure
may be useful in determining outcome efficacy; possibilities include assessments that tap into
reduction of anxiety relative to social situations or changes in peer networks (Locke, Kasari,
Rotheram-Fuller, & Jacobs, 2013).
Finally, expanding the number of participants in general, and including preschool,
secondary school, and specialized educational settings would greatly enhance the generalizability
of results to the broader ASD population.
The results of this study can be useful for parents, psychologists, educators, and
specialists who live and work with children on the autism spectrum. As the prevalence of ASD
increases, the identification of more evidence-based and cost effective methods to augment the
education of children with ASD is warranted. This study demonstrates that CAI is a highly
effective vehicle for helping children with ASD understand the social world.
Conclusions
Our results indicate that by practicing simulated activities addressing eye gaze, joint
attention skills, emotional cognition, and facial recognition skills on the computer, students were
able to significantly increase their affect recognition capabilities and mentalizing skills, and
reduce their teacher-observed social impairment. Although these improvements were based on
standardized assessments of emotion recognition and social cognition skill, the hypotheses that
social interactions in the school environment would also improve were not fully supported.
COMPUTER-ASSISTED IMPROVEMENTS IN ASD 22
This study demonstrates that the use of computer technology in helping ASD children
understand the mental states of others is highly effective. The computer software program
FaceSay improves the ability of children with ASD to recognize emotions and understand
another’s perspective, and shows great promise in enhancing these skills in the more general
school environment. We hope these results can be useful for parents, psychologists, educators,
and specialists who live and work with children with ASD.
COMPUTER-ASSISTED IMPROVEMENTS IN ASD 23
References
Adolphs, R., Sears, L., & Piven, J. (2001). Abnormal processing of social information from faces
in autism. Journal of Cognitive Neuroscience, 13(2), 232-240.
Amenta, S., Ferrari, C., & Balconi, M. (2014). Facial Expression Decoding in Autistic and
Asperger Children. Comprehensive Guide to Autism, 1885-1904.
American Psychiatric Association. (2013). Diagnostic and statistical manual of mental
disorders (5th ed.). Arlington, VA: American Psychiatric Publishing.
Baron-Cohen, S. (1993). From attention-goal psychology to belief-desire psychology: The
development of a theory of mind, and its dysfunction. IN S. Baron-Cohen, H. Tager-
Flusberg, &D. J. Cohen (Eds.), Understanding other minds. Perspectives from autism
(pp.59-82. Oxford: Oxford University Press.
Baron-Cohen, S. (1995). Mindblindness: An essay on autism and theory of mind.
Cambridge, MA: MIT Press.
Baron-Cohen, S. (2003). The essential difference: The truth about the male and female
brain. New York: Basic Books.
Baron-Cohen, S., Leslie, A., & Frith, U. (1985). Does the autistic child have a “theory of
mind?”. Cognition, 21, 37-46.
Baron‐Cohen, S., Wheelwright, S., Hill, J., Raste, Y., & Plumb, I. (2001). The “Reading the
Mind in the Eyes” test revised version: A study with normal adults, and adults with
Asperger syndrome or high‐functioning autism. Journal of child psychology and
psychiatry, 42(2), 241-251.
Bernard-Opitz, V. (1989). Computer assisted instruction for autistic children. Journal of
Child and Adolescent Psychiatry, 17, 125-130.
COMPUTER-ASSISTED IMPROVEMENTS IN ASD 24
Blocher, K., & Picard, R. W. (2002). Affective social quest: Emotion recognition therapy
for autistic children. In K. Dautenhahn, A. Bond, L. Canemero, & B. Edmonds (Eds.),
Socially intelligent agents: Creating relationships with computers and robots (pp. 133-
140). New York: Springer Publishing.
Bolte, S., Hubl, D., Feineis-Matthews, S., Prvulovic, D., Dierks, T., & Poustka, F. (2006).
Facial affect recognition training in autism: Can we animate the fusiform gyrus?
Behavioral Neuroscience, 120, 211-216.
Bosseler, A., & Massaro, D. W. (2003). Development and evaluation of computer-
animated tutor for vocabulary and language learning in children with autism. Journal of
Autism and Developmental Disorders, 33, 653-673.
Buitelaar, J. K., Van der Wees, M., Swaab–Barneveld, H., & Van der Gaag, R. J. (1999). Theory
of mind and emotion-recognition functioning in autistic spectrum disorders and in
psychiatric control and normal children.Development and psychopathology, 11(01), 39-
58.
Calvert, S. L. (1999). Children’s journeys through the information age. Boston: McGraw
Hill.
Calvert, S. L., Watson, J., Brinkley, V., & Penny, J. (1989). Computer presentational
features for poor readers’ recall of information. Journal of Educational Computing
Research, 22, 251-258.
Chen, S. H., & Bernard-Opitz, V. (1993). Comparison of personal and computer-assisted
instruction for children with autism. Mental Retardation, 31, 368-376.
Chevallier, C., Kohls, G., Troiani, V., Brodkin, E. S., & Schultz, R. T. (2012). The social
motivation theory of autism. Trends in cognitive sciences, 16(4), 231-239.
COMPUTER-ASSISTED IMPROVEMENTS IN ASD 25
Constantino, J. & Gruber, C. (2002). Social Responsiveness Scale. Los Angeles, CA:
Western Psychological Services.
Daou, N., Vener, S. M., & Poulson, C. L. (2014). Analysis of three components of affective
behavior in children with autism. Research in Autism Spectrum Disorders, 8(5), 480-501.
Dawson, A. N., Meltzoff, J., Osterling, J., Rinaldi, E., & Brown, A. (1998). Children with
autism fail to orient to naturally occurring social stimuli. Journal of Autism and
Developmental Disorders, 28, 479-485.
Golan, O., Ashwin, E., Granader, Y., McClintock, S., Day, K., Leggett, V., & Baron-Cohen, S.
(2010). Enhancing emotion recognition in children with autism spectrum conditions: An
intervention using animated vehicles with real emotional faces. Journal of autism and
developmental disorders, 40(3), 269-279.
Golan, O., & Baron-Cohen, S. (2006a). Systemizing empathy: Teaching adults with
Asperger syndrome or high functioning autism to recognize complex emotions using
interactive media. Development and Psychopathology, 18, 591-617.
Golan, O., & Baron-Cohen, S. (2006b). Teaching children with Asperger syndrome and
high functioning autism to recognize emotions using interactive multimedia.
Unpublished manuscript. University of Cambridge, UK.
Golan, O., LaCava, P. G., & Baron-Cohen, S. (2007). Assistive technology as an aid in
reducing social impairment in autism. In R. L. Gabriels & D. E. Hill (eds.), Growing up
with autism: Working with school-age children and adolescents (pp. 124-142). New
York: Guilford Press.
COMPUTER-ASSISTED IMPROVEMENTS IN ASD 26
Hadjikhani, N., Joseph, R. M., Snyder, J., & Tager-Flusberg, H. (2006). Anatomical differences
in the mirror neuron system and social cognition network in autism. Cerebral
cortex, 16(9), 1276-1282.
Harms, M. B., Martin, A., & Wallace, G. L. (2010). Facial emotion recognition in autism
spectrum disorders: a review of behavioral and neuroimaging studies.Neuropsychology
review, 20(3), 290-322.
Heiman, M. Nelson, K. E., Tjus, T., & Gillberg, C. (1995). Increasing reading and
communication skills in children with autism through an interactive multimedia computer
program. Journal of Autism and Developmental Disorders, 25, 459-480.
Hobson, R. P., (1986). The autistic child’s appraisal of expressions of emotion. Journal
of Child Psychology and Psychiatry, 27, 321-342.
Hobson, P. (1993). Autism and the development of mind. Hove, U.K.: Lawrence
Erlbaum.
Hopkins, I. M., Gower, M. W., Perez, T. A., Smith, D. S., Amthor, F. R., Wimsatt, F. C., &
Biasini, F. J. (2011). Avatar assistant: Improving social skills in students with an ASD
through a computer-based intervention. Journal of autism and developmental disorders,
41(11), 1543-1555.
Joseph, R. M. & Tanaka, J. (2003). Holistic and part-based face recognition in children
with autism. Journal of Child Psychology and Psychiatry, 44, 529-542.
Klin, A., Jones, W., Schultz, R. T., Volkmar, F. R., & Cohen, D. J. (2002). Visual
fixation patterns during viewing of naturalistic social situations as predictors of social
competence in individuals with autism [Electronic Version]. Archives of General
Psychiatry, 59, 809-816.
COMPUTER-ASSISTED IMPROVEMENTS IN ASD 27
Korkman, M., Kirk, U., & Kemp, S. (2007). NEPSY-II. San Antonio, TX: Harcourt
Assessment, Inc.
LaCava, P. G., Golan, O., Baron-Cohen, S., & Myles, B. S. (2007). Using assistive
technology to teach emotion recognition to students with Asperger syndrome: A pilot
study. Remedial and Special Education, 28, 174 – 181.
Leekman, S., Lopez, B., & Moore, C. (2000). Attention and joint attention in preschool
children with autism. Developmental Psychology, 36, 261-273.
McPartland, J. C., Webb, S. J., Keehn, B., & Dawson, G. (2011). Patterns of visual attention to
faces and objects in autism spectrum disorder. Journal of autism and developmental
disorders, 41(2), 148-157.
Moore, M., & Calvert, S. (2000). Brief report: Vocabulary acquisition for children with autism:
Teacher or computer instruction. Journal of autism and developmental disorders, 30(4),
359-362.
Moore, C. & Corkum, V. (1994). Social understanding at the end of the first year of life.
Developmental Review, 14, 349-372.
Moore, D., McGrath, P. & Thorpe, J. (2000). Computer-aided learning for people with
autism: A framework for research and development. Innovations in Education and
Training International, 37, 218-228.
Moore, D., Cheng, Y., McGrath, P., & Powell, N. J. (2005). Collaborative virtual
environment technology for people with autism. Focus on Autism and Other
Developmental Disabilities, 20, 231-243.
Mundy, P., Gwaltney, M., & Henderson, H. (2010). Self-referenced processing,
neurodevelopment and joint attention in autism. Autism, 14(5), 408-429.
COMPUTER-ASSISTED IMPROVEMENTS IN ASD 28
Muris, P., Steerneman, P., Meesters, C., Merckelbach, H., Horselenberg, R., van den Hogen, T.,
& van Dongen, L. (1999). The TOM test: A new instrument for assessing theory of mind
in normal children and children with pervasive developmental disorders. Journal of
Autism and Developmental Disorders,29(1), 67-80.
Panyan, M. V. (1984). Computer technology for autistic students. Journal of Autism and
Developmental Disorders, 14, 375-382.
Pelphrey, K. A., Sasson, N. J., Reznick, S., Paul, G., Goldman, B. D., & Pevin, J. (2002).
Visual scanning of faces in autism. Journal of Autism and Developmental Disorders, 32,
249-261.
Rogers, S. J., & DiLalla, D. L., (1990). Age of symptom onset in young children with
pervasive developmental disorders. Journal of the American Academy of Child and
Adolescent Psychiatry, 29(6), 863-872.
Rutishauser, U., Tudusciuc, O., Wang, S., Mamelak, A. N., Ross, I. B., & Adolphs, R. (2013).
Single-Neuron Correlates of Atypical Face Processing in Autism. Neuron, 80(4), 887-
899.
Samson, A. C., Huber, O., & Gross, J. J. (2012). Emotion regulation in Asperger's syndrome and
high-functioning autism. Emotion, 12(4), 659.
Schultz, R. T., Grelotti, D. J., Klin, A., Kleinman, J., Van der Gaag, C., Marois, R., &
Skudlarski, P. (2003). The role of the fusiform face area in social cognition: implications
for the pathobiology of autism. Philosophical Transactions of the Royal Society of
London. Series B: Biological Sciences,358(1430), 415-427.
Shic, F., Bradshaw, J., Klin, A., Scassellati, B., & Chawarska, K. (2011). Limited activity
monitoring in toddlers with autism spectrum disorder. Brain research, 1380, 246-254.
COMPUTER-ASSISTED IMPROVEMENTS IN ASD 29
Silver, M. & Oakes, P. (2001). Evaluation of a new computer intervention to teach
people with autism or Asperger syndrome to recognize and predict emotions in others.
Autism, 5(3), 299-316.
Smith, V., & Sung, A. (2014). Computer Interventions for ASD. In V. B. Patel, V. R. Preedy, &
R. Colin (Eds.), Comprehensive Guide to Autism (pp. 2173-2189). New York: Springer.
Tanaka, J. W., Wolf, J. M., Klaiman, C., Koenig, K., Cockburn, J., Herlihy, L., ... & Schultz, R.
T. (2010). Using computerized games to teach face recognition skills to children with
autism spectrum disorder: The Let’s Face It! program.Journal of Child Psychology and
Psychiatry, 51(8), 944-952.
Whalen, C., Liden, L., Ingersoll, B., Dallaire, E., & Liden, S. (2006). Behavioral improvements
associated with computer-assisted instruction for children with developmental
disabilities. The Journal of Speech and Language Pathology-Applied Behavior
Analysis, 1(1), 11-26.
Whalen, C., Moss, D., Ilan, A. B., Vaupel, M., Fielding, P., Macdonald, K., ... & Symon, J.
(2010). Efficacy of TeachTown: Basics computer-assisted intervention for the intensive
comprehensive autism program in Los Angeles unified school district. Autism, 14(3),
179-197.
What Works Clearinghouse (2009). Intervention: SuccessMaker®. Retrieved February
17, 2010, from http://ies.ed.gov.
White, S., Frith, U., Rellecke, J., Al-Noor, Z., & Gilbert, S. J. (2013). Autistic adolescents show
atypical activation of the brain's mentalizing system even without a prior history of
mentalizing problems. Neuropsychologia.
COMPUTER-ASSISTED IMPROVEMENTS IN ASD 30
Wolf, J. M., Tanaka, J. W., Klaiman, C., Cockburn, J., Herlihy, L., Brown, C., ... & Schultz, R.
T. (2008). Specific impairment of face‐processing abilities in children with autism
spectrum disorder using the Let's Face It! skills battery.Autism Research, 1(6), 329-340.
Xu, B., & Tanaka, J. W. (2014). Teaching Children with Autism to Recognize
Faces. Comprehensive Guide to Autism, 1043-1059.
Yamamoto, J., & Miya, T. (1999). Acquisition and transfer of sentence construction in
autistic students: Analysis of computer-based teaching. Research in Developmental
Disabilities, 20, 355-377.
Young, R. L., & Posselt, M. (2012). Using The Transporters DVD as a learning tool for children
with autism spectrum disorders (ASD). Journal of autism and developmental
disorders, 42(6), 984-991.
Zilbovicius, M., Saitovitch, A., Popa, T., Rechtman, E., Diamandis, L., Chabane, N., ... &
Boddaert, N. (2013). Autism, social cognition and superior temporal sulcus. Open
Journal of Psychiatry, 3, 46.
COMPUTER-ASSISTED IMPROVEMENTS IN ASD 31
Figure Captions
Figure 1. Screenshots from three games within FaceSay. a) “Amazing Gazing,” b) “Band Aid
Clinic,” c) “Follow the Leader.”
Figure 2. Post-intervention means for control and experimental groups.
COMPUTER-ASSISTED IMPROVEMENTS IN ASD 32
Figures
Figure 1a top
Figure 1b top
Figure 1c top
COMPUTER-ASSISTED IMPROVEMENTS IN ASD 33
Figure 2 top
*
COMPUTER-ASSISTED IMPROVEMENTS IN ASD 34
Tables
Table 1
Sample characterization means (standard deviations)
Measure FaceSay Control
N
16
15
Chronological age
7.68 (1.45)
7.87 (1.60)
Male:Female
16:0
12:3
IQ
104.8 (15.92)
98.53 (12.43)
SRS-2 65.18 (7.66) 65.40 (9.91)
Table 2
Correlations between difference scores for dependent variables for all groups
1. 2. 3. 4.
1. ΔAffect Recognition
2. ΔTheory of Mind 0.52**
3. ΔSRS-2 -0.53** -0.42*
4. ΔPositive Observations 0.21 0.01 -0.27
5. ΔNegative Observations -0.001 0.21 -0.29 -0.13
Note. *p<.05, ** p<.01.
Table 3
Means (standard deviations) of dependent variables pre- and post-intervention for all groups
Measure
FaceSay
Control
Pre Post Pre Post
Affect Recognition
8.63 (3.36) 12.56 (2.71)
8.73 (2.55) 8.53 (3.18)
Theory of Mind
15.38 (5.83) 21.63 (4.83)
14.80 (7.35) 16.60 (6.90)
SRS-2
65.19 (7.66)
62.25 (9.34)
65.40 (9.91)
67.80 (10.05)
Positive Observations
6.47 (3.73)
6.47 (4.37)
7.20 (3.45)
7.87 (3.53)
Negative Observations
1.00 (.93)
.56 (.95)
.80 (1.00)
.17 (.36)
