From the small screen to the big world: mobile apps for teaching real-world face recognition to children with autism

Abstract

In their everyday situations, individuals with autism spectrum disorder (ASD) encounter problems perceiving and understanding the facial expressions of others. If people with ASD have difficulties interpreting facial emotions, it is not surprising that they would struggle in their daily social interactions. An important question is whether facial emotion skills can be learned through systematic instruction and training. The accessibility, portability, and engagement of mobile devices (ie, smartphones, tablets) afford exciting new opportunities for creating innovative apps in emotional face training. In this article, we review the current crop of facial emotion apps for autism. We evaluate the apps according to the following criteria: face-processing skills, social attributes, and usability. We discuss the key ingredients of face-processing apps that will help a person on the autism spectrum make the transition from the small screen of the mobile device to the big world of real life.

Full text

© 2015 Sung et al. This work is published by Dove Medical Press Limited, and licensed under Creative Commons Attribution – Non Commercial (unported, v3.0)
License. The full terms of the License are available at http://creativecommons.org/licenses/by-nc/3.0/. Non-commercial uses of the work are permitted without any further
permission from Dove Medical Press Limited, provided the work is properly attributed. Permissions beyond the scope of the License are administered by Dove Medical Press Limited. Information on
how to request permission may be found at: http://www.dovepress.com/permissions.php
Advanced Health Care Technologies 2015:1 37–45
Advanced Health Care Technologies Dovepress
submit your manuscript | www.dovepress.com
Dovepress
37
REVIEW
open access to scientific and medical research
Open Access Full Text Article
http://dx.doi.org/10.2147/AHCT.S64483
From the small screen to the big world: mobile
apps for teaching real-world face recognition to
children with autism
AN Sung
A Bai
JG Bowen
B Xu
LM Bartlett
JC Sanchez
MD Chin
LJ Poirier
MR Blinkhorn
AC Campbell
JW Tanaka
Centre for Autism, Research,
Technology and Education (CARTE),
Department of Psychology, University
of Victoria, Victoria, BC, Canada
Abstract: In their everyday situations, individuals with autism spectrum disorder (ASD)
encounter problems perceiving and understanding the facial expressions of others. If people
with ASD have difficulties interpreting facial emotions, it is not surprising that they would
struggle in their daily social interactions. An important question is whether facial emotion skills
can be learned through systematic instruction and training. The accessibility, portability, and
engagement of mobile devices (ie, smartphones, tablets) afford exciting new opportunities for
creating innovative apps in emotional face training. In this article, we review the current crop
of facial emotion apps for autism. We evaluate the apps according to the following criteria:
face-processing skills, social attributes, and usability. We discuss the key ingredients of face-
processing apps that will help a person on the autism spectrum make the transition from the
small screen of the mobile device to the big world of real life.
Keywords: mobile apps, emotion, facial expression, development, social skills, gamification
Introduction
The human face is the gateway to our social world, revealing to others our momentary
thoughts, feelings, and intentions. Although most of us can interpret facial cues in
an instant, people with autism spectrum disorder (ASD) struggle to perceive facial
emotions and interpret the social cues conveyed by the face. An important question
is whether the skills required to read the social cues of a face can be learned through
systematic instruction and training. Mobile devices (eg, smartphones, tablets) with
their flexibility, adaptability, and customizability are promising tools for teaching the
social skills involved in face processing. For autism software developers, it is essential
that the social skills learned on the small screen of a mobile device transfer to the big
world of real life. In our review, we begin by discussing the social consequences of
face-processing deficits in ASD. We then examine the potential of mobile devices for
promoting face-processing skills and provide a systematic review of current mobile
apps that are intended to improve face-processing abilities. Finally, we discuss exciting
new innovations that will help children on the autism spectrum make the transition
from the small screen to the big world.
Challenge: improving social communication in
autism through face processing
The most recent statistics show that approximately one of every 68 children in the
United States are diagnosed with ASD.
1
According to the fifth edition of the Diagnostic
and Statistical Manual of Mental Disorders, individuals with ASD show repetitive
Correspondence: James Tanaka
Centre for Autism Research, Technology
and Education, Department of Psychology,
University of Victoria, Victoria, BC,
V8W 2Y2, Canada
Email jtanaka@uvic.ca
Number of times this article has been viewed
This article was published in the following Dove Press journal:
Advanced Health Care Technologies
23 October 2015

Advanced Health Care Technologies 2015:1
submit your manuscript | www.dovepress.com
Dovepress
Dovepress
38
Sung et al
patterns of behavior, display restricted interests, and have
difficulties interacting with others in a social context.
2
Although not a core symptom of the autism diagnosis, many
children with ASD are impaired in their ability to recognize
facial identity
3
and facial emotion.
4
Studies have shown that
persons with ASD are delayed in attending to faces
5–7
and
have difficulty recognizing facial identities
8–10
and interpret-
ing facial expressions.
11–13
Additionally, individuals on the
autism spectrum show deficits in their ability to maintain eye
contact and follow the eye gaze of others.
14–16
Difficulty in perceiving and understanding facial cues
in eye contact, gaze, and expression can lead to severe
problems in everyday social interactions. For example, people
who experience problems in their face-processing abilities
are found to rank higher in social anxiety
17
and along the
autism phenotype.
18–20
Face-processing deficits correlate
with social communication impairments in coordinating
social attention,
21–23
language development,
24–26
understand-
ing the thoughts of others,
18,27,28
and displaying empathy
with others.
29,30
Atypical patterns in face-processing strategies that are
evident in the first and second years of life can produce a cas-
cade of social difficulties later on in childhood, adolescence,
and adulthood. Young children on the autism spectrum have
problems initiating play with others and show an impaired
ability in maintaining peer friendships.
31,32
For adolescents
with ASD, social communication challenges usually intensify
as teen relationships take on greater social importance.
33
Although individuals with ASD may desire greater social
interaction,
34
they experience greater loneliness, social
rejection, and bullying than their peers.
32,35–37
For adults with
autism, persistent social communication deficits are evident
in their workforce participation.
38,39
While adults with ASD
have little difficulty completing job-related tasks, navigat-
ing the social aspect of the workplace proves overwhelming
for many people with ASD.
40
Unfortunately, the evidence
suggests that face and social communication abilities of
children with ASD do not improve over the course of devel-
opment
41
and may persist over the lifespan.
42,43
Technology, autism intervention, and
mobile devices
Computer-based technologies are promising tools for improv-
ing the social communication skills of individuals with
ASD. For example, several computerized training platforms
have been developed to teach face recognition abilities to
individuals with ASD.
44–46
Although positive training effects
were obtained in a laboratory context, the real test of these
interventions is whether the results transfer to improved social
communication in the child’s everyday life.
47
The generaliza-
tion of face-processing skills from the small screen to the big
world is paramount in training social communication skills
in ASD. Paradoxically, spending too much time in front of
computers has negative effects on the quality of real-life
social interactions in typically developing children.
48
The advent of smart mobile devices provides new learning
opportunities for children with ASD, especially in the training
of social interaction and communication skills. There are a
number of advantages of using mobile devices as assistive
tools. First, smartphones and tablets are easy and intuitive
to use. On a touch screen, the user can launch programs and
access program functions with a simple finger tap or swipe;
tactile motor actions are more direct than the mouse and
trackpad actions required for desktop and laptop computer
programs. More generally, touch screen devices follow the
principles of universal design that stress the importance
of making technological devices assessable to populations
with intellectual disabilities and sensory motor challenges.
49
Second, mobile apps are well suited to the learning needs of
the individuals with ASD because they are visually engag-
ing and provide a consistent, predictable learning environ-
ment.
50–53
Finally, mobile devices are portable. Desktop
computers, although suited to deliver facial recognition
programs, are stationary and socially isolating, while mobile
devices can fit into any individual’s pocket and be used any-
where and at any time. Children with ASD can readily carry
these devices and use them in everyday situations, including
the moments promoting real-life social interactions with their
peers. Mobile devices create new options and opportunities
for programmers to create innovative instructional content
that is appropriate for a diverse group of learners and diverse
learning styles.
54
What makes an effective learning
app?
Training or assistive apps will not be effective if the pro-
gram is cumbersome to use or is not engaging to the user.
An app may have been developed with a valid theoretical
rationale and yet still fail to deliver any benefits due to a
poor implementation. Usability is a critical design factor
for the success of tablet-based mobile apps. According to
the International Standards Organization, usability is the
extent to which a product can be used by the specified users
to achieve specified goals with effectiveness, efficiency, and
satisfaction in a specified context of use.
55
An app is effective
if users can complete the target tasks in the manner in which

Advanced Health Care Technologies 2015:1
submit your manuscript | www.dovepress.com
Dovepress
Dovepress
39
Autism apps
they were intended. Users may spend a great deal of time on
an app, but if they are not performing the critical tasks or if
they are performing them incorrectly, then the app will not
be effective in achieving its goal.
There are several approaches for evaluating the usability
of an app. One approach is to empirically evaluate the app’s
effectiveness through user testing. Following this method, naïve
users are asked to perform set functions on the app, while an
evaluator records his/her errors and problems and afterward,
interviews the users about their opinions and impressions of
the app. Following a heuristic evaluation approach, an inde-
pendent evaluator assesses the effectiveness and quality of a
program according to established indices of usability, such
as ease of use, number of errors, and the cognitive load of
the user.
56
Finally, apps can be assessed with respect to their
gamification qualities, such as their use of graphics and sounds;
design elements such as interface, levels, and badges; and
game-mechanics such as time constraints, limited resources,
and special powers. The immense popularity of app games
such as Candy Crush or Angry Birds demonstrates the efficacy
of game design principles for sustaining a user’s interest over
a long period of time. It is essential that gaming features are
incorporated into the development of academic and social apps
because these abilities require weeks, months and if not years
to acquire.
57
A recent meta-analysis indicates that gamification
techniques are effective in a wide variety of learning contexts
and can produce positive psychological, educational, and
behavioral outcomes.
58
Serious games are designed to promote
learning of specific life and academic skills that are particularly
challenging and less rewarding for participants. In contrast to
nonserious games, the central goal of serious games is that
game learning will translate to improved real-life outcomes.
59
In autism intervention, gamification and serious games provide
the key guidelines for creating better learning outcomes for the
child in a social skill that will improve his/her everyday life.
A survey of mobile apps in autism
face training
Computer developers with a keen interest in autism inter-
vention have recognized the tremendous potential of mobile
devices as powerful instructional tools. The result is that the
market has been flooded with autism-aimed apps of varying
qualities. Many of these apps are costly, so it is important to
establish which apps are worthwhile investments for parents
and educators. To this end, we present a review of apps
targeting face-processing skills in children with ASD.
The search for apps related to face processing in autism
was conducted between March 12 and March 26, 2015, and
utilized the Apple store, the Windows store, and the Google
Play store. Search terms were autism, ASD, faces, emotion,
face affect, and recognition. Such search terms were chosen
to identify apps specifically designed for individuals on the
autism spectrum as well as apps focusing on faces, emo-
tions, face affect, and face recognition, and to exclude apps
focusing on feelings, behavior regulation, self-monitoring,
and emotional regulation.
Exclusion criteria covered: 1) apps that focused on
domains other than face processing (eg, identifying internal
emotions in social situations with social stories); 2) apps
that focused on a single emotion; 3) apps that focused on
body language, such as hand gestures; 4) duplicates or older
versions of apps; 5) apps without gamification aspects; and
6) apps that relied solely on computer-generated avatars
rather than photographs of real people. The primary survey
returned hundreds of autism-related apps, but the exclusion
criteria removed the majority. The final survey contained
13 apps that met the inclusion criteria.
Each app was evaluated with respect to: 1) face- processing
skills; 2) social attributes; and 3) usability characteristics.
Face-processing skills were examined under recognition
of facial expressions, identification of people by faces,
and/or attention to the eyes. Social attributes were denoted
by extent to which each app incorporated content from the
child’s everyday life (eg, people, pets, toys) and enabled
participation of multiple players. Usability characteristics
included playability and gamification, indexed by whether
each app contained a point and reward system and whether it
allowed the user to customize sounds, graphics, and avatars
(refer to Table 1 for a summary of the apps).
App reviews
Autimo (iOS) – Lite (free) or Upgrade fee
Developer: Auticiel; age group: 6–10 years; Autimo is an app
utilizing three multilevel games to help users learn the six
basic facial expressions (happy, sad, angry, fearful, surprised,
and disgusted). Games use both audio and text instructions,
and present real pictures of the emotions. Activities include
a matching expression game (Pairs), selecting the odd
expression activity (Intruder), and a riddle game (Riddle).
This highly customizable app allows user to add additional
pictures, record their own voices, and select unique reward
animations. Feedback is presented in a variety of detailed
charts and graphs and automatically records time spent
playing. In summary, Autimo contains unique games to
practice emotional recognition skills and allows for accurate
tracking of progress.

Advanced Health Care Technologies 2015:1
submit your manuscript | www.dovepress.com
Dovepress
Dovepress
40
Sung et al
Auticiel. (2014). Autimo (Version 3.3.7) [Mobile applica-
tion software]. Retrieved from: http://itunes.apple.com.
CopyMe (iOS) – free
Developer: Games Studio; age group: 9–11 years; CopyMe
is a facial emotion production app employing webcam and
emotion recognition software. Users produce six basic
facial expressions (happy, sad, surprised, angry, fearful, and
disgusted) to match a photo-prompt, and the app judges the
quality of the production. If the expression is correct, feed-
back is given and the user progresses to the next emotion.
If the emotion is not correctly made, the user can try again
or skip to the next expression. Within three levels, ie, basic,
medium, and hard, the user can increase the difficulty or
replay easier levels. The app is moderately difficult, with the
reviewers having significant difficulty producing acceptable
angry and disgusted expressions. With a simple layout and
use of emotion prompt pictures, the app remains accessible
to pre-readers. Additional control over in-app sounds and
music makes the app customizable to different sensory needs.
Although challenging, CopyMe is a unique app practicing
emotion production with real faces, use of one’s own face,
and real-time assessment.
Games Studio. (2014). CopyMe (Version 1.31) [Mobile
application software]. Retrieved from: http://www.gamestu-
dio.org/CopyMe/index.php.
Emotions 2 from I Can Do Apps
(iOS) – Purchase required
Developer: I Can Do Apps, LLC; age group: 4+ years; Emo-
tions 2 contains stock images of real people of different ages
and ethnicities making various facial expressions (calm, tired,
Encourage
peer
interaction
Small
screen
to big
world
Feedback Gamified Ease of
use
Language
accessibility
Custom
settings
Identity
training
Expression
training
Eye
training
Autimo
CopyMe
Expression
Emotions 2 from I
Can Do Apps
Face Read 2
Faces by Whys
Learning
Learning with
Rufus: Emotions
Let’s Learn
Emotions: Emotion
Recognition
LOOK AT ME
My Memory App
Names to Faces
Training Faces
Notes:
Social attributes Face processingUsability
, criterion not met.
, criterion partially met; , criterion met;
Table 1 Evaluation of mobile apps with respect to social attributes, usability, and face processing

Advanced Health Care Technologies 2015:1
submit your manuscript | www.dovepress.com
Dovepress
Dovepress
41
Autism apps
scared, proud, sick, excited, happy, bored, and angry). This
app works on identification of different facial expressions
through five levels: identifying pictures with emotions, iden-
tifying emotions with pictures, identifying pictures based on
scenarios and emotions, identifying pictures with labels based
on scenarios, and identifying pictures based on scenarios.
Voice-over feedback and encouragement is provided. At the
end of each level, limited text feedback is given.
I Can Do Apps, LLC. (2013). Emotions 2 from I Can Do
Apps (Version 1.3) [Mobile application software]. Retrieved
from: https://itunes.apple.com/ca/app/emotions-2-from-i-
can-do-apps/id661208169?mt=8.
Face Read 2 (iOS) – Lite (free) or
Purchase required
Developer: ColorsKit; age group: 6–10 years; Face Read
2 uses stock photographs matched with voice-prompted
emotion labels to train the recognition of six basic emo-
tional expressions (happy, sad, angry, fearful, surprised, and
disgusted). Although various achievements are awarded for
completing the game, specific feedback and analysis are
lacking. There is no customization or pause button within
the app. Agewise, content in Face Read 2 seems geared more
toward younger audiences. Although Face Read 2 uses vivid
animations and a variety of subjects, content and usability
are limited.
ColorsKit. (2014). Face Read 2 (Version 1.0.3) [Mobile
application software]. Retrieved from: http://itunes.apple.
com.
Faces by Whys Learning (iOS) – Purchase
required
Developer: Whys Learning; age group: 5 years and under;
Faces by Whys Learning teaches emotion labeling of illus-
trated faces or real faces of children, teenagers, and adults.
Faces are matched with one of two labels, and voice-over
allows the user to operate the app with limited reading.
Expressions practiced by the app include frustrated, happy,
surprised, relaxed, sad, worried, angry, thoughtful, silly, and
tired. Feedback on correct and incorrect answers is given
during game play, and a summary of learning is provided.
However, the app does not allow the user to correct mistakes
made during play. Overall, Faces by Whys Learning is simple,
easy to use, and can help users practice emotion recognition
skills across age groups.
Whys Learning. (2014). Faces by Whys Learning (Version
1.0) [Mobile application software]. Retrieved from: http://
www.whyslearning.com.
Learning with Rufus: Emotions
(iOS) – Purchase required
Developer: Rufus Robot, Inc.; age group: 6–8 years; Learn-
ing with Rufus uses stock images of people of varied age and
ethnicity using the six basic emotions (happy, sad, angry, fearful,
surprised, and disgusted). The app has a very simple layout with
only one correct emotion and incorrect emotion option. There is
a continuous reward system that is locomotive themed. A timed
doodling break is provided as reward between levels. Text
feedback and encouragement are given. Although the quality
of this app is medium to low, it lays out the basic emotions in
a very simple context with an emphasis on casual play.
Rufus Robot, Inc. (2014). Learning with Rufus: Emotions
(Version 1.0.4) [Mobile application software]. Retrieved
from: http://itunes.apple.com.
Let’s Learn Emotions: Emotion
Recognition (iOS) – Purchase required
Developer: Everyday Speech; age group: 6+ years; Let’s
Learn Emotions uses both stock and uploaded images of real
people making expressions from a variety of ages, ethnicities,
and sex. As the user is able to add custom content, faces with
any expression can be used. To teach emotion identification
and understanding, users can match faces to emotion labels
and use these pairings as flash cards. The discussion aspect
of the app integrates an emotion with an activity, such as
drawing an expression, making an expression, or utilizing an
expression in some way with a friend. Text feedback is given
for all the language components, and app graphics are of high
quality. The integration of social environments, customizable
settings, and photos makes this app educational and fun.
Everyday Speech. (2014). Let’s Learn Emotions: Emotion
Recognition (Version 1.2.1) [Mobile application software].
Retrieved from: https://itunes.apple.com/ca/app/lets-learn-
emotions-emotion/id908762349?mt=8.
LOOK AT ME (Android) – free
Developer: Samsung; age group: 4+ years; LOOK AT ME
is an eye contact and facial expression training program
built around seven missions. There are four different theme
choices with customizable music and voice preferences,
and play is encouraged through character cards, rubies,
and points. Within the game, there are activities of learn-
ing, labeling, identifying, and describing emotions as well
as interactive games. These games can be played with the
user’s own pictures to integrate his/her social environment.
Additionally, any expression can be introduced, making the
app flexible and customizable. App graphics are very good

Advanced Health Care Technologies 2015:1
submit your manuscript | www.dovepress.com
Dovepress
Dovepress
42
Sung et al
and feedback follows immediately. Although this program
is creative, customizable, and interactive for the child, the
app is prone to crashes and is reportedly unable to open on
certain Android devices.
Samsung LOOK AT ME (2014). LOOK AT ME (Version
3.12.20141219) [Mobile application software]. Retrieved
from: https://play.google.com/store/apps/details?id=com.
samsung.lookatme&hl=en.
My Memory App (iOS) – Purchase
required
Developer: Elliot Buczek, WIDTOT Software Solutions;
age group: 3+ years; My Memory App allows users to take
their own photos and videos to use as content in the games.
Owing to the customizable nature, various facial identities
and expressions can be practiced in this game. This is a card
matching game that is accompanied by voice-over and text
of the expressions or identities shown. Users take images or
videos of people and use these images and videos as cards.
Text feedback of correct pairs is shown throughout the game.
The simple layout of this app makes usability easy, though
an esthetic engagement is arguably lacking. The customiz-
ability of the photos/videos and reward system makes this
app distinct. Adding personal photos and videos into the app
extends play into peer interactions.
WIDTOT Software Solutions. (2015). My Memory App
(Version 7.0) [Mobile application software]. Retrieved
from: https://itunes.apple.com/us/app/my-memory-app/
id937370829?mt=8.
Names to Faces (iOS) – Purchase required
Developer: Paul Hudson; age group: 6+ years; Names to Faces
integrates personalized content by having users collect own
photographs of family and friends. Users then sort through their
collection of faces and practice memorizing the names in each
image. This app works on identity recognition, but does not pro-
vide feedback for correct or incorrect choices. It is a simple app
with only camera and labeling functions. The main attraction
of this app is its applicability to real life using familiar faces,
in an attempt to improve the users’ social environment.
Paul Hudson. (2013). Names to Faces (Version 1.0)
[Mobile application software]. Retrieved from: http://itunes.
apple.com.
Training Faces (iOS/Android) – Purchase
required
Developer: Training With Gaming Inc.; age group: 4+ years;
Training Faces features stock images of real people of different
ages and ethnicities making various facial expressions (happy,
excited, tired, sad, angry, confused, afraid, sick, and silly). The
app works on identification of emotional expressions within a
given social context story with varying difficulty levels. The
app employs a train theme where the user identifies train pas-
sengers who display a given target emotion. Text feedback and
encouragement are given at the end of each level. App graphics
are of low-to-medium quality; however, the train theme, use of
social context, and level selection make this app unique.
Training With Gaming Inc. (2012). Training Faces
[Mobile application software]. Retrieved from: https://itunes.
apple.com/ca/app/training-faces/id522989729?mt=8.
Summary of face-processing apps
As shown in Table 1, a wide array of mobile apps teaching
face-processing skills to children on the autism spectrum is
available. To follow, we summarized the main findings of
our evaluation with respect to face-processing skills, social
attributes, and usability characteristics.
Face-processing skills
The vast majority of apps focused on developing facial
expression recognition through naming expressions (eg, Face
Read 2), constructing expressions (eg, Expressions for
Autism, Let’s Learn Emotions), or identifying them in
social scenarios (Emotions 2 and Training Faces). Two of
the reviewed apps targeted the recognition of facial identity
by matching personalized faces to written names (Names to
Faces) or text-to-voice (My Memory App). The LOOK AT
ME app was the only app directly promoting eye contact by
having the player identify a small face embedded in the iris
region of the eye within a larger face.
Most of the facial expression training apps supply a
stock of expressive faces, and the child’s task is to identify a
target expression (such as a smile as happy) across different
people. Some apps, not reviewed here, use avatars or cartoons
to simulate expressions; however, avatars and cartoons can
only approximate real facial expressions. Cartoon faces are
more focused on individual parts, such as the eyes, nose, and
mouth, and are void of more detailed facial features, such as
lines in the corners of a person’s eyes when he/she smiles.
Transfer from the small screen to the big world may be limited
as learning facial cues from avatars and cartoons may not
improve social communication with real-life faces. Neverthe-
less, many apps make sure to incorporate real individuals,
with faces from differing ethnicities and ages emanating
a wide range of emotions. This diversity is important as
children with autism have difficulties recognizing the same

Advanced Health Care Technologies 2015:1
submit your manuscript | www.dovepress.com
Dovepress
Dovepress
43
Autism apps
emotion across multiple individuals.
4
In some apps, labeling
of facial expressions was limited to the six basic emotions,
such as happy, sad, angry, fearful, disgusted, and surprised,
60
while other apps further built on more complex emotions
such as tired and thoughtful.
Few apps involved training identity recognition and atten-
tion to eyes. In the current review, the apps addressing identity
recognition presented faces in scrapbooks that the user labeled
with names. Children with ASD have difficulty establishing
eye contact with others and interpreting their gaze cues,
skills that are essential to everyday social interactions.
3,5,61–63
However, in the current review, only one app provided specific
instruction to eye-gaze training, suggesting that this is a rich
area of remediation for future app developers.
Social attributes
Several of the apps addressed the small screen to the big world
challenge by incorporating user-generated content into the
games. For example, in Autimo, LOOK AT ME, and Names to
Faces, personalized photos can be featured within the app using
the built-in camera functions of mobile devices. In Let’s Learn
Emotions, players have the capacity to upload images from a
personalized photo library. A small number of the apps evaluated
in this article encouraged the production of user-generated con-
tent, while the rest provided generic stock photographs. Although
the majority of apps are single-player games, LOOK AT ME
and Names to Faces have multiplayer modes, thereby promoting
group social interactions among game players.
Another key feature for future face-training apps is
the user’s own production of emotional facial expressions.
Production of emotions has been demonstrated to improve
perceptual recognition of expressions;
64
however, this impor-
tant avenue for app development is certainly in its infancy.
While identification of facial expressions is a key goal in
current emotion learning apps, only a few focus on the actual
production element. Intelligently exploiting the use of face
recognition software, CopyMe is the sole app in our review,
focusing on production of emotional expressions. However
when evaluating CopyMe in this review, we found that some
targeted expressions were almost impossible to emulate,
which may act as a source of frustration for children with
autism. As computerized emotion recognition technology
advances, promoting facial expression production is likely
to become more readily available in app programs.
Usability characteristics
The reviewed apps exhibited varying degrees of efficiency,
satisfaction, and gamif ication. Existing face-training
apps available for download varied greatly in quality and
production value. Some of the apps evaluated were not
always captivating games to play. Many involved repetitive,
simple game elements and did not provide many challenges
or incentives to continue playing. Not all apps provided
real-time feedback for correct and incorrect answers. Apps
should indeed provide a learning experience where users
instantly make connections between their responses and
expected outcomes. Users should immediately see their
successes.
User testing methods were not conducted in this specific
review, though future efforts would be insightful. We chose
not to apply numeric user satisfaction scores to review face-
processing apps as we did not recruit child raters. Although a
detailed usability analysis of each app was beyond the scope
of the current paper, applying a rigorous user testing method
to assess the face-processing apps would be informative for
identifying their specific strengths and weaknesses.
Conclusion
The stage is set for the development of robust next- generation
face-training apps. Future developers should make use of
the growing literature on gamification and serious games
that promote the transfer of social skills in autism from the
small screen to the big world.
59
In addition to advances on
the theoretical front, there are several technological advance-
ments that face-training apps can exploit. Emotion recogni-
tion software, such as the type found in CopyMe, allows for
real-time tracking of facial expressions. Automatic expres-
sion recognition software has been successfully applied for
teaching children with autism spectrum how to produce
more readable facial emotions.
65
Automatic recognition
software opens the door for facial expression training apps
that link facial emotions and everyday social situations. It
may also be possible to implement eye-tracking software
to track eye-gaze patterns and incorporate them into new
game-mechanics.
Another promising trend in mobile devices is the advance
of user-generated content in which the player takes photos
and videos with the device’s camera and incorporates the
personally authored images into the games. User-driven apps
are especially powerful methods for teaching everyday face-
processing skills where the images of people from the child’s
everyday life (eg, friends, siblings, teachers, parents, etc)
are integrated into the intervention program. Thus, the child
learns to identify a smile or angry expression of a familiar
person, such as his/her mother or best friend, rather than the
expression of an unfamiliar stranger.

Advanced Health Care Technologies 2015:1
submit your manuscript | www.dovepress.com
Dovepress
Dovepress
44
Sung et al
Now is an exciting time to be developing mobile face-
processing apps that aim to improve social communication in
autism. There is a developing literature on design principles for
serious games that is readily accessible, and emerging affective
computing technologies are opening the door for virtual reality
applications and more robust interventions. So far, we have seen
only a very small degree of this potential tapped into existing
apps. We leave it to informed developers to take these theoretical
and technological resources and create robust next-generation
face-training apps. The resources and technologies are avail-
able; the challenge is for creative and innovative developers to
put their skills to the purpose of serving the autism community.
Innovative apps that succeed in following these recommenda-
tions, it is hoped, will train useful face-processing skills that
make the transition from the small screen of the mobile device
to the big world of everyday social life.
Acknowledgment
This work was supported by a treatment grant from Autism
Speaks awarded to JWT.
Disclosure
The authors report no conflicts of interest in this work.
References
1. Baio J. Prevalence of autism spectrum disorder among children aged 8
years – autism and developmental disabilities monitoring network, 11
sites, United States, 2010. MMWR Surveill Summ. 2014;63:1–21.
2. American Psychiatric Association. Diagnostic and Statistical Manual
of Mental Disorders. 5th ed. Arlington, VA: American Psychiatric
Association; 2013.
3. Wolf JM, Tanaka JW, Klaiman C, et al. Specific impairment of face-
processing abilities in children with autism spectrum disorder using the
Let’s Face It! skills battery. Autism Res. 2008;1:329–340.
4. Tanaka JW, Wolf JM, Klaiman C, et al. The perception and identification
of facial emotions in individuals with autism spectrum disorders using
the Let’s Face It! Emotion Skills Battery. J Child Psychol Psychiatry.
2012;53:1259–1267.
5. Klin A, Jones W, Schultz R, Volkmar F, Cohen D. Visual fixation
patterns during viewing of naturalistic social situations as predictors
of social competence in individuals with autism. Arch Gen Psychiatry.
2002;59:809–816.
6. Osterling JA, Dawson G, Munson JA. Early recognition of 1-year-old
infants with autism spectrum disorder versus mental retardation. Dev
Psychopathol. 2002;14:239–251.
7. Volkmar F, Chawarska K, Klin A. Autism in infancy and early
childhood. Annu Rev Psychol. 2005;56:315–336.
8. Boucher J, Lewis V. Unfamiliar face recognition in relatively able
autistic children. J Child Psychol Psychiatry. 1992;33:843–859.
9. Dawson G, Webb SJ, McPartland J. Understanding the nature of
face processing impairment in autism: Insights from behavioral and
electrophysiological studies. Dev Neuropsychol. 2005;27:403–424.
10. Weigelt S, Koldewyn K, Kanwisher N. Face identity recognition in
autism spectrum disorders: a review of behavioral studies. Neurosci
Biobehav Rev. 2012;36:1060–1084.
11. Celani G, Battacchi MW, Arcidiacono L. The understanding of
emotional meaning of facial expressions in people with autism. J Autism
Dev Disord. 1999;29:57–66.
12. Golan O, Baron-Cohen S, Golan Y. The ‘reading the mind in films’
task [child version]: complex emotion and mental state recognition in
children with and without autism spectrum conditions. J Autism Dev
Disord. 2008;38:1534–1541.
13. Gross TF. The perception of four basic emotions in human and
nonhuman faces by children with autism and other developmental
disabilities. J Abnorm Child Psychol. 2004;32:469–480.
14. Bedford R, Elsabbagh M, Gliga T, et al. Precursors to social and com-
munication difficulties in infants at-risk for autism: gaze following and
attentional engagement. J Autism Dev Disord. 2012;42:2208–2218.
15. Freeth M, Foulsham T, Chapman P. The influence of visual saliency
on fixation patterns in individuals with autism spectrum disorders.
Neuropsychologia. 2011;49:156–160.
16. Riby DM, Hancock PJB. Looking at movies and cartoons: eye-tracking
evidence from Williams syndrome and autism. J Intellect Disabil Res.
2009;53:169–181.
17. Davis JM, McKone E, Dennett H, O’Connor KB, O’Kearney R, Palermo R.
Individual differences in the ability to recognise facial identity are
associated with social anxiety. PLoS One. 2011;6(12):e28800.
18. Baron-Cohen S, Wheelwright S, Hill J, Raste Y, Plumb I. The “reading
the mind in the eyes” test revised version: a study with normal adults,
and adults with Asperger syndrome or high-functioning autism. J Child
Psychol Psychiatry. 2001;42:241–251.
19. Halliday DWR, MacDonald SWS, Sherf SK, Tanaka JW. A reciprocal
model of face recognition and autistic traits: evidence from an individual
differences perspective. PLoS One. 2014;9:e94013. doi: 10.1371/ -
journal.pone.0094013.
20. Wilson CE, Freeman P, Brock J, Burton AM, Palermo R. Facial identity
recognition in the broader autism phenotype. PLoS One. 2010;5(9):1–7.
doi: 10.1371/journal.pone.0012876.
21. Mundy P, Sigman M, Ungerer J, Sherman T. Defining the social deficits
of autism: the contribution of non-verbal communication measures.
J Child Psychol Psychiatry. 1986;27:657–669.
22. Dawson G, Toth K, Abbott R, et al. Early social attention impairments
in autism: social orienting, joint attention, and attention to distress. Dev
Psychol. 2004;40:271–283.
23. Warreyn P, Roeyers H, Oelbrandt T, De Groote I. What are you looking
at? Joint attention and visual perspective taking in young children with
autism spectrum disorder. J Dev Phys Disabil. 2005;17:55–73.
24. Droucker D, Curtin S, Vouloumanos A. Linking infant-directed speech
and face preferences to language outcomes in infants at risk for autism
spectrum disorder. J Speech Lang Hear Res. 2013;56:567–576.
25. Eisenmajer R, Prior M, Leekam S, et al. Delayed language onset as a
predictor of clinical symptoms in pervasive developmental disorders.
J Autism Dev Disord. 1998;28:527–533.
26. Stagg SD, Davis R, Heaton P. Associations between language
development and skin conductance responses to faces and eye gaze
in children with autism spectrum disorder. J Autism Dev Disord.
2013;43:2303–2311.
27. Campbell R, Lawrence K, Mandy W, Mitra C, Jeyakuma L, Skuse D.
Meanings in motion and faces: developmental associations between the
processing of intention from geometrical animations and gaze detection
accuracy. Dev Psychopathol. 2006;18:99–118.
28. von dem Hagen EAH, Stoyanova RS, Rowe JB, Baron-Cohen S, Calder AJ.
Direct gaze elicits atypical activation of the theory-of-mind network in
autism spectrum conditions. Cereb Cortex. 2014;24: 1485–1492.
29. Golan O, Baron-Cohen S. Systemizing empathy: teaching adults with
Asperger syndrome or high-functioning autism to recognize complex
emotions using interactive multimedia. Dev Psychopathol. 2006;18:
591–617.
30. Clarke TF, Winkielman P, McIntosh DN. Autism and the extraction of
emotion from briefly presented facial expressions: stumbling at the first
step of empathy. Emotion. 2008;8:803–809.
31. Carter AS, Davis NO, Klin A, Volkmar FR. Social development in
autism. In: Volkmar FR, Paul R, Klin A, Cohen D, editors. Handbook
of Autism and Pervasive Developmental Disorders: Vol 1. Diagnosis,
Development, Neurobiology, and Behaviour. Hoboken, NJ: John Wiley
& Sons; 2005:312–334.

Advanced Health Care Technologies
Publish your work in this journal
Submit your manuscript here: http://www.dovepress.com/advanced-health-care-technologies-journal
Advanced Health Care Technologies is an international, peer reviewed,
open access journal that provides a unique forum for articles on:
point-of-care, health care diagnostics and treatment, bioengineer-
ing, biotechnology, biosensing, electronics, clinical/medical science,
chemical engineering, materials science, regenerative medicine, micro-/
nano-technologies, and methods and applications for nanoscience and
nanotechnology. The manuscript management system is completely
online and includes a very quick and fair peer review system, which
is all easy to use. Visit http://www.dovepress.com/testimonials.php to
read real quotes from published authors.
Advanced Health Care Technologies 2015:1
submit your manuscript | www.dovepress.com
Dovepress
Dovepress
Dovepress
45
Autism apps
32. Kasari C, Locke J, Gulsrud A, Rotheram-Fuller E. Social networks
and friendships at school: comparing children with and without ASD.
J Autism Dev Disord. 2011;41:533–544.
33. Williams White S, Keonig K, Scahill L. Social skills development
in children with autism spectrum disorders. J Autism Dev Disord.
2007;37:1858–1868.
34. Bauminger N, Kasari C. Loneliness and friendship in high-functioning
children with autism. Child Dev. 2000;71:447–456.
35. Cappadocia MC, Weiss JA, Pepler D. Bullying experiences among chil-
dren and youth with autism spectrum disorders. J Autism Dev Disord.
2011;42:266–277.
36. Humphrey N, Lewis S. Make me normal: the views and experiences
of pupils on the autistic spectrum in mainstream secondary schools.
Autism. 2008;12:23–46.
37. Locke J, Rotheram-Fuller E, Kasari C. Exploring the social impact of
being a typical peer model for included children with autism spectrum
disorder. J Autism Dev Disord. 2012;42:1895–1905.
38. Szatmari P, Bartolucci G, Bremmer R. Asperger’s syndrome and autism:
comparison of early history and outcome. Dev Med Child Neurol.
1989;31:709–720.
39. Venter A, Lord C, Schopler E. A follow-up study of high-functioning
autistic children. J Child Psychol Psychiatry. 1992;33:489–507.
40. Simone R. Asperger’s on the Job. Arlington, TX: Future Horizons;
2010.
41. Rao PA, Beidel DC, Murray MJ. Social skills interventions for children
with Asperger’s syndrome or high functioning autism: a review and
recommendations. J Autism Dev Disord. 2008;38:353–361.
42. Schultz RT. Developmental deficits in social perception in autism:
the role of the amygdala and fusiform face area. Int J Dev Neurosci.
2005;23:125–141.
43. Sasson NJ. The development of face processing in autism. J Autism
Dev Disord. 2006;36:381–394.
44. Faja S, Aylward E, Bernier R, Dawson G. Becoming a face expert: a
computerized face-training program for high-functioning individuals
with autism spectrum disorders. Dev Neuropsychol. 2007;33:1–24.
45. Rice LM, Wall CA, Fogel A, Shic F. Computer-assisted face processing
instruction improves emotion recognition, mentalizing, and social skills
in students with ASD. J Autism Dev Disord. 2015:1–11.
46. Tanaka JW, Wolf JM, Klaiman C, et al. Using computerized games to
teach face recognition skills to children with autism spectrum disorder:
the Let’s Face It! program. J Child Psychol Psychiatry. 2010;51:
944–952.
47. Hopkins IM, Gower MW, Perez TA, et al. Avatar assistant: improving
social skills in students with an ASD through a computer-based inter-
vention. J Autism Dev Disord. 2011;41:1543–1555.
48. Pea R, Nass C, Meheula L, et al. Media use, face-to-face communication,
media multitasking, and social well-being among 8-to 12-year-old girls.
Dev Psychol. 2012;48:327–336.
49. Fager J. Autism and the iPAD [60 Minutes]. New York: CBS; 2011.
50. Althaus M, de Sonneville LM, Minderaa RB, Hensen LG, Til RB.
Information processing and aspects of visual attention in children
with the DSM-III-R diagnosis “Pervasive Developmental Disorder
Not Otherwise Specified” (PDDNOS): II. Sustained attention. Child
Neuropsychol. 1996;2:17–29.
51. Nally B, Houlton B, Ralph S. The management of television and video
by parents of children with autism. Autism. 2000;4:331–337.
52. Shane HC, Albert PD. Electronic screen media for persons with
autism spectrum disorders: results of a survey. J Autism Dev Disord.
2008;38:1499–1508.
53. Yee HSS. Mobile technology for children with autism spectrum
disorder: major trends and issues. IEEE Symposium on E-learning,
E-Management and E-Services. Washington, DC: IEEE; 2012:1–5.
54. Rose D, Meyer A. Teaching Every Student in the Digital Age: Universal
Design for Learning. Alexandria, VA: ASCD; 2002.
55. International Standards Organization. ISO 9241-11: Ergonomic
Requirements for Office Work with Visual Display Terminals (VDTs)
s-Part II: Guidance on Usability. Geneva: International Standards
Organization; 1998.
56. Nielsen J, Molich R. Heuristic evaluation of user interfaces. Paper presented
at: Proc. ACM CHI’90 Conf; April 1–5, 1990:249–256. Seattle, WA.
57. Deterding S, Dixon D, Khaled R, Nacke L. From game design elements
to gamefulness: defining “gamification”. Paper presented at: MindTrek
‘11 Proceedings of the 15th International Academic MindTrek Confer-
ence: Envisioning Future Media Environments. New York, NY: ACM;
2011:9–15.
58. Hamari J, Koivisto J, Sarsa H. Does gamification work? – A litera-
ture review of empirical studies on gamification. Paper presented at:
“9241-11 Ergonomic Requirements for Office Work with Visual Display
Terminals (VDTs)-Part II Guidance on Usability”; 2014: 3025–3034;
Washington, DC. [ISO/IEC 9241-11, 1998 (E)].
59. Whyte EM, Smyth JM, Scherf KS. Designing serious game interven-
tions for individuals with autism. J Autism Dev Disord. 2014;1–12.
60. Ekman P, Friesen W. Constants across cultures in the face and emotion.
J Pers Soc Psychol. 1971;17:124–129.
61. Joseph RM, Tanaka J. Holistic and part-based face recognition in
children with autism. J Child Psychol Psychiatry. 2003;44:529–542.
62. Tanaka JW, Sung AN. The “Eye Avoidance” hypothesis of face
processing in autism. J Autism Dev Disord. 2013:1–15.
63. Xu, B, Tanaka, JW. Teaching children with autism to recognize faces.
In Patel V, Preedy V, Martin C, editors. Comprehensive guide to autism.
New York: Springer; 2014:1043–1059.
64. Deriso D, Susskind J, Tanaka J, et al. Exploring the facial expression
perception-production link using real-time automated facial expression
recognition. In: Fitzgibbon A, Lazebnik S, Sato Y, Schmid C, editors.
Lecture Notes in Computer Science: European Conference on Computer
Vision, Workshop on What’s in a Face. Berlin, Heidelberg: Springer;
2012:270–279.
65. Gordon I, Bartlett M, Pierce MS, Tanaka JW. Training voluntary facial
expressions using automated, real-time feedback. J Autism Dev Disord.
2014;44:2486–2498.