Remember to smile – design of a mobile affective technology to help promote individual happiness through smiling

Abstract

Wellbeing plays a central role in quality of life and encompasses aspects pertaining to mental and social wellbeing, as well as physical wellbeing and the absence of disease. Building upon the natural human understanding that smiling is an expression of happiness, studies have shown that the process of smiling in a genuine manner can help to improve an individual's happiness. To date, approaches that measure happiness have relied upon subjective self-assessment using one of a wide range of questionnaires. However, more recently, Affective Technology has emerged that provides the potential to move towards a more objective assessment of happiness. This paper describes a proposed study aimed at evaluating a bespoke smartphone-based affective technology that attempts to promote happiness through smiling, by reminding participants to smile on a regular basis.

Full text

Remember to smile – design of a mobile affective
technology to help promote individual happiness through
smiling
George Moore
Ulster University
Belfast, Northern Ireland
g.moore@ulster.ac.uk
orcid.org/0000-0003-1623-0014
Leo Galway
Ulster University
Belfast, Northern Ireland
l.galway@ulster.ac.uk
orcid.org/0000-0002-1881-1501
Mark Donnelly
Ulster University
Belfast, Northern Ireland
mp.donnelly@ulster.ac.uk
orcid.org/0000-0003-1250-265X
ABSTRACT
Wellbeing plays a central role in quality of life
and encompasses aspects pertaining to mental and
social wellbeing, as well as physical wellbeing
and the absence of disease. Building upon the
natural human understanding that smiling is an
expression of happiness, studies have shown that
the process of smiling in a genuine manner can
help to improve an individual’s happiness. To
date, approaches that measure happiness have
relied upon subjective self-assessment using one
of a wide range of questionnaires. However, more
recently, Affective Technology has emerged that
provides the potential to move towards a more
objective assessment of happiness. This paper
describes a proposed study aimed at evaluating a
bespoke smartphone-based affective technology
that attempts to promote happiness through
smiling, by reminding participants to smile on a
regular basis.
Author Keywords
Affective Computing; Happiness; Human
Computer Interaction; Mental Wellbeing; Mobile
Health; Mobile Technology; Positive Computing;
Positive Psychology; Reminding; Smiling.
ACM Classification Keywords
H.1.2 User/Machine Systems, H.5.2 User
Interfaces, I.2.10 Vision and Scene
Understanding, I.4.9 Applications, J.3 Life and
Medical Sciences, J.4 Social and Behavioral
Sciences.
INTRODUCTION
Internationally, governments have long
recognised the importance of maintaining and
promoting wellbeing and quality of life of among
their citizens towards: increasing economic
benefits, improving productivity and reducing
demands on health care. More recently, this
awareness has broadened to include additional
dimensions of wellbeing such as mental and
social wellbeing [7]. This is, in part, prompted
and supported by the World Health
Organization’s longstanding inclusion of mental
and social wellbeing as key indicators of health,
physical wellbeing and the absence of disease
[20].
At an individual level, we all share a fundamental
desire to be happy, although perhaps while not
always fully understanding the wellbeing benefits
that this can bring. Within this natural
understanding of the value of happiness, smiling
has always been viewed as an easily identifiable
indicator of an individual’s happiness.
Accordingly, there exists a long established
scientific understanding of the reciprocal
relationship between the two, with smiling being
understood to help improve an individual’s
happiness through facial feedback mechanisms
[15, 11]. An initial reaction to the understanding
that smiling can improve happiness could simply
Author prepared copy. Paper published in the
proceedings of Health-i-Coach, Intelligent
Technologies for Coaching in Health, workshop
at ACM 11th EAI International Conference on
Pervasive Computing Technologies for
Healthcare, Barcelona, Spain, on May 23-26
2017.
Project updates:
https://www.researchgate.net/project/soHappy

be to encourage people to smile more. However,
a more complex relationship exists between
smiling and happiness. Genuine and forced smiles
are processed through different pathways within
our brains [18], and faking a smile can lead to
emotional dissonance, with links to depression
[9]. As such, care must be taken to encourage
genuine smiles when attempting to improve
happiness. Given the growing popular interest in
happiness and rising public awareness of the
potential benefits for emotional wellbeing, there
have been a number of lifestyle initiatives aimed
at promoting more positive attitudes, such as
Action for Happiness [1]. There has also been an
increase in the use of technology to promote
awareness of happiness or even attempt to
improve individual happiness, a notable example
being GPS for the Soul [6]. Moreover, it has been
shown that the use of smartphone applications to
deliver interventions aimed at promoting
happiness can be effective [10].
The overarching goal of the research outlined in
this paper is to explore the potential role of
affective technology in attempting to improve the
happiness of individuals within society through
the medium of a dedicated positive technology
[4]. In an effort towards this goal, the following
hypothesis has been posed, “Individual happiness
levels can be improved following regular use of a
smartphone application that has been designed to
encourage genuine smiling on a regular basis.”
BACKGROUND
To date, approaches to measuring and quantifying
happiness, at a societal and individual level, have
been predominantly subjective in nature, paper-
based, and differing in format due to their specific
definition of happiness [13]. More recently,
technology advances in the field of Affective
Computing have demonstrated the potential for a
more objective method of measuring emotion
[14,12,16]. Underlying the ability of such
technology to classify emotion through facial
expression is the well understood and established
Facial Action Coding System (FACS) [5]. These
technologies typically support such underlying
approaches with facial feature tracking, skin
detection, data analytics and machine learning
techniques in order to help improve robustness
and accuracy. The use of much of this resulting
technology has initially been focused on
analysing and enhancing the effectiveness of
advertising, through the measurement of
engagement and emotional response [17].
However, there is potential to use this technology
within a dedicated Positive Computing approach
in support of mental wellbeing. That is, to
produce “a technology that is purpose built and
dedicated to fostering wellbeing and/or human
potential in some way” [4]. While there has been
growth in the use of smartphone applications to
support activities aimed at improving happiness,
notably journaling, meditation and mindfulness,
these generally still rely on subjective reflection
prompted by the technology rather than the use of
Affective Computing in a more meaningful way.
Approaches that try to make use of more
objective Affective Computing techniques, such
as HappinessCounter [19], are much less
common, as are technologies that take a dedicated
Positive Computing approach, such as that
proposed in this study.
METHODOLOGY
The proposed study aims to evaluate the efficacy
of a smartphone-based affective technology for
improving participant happiness through genuine
smiling on a regular basis.
Experimental design
This initial study will employ a quasi-
experimental one group pretest-posttest design to
facilitate early evaluation of the hypothesis and of
the corresponding experimental setup;
specifically, the data collected and smartphone
application’s functionality and user journey.
However, we have taken steps to improve the
protocol’s internal validity through the collection
of additional data proximate to each intervention.
Given this approach, it should be possible to
establish if there is any potential causal link that
merits testing with a larger, more controlled,
experiment. The study will involve a group of 45
healthy adults, recruited from a population of
university staff and students, who will use the
smartphone application four times per day over
the course of a 30 day period.

Pre-test design
At the outset of the study, a self-reported
happiness level will be recorded using the
Subjective Happiness Scale [13]. This will form
the baseline happiness assessment against which
the participant’s post-study happiness levels will
be assessed. Demographic data, specifically
gender and age group, will also be collected in
order to help facilitate subsequent participant
modelling.
Main study design
During the study, participants will be reminded to
access the application on their personal
smartphones, to reflect on a stimulus that may
encourage them to smile and to allow themselves
to smile for approximately 30 seconds. It is this
smile that will be analysed using the Affective
Computing component of the smartphone
application. Personalised reminders will be issued
in order to encourage participants to engage with
the application on a regular basis. Additional data
relating to contextual circumstances that are
proximate to each session will also be collected in
order to gain further insight into contextual
circumstances that might have impacted on the
nature of the participant’s smile and engagement
with the application. Participants will also
complete a single-item happiness scale [2] during
each session, to help model ongoing changes in
reported happiness levels.
Post-test design
At the conclusion of the study period participants
will once again complete the Subjective
Happiness Scale. This post-test assessment will
be compared to the baseline measurement, taken
at the outset of the study, in order to determine
any significant change in reported happiness
levels that may be as a result of having engaged
in the experiment. Additionally, participants will
complete a post-study usability assessment of the
smartphone application using the System
Usability Scale [3]. This will help to determine if
any usability factors may have effected the
results. Moreover, it will help to inform the
design of future iterations of the smartphone
application.
STUDY DELIVERY
The study to be delivered through the soHappy
affective technology, which manages all aspects
of pre-test, intervention and post-test experiment
delivery and data recording, as illustrated in
Figure 1. Reminders will be issued as push
notifications to promote participant engagement
in the study. Participants will be able to
personalise the time of the reminders, if desired,
to help to promote adherence. The Video Masker
component partially obscures the live video feed
from the smartphone’s front-facing camera. This
was included as it was felt that participants might
feel self-conscious smiling at their own image,
particularly as application engagement may, at
times, take place in a public setting. However,
fully obscuring the camera feed was not an option
as the participant will need to be able to frame
their face within the camera’s field of view in
order to facilitate affect analysis. The Affect
Analyser makes use of Affectiva’s Affdex
affective computing software development kit
[14] to analyse the video feed and determine the
probability of a given expression and emotion
being present, as well as recording the
corresponding set of facial landmarks. In
Figure 1: soHappy affective technology system architecture.

addition, the Affect Analyser also provides
feedback to the participant on how successful the
smile detection process is progressing during
smile acquisition. The Forms Manager delivers
the pre-test, post-test and engagement session
questionnaire-style question and response items.
The application maintains a Client-side database
of all data recorded during each session, along
with a Client-side Database Management System
(DBMS) that manages data transfer to the main
server-sided database on completion of each
engagement session. A Server-side DBMS will
interact with the Client-side DBMS to undertake
the transfer of study session data, as well as
facilitating researcher access to the study data for
post-study analysis purposes.
Pre-test and post-test question and response
delivery both take the same form and are a simple
onscreen presentation of the four items and
associated Likert scales that make up
Lyubomirsky’s Subjective Happiness Scale [13].
This is carried out by participants once at the
outset of the study and once again on completion
of the study, and is delivered by the smartphone
application. It was felt important to keep the
presentation of this scale as simple as possible so
as not to interfere with the validity of the measure
when moving it to delivery using a smartphone.
This is also the case with the delivery of the
System Usability Scale, which is delivered
following the post-test subjective happiness
assessment, and for collection of participant
demographic data at the outset of the study. Our
attention here is centred on the delivery of the
study intervention, as it presents a more complex
technology delivery challenge and the novel
aspect of the work presented in this paper.
Consideration of this work is perhaps best
presented as a high-level walkthrough of the user
journey that takes place during one of the four
daily engagement sessions, which across time go
to make up the intervention stage of the study.
USER JOURNEY
At each of the four daily reminder times, the
smartphone application will issue a reminder to
the participant that it is time to engage with the
soHappy application. Figure 2 illustrates some of
the key screens employed during an engagement
session. On launching the soHappy smartphone
application, the participant will be prompted to
orient the smartphone so that their face is
positioned centrally onscreen (Figure 2a). Once
the Affect Analyser has detected the presence of a
human face within the camera’s field of view, a
brief countdown is displayed along with a prompt
for the participant to take three relaxing breaths
(Figure 2b). Following this, a stimulus in the
form of a text message is presented onscreen to
suggest that the participant should try to recall a
memory that makes them happy (Figure 2c).
These suggestions will come from a generic set of
stimuli, such as “Remember a time when
Figure 2: soHappy smiling a ctivity walkthrough, from left to right: guiding prompting the participant to correctly
frame their face, encouraging the participant to relax in preparation for smiling, prompting the participant to recall
a happy memory to a genuine smile, providing feedback that a smile has been detected.

someone made you laugh out loud unexpectedly”,
however it is eventually intended to allow
participants to contribute their own prompts in
future iterations of the smartphone application.
Once a smile is detected by the smartphone’s
front-facing camera the screen will provide
feedback to the participant that this has been
detected (Figure 2d). Once the recommended
thirty second smiling period has elapsed the
participant will be informed that this is happened,
although they can continue to smile for longer if
desired. Should the participant not continue
smiling, or detection of their smile fails for some
reason, they will be offered the opportunity to
smile again. However, there is no requirement to
do so before moving on to the next stage. During
the smiling exercise the Affect Analyser
measures and records the likelihood of a smile
having occurred along with its duration.
Additionally, the participant’s most probable
expression and emotion, as determined by the
Affect Analyser, are recorded, as well as a set of
facial landmarks that relate to FACS Action Units
[5], which are sampled every second during smile
detection.
Following this attempt to smile naturally, the
participant is invited to provide a subjective
measure of their happiness using Abdel-Khalek’s
single-item happiness scale [2]. This scale first
poses the question “Do you feel happy?”, before
directing the participant to consider three points,
designed to encourage the participant to base their
response on their overall feelings, rather than
their current feelings. This direction promotes
responses that relate to eudaemonic wellbeing,
rather than hedonistic happiness. Participants are
then guided to interact with the screen to rate
their overall happiness level on an eleven point
scale that ranges from Minimum (0) to Maximum
(10).
Next, the participant is invited to provide details
of a range of contextual circumstances proximate
to the ongoing session. First, an indication of the
nature of the location that the session took place
within is requested, by asking the participant to
select a label for their current location from one
of the following: Home, Work, University,
Shopping, Café or Restaurant, Bar or Club,
Outdoors Rural, or Outdoors City or Town.
Following this, the amount of social interaction
since the application was last used is recorded;
this is prompted using the question “How much
social interaction have you had since last using
the app?” and the response recorded using an
eleven point scale, which ranges from None (0) to
Lots (10). This is further explored using the
question “Who was the most recent social
interaction with?” and the set of labelled
responses: Family, Friend, Colleague,
Acquaintance, or Stranger. Once social
interaction-related factors have been recorded,
physiological factors are explored to establish the
participant’s physical activity and fatigue levels.
The physical activity measure is prompted using
the question “How physically active have you
been since last using the app?”, again recorded
using an eleven point scale, ranging from Not
Active (0) to Very Active (10). The fatigue
measure is prompted using the question “How
tired do you feel at present?”, and recorded using
an eleven point scale ranging from Not Tired (0)
to Very Tired (10). Subsequently, the participant
is thanked for their engagement and can use the
app to prompt unrecorded smiling until the next
planned engagement session.
EVALUATION METHODS
Descriptive statistics will be applied to explore
the data arising from the study. Initial evaluation
of the appropriately normalized data will take the
form of Pearson’s Correlation in order to find any
significant relationships within each participant’s
data points and across the full set of study data.
Box plots and scatter graphs will also be utilized
to further investigate the underlying relationships
discovered and the significance of the associated
data points. Additionally, an analysis of variance
of results will also be conducted to explore any
insights into changes in individual self-reported
and automated happiness levels recorded during
the study. Furthermore, feature extraction will be
performed on the recorded data, with the resulting
feature vectors utilized by Machine Learning
techniques, such as Support Vector Machines,
Bayesian Networks and Linear Discriminant
Analysis, in order to generate a classification

model that will be employed to investigate if
happiness levels can be predicted though the
contextual circumstances measured. The usability
of the smartphone application will also be
evaluated at the end of the study using the set of
responses to the System Usability Scale [3].
DISCUSSION
This study, and associated technology, have been
designed to test the hypothesis that “Individual
happiness levels can be improved following
regular use of a smartphone application that has
been designed to encourage genuine smiling on a
regular basis.” To this end, establishing if there is
a change in participant self-assessed happiness,
using Lyubomirsky’s validated Subjective
Happiness Scale [13], will be central to the
findings. While it is acknowledged that the
absence of a control group will weaken internal
validity, care has been taken to mitigate this.
Specifically, the study protocol has been designed
to allow the recording of additional data during
periods of engagement, both in relation to
subjective happiness and surrounding,
environmental, social and physiological,
contextual circumstances. These should help
facilitate the identification of any confounding
factors that might affect internal validity. Perhaps
most significant of these measures is the
continuous assessment of happiness during the
study using two separate mechanisms; one
subjectively self-assessed and the other objective
assessed using the Affect Analyser.
These additional measures provide additional
benefit beyond the sole purpose of helping to
assess the validity of the study. Recording data
relating to the contextual circumstances at the
point of intervention will also allow for work to
be performed that explores the potential to model
and predict when a participant is most likely to be
able to constructively engage in the smiling
exercise. This could afford the automatic
personalisation of reminders to help to encourage
participants to engage in the smiling exercise at
the most beneficial times. This is an important
consideration, given the understanding reported
earlier in this paper that it is likely to prove more
conducive to happiness if participants smile
naturally. Moreover, such a model may also allow
for the number of instances of engagement to be
minimised, while retaining any beneficial effect,
by reducing the amount of disruption to daily life
caused by delivery of any ongoing intervention
based on this work.
Finally, the usability assessment of the
smartphone application will help to understand
any aspects of the application’s user experience
design that could have influenced the data
collected, and will help to inform future revisions
to the application.
FURTHER WORK
Clearly, the next step in this work is to deliver the
study in order to determine if the hypothesis can
be tested using the soHappy affective technology.
Notwithstanding this, a number of additional
features for this technology have already been
planned. At an early stage, the inclusion of a
feature to allow participants to contribute their
own happiness prompts to the set of stimuli
presented could be highly beneficial. This
additional personalization could afford a greater
degree of meaningfulness to the prompts and will
also allow the application to become a form of
journaling tool that could further prove conducive
to improved happiness. It is also proposed that a
Happiness Dashboard feature be included, from
which participants could view their past activity
within the application and perhaps engage in
socially sharing of their activity. This would
allow for aspects such as social feedback
mechanisms and the effects of social integration
to be explored within future studies. Finally, the
delivery of a larger scale, more controlled study
would be important to validating any findings.
REFERENCES
1. Ahmed M Abdel-Khalek. 2006.
Measuring happiness with a single-item
scale. Social Behavior and Personality: an
international journal 34, 2: 139–150.
2. Action for Happiness. Action for
Happiness. Retrieved March 16, 2017
from http://www.actionforhappiness.org
3. John Brooke. 2013. SUS: a retrospective.
Journal of usability studies 8, 2: 29–40.

4. Rafael Calvo and Dorian Peters. Positive
Computing: Technology for Wellbeing
and Human Potential. 2016. The MIT
Press, Cambridge.
5. Paul Ekman, Wallace V Friesen, and
Joseph C Hager. 1978. Facial action
coding system (FACS). A technique for
the measurement of facial action.
Consulting, Palo Alto 22.
6. John F Helliwell, Richard Layard, and
Jeffrey Sachs. 2014. World happiness
report 2013. The Earth Institute, Columbia
University.
7. Peter Hills and Michael Argyle. 2002. The
Oxford Happiness Questionnaire : a
compact scale for the measurement of
psychological well-being. 33: 1073–1082.
8. Arlie Russell Hochschild. 2003. The
managed heart: Commercialization of
human feeling. Univ of California Press.
9. Annika Howells, Itai Ivtzan, and
Francisco Jose Eiroa-Orosa. 2016. Putting
the “app” in Happiness: A Randomised
Controlled Trial of a Smartphone-Based
Mindfulness Intervention to Enhance
Wellbeing. Journal of Happiness Studies
17, 1: 163–185.
10. Arianna Huffington. 2012. GPS for the
Soul: A Killer App for Better Living. The
Huffington Post. Retrieved March 16,
2017 from
http://www.huffingtonpost.com/arianna-
huffington/gps-for-the-
soul_b_1427290.html
11. Chris L Kleinke, Thomas R Peterson, and
Thomas R Rutledge. 1998. Effects of self-
generated facial expressions on mood.
Journal of Personality and Social
Psychology 74, 1: 272.
12. Tim Llewellynn and Matteo Sorci. 2011.
Method and System for Assessing and
Measuring Emotional Intensity to a
Stimulus. U.S. Patent Application No.
13/082,758.
13. Sonja Lyubomirsky and Heidi S Lepper.
1999. A measure of subjective happiness:
preliminary reliability and construct
validation. Social indicators research 46,
2: 137–155.
14. Daniel McDuff, Abdelrahman Mahmoud,
Mohammad Mavadati, May Amr, Jay
Turcot, and Rana el Kaliouby. 2016.
AFFDEX SDK: A Cross-Platform Real-
Time Multi-Face Expression Recognition
Toolkit. In Proceedings of the 2016 CHI
Conference Extended Abstracts on
Human Factors in Computing Systems,
3723–3726.
15. Daniel N McIntosh. 1996. Facial feedback
hypotheses: Evidence, implications, and
directions. Motivation and emotion 20, 2:
121–147.
16. Microsoft. Microsoft Cognitive Services -
Emotion API. Microsoft Cognitive
Services. Retrieved March 16, 2017 from
https://www.microsoft.com/cognitive-
services/en-us/emotion-api
17. Gregory Mone. 2015. Sensing emotions.
Communications of the ACM, 9: 15–16.
18. Rosalind W. Picard. 1995. Affective
Computing. MIT press.
19. Hitomi Tsujita and Jun Rekimoto. 2011.
HappinessCounter: smile-encouraging
appliance to increase positive mood.
Proceedings of the 2011 annual
conference extended abstracts on Human
factors in computing systems (CHI ’11).:
117–126.
20. World Health Organization and others.
1948. Preamble to the Constitution of the
World Health Organization as adopted by
the International Health Conference, New
York.