Conference PaperPDF Available

AI-Powered Tangible Interfaces to Transform Children's Mental Well-Being

Authors:

Abstract and Figures

Mental health is placing increasing pressure on global health organisations with modern lifestyles significantly contributing to worsening mental well-being. Mental health challenges do not only affect adults but also children with increasing social and academic pressure. Technologies for mental well-being possess many qualities that offer the potential for people who may otherwise not receive help due to fear of stigma or lack of resources to improve how they feel. In this work, we pursue the design of multiple tangible interfaces for children containing sensors with the aim of using artificial intelligence (AI) to automatically infer their mental well-being and custom buttons to enable the real-world data to be labelled. Tangible interfaces for mental well-being are ideal as they provide a non-intrusive means to infer and communicate mental well-being through play
Content may be subject to copyright.
AI-Powered Tangible Interfaces to Transform Children’s
Mental Well-being
Kieran Woodward
Eiman Kanjo
David Brown
School of Science and Technology
School of Science and Technology
School of Science and Technology
Nottingham Trent University
Nottingham Trent University
Nottingham Trent University
kieran.woodward@ntu.ac.uk
eiman.kanjo@ntu.ac.uk
david.brown@ntu.ac.uk
Abstract - Mental health is placing increasing pressure
on global health organisations with modern lifestyles
significantly contributing to worsening mental well-being.
Mental health challenges do not only affect adults but also
children with increasing social and academic pressure.
Technologies for mental well-being possess many qualities
that offer the potential for people who may otherwise not
receive help due to fear of stigma or lack of resources to
improve how they feel. In this work, we pursue the design of
multiple tangible interfaces for children containing sensors
with the aim of using artificial intelligence (AI) to
automatically infer their mental well-being and custom
buttons to enable the real-world data to be labelled. Tangible
interfaces for mental well-being are ideal as they provide a
non-intrusive means to infer and communicate mental well-
being through play.
Index Terms - Affective computing, Machine learning,
Artificial intelligence, Pervasive computing
I. INTRODUCTION
The mental well-being of young people is increasingly
important as school and social pressures are resulting in
young people experiencing more stress than ever before.
The World Health Organisation [1] defines the well-being
of an individual as being encompassed in the realisation of
their abilities, coping with the normal stresses of life,
productive work and contribution to their community. This
presents a problem especially for young people who often
do not know how to express their mental well-being and
emotions to either their peers or family members.
Although a limited number of tangible interfaces to infer
mental well-being have previously been developed many
of these were not aimed at children and often contain
physiological sensors which have to be used in specific
ways which young people will not always conform to.
Additionally, advances in artificial intelligence (AI) such
as deep learning enable the more accurate classification of
stress and mental well-being directly from raw data, which
has not yet been fully explored. However, one of the most
significant issues is having sufficient real-world data to
train the models accurately. The tangible interfaces
developed aim to alleviate this problem by including novel
labelling tools that allow young people to tag the sensor
data with their self-reports as they interact with the toys.
By collecting sensory data paired with mental well-being,
it is possible to communicate emotions naturally and in
real-time.
This research introduces multiple naturalistic tangible
interfaces in the form of children’s toys with the aim of
inferring mental well-being automatically from the
embedded sensors using deep learning models on a
connected device. Initial prototypes contain sensors used
to assess mental well-being, buttons to enable the labelling
of the data collected and wireless communication to infer
and convey children’s emotions in real-time and in-situ
settings. Furthermore, this work highlights key directions
for the continued renement of tangible devices for
tacking mental well-being.
II. RELATED WORK
Mental well-being challenges can impact all people as
work-related stress affects 59% of the UK population [2].
However, it is not just adults that experience increased
stress children are also experiencing more stress than ever
before [3] [4].
Many existing tangible devices have been utilised to
promote communication and provide an easy method to
express emotions and mental health state that can often
be difficult to communicate [5] [6].
Younis & Kanjo, introduced novel techniques to self-
report emotions that can go beyond a mobile screen [7] [8].
Similarly, Emoball [9] is a tangible device that enables
users to record their emotions by squeezing an electronic
ball. Subtle Stone [10] also allows users to express their
current emotions represented as a unique colour displayed
on the stone. The use of colours to represent emotions
limits the number of people to whom users exposed their
emotions to. Eight school students tested Subtle stone in
their language class where the teacher was able to view the
data in real-time using the mobile app. Overall, the use of
colours to represent emotions was found to be well
received with students liking the anonymity it provided
along with students finding it more comfortable to express
their emotions as colours.
Another self-reporting tangible interface, Mood TUI [11]
not only allows users to record their emotions but also
collects data from the users smartphones including
location data and physiological data such as heart rate
from smartwatches. Participants found the tangible
interface very exciting, although the device was felt to be
too large leading to participants believing they would lose
motivation to continue using it over an extended period.
This feedback shows that while tangible interfaces excite
users the design and functionality of the devices must be
prioritised.
A tangible interface for mental well-being used to detect
stress in real-time automatically is Grasp, which has been
tested with anxious participants in a dentist’s office [12].
Participants were told to squeeze Grasp whenever they felt
stressed and the device detected how much force was
exhorted and displayed this data on a mobile app. By
utilising force resistive sensors Grasp allowed users to
quickly and easily record their stress levels in real-time
which could be useful for monitoring stress as it does not
rely on participants manually recording stressful events.
Force resistive sensors have also been used to create a
tangible ball allowing for music to be manipulated by
squeezing different areas and moving the ball [13]. This
research concluded that the ball could successfully be used
for music therapy with children as it promoted positive
emotions through tactile input and music. Sensors such as
force sensors and accelerometers to detect touch and
motion have been shown to provide an intuitive method of
interaction for tangible interfaces. These sensors show the
increased opportunities tangible interfaces present as they
can embed a variety of sensors to infer mental well-being
which is not possible with other technological solutions
such as mobile apps.
Machine learning is vital to accurately infer mental health
state from the sensory data collected. Numerous sensors
can be used to determine mental well-being and when
paired with sufficiently trained machine learning
classifiers can result in devices that can accurately infer
mental well-being in real-time.
Physiological sensors present a significant opportunity to
infer mental health state using Electrodermal activity
(EDA) and heart rate variability (HRV) sensors [14] [15]
[16].
Both EDA and HRV were used in a wearable device
aiming to measure stress [17]. The wearable device took
measurements over a 5 minute period to detect stress
levels with an accuracy of 97.4% and found that HRV and
EDA are highly reliable making them extremely useful in
detecting mental well-being.
Once mental well-being has been inferred, interventional
feedback can automatically be applied if poor mental well-
being is detected. Good vibes [18], the Relax! pen [19]
and Doppel [20] are wearable devices that use haptic
feedback to reduce stress; the studies show participants
remained less stressed than control groups when in
stressful situations showing haptic feedback can have a
substantial positive impact in improving mental well-being.
A novel approach to provide feedback is through the use
of robotics such as therapy animals which are most
commonly used to reduce loneliness in the elderly. Paro is
a robot used for therapy; it has been designed as an easy to
use robotic seal that encourages user interaction with its
large eyes and soft fur [21]. Force sensors allow Paro to
know the location and force of users’ touch allowing for
the responses to be relevant to the input. Paro has been
shown to provide effective therapy by helping to reduce
stress in a day service centre for elderly adults [22],
increasing users social interactions and improving their
reactions to stress [21].
Although most therapy animals such as Paro target the
elderly, a robotic teddy aimed at reducing stress in young
children at the hospital has been developed [23]. Rather
than relying upon tactile interaction like Paro, this teddy
uses vocal interactions. The children who used the robotic
teddy spent more time playing with it than the comparative
virtual or traditional plush teddy and their behaviours
conveyed they were not stressed.
This research shows the powerful impact of tangible
devices can have on both the young and the elderly by
utilising on-device processing and novel labelling
techniques. Little research has been conducted with the
aim of pairing artificial intelligence with tangible
interfaces however a vast magnitude of reliable data must
first be collected to train machine learning classifiers.
III. SYSTEM DESIGN
A. Design approach
While there are a range of tangible interfaces designed to
reduce stress and loneliness in the elderly, little focus has
been given to children who are experiencing more social
and academic-related stress than ever before [3] and often
find it difficult to communicate their mental well-being
[24] [25].
Children’s soft toys are ideal interfaces to contain the
required sensors as they contain ample space and are
omnipresent with children. As the devices are to be used
by children, they must remain physically small while
securely containing all of the required electronics to
ensure safety.
The prototypes developed include soft teddys, small balls
and cushions, these devices all have enough space to
contain the required microcontroller and sensors while
remaining enticing for children to use. The teddy was
chosen to target younger children, the ball was aimed at
active children as the embedded sensors are additionally
protected allowing the ball to be thrown without causing
damage and the cushion was designed for older children
who may still find it difficult to express their mental well-
being but no longer use toys.
Accessibility was a key design requirement and by
developing a range of devices that remain similar to
existing children’s toys, it will allow the device to be used
in all locations where children may experience different
mental well-being states.
The current devices developed provide concrete examples
of what can be achieved when developing mental well-
being toys but in the near future new devices will be co-
designed with adults and students with learning
disabilities. Former projects show these end users are very
effective when they have concrete designs that they can
use as examples and helps compensate for their cognitive
impairment.
B. System architecture
As the devices need to remain physically small the system
employs Arduino Nanos as the microcontroller due to its
small size of only 43mm x 19mm, compatibility with an
extensive array of sensors due to its analogue and digital
inputs and its open source nature.
A variety of sensors were considered to be embedded
within the prototypes including both physiological and
environmental sensors. However, when prototypes with
physiological sensors such as heart rate variability were
tested with children it was revealed that it was challenging
for them to touch the sensors in the required manner for
accurate data to be collected as they would frequently
move the toy and adjust their grip. This resulted in the
omission of physiological sensors due to the poor accuracy
of the physiological data being collected when the devices
were used.
Fig. 1. Sensors, microcontroller and feedback embedded into the
devices
The ball, teddy and cushion devices instead contain force
resistive sensors to measure the force and location of
which the toys are touched, flexi sensors to measure the
angle the toy is bent during use and accelerometers to
measure motion. These sensors combined show the
manner of which the devices are being interfaced with
ranging from soft touches with slow, gentle movements to
harsh touches with fast, aggressive movements. The
devices also contain Bluetooth low energy chips allowing
them to communicate with each other and communicate
with a central device such as a smartphone without using
excessive power. All of the electronics can be directly
soldered to the Arduino’s pins allowing for a small device
to developed while also being highly functional.
Fig. 2. Ball prototype containing force and motion sensors and
interventional feedback
As well as the ability to sense, the devices can also provide
feedback; visual, auditory and haptic feedback have been
embedded into the prototypes to enable this s multi-colour
LED, haptic motor controller with vibration motor and
speakers are embedded into the devices. The haptic motor
controller allows for varying haptic feedback patterns to
be played rather than just turning the digital motor on or
off. Additionally, buttons enable children to record their
mental well-being and an SD card reader allows for all of
the sensory data to be recorded.
One of the most significant issues faced was battery life as
the device would be required to run for multiple hours at a
time between charges. 9V alkaline batteries were
considered due to their small size and low cost but the
inability to recharge these batteries resulted in their
dismissal, instead LiPo batteries are used as they also
remain physically small but can be recharged. The
batteries used within the prototypes range from 500mAh to
1200mAh providing a battery life of between 5 and 10
hours.
C. Classifying stressful behaviour
Detecting stressful behaviour is a challenging proposition
as machine learning classifiers will be required to analyse
the data. However, before machine learning classifiers can
be trained a vast amount of real-world data is required
from the sensors to ensure the model is accurately trained.
When the devices are used sensory data is automatically
logged to a Micro SD card but simply collecting sensory
data is not sufficient as the data must also be labelled with
the user’s well-being to enable the model to learn which
behaviours correlate with poor mental well-being.
Different coloured buttons have been incorporated into the
prototypes allowing children to label their emotional well-
being as sensory data is collected. Majority of the
prototypes contain two buttons (one red for sad or stressed
emotions and one green for happy emotions) making it
simple for children to label their emotions without the
need for them to express their well-being in words which
children traditionally find difficult.
Example motion and touch data collected from the sensors
embedded into the teddy device has been normalised to
enable possible correlations to be observed. Figure 3
shows that when poor mental well-being was reported the
device was moved quicker and more frequently compared
to when positive mental well-being was reported.
0
0.5
1
1.5
020 40 60 80 100 120 140 160 180
Normalised accelerometer
Duration (s)
Motion
Poor well-being Positive well-being
Fig. 3. Example motion sensor data comparing poor mental well-being
and positive mental well-being
Figure 4 compares example touch data from the teddy
prototype showing that when poor mental well-being was
reported the device was consistently touched harder than
when positive mental well-being was recorded. While this
data shows clear correlations between touch, motion and
mental well-being more real-world data is required from
end users to confirm such trends exist and can be
generalised to the wider population.
0
0.5
1
1.5
020 40 60 80 100 120 140 160 180
Normalised force
Duration (s)
Touch
Poor well-being Positive well-being
Fig. 4. Example force resistive sensor data comparing poor mental well-
being and positive mental well-being
Once sufficient data has been collected, machine learning
models can be trained with the labelled data, it will not be
possible to run the model directly from the device due to
additional processing power being required but as the
devices already contain Bluetooth for communicating the
model could be incorporated into the mobile app. This
would allow for the mobile app to receive data in real-
time, classify the data and then automatically inform
parents or teachers of the mental well-being of children.
D. Interventions
Once mental well-being has either been communicated via
the child pressing a button or automatically inferred,
interventional feedback can be applied. The devices
developed contain a variety of different feedback
mechanisms including haptic feedback, auditory feedback
and multi-colour changing LEDs to provide visual
feedback.
Fig. 5. Teddy prototype containing force and motion sensors, labelling
buttons and interventional feedback
Currently when a child expresses they are feeling low (e.g.
sad, or stressed) using the ball or teddy prototypes and a
higher than average level of force is exhorted, the device is
bent at a steep angle or the device is shaken then gradually
reducing haptic feedback is issued as well as the
embedded LED changing to blue. This feedback is hoped
to provide a distraction allowing the child to improve their
mental wellbeing as distractions can often result in mood
improvement. When a child expresses they are happy but
still shakes the device or presses the device harshly, the
LED remains green but sharp short haptic feedback
patterns are issued to alert users of their actions and again
provide a distraction.
The cushion prototype also contains a multi-colour
changing LED but additionally embeds a small speaker
thus when a child records poor mental well-being and
shakes or forcefully touches the cushion, short stories or
calming sounds such as wind and water sound effects can
be played to provide high quality auditory feedback
improving mental well-being.
Future iterations that contain the trained machine learning
model could evaluate which interventional feedback
mechanism is most effective for each individual user. This
would allow for different colours, feedback patterns and
sounds to be played dependant on the user’s reaction to the
different feedback providing more effective personalised
real-time feedback.
IV. DISCUSSION
The tangible interfaces developed can be used individually
enabling parents and guardians to monitor their children’s
emotional well-being in real-time through an Android
mobile application. Alternatively, the devices are also
suited to being used in groups such as in schools as each
device contains Bluetooth communication enabling them
to communicate with one another creating a pair to pair
network. This peer to peer network of tangible interfaces
could enable devices to react differently when someone
nearby is experiencing poor mental well-being creating a
local support network. The feedback mechanisms
incorporated into the devices could change dependant on
the user’s mental well-being as well as those nearby. A
peer to peer network also enables the mental well-being
impact of social interactions to be examined as research
suggests that play increases happiness [26] [27]. As these
tangible interfaces enable children to quickly log their
emotional well-being throughout the day in real-world
environments, this data can be used to further explore the
impact social interactions have on children’s mental well-
being.
Alternatively, the interfaces can communicate with a
central device such as a computer or mobile device. The
central device can collect data from all of the connected
devices, this could be used to allow schools to understand
the mental well-being of their students with the devices
responding in real-time to this data.
Collecting mental well-being data both enables parents
and teachers to ensure the positive mental well-being of
children while also providing the vast real-world data to
train future deep learning models enabling more accurate
automatic inference of mental well-being.
V. FINDINGS
A. Concept
The developed interfaces were discussed at a focus group
in Oakfield school, Nottingham, UK with teachers and
adults whose mental well-being issues is often
diagnostically overshadowed.
The feedback shows the devices were considered suitable
for children especially those suffering from mental health
challenges as it enables children to easily communicate
how they are feeling without having to explicitly discuss
their emotions which many find difficult. Participants
believed children would like the concept of embedding
sensors within traditional toys as it provides new
interaction methods while also allowing parents or teacher
to monitor their well-being.
Participants liked the ability that children could
continuously update their mental well-being by simply
pressing different buttons as this has traditionally been
difficult as children can often find it challenging to express
their emotions verbally.
B. Design
Participants liked the soft design of the teddy device and
believed that it was accessible for children with its small
and familiar design. Similarly, participants believed
children would appreciate the design the ball and cushion
as they appear familiar and unlikely to draw attention.
The inconspicuous nature of the devices was appreciated
as they appears the same as other toys making them less
stigmatised than other mental well-being devices.
Fig. 6. Teddy prototype being used at the focus group
Participants stated they believed the design of the devices
would encourage children to use such devices but did
question the durability as children may drop or throw the
toys. Durability is a key design requirement that currently
requires more consideration to ensure children are not able
to access or damage the embedded microcontroller or
sensors.
C. Sensor and Feedback
Participants liked the sensors being used to measure touch,
the angle at which the device is bent and motion; they
believed the sensors would collect a wide range of useful
data on how the devices are used which they believed
would correlate to their mental well-being.
Participants believed the sensors were ubiquitously
embedded into the devices as they felt like and were
interacted with like regular toys. Designing the devices to
look and feel like traditional toys and cushions is vital to
ensure they are used normally by children so that real-
world usage data can be collected and any correlations
between device use and mental well-being can be
established.
Visual, haptic and audio feedback was demonstrated at the
focus group. The haptic feedback proved extremely
popular as the participants found the different vibration
patterns to be relaxing and teachers believed it could be
used to help improve students hand grip and motor
control.
Visual feedback in the form of multi-colour changing
LEDs was also demonstrated as the colours would change
depending on how the device was used. Participants
believed children would like the different colours but
depending on their age may not understand different
colours represent different actions and emotions.
Finally, auditory feedback is incorporated into the cushion
which allows for calming sounds of nature to be played
although due to the small size of the cushion the speaker
has had to remain small leading to poor sound quality.
Participants believed that better sound quality would be
necessary for children to clearly hear the sounds and help
them relax.
D. Machine learning
Participants were intrigued by the idea of training a
computational model to enable the device to automatically
infer the mental well-being of the user based on the
sensory data collected. A computational model would
make it easier to understand how children are feeling
automatically with no data labelling action required from
the children or teachers. Children will simply be able to
interact with the devices normally but the school could
apply interventions if it detects a child has poor mental
well-being.
However, before this can be achieved participants
understood that the current prototypes must first be trialled
to collect the data to train the model. Participants liked the
ease of which children could record their mental well-
being by simply pressing either the red button for negative
emotions or the green button for positive emotions. They
believed the simplicity of this process would encourage
children to click the relevant button whenever their well-
being changes which is vital to ensure the model is trained
accurately.
One teacher noted that not all students, for example, those
with cognitive impairments would be able to recognise
their mental well-being state and press the appropriate
button. A possible solution discussed to enable labelled
data to be collected for all children is for the paired mobile
app to contain multiple icons representing the different
emotions children can experience, then the teacher or
parent who understands the child’s emotions can use the
app to label the data collected from the toy. This solution
could be extremely valuable as it is easy to implement as
the devices already contain Bluetooth to connect to an
existing Android application and it ensures the devices are
inclusive by recording data from all users especially those
who may experience mental well-being challenges.
VI. DESIGN PROPOSAL
In the near future, a co-design workshop will be held with
adults and students with learning disabilities. Designing
with people with cognitive impairments is vital because
they are often overlooked as if a mental health problem
presents it is more likely to be attributed to their
learning disability (diagnostic overshadowing) or classed
as challenging behaviour [28]. The use of existing
tangible examples will help the understanding of what
such interfaces can look like and function as. This
enhanced understanding of the tangible interfaces will
enable the participants to devise new realistic designs that
could then be developed and trialled.
While there are ethical issues when co-designing with
people with learning disabilities such as legality issues
regarding informed consent and the impact of participation
as some participants may find it challenging when
confronted by their own limitations, many of these issues
can be addressed with the use of method stories [29].
Method stories are defined as the behind stories of
methods [30] and describe how methods are made to
work in reality, instead of how they ought to work in
theory.
After the co-design workshop has been conducted, the
Interpersonal Process Recall (IPR) [31] method could be
utilised to gain additional feedback from participants that
did not show up in the original co-design workshop. The
IPR method involves video recording the co-design
workshop and then playing back the video to participants
as a stimulus for recall and reflection allowing participants
to expand upon or clarify their comments. This allows for
further insights to be gained from participants who may
find it difficult to express themselves in the moment due to
their cognitive impairment.
The use of method stories and Interpersonal Process
Recall could be utilised to ensure an individual approach
is used to co-design new devices and gain all possible
insights.
VII. LIMITATIONS AND FUTURE WORK
A current limitation of this research is the limited testing
of the devices with children although the devices have
been discussed at a focus group with people who
experience mental health challenges. In the future, the
devices will be trialled with children and the positive
feedback from the user group shows minimal changes are
required before trials can begin.
Currently, the existing devices are based on existing toys
and cushions to provide examples of what can be achieved
but potential users were not involved in the design process.
In the near future a co-design workshop is being held to
address this issue as children and adults who experience
mental well-being challenges will be able to share their
thoughts and design ideas to ensure the devices are
accessible and meet the requirements of both the children
who use the device and the adults who will view the
recorded data.
Another limitation is the current lack of data from the
devices, although when the devices are trialled with
children, this will allow for real-world data to be collected.
In the future, this data can be used to train deep learning
classifiers enabling mental well-being state to
automatically be inferred and provide real-time
interventions when necessary. Additionally, data collected
from the device could be used to analyse the impact the
interventional feedback has on users’ behaviours. The
impact different feedback has on individuals could be
utilised so that the most effective feedback can be applied
on an individual basis.
VIII. CONCLUSION
Tangible user interfaces present a wide array of
opportunities to both sense and intervene in children’s
mental well-being. This research has shown the possibility
to develop well-being devices for children that embed
sensors, feedback and buttons.
The devices enable vast amounts of real-world data to be
collected through the novel use of buttons allowing
children to label the real-world data collected in real-time.
This data along with advancements in AI will ensure
classification models developed for future devices will be
able to accurately infer children’s mental well-being in
real-time from the sensory data collected.
Furthermore, the ability for the interfaces to communicate
with one another as well as with a central device enables a
peer to peer network of devices to be created allowing
children to support each other as well as alerting parents
and teachers to the mental well-being state of children in
real-time.
Overall, the tangible interfaces developed show the
potential for mental well-being toys containing suitable
sensors to unobtrusively allow children to record their
mental well-being and automatically share this data with
parents and teachers.
REFERENCES
[1] WHO, “WHO | Mental health: a state of well-being,” WHO,
2014. .
[2] Perkbox, “THE 2O18 UK WORKPLACE STRESS
SURVEY,” 2018.
[3] American Psychological Association, “Stress in America
2009,” 2009.
[4] American Psychological Association, “Stress in America TM
Are Teens Adopting Adults’ Stress Habits?”
[5] K. Woodward, E. Kanjo, S. Burton, and A. Oikonomou,
“EmoEcho : A Tangible Interface to Convey and Communicate
Emotions,” Proc. ACM Int. Jt. Conf. Pervasive Ubiquitous
Comput., 2018.
[6] K. Woodward and E. Kanjo, “Things of the Internet (ToI),” in
Proceedings of the 2018 ACM International Joint Conference
and 2018 International Symposium on Pervasive and
Ubiquitous Computing and Wearable Computers -
UbiComp ’18, 2018, pp. 12281233.
[7] E. Kanjo, E. M. G. Younis, and C. S. Ang, “Deep Learning
Analysis of Mobile Physiological, Environmental and Location
Sensor Data for Emotion Detection,” J. Inf. Fusion, pp. 133,
2018.
[8] E. Kanjo, E. M. G. Younis, and N. Sherkat, “Towards
unravelling the relationship between on-body, environmental
and emotion data using sensor information fusion approach,”
Inf. Fusion, vol. 40, pp. 1831, Mar. 2018.
[9] J. Bravo, R. Hervás, and V. Villarreal, “Ambient intelligence
for health first international conference, AmIHEALTH 2015
Puerto Varas, Chile, December 1–4, 2015 proceedings,” Lect.
Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell.
Lect. Notes Bioinformatics), vol. 9456, pp. 189200, 2015.
[10] M. Balaam, G. Fitzpatrick, J. Good, and R. Luckin, “Exploring
Affective Technologies for the Classroom with the Subtle
Stone,” Proc. 28th Int. Conf. Hum. factors Comput. Syst. -
CHI ’10, p. 1623, 2009.
[11] F. Sarzotti, “Self-Monitoring of Emotions and Mood Using a
Tangible Approach,” Computers, vol. 7, no. 1, p. 7, Jan. 2018.
[12] F. Guribye and T. Gjøsæter, “Tangible Interaction in the
Dentist Office,” in Proceedings of the Twelfth International
Conference on Tangible, Embedded, and Embodied
Interaction - TEI ’18, 2018, pp. 123130.
[13] D. Beattie, “SqueezeMusic- HCI & Audio Interaction
Research,” 2017. .
[14] N. Sharma and T. Gedeon, “Objective measures, sensors and
computational techniques for stress recognition and
classification: A survey,” Comput. Methods Programs
Biomed., vol. 108, no. 3, pp. 12871301, Dec. 2012.
[15] J. Schumm et al., “Discriminating stress from cognitive load
using a wearable EDA device. Discriminating Stress From
Cognitive Load Using a Wearable EDA Device,” IEEE Trans.
Inf. Technol. Biomed., vol. 14, no. 2, 2010.
[16] N. Alajmi, E. Kanjo, N. El Mawass, and A. Chamberlain,
“Shopmobia: An Emotion-Based Shop Rating System,” in
2013 Humaine Association Conference on Affective
Computing and Intelligent Interaction, 2013, pp. 745750.
[17] J. A. Healey and R. W. Picard, “Detecting Stress During Real-
World Driving Tasks Using Physiological Sensors,” IEEE
Trans. Intell. Transp. Syst., vol. 6, no. 2, pp. 156166, Jun.
2005.
[18] C. Kelling, D. Pitaro, and J. Rantala, “Good vibes,” in
Proceedings of the 20th International Academic Mindtrek
Conference on - AcademicMindtrek ’16, 2016, pp. 130136.
[19] N. Chaitanya M., S. Jayakkumar, E. Chong, and C. H. Yeow,
“A wearable, EEG-based massage headband for anxiety
alleviation,” in 2017 39th Annual International Conference of
the IEEE Engineering in Medicine and Biology Society
(EMBC), 2017, vol. 2017, pp. 35573560.
[20] R. T. Azevedo, N. Bennett, A. Bilicki, J. Hooper, F.
Markopoulou, and M. Tsakiris, “The calming effect of a new
wearable device during the anticipation of public speech,” Sci.
Rep., vol. 7, no. 1, p. 2285, 2017.
[21] T. Shibata and K. Wada, “Robot Therapy: A New Approach
for Mental Healthcare of the Elderly A Mini-Review,”
Gerontology, vol. 57, no. 4, pp. 378386, 2011.
[22] B. D. Argall and A. G. Billard, “A survey of Tactile
HumanRobot Interactions,” Robotics and Autonomous
Systems. 2010.
[23] S. Jeong et al., “A Social Robot to Mitigate Stress, Anxiety,
and Pain in Hospital Pediatric Care,” in Proceedings of the
Tenth Annual ACM/IEEE International Conference on
Human-Robot Interaction Extended Abstracts - HRI’15
Extended Abstracts, 2015, pp. 103104.
[24] Ons, “Menta lhealth of children and young people in Great
Britain,” 2004.
[25] “Mental health of children in England,” 2016.
[26] S. C. Bratton, D. Ray, T. Rhine, and L. Jones, “The Efficacy of
Play Therapy With Children: A Meta-Analytic Review of
Treatment Outcomes.,” Prof. Psychol. Res. Pract., vol. 36, no.
4, pp. 376390, 2005.
[27] C. Hendon and L. M. Bohon, “Hospitalized children’s mood
differences during play and music therapy,” Child. Care.
Health Dev., vol. 34, no. 2, pp. 141144, Mar. 2008.
[28] A. Giraud-Saunders, “Mental health in people with learning
disabilities.”
[29] N. Hendriks, K. Slegers, and P. Duysburgh, “Codesign with
people living with cognitive or sensory impairments: a case for
method stories and uniqueness,” Int. J. CoCreation Des. Arts,
2015.
[30] J.-J. Lee, “The True Benefits of Designing Design Methods,”
Artifact, vol. 3, no. 2, p. 5, Aug. 2014.
[31] D. Brown, R. Kettley, and S. Kettley, “Interpersonal Process
Recall a method for reflecting on co-design experiences,”
2016.
... Of the 42 studies in our corpus, 24% (10) employed virtual reality for delivering wellbeing solutions [13-16, 19, 23, 24, 34, 36, 38], 45% (19) utilized artificial intelligence [17, 18, 20, 22, 25-27, 30, 35, 37, 42, 44, 45, 47, 51, 53], 42% (18) used mobile app [52,51,49,48,47,46,43,60,41,60,40,39,33,32,31,29,28,21,20] ,(1) were used web app [50] , (3) were used sensors as input technology [24,23,31], and (1) used tangible interface for interaction [42]. ...
... [17, 25, 27-29, 35, 37, 39, 42, 52]. [13,17,24,26,32,41,42,46], and 23% (10) articles discussed only the evaluation of digital interventions [53,52,51,49,43,37,33,25,21,20]. ...
... Design and Evaluation Methodology While in some articles, design, and evaluation methodologies are not discussed explicitly, the authors tried to recognize the methodology of articles by reading the details of articles. Four articles proposed design ideas but did not express design or evaluation methodologies [13,17,42,37]. The most commonly reported design and evaluation methods were iterative development (n=14) [53,51,49,48,47,41,39,36,35,30,28,27,26,24] Iterative development involves repeated cycles of designing, building and testing prototypes in order to refine and improve a system. ...
Chapter
As emerging technologies such as Virtual Reality (VR) and Artificial Intelligence (AI) extend beyond research and development centers, their application in sensitive domains such as mental health is on the rise. In recent years, there has been a rapid growth in the development of digital mental health interventions. However, there is a lack of comprehensive understanding regarding the current state of these interventions, especially with respect to the technologies employed, design practices utilized, and evaluation methods applied. To address this knowledge gap, we followed PRISMA 2020 statement guidelines to identify the technologies, design practices, and usability issues discussed in scientific publications pertaining to the development of interactive systems for mental wellness. Out of 500 articles screened, this work analyzed total of 42. The findings highlight characteristics of the studies, technologies employed, intervention focus, design and evaluation methodologies, and contributions to knowledge in this domain. The review underscores the need for understanding and accommodating the diverse needs in designing digital mental wellness interventions. Based on these findings, we propose the development of a crowd-sourced persona repository as a promising direction for future research. It would enable designers to create more tailored and inclusive mental wellness technologies, thereby enhancing their effectiveness and reach.
... Emotion regulation was the second most frequently mentioned topic, encompassing tangible systems designed for various groups such as office workers [5], patients during or after treatment [63,151], disabled individuals [26], the youth [137], and general users [165]. In addition, other purposes were identified in the review: emotion recognition [107,135,164,165,167], emotional education [38,56,121,137,144,153] (two papers focusing the same system was grouped together [38,144]), the elicitation of emotions or moods [26,129,133,142], assessment of affective states or emotional attitudes [59,64,94,114], and exploration of the affective impacts of tangible interactions [4,57,84,95]. ...
... They integrated TUIs for inconspicuous peripheral interaction during daily routines, using non-intrusive sensors for seamless integration into the environment. Woodward and colleagues [164] developed an interactive stuffed animal, a ball, and a cushion to collect pressure data and the labels of the mental well-being statuses of children in real life. These systems have the appearance of everyday objects, thus allowing children to record their mental health data during everyday activities. ...
... Figure 3: Method of user study. Example papers for each category: survey [29], interview [153], observation [28], controlled experiment [94], physiological measurement [109], log analysis [74], task completion time [111], pilot study [5], and accuracy of the classifiers [107] (Others includes usability metrics [56], heuristic evaluation [121], and focus group [164]). ...
Conference Paper
Full-text available
Tangible user interfaces (TUIs) have been applied to assess, communicate, or regulate human core affect, emotions, or moods. Previous studies identified TUIs as an innovative way to serve people’s affective needs. This review examines the design and evaluation of tangible interactive systems that focus on human core affect, emotions, and moods. We provide an overview of current studies. We summarize how tangibility can be leveraged to support affective interaction, and we propose the dimensions of tangible affective interaction, deriving guidelines for design. We highlight three main challenges: understanding tangible affective interaction within real-life scenarios, utilizing embodied interaction to express or influence affective states, and establishing benchmarks for evaluating tangible affective interfaces.
... In recent years, different context-aware affective computing systems have been proposed (McStay, 2020;Sun et al., 2018;Woodward et al., 2019;Zhong et al., 2020). However, most studies do not discuss the relation and relevance of context variables with the analyzed affective states and, instead, examine the results of the identified affective phenomena. ...
Article
Studies indicate the growth and relevance of systems that can recognize affective states in the most different areas. A tendency to more natural environments could be noticed that considering physical or physiological signs and the individual’s relationship with the environment, other people, and daily activities. Therefore, what is the importance of combining ubiquitous computing, affective computing, and the contributions of context-aware information to provide more accurate and intelligent affective systems? This study presents a systematic literature mapping about the use of contextual information to identify affective states, which covered articles published between 2010 and October 2021, resulting in 1.638 studies. After applying filters, which we explain further in this article, we selected 49 works to answer a set of research questions. The results indicate that physiological data was the main parameter for recognizing affective signs (62.3%, 33/53), followed by visual data (32.1%, 17/53). The links between context and affective signs presented a more significant occurrence in the combination of contexts related to activities and physiological data (34%, 18/53).
... Participants found the IMU and FSR sensors the easiest and most natural to use. However, not all sensors would be appropriate for children, such as the heart rate sensor, as children would not be able to keep their finger continuously touching the sensor for the device to accurately measure changes in heart rate [74]. It was suggested that physiological sensors should be reserved for the 3D printed interfaces and not the soft interfaces, where children in particular may continuously move their hands preventing accurate readings. ...
Preprint
Full-text available
Involving and engaging people with learning disabilities on issues relating to their mental wellbeing can bechallenging. This research explores how participatory design techniques and principles can be used to engagepeople with learning disabilities in designing technological solutions relevant to them that could monitorand aid their mental wellbeing. Specifically, we explore methods used in a series of co-design workshopsto engage people with learning disabilities in the use of tangible interfaces for mental wellbeing. A varietyof designs, sensors and interventions were explored during the workshops resulting in the development ofmultiple mental wellbeing interfaces. Furthermore, data collection trials using the developed interfaces havedemonstrated the potential to collect real-world labelled data that can be used to train machine learningclassifiers. The co-design approach adopted for the design of the mental wellbeing tangible interfaces ensuredthat effective and suitable devices have been developed
... Participants found the IMU and FSR sensors the easiest and most natural to use. However, not all sensors would be appropriate for children, such as the heart rate sensor, as children would not be able to keep their finger continuously touching the sensor for the device to accurately measure changes in heart rate [74]. It was suggested that physiological sensors should be reserved for the 3D printed interfaces and not the soft interfaces, where children in particular may continuously move their hands preventing accurate readings. ...
Preprint
Full-text available
Involving and engaging people with learning disabilities on issues relating to their mental wellbeing can bechallenging. This research explores how participatory design techniques and principles can be used to engagepeople with learning disabilities in designing technological solutions relevant to them that could monitorand aid their mental wellbeing. Specifically, we explore methods used in a series of co-design workshopsto engage people with learning disabilities in the use of tangible interfaces for mental wellbeing. A varietyof designs, sensors and interventions were explored during the workshops resulting in the development ofmultiple mental wellbeing interfaces. Furthermore, data collection trials using the developed interfaces havedemonstrated the potential to collect real-world labelled data that can be used to train machine learningclassifiers. The co-design approach adopted for the design of the mental wellbeing tangible interfaces ensuredthat effective and suitable devices have been developed
... Additionally, visual feedback in the form of multi-coloured LEDs has been included within the soft ball and teddy prototypes. TangToys have been presented in focus groups to teachers of young students with mild to moderate learning disabilities to provide feedback on the design and functionality of the interfaces [15]. Teachers considered the methods used to interact with TangToys suitable for children and believed the way in which children interact with the toys will indicate their wellbeing. ...
Article
Full-text available
Children can find it challenging to communicate their emotions especially when experiencing mental health challenges. Technological solutions may help children communicate digitally and receive support from one another as advances in networking and sensors enable the real-time transmission of physical interactions. In this work, we pursue the design of multiple tangible user interfaces designed for children containing multiple sensors and feedback actuators. Bluetooth is used to provide communication between Tangible Toys (TangToys) enabling peer to peer support groups to be developed and allowing feedback to be issued whenever other children are nearby. TangToys can provide a non-intrusive means for children to communicate their wellbeing through play. CCS CONCEPTS • Human-centered computing → Ubiquitous and mobile computing systems and tools.
... TangToys have been presented in focus groups to teachers of young students with mild to moderate learning disabilities to provide feedback on the design and functionality of the interfaces [15]. Teachers considered the methods used to interact with TangToys suitable for children and believed the way in which children interact with the toys will indicate their wellbeing. ...
Preprint
Full-text available
Children can find it challenging to communicate their emotions especially when experiencing mental health challenges. Technological solutions may help children communicate digitally and receive support from one another as advances in networking and sensors enable the real-time transmission of physical interactions. In this work, we pursue the design of multiple tangible user interfaces designed for children containing multiple sensors and feedback actuators. Bluetooth is used to provide communication between Tangible Toys (TangToys) enabling peer to peer support groups to be developed and allowing feedback to be issued whenever other children are nearby. TangToys can provide a non-intrusive means for children to communicate their wellbeing through play.
Conference Paper
Full-text available
In this work, we would like to discuss our findings obtained from the newly proposed hardware design for anxiety detection and its mitigation where a subject's state of mind is identified from the neurological data retrieved. A feedback-based network implemented for stress alleviation mainly comprises of a pair of massage motors and RGB light-emitting diode (LED) that activate during conditions of stress detected, a custom-made Electroencephalography (EEG)-sensor and a massage motor circuit both functioning on an Arduino driven platform. The skin electrodes facilitate a hassle-free retrieval of beta waves from the frontal areas that are transmitted wirelessly by a Bluetooth console to a computer post signal amplification and filtration. Rising amplitudes of beta signals that are associated to anxiety have been successfully tackled in three out of four subjects by suppressing the high values due to the massage motor therapy introduced. The motors on sensing high amplitude values exceeding the pre-set threshold limits during the three experiment trials rotate smoothly thus helping one to relax and guaranteeing a higher work performance.
Article
Full-text available
The detection and monitoring of emotions are important in various applications, e.g. to enable naturalistic and personalised human-robot interaction. Emotion detection often require modelling of various data inputs from multiple modalities, including physiological signals (e.g.EEG and GSR), environmental data (e.g. audio and weather), videos (e.g. for capturing facial expressions and gestures) and more recently motion and location data. Many traditional machine learning algorithms have been utilised to capture the diversity of multimodal data at the sensors and features levels for human emotion classification. While the feature engineering processes often embedded in these algorithms are beneficial for emotion modelling, they inherit some critical limitations which may hinder the development of reliable and accurate models. In this work, we adopt a deep learning approach for emotion classification through an iterative process by adding and removing large number of sensor signals from different modalities. Our dataset was collected in a real-world study from smart-phones and wearable devices. It merges local interaction of three sensor modalities: on-body, environmental and location into global model that represents signal dynamics along with the temporal relationships of each modality. Our ap proach employs a series of learning algorithms including a hybrid approach using Convolutional Neural Network and Long Short-term Memory Recurrent Neural Network (CNN-LSTM) on the raw sensor data, eliminating the needs for manual feature extraction and engineering. The results show that the adoption of deep-learning approaches is effective in human emotion classification when large number of sensors input is utilised (average accuracy 95% and F Measure=%95) and the hybrid models outperform traditional fully connected deep neural network (average accuracy 73% and F-Measure=73%). Further more, the hybrid models outperform previously developed Ensemble algorithms that utilise feature engineering to train the model average accuracy 83% and F-Measure=82%)
Conference Paper
Full-text available
An interactive tangible interface has been developed to capture and communicate emotions between people who are missing and longing for loved ones. EmoEcho measures the wearer’s pulse, touch and movement to provide varying vibration patterns on the partner device. During an informal evaluation of two prototype devices users acknowledged how EmoEcho could help counter the negative feeling of missing someone and liked the range of feedback offered. In general, we believe, tangible interfaces appear to offer a non-obtrusive means towards interpreting and communicating emotions to others.
Conference Paper
Full-text available
When it comes to attention and notification management, most of the previous attempts to visualise notifications and smart phone usage have focused on digital representation on a screen that is not fully embedded in users’ environment. Today, the constant development in hardware and embedded systems including mini displays, LEDs, actuation as well as digital fabrication, have begun to provide new opportunities for representing data physically in the surrounding environments In this paper, we introduce a new way of visualising notification data using physical representations that are deeply integrated with the physical space and everyday objects. Based on our preliminary design and prototype, we identify a variety of design challenges for embedded data representation, and suggest opportunities for future research.
Article
Full-text available
Nowadays Personal Informatics (PI) devices are used for sensing and saving personal data, everywhere and at any time, helping people improve their lives by highlighting areas of good and bad performances and providing a general awareness of different levels of conduct. However, not all these data are suitable to be automatically collected. This is especially true for emotions and mood. Moreover, users without experience in self-tracking may have a misperception of PI applications’ limits and potentialities. We believe that current PI tools are not designed with enough understanding of such users’ needs, desires, and problems they may encounter in their everyday lives. We designed and prototype the Mood TUI (Tangible User Interface), a PI tool that supports the self-reporting of mood data using a tangible interface. The platform is able to gather six different mood states and it was tested through several participatory design sessions in a secondary/high school. The solution proposed allows gathering mood values in an amusing, simple, and appealing way. Users appreciated the prototypes, suggesting several possible improvements as well as ideas on how to use the prototype in similar or totally different contexts, and giving us hints for future research.
Article
Full-text available
Abstract Over the past few years, there has been a noticeable advancement in environmental models and information fusion systems taking advantage of the recent developments in sensor and mobile technologies. However, little attention has been paid so far to quantifying the relationship between environment changes and their impact on our bodies in real-life settings. In this paper, we identify a data driven approach based on direct and continuous sensor data to assess the impact of the surrounding environment and physiological changes and emotion. We aim at investigating the potential of fusing on-body physiological signals, environmental sensory data and on-line self-report emotion measures in order to achieve the following objectives: 1) model the short term impact of the ambient environment on human body, 2) predict emotions based on-body sensors and environmental data. To achieve this, we have conducted a real-world study ‘in the wild’ with on-body and mobile sensors. Data was collected from participants walking around Nottingham city centre, in order to develop analytical and predictive models. Multiple regression, after allowing for possible confounders, showed a noticeable correlation between noise exposure and heart rate. Similarly, UV and environmental noise have been shown to have a noticeable effect on changes in ElectroDermal Activity (EDA). Air pressure demonstrated the greatest contribution towards the detected changes in body temperature and motion. Also, significant correlation was found between air pressure and heart rate. Finally, decision fusion of the classification results from different modalities is performed. To the best of our knowledge this work presents the first attempt at fusing and modelling data from environmental and physiological sources collected from sensors in a real-world setting.
Article
Full-text available
We assessed the calming effect of doppel, a wearable device that delivers heartbeat-like tactile stimulation on the wrist. We tested whether the use of doppel would have a calming effect on physiological arousal and subjective reports of state anxiety during the anticipation of public speech, a validated experimental task that is known to induce anxiety. Two groups of participants were tested in a single-blind design. Both groups wore the device on their wrist during the anticipation of public speech, and were given the cover story that the device was measuring blood pressure. For only one group, the device was turned on and delivered a slow heartbeat-like vibration. Participants in the doppel active condition displayed lower increases in skin conductance responses relative to baseline and reported lower anxiety levels compared to the control group. Therefore, the presence, as opposed to its absence, of a slow rhythm, which in the present study was instantiated as an auxiliary slow heartbeat delivered through doppel, had a significant calming effect on physiological arousal and subjective experience during a socially stressful situation. This finding is discussed in relation to past research on responses and entrainment to rhythms, and their effects on arousal and mood.
Conference Paper
Full-text available
Stress and anxiety impact almost everyone at some point in life. In the most severe cases the effects can be extremely detrimental to overall health and everyday life. However, two thirds of those suffering go untreated. Good Vibes was created to aid those who suffer from stress regardless of whether they receive professional help. The system consists of three components: a sleeve embedded with actuators, a heart rate monitor, and a smartwatch application. Using dynamic vibration patterns, the system simulates a cascading sensation and soothes the user during stressful situations. The Good Vibes system was tested on eight participants who were put through a stress test. The findings of our evaluations showed that participants who received haptic stimulation experienced a decrease in heart rate as compared to those who did not.
Conference Paper
This paper presents the design efforts involved in making a system for supporting haptic communication between dentist and patient during dental treatment. We describe Grasp Live, a haptic interaction technology consisting of a tangible stone-like object connected to a vibro-tactile feedback device.