PreprintPDF Available

Abstract and Figures

Children can find it challenging to communicate their emotions especially when experiencing mental health challenges. Technological solutions may help children communicate digitally and receive support from one another as advances in networking and sensors enable the real-time transmission of physical interactions. In this work, we pursue the design of multiple tangible user interfaces designed for children containing multiple sensors and feedback actuators. Bluetooth is used to provide communication between Tangible Toys (TangToys) enabling peer to peer support groups to be developed and allowing feedback to be issued whenever other children are nearby. TangToys can provide a non-intrusive means for children to communicate their wellbeing through play.
Content may be subject to copyright.
TangToys: Smart Toys that can Communicate and Improve
Children’s Wellbeing
Kieran Woodward
kieran.woodward@ntu.ac.uk
Nottingham Trent University
Nottingham, UK
Eiman Kanjo
eiman.kanjo@ntu.ac.uk
Nottingham Trent University
Nottingham, UK
David J Brown
Nottingham Trent University
Nottingham, UK
Becky Inkster
University of Cambridge
Cambridge, UK
ABSTRACT
Children can nd it challenging to communicate their emotions
especially when experiencing mental health challenges. Technolog-
ical solutions may help children communicate digitally and receive
support from one another as advances in networking and sensors
enable the real-time transmission of physical interactions. In this
work, we pursue the design of multiple tangible user interfaces
designed for children containing multiple sensors and feedback
actuators. Bluetooth is used to provide communication between
Tangible Toys (TangToys) enabling peer to peer support groups to
be developed and allowing feedback to be issued whenever other
children are nearby. TangToys can provide a non-intrusive means
for children to communicate their wellbeing through play.
CCS CONCEPTS
Human-centered computing Ubiquitous and mobile com-
puting systems and tools.
KEYWORDS
Tangible User Interfaces, Children, Communication, Mental Well-
being, Emotion, Sensors
ACM Reference Format:
Kieran Woodward, Eiman Kanjo, David J Brown, and Becky Inkster. 2020.
TangToys: Smart Toys that can Communicate and Improve Children’s Well-
being. In Proceedings of UbiComp ’20. ACM, New York, NY, USA, 3 pages.
https://doi.org/10.1145/1122445.1122456
1 INTRODUCTION
The mental wellbeing of children is increasingly important as more
young people than ever before are experiencing high levels of stress
[
2
]. Tangible User Interfaces (TUIs) present new opportunities to
digitise physical interfaces to help children communicate their well-
being. Recent advances in microcontrollers and sensors enable small
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for prot or commercial advantage and that copies bear this notice and the full citation
on the rst page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specic permission and/or a
fee. Request permissions from permissions@acm.org.
UbiComp ’20, September 12–16, 2020, Cancun, Mexico
©2020 Association for Computing Machinery.
ACM ISBN 978-1-4503-XXXX-X/18/06.. .$15.00
https://doi.org/10.1145/1122445.1122456
interfaces to be developed that can process and communicate sensor
data in real-time. Children’s toys represent an ideal embodiment
for TUIs as they provide sucient space for the electronics and en-
courage tactile interactions. Although a limited number of TUIs for
mental wellbeing have previously been developed, many of these
were not engaging for children and often contained physiological
sensors which prevent physical interactions commonly used by
children to interact with objects such as toys.
While there are many challenges in developing mental wellbeing
interfaces, the decreasing cost and increasing capability of network-
ing, sensors and microcontrollers is enabling new forms of inter-
faces to be developed [
16
]. An interface that could actively monitor
and enable the communication of a user’s physical interactions and
mood would be benecial for all. Through the use of Bluetooth Low
Energy (BLE), TUIs can communicate with one another enabling
real-time communication networks to be developed.
TUIs have previously been used to provide a method to com-
municate emotions and mental health states [
17
]. Emoball [
5
] is a
physical ball that enabled users to report their emotions by squeez-
ing the device. Similarly, Subtle Stone [
4
] allowed users to express
their emotions as a colour on the stone. Using colours to represent
emotions enabled the private communication of emotions to only
those who understood such colour representations. Mood TUI [
13
]
also enabled users to self report their emotions but additionally
collected data from the usersâĂŹ smartphones such as location and
heart rate. Overall, participants in this study found TUIs exciting to
use, and that a small sized device was key for sustained interactions.
The vast majority of previously developed wellbeing interfaces
have utilised self-reporting, which children in particular may nd
challenging. Recent developments in non-invasive sensors intro-
duce the possibility to objectively and intuitively measure physical
interactions and physiological changes in real-time. Motion data
collected through accelerometers, gyroscopes and magnetometers
could be used to measure physical interactions in addition to phys-
iological sensors to monitor indicators of wellbeing. Previously,
motion data has been used to infer emotions with 81.2% accuracy
across 3 classes [
19
]. However, similar studies reported lower lev-
els of accuracy between 50% - 72% [
12
] [
10
] [
7
] when inferring
emotions from motion data.
The ability to measure and communicate wellbeing through non-
invasive sensors presents many opportunities. This research intro-
duces Tangible Toys (TangToys) with the aim of communicating
arXiv:2007.05286v1 [cs.HC] 10 Jul 2020
UbiComp ’20, September 12–16, 2020, Cancun, Mexico Woodward et al.
mental wellbeing inferred from the embedded sensors. Initial proto-
types embed sensors used to measure interactions and well-being
and Bluetooth to enable real-time communication in-situ settings.
The ability for devices to communicate with one another enables
friends to communicate when socially distant and the ability to
discover other nearby users. Furthermore, this work highlights key
directions for the continued renement of TangToys.
2 TANGIBLE TOYS (TANGTOYS)
Few sensor based interfaces have been designed for children even
though children traditionally nd it challenging to communicate
their mental wellbeing [
11
]. We introduce the concept of TangToys
as children’s toys that embed electronics to measure tangible inter-
actions. The interfaces can vary in shape, size and material ranging
from soft balls and teddies designed for younger children, to 3D
printed dgeting cubes designed for older children. As children
physically interact with TangToys in the same way as traditional
toys all of the interfaces are suitable for children and encourage
engagement by resembling similar toys.
2.1 Communication Framework
Embedding sensors within toys that can communicate with one
another through Bluetooth oers many new opportunities for real-
time interactions. BLE 4.2 has a range of around 50m allowing
TangToys to communicate with one another in locations such as
playgrounds. In the following sections we present two opportunities
for real-time digital social interaction between TangToys.
Table 1 shows the ve TangToys developed during a co-design
and co-creation workshop including 2 soft teddies, a soft ball and a
3D printed cube and torus. Each TangToy includes a microcontroller
and micro SD card to record all interactions along with bluetooth
4.2 for communication. A range of sensors can be used to monitor
children’s interactions with the toys including capacitive sensors
to measure touch and 9-Degree Of Freedom Inertial Measurement
Unit (9-DOF IMU) to measure motion. Physiological sensors can
also be embedded within the toys such as Heart Rate (HR) sensors
as they directly correlated with the sympathetic nervous system
helping to monitor mental wellbeing [
14
] [
1
]. All of the TangToys
measure motion and touch interactions while only the 3D printed
interfaces designed for older children include HR sensors to mea-
sure physiological changes.
In addition to the sensors, TangToys can provide real-time inter-
ventional feedback [
18
]. Haptic feedback to provide the sensation
of touch has been included within some of the developed proto-
types. Haptic feedback provides a physical sense resembling touch
which can improve mental wellbeing [
9
] [
6
]. Additionally, visual
feedback in the form of multi-coloured LEDs has been included
within the soft ball and teddy prototypes. These forms of feedback
can function as real-time interventions if poor mental wellbeing
can be detected, helping to alert the user.
TangToys have been presented in focus groups to teachers of
young students with mild to moderate learning disabilities to pro-
vide feedback on the design and functionality of the interfaces [
15
].
Teachers considered the methods used to interact with TangToys
suitable for children and believed the way in which children interact
with the toys will indicate their wellbeing. Additionally, teachers
Table 1: Initial TangToys prototypes.
Device Image Description
Ball
A soft ball embedding 9-DOF
IMU to measure motion, capac-
itive sensors to measure touch
and Multi-coloured LEDs to per-
form visual feedback.
Cube
A 3D printed cube embedding 9-
DOF IMU, capacitive touch, HR,
EDA sensors and haptic feed-
back
Teddy
A soft teddy embedding 9-DOF
IMU, capacitive touch sensor
and visual feedback
Torus
A 3D printed tours embedding
HR, EDA, 9-DOF IMU, capac-
itive touch sensor and haptic
feedback
Teddy
A soft teddy embedding 9-DOF
IMU, capacitive touch sensor,
haptic and visual feedback
liked the design of the toys as they appear similar to other toys
helping to reduce stigma. Overall, the teachers reported the design,
sensors and communication capabilities were all suitable for chil-
dren and believed would promote the communication of wellbeing
between friends.
2.1.1 Peer to Peer Communication (P2P). By utilising P2P commu-
nication it is possible for two connected devices to communicate
with one another. This method of communication helps friends
who may be nearby but socially distanced to provide physical com-
munication that is not possible with other devices. When one user
touches or moves their toy, the paired interface can react through
the embedded visual or haptic feedback, providing a sense of phys-
ical interaction. The visual feedback and haptic patterns can dier
dependent on the way in which the partner device is interacted
with. For example, if a child is aggressively shaking their TangToy
or touching it harshly this can result in prolonged sharp haptic
feedback patterns being played on the paired device and red visual
TangToys: Smart Toys that can Communicate and Improve Children’s Wellbeing UbiComp ’20, September 12–16, 2020, Cancun, Mexico
feedback. This enables friends to physically communicate how they
are feeling and provide comfort to one another (see gure 1).
Figure 1: Two children playing using TangToys.
The range of feedback that can be oered allows for emotions
to be wirelessly communicated with haptic feedback providing a
sense of presence as it simulates touch. Therefore, capacitive sensor
data measuring touch can be actuated on the partner device using
haptic feedback to simulate physical communication. Furthermore,
haptic feedback can be used to comfort as previous work has shows
the potential of haptic feedback to improve wellbeing [8], [3].
2.1.2 Wireless Scanning. Each TangToy can also use its Bluetooth
capabilities to broadcast its presence to other TangToys. When a
TangToy detects another device nearby this can initiate feedback
being issued to alert the child of other nearby children. This allows
a child to nd other children who may require support when not
near their friends to facilitate peer to peer communication. These
children can then interact with the devices to form a support group
to communicate their wellbeing to each other. The feedback actu-
ated when detecting other devices can be impacted by the number
of nearby interfaces. For example, if a single child is detected nearby
more subtle haptic feedback can be issued compared with more
pronounced feedback when multiple children are nearby. Similar,
the colour displayed on the TangToy can change dependent on the
number of users located nearby to alert the user visually. Using
this method of interaction would not enable the same capabilities
as the P2P communication, but would enable each device to inter-
act automatically with other nearby devices, and aord a sense of
’togetherness’.
3 CONCLUSION AND FUTURE WORK
We have presented TangToys, a new concept to combine tangible
user interfaces with traditional toys. Various sensors can be embed-
ded within TangToys to communicate physical interactions such
as movement and touch in real-time with other TangToys. Using
a peer to peer communication system enables friends to commu-
nicate their wellbeing through device interactions that can then
be actuated using haptic and visual feedback on a friend’s device.
Alternatively, TangToys can simultaneously broadcast and scan for
nearby devices allowing for TangToys to discover other TangToys
and create local support networks.
In the future, TangToys should be trialled with children, poten-
tially in schools where they will be able to communicate with each
other through touch and haptic feedback. The impact of the com-
munication networks can be measured along with the duration in
which children use the interfaces. Parental monitoring could also
be included through the use of a mobile app enabling parents to
view previous interactions with the toys.
4 ACKNOWLEDGEMENT
This project has been funded by the Nurture Network (eNurture). eNurture is funded
by UK Research and Innovation (UKRI) and their support is gratefully acknowledged
(Grant reference: ES/S004467/1). Any views expressed here are those of the project
investigators and do not necessarily represent the views of eNurture or UKRI.
REFERENCES
[1]
Nouf Alajmi, Eiman Kanjo, Nour El Mawass, and Alan Chamberlain. 2013. Shop-
mobia: An Emotion-Based Shop Rating System. In 2013 Humaine Association
Conference on Aective Computing and Intelligent Interaction. IEEE, 745–750.
[2]
American Psychological Association. 2013. Stress in America Are Teens Adopting
Adults’ Stress Habits? https://www.apa.org/news/press/releases/stress/2013/
stress-report.pdf
[3]
Ruben T Azevedo, Nell Bennett, Andreas Bilicki, Jack Hooper, Fotini
Markopoulou, and Manos Tsakiris. 2017. The calming eect of a new wear-
able device during the anticipation of public speech. Sci Rep 7, 1 (2017), 2285.
[4]
Madeline Balaam, Geraldine Fitzpatrick, Judith Good, and Rosemary Luckin.
2009. Exploring Aective Technologies for the Classroom with the Subtle Stone.
Proceedings of the 28th international conference on Human factors in computing
systems - CHI ’10 (2009), 1623.
[5]
José Bravo, Ramón Hervás, and Vladimir. 2015. Ambient intelligence for health
rst international conference, AmIHEALTH 2015 Puerto Varas, Chile, December
1âĂŞ4, 2015 proceedings. Lecture Notes in Computer Science 9456 (2015), 189–200.
[6]
Brendan Corbett, Chang S. Nam, and Takehiko Yamaguchi. 2016. The Eects of
Haptic Feedback and Visual Distraction on Pointing Task Performance. Interna-
tional Journal of Human-Computer Interaction 32, 2 (feb 2016), 89–102.
[7]
Rasam Bin Hossain, Mefta Sadat, and Hasan Mahmud. 2014. Recognition of
human aection in smartphone perspective based on accelerometer and user’s
sitting position. In 2014 17th International Conference on Computer and Information
Technology, ICCIT 2014.
[8]
Chelsea Kelling, Daniella Pitaro, and Jussi Rantala. 2016. Good vibes. In Proceed-
ings of the 20th International Academic Mindtrek Conference on - AcademicMindtrek
’16. ACM Press, New York, New York, USA, 130–136.
[9]
Josephin Klamet, Matthies, and Denys J. C. 2016. WeaRelaxAble. In Proceedings of
the 3rd International Workshopon Sensor-base d ActivityRe cognition and Interaction
- iWOAR ’16. ACM Press, New York, New York, USA, 1–6.
[10]
Andreas Færøvig Olsen and Jim Torresen. 2017. Smartphone accelerometer
data used for detecting human emotions. In 2016 3rd International Conference on
Systems and Informatics, ICSAI 2016.
[11]
Public Health England. 2016. Mental health of children in England.
https://assets.publishing.service.gov.uk/government/uploads/system/uploads/
attachment{_}data/le/575632/Mental{_}health{_}of{_}children{_}in{_}England.
pdf
[12]
Juan C. Quiroz, Min Hooi Yong, and Elena Geangu. 2017. Emotion-recognition
using smart watch accelerometer data: Preliminary ndings. In 2017 ACM Inter-
national Joint Conference on Pervasive and Ubiquitous Computing and Proceedings
of the 2017 ACM International Symposium on Wearable Computers (UbiComp ’17).
https://doi.org/10.1145/3123024.3125614
[13]
Federico Sarzotti. 2018. Self-Monitoring of Emotions and Mood Using a Tangible
Approach. Computers 7, 1 (jan 2018), 7.
[14]
Nandita Sharma and Tom Gedeon. 2012. Objective measures, sensors and compu-
tational techniques for stress recognition and classication: A survey. Computer
Methods and Programs in Biomedicine 108, 3 (dec 2012), 1287–1301.
[15]
Kieran Woodward, Eiman Kanjo, David Brown, and T. M. McGinnity. 2019. AI-
Powered Tangible Interfaces to Transform Children ’ s Mental Well-being. In The
5th IEEE International Conference on Internet of People. Leicester.
[16]
Kieran Woodward, Eiman Kanjo, David Brown, T. M. McGinnity, Becky Inkster,
and Donald J Macintyre. 2019. Beyond Mobile Apps: A Survey of Technologies
for Mental Well-being. IEEE Transactions on Aective Computing (2019).
[17]
Kieran Woodward, Eiman Kanjo, Andreas Oikonomou, and Samuel Burton. 2018.
Emoecho: A tangible interface to convey and communicate emotions. In Ubi-
Comp/ISWC 2018 - 2018 ACM International Joint Conference on Pervasive and
Ubiquitous Computing and Proceedings of the 2018 ACM International Symposium
on Wearable Computers.
[18]
Kieran Woodward, Eiman Kanjo, Muhammad Umir, and Corina Sas. 2018. Har-
nessing Digital Phenotyping to Deliver Real-Time Interventional Bio-Feedback.
In WellComp’19: 2nd International Workshop on Computing for Well-Being - UBI-
COMP 2019. https://doi.org/10.1145/1234567890
[19]
Zhan Zhang, Yufei Song, Liqing Cui, Xiaoqian Liu, and Tingshao Zhu. 2016. Emo-
tion recognition based on customized smart bracelet with built-in accelerometer.
PeerJ 4 (2016), e2258.
ResearchGate has not been able to resolve any citations for this publication.
Article
Full-text available
Mental health problems are on the rise globally and strain national health systems worldwide. Mental disorders are closely associated with fear of stigma, structural barriers such as financial burden, and lack of available services and resources which often prohibit the delivery of frequent clinical advice and monitoring. Technologies for mental well-being exhibit a range of attractive properties which facilitate the delivery of state of the art clinical monitoring. This review article provides an overview of traditional techniques followed by their technological alternatives, sensing devices, behaviour changing tools, and feedback interfaces. The challenges presented by these technologies are then discussed with data collection, privacy, and battery life being some of the key issues which need to be carefully considered for the successful deployment of mental health toolkits. Finally, the opportunities this growing research area presents are discussed including the use of portable tangible interfaces combining sensing and feedback technologies. Capitalising on the data these ubiquitous devices can record, state of the art machine learning algorithms can lead to the development of robust clinical decision support tools towards diagnosis and improvement of mental well-being delivery in real-time.
Conference Paper
Full-text available
An interactive tangible interface has been developed to capture and communicate emotions between people who are missing and longing for loved ones. EmoEcho measures the wearer’s pulse, touch and movement to provide varying vibration patterns on the partner device. During an informal evaluation of two prototype devices users acknowledged how EmoEcho could help counter the negative feeling of missing someone and liked the range of feedback offered. In general, we believe, tangible interfaces appear to offer a non-obtrusive means towards interpreting and communicating emotions to others.
Article
Full-text available
Nowadays Personal Informatics (PI) devices are used for sensing and saving personal data, everywhere and at any time, helping people improve their lives by highlighting areas of good and bad performances and providing a general awareness of different levels of conduct. However, not all these data are suitable to be automatically collected. This is especially true for emotions and mood. Moreover, users without experience in self-tracking may have a misperception of PI applications’ limits and potentialities. We believe that current PI tools are not designed with enough understanding of such users’ needs, desires, and problems they may encounter in their everyday lives. We designed and prototype the Mood TUI (Tangible User Interface), a PI tool that supports the self-reporting of mood data using a tangible interface. The platform is able to gather six different mood states and it was tested through several participatory design sessions in a secondary/high school. The solution proposed allows gathering mood values in an amusing, simple, and appealing way. Users appreciated the prototypes, suggesting several possible improvements as well as ideas on how to use the prototype in similar or totally different contexts, and giving us hints for future research.
Conference Paper
Full-text available
This study investigates the use of accelerometer data from a smart watch to infer an individual’s emotional state. We present our preliminary findings on a user study with 50 participants. Participants were primed either with audio-visual (movie clips) or audio (classical music) to elicit emotional responses. Participants then walked while wearing a smart watch on one wrist and a heart rate strap on their chest. Our hypothesis is that the accelerometer signal will exhibit different patterns for participants in response to different emotion priming. We divided the accelerometer data using sliding windows, extracted features from each window, and used the features to train supervised machine learning algorithms to infer an individual’s emotion from their walking pattern. Our discussion includes a description of the methodology, data collected, and early results.
Article
Full-text available
We assessed the calming effect of doppel, a wearable device that delivers heartbeat-like tactile stimulation on the wrist. We tested whether the use of doppel would have a calming effect on physiological arousal and subjective reports of state anxiety during the anticipation of public speech, a validated experimental task that is known to induce anxiety. Two groups of participants were tested in a single-blind design. Both groups wore the device on their wrist during the anticipation of public speech, and were given the cover story that the device was measuring blood pressure. For only one group, the device was turned on and delivered a slow heartbeat-like vibration. Participants in the doppel active condition displayed lower increases in skin conductance responses relative to baseline and reported lower anxiety levels compared to the control group. Therefore, the presence, as opposed to its absence, of a slow rhythm, which in the present study was instantiated as an auxiliary slow heartbeat delivered through doppel, had a significant calming effect on physiological arousal and subjective experience during a socially stressful situation. This finding is discussed in relation to past research on responses and entrainment to rhythms, and their effects on arousal and mood.
Article
Full-text available
Background: Recently, emotion recognition has become a hot topic in human-computer interaction. If computers could understand human emotions, they could interact better with their users. This paper proposes a novel method to recognize human emotions (neutral, happy, and angry) using a smart bracelet with built-in accelerometer. Methods: In this study, a total of 123 participants were instructed to wear a customized smart bracelet with built-in accelerometer that can track and record their movements. Firstly, participants walked two minutes as normal, which served as walking behaviors in a neutral emotion condition. Participants then watched emotional film clips to elicit emotions (happy and angry). The time interval between watching two clips was more than four hours. After watching film clips, they walked for one minute, which served as walking behaviors in a happy or angry emotion condition. We collected raw data from the bracelet and extracted a few features from raw data. Based on these features, we built classification models for classifying three types of emotions (neutral, happy, and angry). Results and discussion: For two-category classification, the classification accuracy can reach 91.3% (neutral vs. angry), 88.5% (neutral vs. happy), and 88.5% (happy vs. angry), respectively; while, for the differentiation among three types of emotions (neutral, happy, and angry), the accuracy can reach 81.2%. Conclusions: Using wearable devices, we found it is possible to recognize human emotions (neutral, happy, and angry) with fair accuracy. Results of this study may be useful to improve the performance of human-computer interaction.
Article
Full-text available
Previous research has not fully examined the effect of additional sensory feedback, particularly delivered through the haptic modality, in pointing task performance with visual distractions. This study examined the effect of haptic feedback and visual distraction on pointing task performance in a 3D virtual environment. Results indicate a strong positive effect of haptic feedback on performance in terms of task time and root-mean square error of motion. Level of similarity between distractor objects and the target object significantly reduced performance, and subjective ratings indicated a sense of increased task difficulty as similarity increased. Participants produced the best performance in trials where distractor objects had a different color but the same shape as the target object and constant haptic assistive feedback was provided. Overall, this study provides insight towards the effect of object features and similarity and the effect of haptic feedback on pointing task performance.
Conference Paper
Mental health is placing increasing pressure on global health organisations with modern lifestyles significantly contributing to worsening mental well-being. Mental health challenges do not only affect adults but also children with increasing social and academic pressure. Technologies for mental well-being possess many qualities that offer the potential for people who may otherwise not receive help due to fear of stigma or lack of resources to improve how they feel. In this work, we pursue the design of multiple tangible interfaces for children containing sensors with the aim of using artificial intelligence (AI) to automatically infer their mental well-being and custom buttons to enable the real-world data to be labelled. Tangible interfaces for mental well-being are ideal as they provide a non-intrusive means to infer and communicate mental well-being through play