Content uploaded by Kieran Woodward
Author content
All content in this area was uploaded by Kieran Woodward on Jul 12, 2023
Content may be subject to copyright.
Co-Creating an Object Recognition Exergame
with Hospital Service Users to Promote Physical
Activity
Kieran Woodward, Eiman Kanjo, and Will Parker
Department of Computer Science, Nottingham Trent University, Nottingham, UK
{kieran.woodward, eiman.kanjo, william.parker}@ntu.ac.uk
Abstract. It is challenging to encourage hospital service users with
mental health conditions to engage in physical activity despite its proven
benefits. Technological implementation, such as mobile games could of-
fer a method of improving healthcare delivery without significant cost.
Co-design helps create a sense of empowerment and a feeling of com-
petence, which benefits the participants as they derive satisfaction and
fun while feeling useful through their participation. Therefore, we pro-
pose the co-creation of objects using papier-mˆach´e for use within an AI
exergame whereby hospital service users and staff construct co-created
models that are placed along a mile long walk around the grounds for
use within the game. This enables a novel approach to gamify exercise
and promote physical activity by developing a smartphone app that en-
courages hospital service users to search for and scan co-created objects
around the hospital grounds. The game has successfully engaged hospi-
tal service users as they both enjoyed the active gamified experience and
took ownership of the co-designed objects used within the game.
Keywords: Co-Deign ·Mental Wellbeing ·App ·Object Recognition ·
AI ·Exergame ·Physical Activity ·Gamifiction.
1 Introduction
Mental health problems constitute a global challenge that affects a large number
of people of all ages and socioeconomic backgrounds. According to the Mental
Health Foundation, about a quarter of the population will experience some form
of mental health problem in the course of a year [22]. Physical activity’s (PA)
health benefits are proven and wide-ranging. However, people with mental health
challenges tend to be less physically active and it is even more so challenging to
encourage hospital service users with mental health conditions to engage in PA
[4]. Technological implementation, such as mobile health (mHealth), could offer
a method of improving healthcare delivery without significant cost. Specifically,
Exergames are video games that require physical movement to play. Exergames
make physical activity more enjoyable for users by providing a fun gaming en-
vironment with gamification that motivates and engages people to participate
in physical activity. The goal of exergames is to create games that are fun but
2 K. Woodward et al.
also motivate players to be more physically active therefore the development of
fun, sustainable physical activities that people with mental health conditions
will be motivated to participate in consistently and frequently would be highly
beneficial.
Technological advancements such as Artificial intelligence (AI), in particular
object recognition, is becoming an increasingly popular but its uses for enter-
tainment have thus far been limited. With the ubiquity of smartphones capable
of running AI models, interactive apps utilising object recognition is an ideal
method to blend the virtual and physical worlds whilst promoting exercise. AI
applications present many opportunities for the future of Human-Computer In-
teraction (HCI), by revolutionizing the way users seamlessly interact with the
real world using a real-time camera feed. Object recognition presents many op-
portunities for promoting exercise as it can be used to engage players and en-
courage exploration. The vast majority of existing implementations of object
recognition are not for entertainment purposes. This may be because that un-
til recently small devices such as smartphones were not capable of classifying a
real-time camera feed.
Co-designing an AI exergame is a vital step for actively engaging people who
have mental health conditions, allowing them to make a meaningful contribution
to the design process [2], [1], [8]. Co-design helps to solve real-world problems
by bringing together people from different backgrounds into the design process,
resulting in more inclusive solutions. We propose the co-creation of sculptures
and objects using papier-mˆach´e whereby hospital service users and staff con-
struct co-created models that are utilised within an object detection treasure
hunt exergame.
This research outlines the development of a smartphone app that utilises
advances in deep learning to detect the co-designed objects in real-time. We
have worked with occupational therapists at Highbury mental health Hospital,
Nottingham to help engage patients in the design and development of the casual
exergame and co-designed models to promote physical activity. The hospital cur-
rently have a mile long walk around the grounds which service users are actively
encouraged to engage with, but a lack of motivation and the mundane nature
of walking around hospital grounds prevented many from exercising despite the
benefits of PA. Therefore, this research focuses on the co-design of objects with
hospital service users for use within an exergame demonstrating the ability to
engage people with SMI in the solutions to help them become more physically
active.
Title Suppressed Due to Excessive Length 3
2 Background of Exergames
Physical inactivity is a significant issue for people with serious mental illness
(SMI), including psychotic disorders such as schizophrenia spectrum disorders,
bipolar disorder, and major depressive disorder [4]. Lack of motivation is a key
factor contributing to this inactivity, and therefore, new ways to encourage indi-
viduals with mental health conditions to engage in physical activity are needed.
Smartphone apps, specifically game-inspired apps, have emerged as a potential
solution to this issue.
Exergames, or games that require physical activity to play, have been devel-
oped and tested by researchers in the past. Examples include World of Work-
out [5], a mobile exergame that uses a user’s steps to play a role-playing game
and complete quests by walking a certain number of steps, and PiNiZoRo [18],
in which players walk to locate enemies and then complete puzzle games to
defeat them. Most popularly, the Pokemon Go platform [11], which combines
augmented reality, exergaming, and location-based multiplayer features, gained
widespread attention on smartphones in the summer of 2016.
Most previous exergames have been developed for individuals without SMI.
However, a study using a Wiimote with individuals who have schizophrenia [13]
found that an exergame had a positive impact on patients’ mood and motiva-
tion. Another study using a Microsoft Kinect-based exergame with individuals
suffering from schizophrenia [3] also found that exergames can have a positive
effect on physical activity levels. These findings suggest that exergames may be
a promising solution for promoting physical activity in individuals with mental
illness.
In order to encourage players with SMI to be physically active on a regu-
lar basis, it is important to engage players whilst being physically active. An
approach should also be utilised that does not block the player if they are not
sufficiently active, to avoid discouragement. However, more research is needed
to determine the effectiveness and feasibility of using smartphone apps and ex-
ergames specifically for this population.
3 Methodology
The concept of the developed game is a virtual treasure hunt completed using
a smartphone in which users must walk to find the indicated co-designed object
and then scan the object using the smartphone camera. The exergame has been
implemented at Highbury hospital, Nottingham, where players will walk around
the hospital grounds as shown in Figure 1 searching for the locations. However,
In order to develop the exergame, a number of unique objects are first required
for the object recognition activity. A co-design approach has been adopted as
it offers a unique opportunity to engage service users and enable them to take
ownership in the objects and exergame which will hopefully promote continued
engagement.
4 K. Woodward et al.
Fig. 1. Map of the Highbury Mile route.
3.1 Co-designing Objects
Co-design is the methodology for actively engaging people directly involved in
an issue, place or process in its design, allowing them to make a meaningful
contribution to the design process [2], [1], [8]. Co-design enables the reduction of
the gap in knowledge between end users and researchers, allowing non-designers
to become equal members of the design team, ensuring designer subjectivity
is removed and the technologies developed are suitable for the target popula-
tion[24], [14]. During the process, design tools are used to empower all of the
participants to facilitate a ‘joint inquiry’ where ‘problem and solution co-evolve’
[19]. Co-design brings many benefits to the design of the project by helping the
researcher better understand the challenges faced by users and any potential
solutions [20], [17].
We have adopted a participatory approach to co-design objects to use within
the exergame. This enables the design of products directly with the final users
including those with SMI in order to help them take ownership of the product.
Co-design helps create a sense of empowerment and a feeling of competence,
which benefits the participants as they derive satisfaction and fun while feeling
useful through their participation [6], [10]. We therefore involved adults with
SMI throughout the co-design process with the assumption that their inclusion
will positively impact the quality of the final product in addition to their own
experience. The co-design process was conducted with the same occupational
therapists and researchers. All participants were service users at Highbury hos-
Title Suppressed Due to Excessive Length 5
Fig. 2. The co-design process where the papier-mˆach´e models were created.
pital, Nottinghamshire with varying mental illnesses but no participants had
significant motor skill impairments that would impact their participation.
With very little around the Highbury mile to engage the users, a co-design
element was the most engaging method to support and develop the game. Various
papier-mˆach´e sessions were set-up by staff members to involve the service users
with the creation of multiple 3D objects that would later be water-proofed and
placed securely in designated areas. Over 10 service users participated in the
activity designing their own unique 3D models with each having a different shape
and colour in order for the AI models to be trained with increased accuracy.
Service users were divided into small groups and guidance was provided on how
to create the papier-mˆach´e models. The service users were then able to design
and create their own models as shown in Figure 2. After the initial models were
created they were then painted by the service users under supervision.
6 K. Woodward et al.
With the co-design models in place, a large number of photos were captured
for each of the objects. These photos were specifically taken to capture different
angles ensuring the object recognition would function reliably for users. Each of
the initial objects are displayed in Table 1.
Table 1. Description of the 5 models that were co-designed by hospital service users.
Model Name Image Description
Hot Air Bal-
loon
One of the first papier-mˆach´e objects implemented
in the the Highbury Mile was a hot air balloon
created by a service user.
Pinocchio Pinocchio is another creation by one of the ser-
vice users. This object was placed in an area that
would get the user to search for it before they
could complete the activity.
Robin A robin was also created. The service user sug-
gested that this model to be placed in a tree.
Mars One of the service users wanted to create a more
simple object that represented Mars.
Triceratops One of the final objects to be created by the
service users was a Triceratops. Similarly to the
model of Pinocchio, this object was placed in an
area that would require the user to have a look
around in order to complete the activity.
Title Suppressed Due to Excessive Length 7
3.2 Object Recognition
Advances in AI, in particular deep learning have resulted in the capability to
classify images with increasing accuracy. However, the use of AI and object recog-
nition for entertainment purposes within a game to promote physical activity
has not had much consideration. Object recognition presents many opportunities
and is becoming increasingly used in consumer apps such as enabling automated
photo tagging [7]. The developed exergame utilises advances in object recogni-
tion as players are tasked with finding the co-designed objects identified through
AI.
Convolutional Neural Networks (CNNs) are frequently used for object recog-
nition. These are feed forward networks that are constructed of numerous layers
including an input layer, an output layer and hidden layers that includes con-
volutional layers making use of a set of learnable filters, pooling layers, fully
connected layers and normalisation layers. Transfer Learning (TL) [12] is a com-
mon approach in machine learning to improve the learning of a target task by
improving initial performance, producing sharp performance growth and result-
ing in higher training performance [23]. It is based on the ability to learn new
tasks relatively fast, alleviating the need for large datasets by relying on previous,
similar data from related problems. TL capitalises on a large dataset stemming
from a related problem to pre-train a model, and then adapting the model for
the smaller target dataset [25]. CNNs are commonly used in TL approaches,
being initially trained on a vast dataset and then having the last fully-connected
layer removed and re-trained on a smaller target dataset. A pre-trained CNN
alleviates the need for a large dataset while simultaneously decreasing the time
required to train the model.
TL has most commonly been used to train images as large ImageNets have
been used to developed pre-trained models such as VGGNet [16], Inceptionv3
[21] and mobileNetv3 [9] that contain pre-trained object classification models.
TL has facilitated training new models in the visual domain using pre-trained
CNNs [15] and has been utilised to learn the papier-mˆach´e objects for the AI
object recognition activity. The accuracy of the model (using hold-out validation)
is 99%, demonstrating the ability of the model to infer each of the objects with
high precision. The model was exported as a tflite file and embedded within the
mobile app.
3.3 Smartphone Exergame
A mobile App that integrates the object recognition of the co-designed objects
has been developed to increase physical activity with hospital service users. The
app was developed using Flutter, a cross-platform programming language and
is compatible with Android and iOS operating systems. The app is currently
available on both Google Play and Apple Store.
As users approach different objects, the camera turns on for the players to
find the nearby sculpture which then gets identified by the pre-trained AI model
8 K. Woodward et al.
embedded in the app, enabling frames from the camera to be classified in real-
time, if the accuracy is above 85% the location has then been ’found’ and the
user moves on to find the next object. During the gameplay players can view
the current inference accuracy displayed as a progress bar. Players are able to
find the objects in any order with the aim of the game being for players to find
as many objects as possible. Figure 3 shows screenshots of the camera-based AI
activity and the progress of the AI model detecting the sculpture.
Fig. 3. Screenshots of the developed exergame showing the inference of a co-designed
object using the camera feed.
To ensure the app has high replayability, in some areas of the Highbury mile
multiple co-designed objects are placed within a small area and the app ran-
domly selects one of the objects for the user to find and scan. This random
element ensures walks around the mile are dynamic and more engaging for hos-
pital service users who may complete this walk daily. The game remains simple
to play in nature with users simply having to point the app to the correct object
as outlined within the app. This ensures the vast majority of service users will be
able to use the app and by having a non-punishing gameplay whereby there are
no punishments for scanning the incorrect object along with no time pressure
it enables the game to be relaxing and not stressful which may otherwise harm
service users’ mental health.
Title Suppressed Due to Excessive Length 9
4 Discussion and Conclusion
We completed the co-creation of objects with hospital service users and im-
plemented an AI exergame at Highbury hospital in Nottingham. Although PA
has been proven to be beneficial for all, motivating it for people with disabil-
ities, especially people with intellectual disabilities, is challenging and requires
specialised and individualised intervention. After the co-design sessions the oc-
cupational therapists who led the sessions provided their feedback. One said “the
freedom in design was encouraging for the service users and it really helped them
be more creative”, another said “the sense of ownership meant that they took
pride in their creations and they want to see users interacting with their model
when development is complete”. This feedback shows the co-design approach
was successful in engaging service users in the creation of the exergame.
The designed objects were all unique and had clear designs ranging from
dinosaurs to birds. This shows the time and effort service users placed when
designing their objects resulting in the pride they felt when the objects were
displayed and used within the app. The outstanding quality of the objects further
shows the benefits of involving users in the co-design process as it creates a
unique, rewarding experience that benefits the service users and the exergame.
Participants stated they would like to design additional objects with possible
new objects being created for different seasons of the year, showing how much
they enjoyed this creative element and the invaluable addition of being part of
the design process.
A preliminary trial was conducted where hospital staff took group of 3-5 ser-
vice users around the Highbury mile to use the app. All service users successfully
managed to use the app and found the Highbury Mile much more engaging than
previously, making sure to find the objects as the app instructed. Participants
took ownership in the co-designed objects making the app and associated phys-
ical activity moreso appealing. Participants both enjoyed the visual nature of
having colourful co-design objects placed around the walk as well as the gamified
element of using the app to search for and scan the correct object. Technically,
the object recognition worked well with the phones correctly identifying the ob-
jects quickly. Overall, this shows the potential for the developed exergame and
co-design process to successfully help increase PA in service users with SMI.
The co-design element of the production of the app enabled Highbury Hospi-
tal to host a variety of sessions that engaged the service users in creating objects
that would later be implemented within the game. Although the co-design had
benefits such as increasing creativity and engagement with the exergame, there
was one issue that arose after some time of testing. Due to the quality of the
materials used and the UK’s adverse weather conditions, some of the models
struggled to stay in tact when placed into the environment. At first thought,
this was a concerning issue for the integrity of the treasure hunt, but having
discussed the issues, it turned out to be a minor issue, and it was suggested to
be positive as it will encourage the continuation of the papier-mˆach´e co-design
sessions. These constant sessions will keep benefiting the services users as previ-
ously mentioned, but will also regularly improve the treasure hunt. It will ensure
10 K. Woodward et al.
that the content get’s less repetitive as new models can be implemented regularly
and it also opens up the opportunity to offer specified content such as seasonal
or holiday features.
Overall, we have conducted a co-creation process with hospital service users
and developed a novel object recognition exergame to promote physical activity.
This research provides an overview of the developed AI exergame from a co-
design perspective. It also discusses how the game extends the current scope
of object recognition to gamify the walking experience. These advances help to
promote the mass adoption of AI technologies for social good.
In the future we aim to continue trialling the exergame to gather further
feedback and explore the impact the app has on service users physical activity
as well as mental wellbeing. We will also continue working with Highbury hospital
to include new co-designed objects within the app.
References
1. Binder, T., De Michelis, G., Ehn, P., Jacucci, G., Linde, P., Wag-
ner, I.: Participation in Design Things. In: Design Things (2019).
https://doi.org/10.7551/mitpress/8262.003.0011
2. Burkett, I.: An Introduction to Co-design. Tech. rep. (2016),
http://www.csi.edu.au/
3. Campos, C., Mesquita, F., Marques, A., Trigueiro, M.J., Orvalho, V.,
Rocha, N.B.: Feasibility and acceptability of an exergame intervention for
schizophrenia. Psychology of Sport and Exercise 19, 50–58 (jul 2015).
https://doi.org/10.1016/J.PSYCHSPORT.2015.02.005
4. Daumit, G.L., Goldberg, R.W., Anthony, C., Dickerson, F., Brown, C.H., Kreyen-
buhl, J., Wohlheiter, K., Dixon, L.B.: Physical activity patterns in adults with
severe mental illness. Journal of Nervous and Mental Disease 193(10), 641–646
(oct 2005). https://doi.org/10.1097/01.NMD.0000180737.85895.60
5. Doran, K., Pickford, S., Austin, C., Walker, T., Barnes, T.: World of Workout:
Towards pervasive, intrinsically motivated, mobile exergaming. In: Meaningful Play
2010 Conference (2010)
6. Frauenberger, C., Good, J., Alcorn, A., Pain, H.: Supporting the design contribu-
tions of children with autism spectrum conditions. ACM International Conference
Proceeding Series pp. 134–143 (2012). https://doi.org/10.1145/2307096.2307112
7. Fu, J., Mei, T., Yang, K., Lu, H., Rui, Y.: Tagging personal pho-
tos with transfer deep learning. WWW 2015 - Proceedings of the 24th
International Conference on World Wide Web pp. 344–354 (may 2015).
https://doi.org/10.1145/2736277.2741112
8. Holmlid, S.: Participative, co-operative, emancipatory: From participatory design
to service design. First Nordic Conference on Service Design and Service Innovation
(2009)
9. Howard, A., Sandler, M., Chu, G., Chen, L.C., Chen, B., Tan, M., Wang, W., Zhu,
Y., Pang, R., Vasudevan, V., Le, Q.V., Adam, H., Ai, G., Brain, G.: Searching for
MobileNetV3. Tech. rep.
10. Malinverni, L., Mora-Guiard, J., Padillo, V., Mairena, M.A., Herv´as, A., Pares,
N.: Participatory design strategies to enhance the creative contribution of children
with special needs. In: ACM International Conference Proceeding Series (2014).
https://doi.org/10.1145/2593968.2593981
Title Suppressed Due to Excessive Length 11
11. Niantic: Pok´emon GO (2021), https://pokemongolive.com/en/
12. Pan, S.J., Yang, Q.: A survey on transfer learning (2010).
https://doi.org/10.1109/TKDE.2009.191
13. Patsi, C., Antoniou, P., Batsiou, S., Bebetsos, E., Lagiou, K.: Exergames and their
effect on emotional state in people with Schizophrenia. Balkan Military Medical
Review 15(4), 275–281 (2012)
14. Sanders, L.: An evolving map of design practice and design research. Interactions
(2008). https://doi.org/10.1145/1409040.1409043
15. Shah Singh, M., Pondenkandath, V., Zhou, B., Lukowicz, P., Liwicki, M., Kaiser-
slautern, T.: Transforming Sensor Data to the Image Domain for Deep Learning-an
Application to Footstep Detection . https://doi.org/10.1109/IJCNN.2017.7966182
16. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale
image recognition. In: 3rd International Conference on Learning Representations,
ICLR 2015 - Conference Track Proceedings (2015)
17. Skliarova, I., Sklyarov, V.: Hardware/software co-design. In: Lecture Notes in Elec-
trical Engineering (2019). https://doi.org/10.1007/978-3-030-20721-26
18. Stanley, K.G., Livingston, I., Bandurka, A., Kapiszka, R., Mandryk, R.L.: PiNiZoRo:
A GPS-based exercise game for families. Future Play 2010: Research, Play, Share -
International Academic Conference on the Future of Game Design and Technology pp.
243–246 (2010). https://doi.org/10.1145/1920778.1920817
19. Steen, M.: Co-design as a process of joint inquiry and imagination. Design Issues (2013).
https://doi.org/10.1162/DESIa00207
20. Steen, M., Manschot, M., de Koning, N.: Benefits of co-design in service design projects.
International Journal of Design (2011)
21. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the In-
ception Architecture for Computer Vision. In: Proceedings of the IEEE Com-
puter Society Conference on Computer Vision and Pattern Recognition (2016).
https://doi.org/10.1109/CVPR.2016.308
22. The Mental Health Foundation: Fundamental facts about mental health. Tech. rep.
(2015)
23. Tommasi, T., Orabona, F., Caputo, B.: Safety in numbers: Learning categories from
few examples with multi model knowledge transfer. In: Proceedings of the IEEE
Computer Society Conference on Computer Vision and Pattern Recognition (2010).
https://doi.org/10.1109/CVPR.2010.5540064
24. Vines, J., Clarke, R., Wright, P., McCarthy, J., Olivier, P.: Configuring participation:
On how we involve people in design. In: Conference on Human Factors in Computing
Systems - Proceedings (2013). https://doi.org/10.1145/2470654.2470716
25. Wang, J., Chen, Y., Zheng, V.W., Huang, M.: Deep Transfer Learning for Cross-
domain Activity Recognition. In: ACM International Conference Proceeding Series
(2018). https://doi.org/10.1145/3265689.3265705
