Content uploaded by Jorge Lains
Author content
All content in this area was uploaded by Jorge Lains on Sep 12, 2019
Content may be subject to copyright.
DEPARTMENT: Applications
Using Virtual Reality to
Increase Motivation in
Post-Stroke Rehabilitation
VR therapeutic mini-games help in post-stroke recovery
VR applications meet fundamental principles of
rehabilitation: intensity, task oriented training,
biofeedback, environments rich in stimuli, and
motivation, all pivotal factors for the success of
rehabilitation programs. This paper describes the
development process of a set of VR mini-games
developed to increase the motivation of stroke
patients while performing repetitive upper limb
movements.
Technology may significantly improve the lives of people suffering from incapacity or deficien-
cy affecting millions worldwide. Virtual Reality (VR) is already used to help patients endure
pain and disease treatment1,2,3, 4 as well as recover from stroke,5,6 among other applications in
medicine. VR has a significant potential for rehabilitation7,8 as it allows the creation of Virtual
Environments (VEs) providing multiple stimuli and fostering the improvement of motor and
cognitive capacities while motivating and engaging the patients. Moreover, VR applications
may meet the four basic principles of rehabilitation: intensity, task oriented training, biofeedback
and motivation, all pivotal factors for the success of rehabilitation programs. 6,9,10 ,11
The following benefits of using VR in rehabilitation have been reported in the literature:7 better
performance, improvement of the affected limb and cognitive functions, neuroplasticity stimula-
tion, and greater autonomy in the daily life activities, while increasing the patients’ motivation
and collaboration during the rehabilitation program. In particular, some authors have ‘found
evidence that the use of virtual reality and interactive video gaming may be beneficial in improv-
Paulo Dias, Ricardo Silva
DETI/IEETA Universidade de
Aveiro, Aveiro, Portugal
Paula Amorim, Jorge Laíns,
Eulália Roque, Inês Serôdio
Fátima Pereira
Centro de Medicina de
Reabilitação da Região Centro -
Rovisco Pais, Tocha, Portugal
Beatriz Sousa Santos
DETI/IEETA Universidade de
Aveiro, Aveiro, Portugal
Editor:
Mike Potel
potel@wildcrest.com
Beatriz Sousa Santos
DETI/IEETA Universidade de
Aveiro Aveiro, Portugal
COMPUTER GRAPHICS AND APPLICATIONS
ing upper limb function and ADL (Activities of Daily Living) function when used as an adjunct
to usual care (to increase overall therapy time) or when compared with the same dose of conven-
tional therapy'. 7 This makes VR an exciting tool in the future of therapy, ‘not only because it
was proven to be effective among sick and healthy subjects but also because it had very little
side-effect and was much safer than other aggressive or offensive therapies’. 8
Recently, affordable sensors developed by the gaming industry have been explored for rehabili-
tation. 6,12 This synergy between benefits and affordable technology makes VR applications a
natural approach for stroke rehabilitation, one of the main causes of incapacity worldwide.
Aware of this potential, and concerned with the lack of motivation of stroke patients while per-
forming repetitive upper limb movements in acute, subacute, and chronic phases, a group of
professionals at ‘Centro de Medicina de Reabilitação da Região Centro – Rovisco Pais’, a na-
tional Rehabilitation Center in Portugal, contacted the Universidade de Aveiro to develop VR
therapeutic serious games aimed at increasing motivation by providing everyday life context to
the movements. Several VR applications were developed using a Leap Motion sensor
(www.leapmotion.com) to track upper limb movements. These applications help patients per-
form relevant shoulder, arm and hand movements, while immersing them in an informal game-
like Virtual Environment (VE). This paper describes the development of the applications and the
main results of a study involving a group of 12 patients of the Rehabilitation Center.
VR APPLICATIONS
With the goal of maximizing the usefulness and efficacy of
the applications and taking into consideration the specific
nature of their users and context of use, the initial phase of
the process involved a series of visits to the Rehabilitation
Center and meetings, first with a group of interested physi-
atrists, and later also with physical and occupational thera-
pists. These meetings helped establish a common ground of
mutual understanding of what patients need and what the
technology can provide, thenceforth facilitating the com-
munication between the teams. The first outcome of these
meetings was the awareness that the ideal VR platform
should encompass not only a set of ‘mini-games’ to moti-
vate patients during the essential but tedious sessions of
upper limb rehabilitation (the initial goal), but also the
possibility of personalization of the games as well as re-
mote monitoring of the patients’ progress, allowing a better
follow-up of the patients’ evolution beyond the Rehabilita-
tion Center. This is a very important feature allowing
patients to actively participate in their program at home. As
a result of this initial phase, a set of decisions concerning
the design and implementation of the applications were
made, and the physiatrists and therapists stayed involved
during the process, regularly giving feedback and helping
establish intermediate goals.
The Leap Motion controller was selected as the sensor to monitor both coarse gestures (shoulder
or elbow movement, detected due to change of hand position) and fine movements (finger pinch-
es) since it detects the position, orientation and current state of the hand. The games were devel-
oped in Unity3D (unity3d.com). This platform allows the creation of VEs as well as game logic
and facilitates the virtual world creation interface as well as native integration with an Oculus
Rift DK2 Head Mounted Display (www.oculus.com/rift) and the official Leap Motion SDK
package.
The system includes a backend server controlling access to the database and the front-end 3D
applications used by patients, as well as a configuration web page. This allows for storage and
management of game configuration data (game instance, number of iterations, maximum com-
The VR platform
should encompass
not only a set of
mini-games to
motivate patients,
but also the
possibility of
personalization
and remote
monitoring of
patient progress
APPLICATIONS
pletion time, difficulty level and other aspects of the game), and game results (task completion,
time elapsed and specific values concerning the patient’s movements as the longest distance
reached).
Our first goal was to define which gestures were relevant for the exercises to be performed by
patients during the games. The ‘Enjalbert Test’ was selected as the basis for the applications to
be developed since it was already used to evaluate patients’ progress at the Rehabilitation Center.
13 The test, a six-level scale, is used to access the current state of the upper limb movement re-
covery for a post-stroke victim and includes different movements, ranging from 0 (no upper limb
movement) to 6 (fine pincer movements with all fingers):
• Lifting and holding the hand in place (shoulder)
• Bringing the hand to the mouth (shoulder and elbow)
• Opening and closing the hand (hand)
• Executing fine pinch movements with the index and middle fingers (hand)
• Executing fine pincer movements with the ring and pinky fingers (hand)
An important requirement was that the games should
evoke real life situations and be aimed at helping patients
recover capacities for an independent life. Thus, it was
decided to develop five mini-games, focused on move-
ments involved in progressing through the Enjalbert scale.
The games developed to exercise the first three gestures
passed a first round of tests with patients (in the same
order as the list above):
• Lift: the patient should lift a dumbbell above a
specified line (Figure 1) and hold it for a prede-
fined time before bringing it back down. This
action should be repeated for a predefined num-
ber of times;
• Apple Eater: the patient should reach one of the
2 apples (Figure 2) on a table and bring it to the
mouth;
• Dish Washer: the patient should wash the dish-
es, opening and closing their hand to turn on and
off the sink’s faucet (Figure 3). The patient must
keep the hand open until the dish is entirely clean;
Two more games were developed to exercise ‘finger pinch’ movements that required users to
pick objects from a box and drop them on a table using different pinch gestures. However, due
to the unreliability of the Leap Motion controller for very fine gestures, doctors concluded these
games were not responsive enough to be tested with patients.
Data such as the duration of each movement, number of repetitions, height of the barbell line,
number of apples on each side of the table, number of dishes, or what is considered an open hand
are configured through a backend web page. A calibration application was also developed to
configure the games according to the patient’s condition, essential for allowing the patients to
accomplish the task. With this application the limits for values such as ‘maximum height when
lifting arm’ or ‘maximum hand opening’ can be set for each patient and updated according to the
patients progress along their rehabilitation program.
The games should
evoke real life
situations and be
aimed at helping
patients recover
capacities for an
independent life.
COMPUTER GRAPHICS AND APPLICATIONS
Figure 1: The ‘Lift’ game: user lifts barbell to a specified height a target number of times.
.
Figure 2: The ‘Apple Eater’ game: user takes an apple to the mouth a number of times.
Figure 3: ‘Dish Washer’ game: user opens and closes the hand to wash the dish a number of times.
As part of the development process, several rounds of preliminary tests were performed at the
Rehabilitation Center with the help of doctors, therapists and volunteer patients who played the
games. This formative evaluation phase had a twofold purpose: identify and correct possible
limitations of the applications and assess whether the patients liked and were motivated by the
mini-games. Some modifications were made, mostly regarding the distance between the virtual
hand resting position and the interaction objects, since in an initial phase applications were only
tested by users with full control of their upper limb and these issues were not noticed. Another
relevant improvement was the addition of a score and a ‘success’ sound effect at the completion
of the task, in a way to provide positive feedback and encouragement, and allow for competition
among patients, features that were considered important to increase motivation. On the other
APPLICATIONS
hand, when patients did not attain the goal, discouraging sounds or negative messages were not
given so as to avoid patient frustration.
Beyond testing the mini-games, these preliminary testing sessions were also meant to instruct the
therapists on how to use the system, especially the configuration settings, as they would be the
main users.
USER STUDY
A VR system was installed at the Rehabilitation Center to enable its patients to use the devel-
oped applications. The VR setup is composed by the following elements, shown in Figure 4;
• A desktop computer to run the applications and local backend server (marked “1” in
the figure).
• A 4k definition monitor to display the VE, when running the applications in a non-
immersive setting (“2”).
• An Oculus Rift DK2 HMD (Head Mounted Display) to display the VE, when running
the applications in a fully immersive setting (“3”).
• A Leap Motion controller to track the position and orientation of the patient’s hands,
so they can be represented and used in the VE (“4”).
• A speaker positioned in front of the patient to provide audio feedback (“5”).
Figure 4: VR system used at the Rehabilitation Center: 1) computer, 2) monitor, 3) Oculus Rift head
Mounted display, 4) Leap Motion controller, 5) speaker.
To evaluate the developed mini-games, a pilot study was conducted after a formal authorization
by the Rehabilitation Center Ethics Committee and a careful selection of the patients that should
participate. The aim of this study was to establish which selection standards should be applied
regarding which patients could use the applications and benefit from them, as well as to obtain
data regarding the patients’ satisfaction with the games.
The main questions to be answered by our study were: 1) at what level of recovery could the
patients start using the mini-games? 2) which exclusion criteria should be used? 3) which partic-
ular stroke sequelae cause unusual results in a patient’s capability and enjoyment when playing?
4) is this type of treatment well accepted by the patients? 5) is there a preference regarding the
level of immersion (non-immersive vs. full immersion)?
A group of 12 patients (six female) aged between 39 and 71 in several phases of recovery and
suffering from different stroke sequelae were selected to test the applications, using both the
immersive and non-immersive versions of the games. The patients used the applications while
seated and then answered a questionnaire regarding their satisfaction with the mini-games, al-
ways accompanied by a developer and a therapist.
First, the patient was instructed about the test and the equipment he/she would be using. The
patient then played the mini-games twice, using the computer screen as the display and the Ocu-
COMPUTER GRAPHICS AND APPLICATIONS
lus Rift DK2 HMD. To prevent bias, half the patients used the non-immersive version of the
applications first, while the other half started by using the HMD.
After the games were concluded (successfully or not) the patient answered orally part of the
questionnaire (concerned with familiarity with technology; non-immersive versus fully immer-
sive and general questions). The remaining sections of the questionnaire (doctor/therapist cre-
dentials; patient information; occupation therapy) were answered by the doctors and therapists.
RESULTS AND DISCUSSION
Although most patients were not familiar with computer games (9 out of 12 had never played
videogames) or VR (10 out of 12), the mini-games were well accepted both in the rate of success
and in satisfaction. Only 2 patients were not able to successfully complete all 3 mini-games and
only one was not able to complete any of them. And all the patients who were able to play came
away satisfied, claiming to have enjoyed the experience and expressing interest in including VR
as part of the rehabilitation therapy. ‘Lift’ was the preferred game, followed by ‘Dish Washer’.
Although no significant differences in performance or acceptance between the fully immersive
and non-immersive games were noted, when asked, most patients claimed to prefer the fully
immersive version of the system. Two patients preferred the non-immersive version of the mini-
games. One of the patients had never used a computer before and found full immersion to be too
invasive. The other patient suffered from proprioceptive sensitivity and, as explained by the
doctors, stroke victims with this particular sequela feel the need to look at their hand in order to
execute the movements. Thus, not being able to see the real hand when using the HMD may
have caused the patient to feel less attracted to the fully immersive version, despite being able to
successfully play in both versions.
No patients expressed feeling any type of cybersickness during or after playing. This was ex-
pected as none of the mini-games involves any virtual full body movements, or rapidly changing
images, which are important causes for this kind of side effect.
When asked in which setting, individual or social, they would prefer to play the games, the
results were approximately the same for both. It was also noted that one patient participating in
the study had previously played the mini-games, during the preliminary tests, with greater suc-
cess. Although being able to complete all three games both times, during the study the patient
was suffering from depression, which was considered to be a plausible cause for the decrease in
performance.
Some limitations were found regarding the use of the Leap Motion sensor as a tracker. Two
specific issues were considered relevant: the position of the sensor on top of the table proved too
hard to reach by patients in early phases of recovery. This obstacle was overcome by placing a
board on the patient’s lap and positioning the sensor on it. In the ‘Apple Eater' game, because the
patient’s mouth position in the virtual environment was static, unless the patient kept his/her
back straight throughout the full exercise, this position would no longer correspond to the actual
mouth area of the patient. This issue was amplified by the fact that the patients would lean for-
ward to reach the objects, and was alleviated by reminding the patients to keep their back straight
during the procedure.
CONCLUSION
Overall, the potential use of the mini-games in occupational therapy in post-stroke rehabilitation
was very well received by patients, doctors and therapists, with its major benefit being the in-
crease in a patient’s motivation for recovery through the use of fun and relaxed environments,
which successfully distract the patient from the dull clinical setting at an affordable cost.
The collaboration continues with the development of more applications, both aimed at upper
limb movement recovery and rehabilitation for other stroke sequelae and further tests to intro-
duce this approach in the routine therapy of the Rehabilitation Center at least in some phases of
their recovery. The next phase will be the evaluation of the efficacy of this approach as an addi-
APPLICATIONS
tional therapeutic instrument in the rehabilitation of post-stroke patients in acute, sub-acute and
chronic phases, through a longitudinal study involving a larger number of patients with a wider
variety of conditions, both at the Rehabilitation Center and at home.
Augmented Reality (AR) based physical therapy games using smartphones might also be a
promising direction as they lower the barrier to greater home-based use and technological litera-
cy of the population is increasing. Compared to conventional approaches, AR alternatives allow
adapting the exercises to the patients’ interests and habits potentially increasing their motivation.
Nevertheless, immersive VR-based games may be ultimately more engaging.
ACKNOWLEDGMENTS
We thank the students and patients who participated in the tests and user study. This work
was partially supported by FCT, under its project UID/CEC/00127/2013.
REFERENCES
1. Rosa M. Baños, Macarena Espinoza, Azucena García-Palacios, et al. 2013. A positive
psychological intervention using virtual reality for patients with advanced cancer in a hospital
setting: a pilot study to assess feasibility. Supportive Care in Cancer 21, 1: 263–270.
2. Christopher V. Maani, Hunter G. Hoffman, Michelle Morrow, et al. 2011. Virtual Reality Pain
Control During Burn Wound Debridement of Combat-Related Burn Injuries Using Robot-Like
Arm Mounted VR Goggles. The Journal of Trauma: Injury, Infection, and Critical Care 71,
supplement: S125–S130.
3. David R. Patterson, Mark P. Jensen, Shelley A. Wiechman, and Sam R. Sharar. 2010. Virtual
Reality Hypnosis for Pain Associated With Recovery From Physical Trauma. International Journal
of Clinical and Experimental Hypnosis 58, 3: 288–300.
4. Susan M Schneider and Linda E Hood. 2007. Virtual reality: a distraction intervention for
chemotherapy. Oncology nursing forum 34, 1: 39–46.
5. Sangwoo Cho, Jeonghun Ku, Yun Kyung Cho, et al. 2014. Development of virtual reality
proprioceptive rehabilitation system for stroke patients. Computer Methods and Programs in
Biomedicine 113, 1: 258–265.
6. Mario Covarrubias, Monica Bordegoni, Mauro Rosini, Eleonora Guanziroli, Umberto Cugini, and
Franco Molteni. 2015. VR system for rehabilitation based on hand gestural and olfactory
interaction. Proceedings of the 21st ACM Symposium on Virtual Reality Software and Technology -
VRST ’15, ACM Press, 117–120.
7. Kate E Laver, Belinda Lange, Stacey George, Judith E Deutsch, Gustavo Saposnik, and Maria
Crotty. 2017. Virtual reality for stroke rehabilitation. Cochrane Database of Systematic Reviews
2017. Retrieved February 26, 2018 from http://doi.wiley.com/10.1002/14651858.CD008349.pub4.
8. Zhejun Liu, Sijia Wangluo, and Hua Dong. 2016. Advances and tendencies: A review of recent
studies on virtual reality for pain management. Lecture Notes in Computer Science (including
subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Springer,
Cham Switzerland, 512–520.
9. Mario Covarrubias, Alessandro Mansutti, Monica Bordegoni, and Umberto Cugini. 2014.
Interacting Game and Haptic System Based on Point-Based Approach for Assisting Patients after
Stroke. In Springer, Cham Switzerland, 289–296.
10. Paweł Kiper, Lamberto Piron, Andrea Turolla, Joanna Stożek, and Paolo Tonin. The effectiveness
of reinforced feedback in virtual environment in the first 12 months after stroke. Neurologia i
neurochirurgia polska 45, 5: 436–44.
11. L Piron, A Turolla, M Agostini, et al. 2009. Exercises for paretic upper limb after stroke: A
combined virtual-reality and telemedicine approach. Journal of Rehabilitation Medicine 41, 12:
1016–102.
12. Alana Da Gama, Pascal Fallavollita, Veronica Teichrieb, and Nassir Navab. 2015. Motor
Rehabilitation Using Kinect: A Systematic Review. Games for Health Journal 4, 2: 123–135.
13. M. Enjalbert, J. Pelissier, D. Blind. 1998. Classification fonctionnelle de la préhension chez
l'hemiplégic adulte, Hémiplégie vasculaire de l'adulte et medicine de rééducation, J. Pélissier, ed.,
Masson, Paris, 212-23.
COMPUTER GRAPHICS AND APPLICATIONS
ABOUT THE AUTHORS
Paulo Dias is assistant professor in the Department of Electronics Telecommunications and
Informatics (DETI) at the University of Aveiro and a researcher at the Institute of Electron-
ics and Informatics Engineering of Aveiro (IEETA). His research interests include visual
computing and robotics. Contact him at paulo.dias@ua.pt.
Ricardo Silva obtained an MSc in Computer Engineering from the University of Aveiro,
Portugal, and he developed this work in the scope of his dissertation. Contact him at ri-
cardojsilva@ua.pt.
Paula Amorim is a Physical and Rehabilitation Physician at the Portugal Centre Region Re-
habilitation Medicine Centre Rovisco Pais (CMRRC-RP). Her current research interests in-
clude Telereabilitation, Virtual and Augmented Reality. Contact her at
pamorim@roviscopais.min-saude.pt.
Jorge Lains is a Physical and Rehabilitation Physician at the Portugal Centre Region Reha-
bilitation Medicine Centre Rovisco Pais (CMRRC-RP). Contact him at
jorgelains@roviscopais.min-saude.pt.
Eulália Roque, Fátima Pereira e Inês Serôdio are occupational therapists at the Portugal
Centre Region Rehabilitation Medicine Centre Rovisco Pais (CMRRC-RP). Contact them
at toc@roviscopais.min-saude.pt.
Beatriz Sousa Santos is associate professor in the Department of Electronics Telecommuni-
cations and Informatics (DETI) at the University of Aveiro and a researcher at the Institute
of Electronics and Informatics Engineering of Aveiro (IEETA), Portugal. Her research in-
terests include virtual and augmented reality. Contact her at bss@ua.pt.
Contact editor Mike Potel at potel@wildcrest.com.
