ArticlePDF Available

Abstract and Figures

Immersive Virtual Reality (iVR) Head-Mounted Display (HMD) systems paired with serious exercise games can positively augment physical rehabilitation process from both engagement and analytics perspectives. This paper presents a serious game for iVR HMD based long term upper-extremity exercise. We demonstrate the capabilities of our game through a case study with five users recovering from upper-extremity injuries. We examine how our program maintains engagement and motivation over eight weeks, where users completed bi-weekly prescribed movements framed as protecting a virtual butterfly. We assess user experiences through a mixture of biomarkers from brainwave, heart rate, and galvanic skin response recorded at runtime as well as motion capture and behavioral game data. Our results suggest that the iVR game was an effective medium in inducing high compliance, physical performance, and biometric changes even with increasing difficulty beyond the novelty effect period. We conclude with considerations of future work for iVR physical therapy games that adapt to biometric response.
Content may be subject to copyright.
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TG.2021.3069445, IEEE
Transactions on Games
Gaming Beyond the Novelty-Effect of Immersive
Virtual Reality for Physical Rehabilitation
Aviv Elor, Michael Powell, Evanjelin Mahmoodi, Mircea Teodorescu, Sri Kurniawan
University of California - Santa Cruz
Jack Baskin School of Engineering
Santa Cruz, California, United States
Abstract—Immersive Virtual Reality (iVR) Head-Mounted Dis-
play (HMD) systems paired with serious exercise games can
positively augment physical rehabilitation process from both
engagement and analytics perspectives. This paper presents a
serious game for iVR HMD based long term upper-extremity
exercise. We demonstrate the capabilities of our game through
a case study with five users recovering from upper-extremity in-
juries. We examine how our program maintains engagement and
motivation over eight weeks, where users completed bi-weekly
prescribed movements framed as protecting a virtual butterfly.
We assess user experiences through a mixture of biomarkers
from brainwave, heart rate, and galvanic skin response recorded
at runtime as well as motion capture and behavioral game data.
Our results suggest that the iVR game was an effective medium in
inducing high compliance, physical performance, and biometric
changes even with increasing difficulty beyond the novelty effect
period. We conclude with considerations of future work for iVR
physical therapy games that adapt to biometric response.
Index Terms—Serious Games, Immersive Virtual Reality, Phys-
ical Rehabilitation, User Studies, Head-Mounted Display, Games
for Health
With the mass commercial adoption of immersive Virtual
Reality (iVR) based Head-Mounted Display systems (HMDs)
and over 200 million headsets sold since 2016, iVR systems
have almost become a common household gadget [1]. On a
parallel track, rehabilitation research, including physical and
cognitive work that incorporates VR based interventions, has
been on the rise due to the ability to create programmable
immersive experiences that can directly influence human be-
havior [2]–[4]. The ability to run conventional therapy in a
virtual environment can be paired with high-fidelity motion
capture, telepresence capabilities, and accessible experiences
[5], [6]. Through serious games, immersive environments with
commercial HMDs can be programmed to translate therapeutic
goals into game mechanics, making the therapies more enjoy-
able [7]–[12].
It has been shown that iVR can be successfully used for
treating Post Traumatic Stress Disorder [13], Borderline Per-
sonality Disorder [14], various phobias [4], [15], schizophrenia
[16], and others. The detachment from reality and immersion
in a virtual world can reduce discomfort, even as far as
minimizing pain when compared to clinical analgesic treat-
ments [17]. Strong immersive stimuli through a VR system
together with the ability to combine presence and emotion
in a virtual world is key to influencing user behavior [13].
Fig. 1. A user plays Project Butterfly during their bi-weekly exercise session:
a) The user physically matches the butterfly’s position to protect it while also
catching crystals to infer the motion path. b) The user’s view of the game
through the Vive HMD. Score is visible on the user’s bubble shield, with time
and repetitions visible for the evaluator interface. c) The scene view of the
user in unity. The bottom half of the figure depicts sensor peripherals utilized
during Project Butterfly user testing.
However, quantifying this success is often difficult due to
system constraints and a lack of computational power [18].
Similarly to iVR, research in biometric sensing has seen
explosive growth over the past decade. We argue that biofeed-
back may serve as a versatile tool to quantify the success of an
iVR based physical therapy experience. Brain-Computer Inter-
faces incorporating Electroencephalogram (EEG) devices have
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TG.2021.3069445, IEEE
Transactions on Games
become more affordable and user-friendly – they come with
computational techniques for understanding user engagement
and intent in the fields of medical, entertainment, education,
gaming, and more [19]. The analysis of different brainwave
frequencies has been correlated to different psychological
functions, such as the 8-13 Hz Alpha band relating to stress
[20], the 13-32 Hz Beta band relating to focus [21], [22], the
0.5-4 Hz Delta band relating to awareness [23], [24], the 4-8
Hz Theta band relating to sensorimotor processing [25], [26],
and the Gamma band of 32-100 Hz related to cognition [27],
Researchers are combining these interfaces with other forms
of multi-modal biometric data collection such as Galvanic Skin
Response (GSR) and Heart Rate (HR) to increase affective
inference. Through measuring the changes in skin resistance,
GSR has been linked to emotional arousal [29]. Quantifying
the intensity of GSR may enable researchers to record emo-
tional response with stimuli such as music, television, and
gaming [30], [31]. Additionally, it has been shown that the
accuracy of the non-invasive prediction of the psychological
response can be increased by using data fusion algorithms
that integrate GSR, HR and EEG readings [32]–[35]. These
biometric measures paired with iVR therapy may yield much
potential for understanding physical exercise experiences, but
are not often incorporated in the traditional physical therapy
Traditional outpatient physical therapy usually involves clin-
ical visits where patients perform exercises, receive evalua-
tions, and undergo in-person manual therapy interventions.
Better clinical outcomes are often associated with a well estab-
lished schedule of at-home exercises following the clinical vis-
its [36]. Many meta-analyses and reviews have suggested that
virtual reality-based rehabilitation techniques, some with bio-
metric sensing [37]–[39], could outperform traditional physical
therapy [40]–[42]. Some of these studies have shown that
iVR therapy can be effective for weeks or months [43]–[45].
However, although there is mounting evidence that VR HMD-
based rehabilitation systems could benefit from integration
with with biometric, physical, and user-reported analysis, there
is a lack of studies with extended period investigating this
The study reported in this paper aims to answer the question
of: Can an iVR HMD experience maintain engagement beyond
the novelty period and show continued rehabilitative improve-
ment using multi-modal analysis when used as a physical ther-
apy environment? To answer this question, we expand upon
an iVR serious game for controlled physical exercise. The
purpose of the updated design is to investigate improvements
in physical performance using an iVR system by following
protocols that are similar to conventional physical rehabilita-
tion. Three outpatient Doctoral Physical Therapists, with over
40 years of combined professional experience, helped design
the protocols used in this study to match exercises used in
clinical settings. Through these three consultants, we learned
that the principles of functional shoulder rehabilitation for late-
phase recovery usually extend from 6 to 12 weeks of treatment
to “(1) restore full range of motion and flexibility... and (2)
increase strength, power, and endurance with exercises that
stress core-based muscle synergy” [46], [47]. In this study,
we extend these principles to stimulate range of motion in the
first four weeks and increase strength in the last four weeks.
Specifically, the contributions of this study are:
1) A demonstration that our iVR HMD based serious game
system can be effective for physical rehabilitation.
2) An examination of methods towards maintaining en-
gagement and motivation over extended period of time.
3) An assessment of the feasibility of using biometrics to
complement the iVR game.
In reflecting on Rego et al.’s taxonomy of serious games for
rehabilitation, we present a two-month user study on a motor
game that utilizes motion-tracking for a head-mounted display
based 3D single-player action experience with adaptability,
progress monitoring, performance feedback, and clinic-to-
home portability [48]. We define serious games as games
whose purposes are beyond entertainment only [48]. We
define iVR-based serious games as immersive experiences that
incorporate 3D user interaction and motion capture, often with
an HMD. To the best of our knowledge, this study is one of the
first to leverage an immersive VR HMD based serious game
for an extended physical therapy period and that examines
multi-modal biometric feedback and physical performance.
Our physical therapy system is based on a game called
“Project Butterfly” (PBF) that was proposed by Elor et al
[49], [50]. Previously, PBF explored the feasibility of a virtual
reality enhanced exo-skeleton for post-stroke and elderly as-
sistance through two exercises, but was not designed or tested
for upper extremity physical therapy over a extended period
of time with varying custom exercise movements as reported
in this study. The game was built using the Unity 2018.2.11f1
Game Engine with SteamVR and incorporates the HTC Vive
Pro 2018 (Vive). Vive uses outside-in tracking through a
constellation of “lighthouse” laser systems for pose collection
in a 3D 4x4m space [1]. It has been verified in previous studies
to analyze therapeutic gamification [51], [52], postural analysis
[53], and accuracy for research data collection [54].
To explore the effects of extended upper-limb rehabilitation,
we heavily modified PBF to create a new gaming experience.
These improvements were designed and suggested by the
collaborating therapists to make the game customizable and
aid best in the rehabilitation process. The updated goal of
the PBF is to safeguard a virtually flying butterfly from
adverse weather and flying projectiles using a translucent
protective “bubble shield” that is controlled by the player
through the Vive Hand Controller. The flying pattern of the
butterfly recreates the rehabilitation exercises, and the game
was modified to allow dynamic adjustment of movements
through therapist-led motion capture, enabling the dynamic
control of pace and position for the prescribed exercise.
Specifically, using the HTC Vive Controllers in PBF, therapists
can record motion paths by entering a custom unity scene
and performing exercises by holding the controller trigger
while performing the desired movement. Movement paths
are remotely uploaded into the unity environment through
comma-separated value format. The 3D motion vector path
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TG.2021.3069445, IEEE
Transactions on Games
is then previewed, confirmed, and named by the therapist,
which is then adjusted to user arm length during gameplay.
The user is required to follow the path of the butterfly with
a 0.1 meters error, where they are awarded a score point
(accompanied by audio and haptic feedback) for each half a
second they successfully protect the butterfly. This includes a
scoring system for the user to engage in self-competition.
Gameplay sessions were updated to provide feedback to
both the user and the therapist by collecting EEG, GSR, HR,
motion capture, and player behavior datum. Each exercise is
prescribed by our collaborating physical therapists to establish
range of motion in the first stage of rehabilitation and complex
movement in the last stage. The user starts following the
path by activating an animation with the controller placed on
the stationary butterfly for three seconds at the start of the
game. Collectible game projectiles were added in the shape of
crystals, and spawn from a distance while veering towards the
butterfly as a motion path indicator for the user to follow their
exercise. The gameplay of the new PBF version is shown in
Figure 1.
Additionally, the updated project applies concepts from
Self-Determination Theory (SDT [55]) to intrinsically moti-
vate users to become the guardian of a butterfly (relatedness),
gain mastery of controlling a virtual shield in following
the butterfly’s path (competence), and be in full control of
their upper extremity movement with progressive difficulty
(autonomy). Previous studies have examined motivation in
exergames through gamification (challenge and score) [56],
self-competition and virtual trainers [57], music [58], sensory
feedback [59], rhythmic clapping [60]. PBF differs from these
exergames in a gameplay mechanic aimed towards relatedness:
enacting a helping behavior on the user to protect the butterfly,
building a gradual sense of attachment and belonging to the
butterfly through becoming it’s guardian. Research has shown
that helping behavior can provide a variety of material, social,
and self-rewards such as mood enhancement to the helper
[61]. Subsequently, the care-helping relationship was found to
be strong mediator for stimulating planned, long-term helping
behaviors in user participation [62]. Through iVR, we extend
these concepts into an immersive environment. Given that
iVR therapeutic intervention’s success is often attributed to
the influence of immersion on users in terms of enhancing
the relationship between presence and emotion [4], [13], the
system proposed in this paper provides a mixture of visual,
audio, and haptic feedback for upper body movements helping
the user to overcome the adversity of physical task-based
The methods used in the study reported in this paper
received Institutional Review Board’s approval from the Uni-
versity of California - Santa Cruz Office of Research Com-
pliance Administration. Five college students recovering from
upper limb injuries (including shoulder dislocation, shoulder
impingement, and rotator cuff tears) consented to volunteer.
These participants consisted of one female and four males
ranging from 21 to 28 years old. Additionally, the participants
were no longer performing in-person physical therapies, nor
continuing at-home therapy exercises, thereby mimicking our
target population. These users agreed to volunteer for two
weekly sessions for eight weeks, although, as the consent form
stated, they were free to drop out of the study at any time.
Participants were offered a $50 USD gift card upon completing
the study (or were paid about $3.12 USD per session). All
users completed all sessions of the study.
Fig. 2. Movements tested in the Foundation and Challenge protocols. The
dotted line shows the butterfly’s path, while the arrows indicate the axis of
rotation for relevant movements.
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TG.2021.3069445, IEEE
Transactions on Games
A. Data Collection
Figure 1 shows the experimental setup. Through Vive and
the Unity Game Engine, motion capture and behavior game
data were recorded and stored during runtime at 90 Hz using
a data exportation method developed in previous studies by
Elor et al. [11], [49]. Motions were captured through utilizing
the HTC Vive Pro 2018’s outside-in tracking algorithm by
a constellation of “lighthouse” laser systems [1]. Runtime
recording of brainwave (EEG), GSR, and HR data was also
performed to complement the Vive data. EEG was sensed
from the pre-frontal cortex (TP9/10, AF1/7 locations) and
recorded using InteraXon Muse 2, a commercially available
brain sensing headband [63]. While Muse is a relatively new
EEG commercial device, it has successfully been used in
other studies to infer mental state, analyze event potentials,
and record biofeedback [64]–[66]. For GSR, we utilized the
Neulog GSR logger sensor NUL-217 [67] to measure the
skin’s conductivity between the fingers of the non-dominant
hand. HR was recorded through the LED optical sensor Polar
OH1 [68]. Every sensor was chosen with accessibility and
cost as a factor. These devices are easy to set up for a
user at home, with dry contact after wiping with saline to
reduce the costs of any EEG gels or sticktrodes for HR and
GSR that are usually required when using many clinical-grade
sensors. Sensor locations and equipment setup used for each
user testing session are seen in Figure 1.
At the end of each testing session, users were asked a series
of Likert scale questions regarding their rehabilitation session
experience with PBF. Survey questionnaires were inspired
from a Jennett et al. survey, which measures immersion
in games [69]. The survey was implemented to focus on
behavioral engagement and a self-reported emotional response
between multiple exercise sessions over many weeks. Reflect-
ing on Doherty and Doherty’s review of engagement in human-
computer interaction, we define engagement assessment as a
behavioral understanding of whether a user desires to and can
effectively use a system [70]. For understanding emotional
response, we utilized the circumplex model of affect to assess
self-reported emotions from users between exercise sessions
[71], [72]. The goals of these surveys were to understand
if users would self-perceive PBF as engaging, immersive,
and positive throughout the two months as the users become
acclimated to the game during rehabilitation. An exit interview
was also conducted at the end of the two months to better
understand participants’ subjective feedback.
B. Protocol
The study followed each of the five users during their pre-
scribed rehabilitative exercises through two protocol phases:
Foundation and Challenge. The Foundation Protocol instigated
basic motion primitives that emphasized recovery of Range of
Motion (ROM) and shoulder strength through three simple
exercises: Forward Arm Raise [FAR], Side Arm Raise [SAR],
and Shoulder Rotation [SR]. The goal of the Foundation Proto-
col was to acclimate the user into performing physical therapy
in iVR with minimal weight resistance to reduce the possibility
of injury as recommended by our collaborating therapists,
and see if the user could maintain compliance over an ex-
Game Performance
Foundation Session
Compliance Rate [%]
Foundation Protocol
Challenge Session
Compliance Rate [%]
Challenge Protocol
Foundation Session
Weak Displacement [movement/m]
Challenge Session
Weak Displacement [movement/m]
Foundation Session
Weight Resistance [lbs]
Challenge Session
Weight Resistance [lbs]
Fig. 3. Game performance between Foundation Protocol (in red of 225
recorded exercises) and Challenge Protocol (in green of 350 recorded exer-
cises). Row one shows compliance, where compliance is defined as the total
time protecting the butterfly over the game’s total time. Row two shows the
mean upper-limb displacement between all exercises required in that session.
Row three indicates the mean weight used between all exercises of that
session. Error bars indicate standard error (note the Foundation Protocol had
less variability between users, so error bars appear substantially smaller than
Challenge Protocol due to shared scale).
Physiological Response
1 2 3 4 5
Foundation Session
HR [bpm]
Foundation Protocol Change
1 2 3 4 5
Challenge Session
HR [bpm]
Challenge Protocol Change
1 2 3 4 5
Foundation Session
GSR [uS]
1 2 3 4 5
Challenge Session
GSR [uS]
Fig. 4. Physiological HR and GSR responses from gameplay are shown. Row
one illustrates mean change from resting state of heart rate. Row two illustrates
mean change from resting state of galvanic skin response. Biometric change
is calculated as the offset between gameplay biometrics against resting-state
biometrics. Error bars indicate standard error.
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TG.2021.3069445, IEEE
Transactions on Games
tended period of iVR exposure. The Challenge Protocol added
four complex motions that further pushed ROM recovery
while including the original Foundation Protocol’s exercises
using increasingly higher weights. The goal of the Challenge
Protocol was to challenge the user with continuing weight
increases and complex movements to investigate the effects of
more complex exercises on compliance and engagement. The
complex movement exercises added were External Rotation
[EXR], Abducted Rotation [ABR], Mixed Press [MXDPR],
and Mixed Circles [MXDPC]. These movements and their
protocol inclusions can be seen in Figure 2.
Each protocol consisted of five sessions, excluding the
initial tutorial session. Two evaluators were always present
to monitor the user for irregular activity and record qualitative
observations. Exercises were played in circuit rotation and can
be seen in Figure 2. Weighted straps were used for the motions
FAR, SAR, and SR and the weights were gradually increased
throughout the study to match the participant’s capability.
The protocol for increasing weight was determined with our
collaborating therapists to minimize the chance of injury.
Participants usually started at 1-2 pounds and weights were
increased by 1-1.5 pounds after comfortably completing two
rounds of gameplay. The order of the movement was kept
consistent to monitor biometric results for examination of data
between sessions, as illustrated in Figure 2.
Users performed three sets of the exercises for the Foun-
dation Protocol and two sets of the seven exercises during
the Challenge Protocol. A break of approximately 90 seconds
was given between each exercise. The user then performed 60
seconds of the prescribed movements with biofeedback, game
data, and video recorded. The entire sessions lasted forty-
five minutes to an hour, including post-user surveys. For our
analysis, we examined the change from the resting baseline
to the gameplay data. A biometric baseline was recorded
at the beginning of each session where the user wore all
of the sensors shown in Figure 1 while sitting in a relaxed
state with the main screen of PBF displayed on the Vive.
The first session, dubbed session “zero”, of each protocol
phase consisted of a basic range of motion test and tutorial
gameplay where the participant performed all new exercises
of the protocol. By playing the games on this initial session, we
hoped to limit the novelty effect. It should be noted that data
was not collected on these tutorial sessions, as the protocol
was unique to each individual to answer questions and help
with acclimation to the iVR environment.
Results from session data were post-processed using the
Mathworks Matlab 2018b Statistics and Machine Learning
Toolbox [73]. We examined user performance between every
session for mean and standard error. In total, we collected
225 session exercises for the Foundation protocol and 350
session exercises for the Challenge protocol for every data
type. Biometric signals were normalized from each user’s
baseline resting state to examine the changes induced by
The game performance data shows general improvements in
weight resistance over time with maintenance of compliance
Neural Response
Foundation Session
Stress [ bels]
Foundation Protocol Change
Challenge Session
Stress [ bels]
Challenge Protocol Change
Foundation Session
Focus [ bels]
Challenge Session
Focus [ bels]
Foundation Session
Awareness [ bels]
Challenge Session
Awareness [ bels]
Foundation Session
Motor [ bels]
Challenge Session
Motor [ bels]
Foundation Session
Cognition [ bels]
Challenge Session
Cognition [ bels]
Fig. 5. EEG responses between Foundation Protocol (in red of 225 recorded
exercises) and Challenge Protocol (in green of 350 recorded exercises). Rows
1-5 show Alpha, Beta, Delta, Theta, and Gamma bands resting state change
respectively. Error bars indicate standard error.
Facial Movement Response
Foundation Session
Blinks [per s]
Foundation Protocol Change
Challenge Session
Blinks [per s]
Challenge Protocol Change
Foundation Session
Jaw Clenches [per s]
Challenge Session
Jaw Clenches [per s]
Fig. 6. Facial muscle movements recorded with Muse between Foundation
Protocol (in red of 225 recorded exercises) and Challenge Protocol (in green
of 350 recorded exercises). Row one shows the mean resting state change
of blinks per second. Row two shows the mean change of jaw clenches per
second from resting state.
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TG.2021.3069445, IEEE
Transactions on Games
Engagement Survey Response
Fig. 7. Survey responses on engagement from 5 subjects, with 1=strongly disagree and 5=strongly agree.
and exercise movement, as shown in Figure 3. On average,
users were able to handle more weight resistance per exercise
than the initial session, and while the compliance remains
almost constant in the Foundation protocol, it increases signifi-
cantly in the Challenge protocol. In both protocols, users were
able to perform the same movements with a gradual weight
For physiological performance, PBF was able to record and
monitor elevated HR and GSR measurements when compared
to resting-state for all sessions of each protocol. Figure 4
shows the changes from resting baselines and indicates that
PBF always induced an elevated HR (indicating physical
engagement) and stimulated GSR by 1uS or higher (indicating
induced arousal). In the Foundation Protocol, users maintained
a constant level of increased physical activity with a slow de-
cline of arousal. In the Challenge Protocol, users had increased
intensity of physiological activity with a considerable decline
of arousal that eventually stabilized.
For brainwave response, neural activities were measures
at all sessions and all protocols, as shown in Figure 5. All
wavebands were found to be at a positive increase from
resting-state change which indicates that Alpha, Beta, Delta,
Theta, and Gamma waves were elevated during PBF usage. In
the Foundation Protocol, all brainwave responses from users
generally increased in the middle of the sessions and began
declining towards the last sessions. In the Challenge Protocol,
all brainwave activities had a more substantial initial session
than the Foundation Protocol and generally declined overtime
to nearly the same level as the Foundation Protocol’s last
Additionally, Muse [63] holds the capability to detect facial
muscle movements to determine a Boolean response of eye
blinks and jaw clenches. This data was recorded during
runtime gameplay, and converted to facial movements per
second based on changes from the baseline, as seen in Figure
6. While playing PBF, users in the Foundation Protocol tended
to blink less than their resting state for every session (with
the exception of Session #3). Jaw clenches do not vary much
between sessions. In the Challenge Protocol, users tended
to blink and clench their jaw much more between every
session than their baseline resting state. Unlike the Foundation
Protocol, these blinks were always at a positive increase when
compared to resting state, except for the first session, and were
more rapid. Lastly, jaw clenches tended to decline as time
progressed between sessions.
For user’s self-reported responses, the qualitative survey
questions can be seen in Figure 7 (engagement based) and
Figure 8 (emotion based). For both protocols on each session,
Emotion Survey Responses
Fig. 8. The self reported emotions ratios felt by users from post-gameplay
most users agreed that the game remained more engaging than
their traditional therapy routine and that the game provided a
distraction for them during their exercise, as shown in Figure
7. Similarly, the majority of users reported a positive range
of emotions for each exercise ranging from Happy/Joyful,
Excited/Motivated, and Relaxed, as shown in Figure 8. Q3-
4 show the largest differences in survey responses between
protocols. Specifically on Q4, the Foundation Protocol saw
a transition from unanimous disagreement with noticing “the
outside world while playing the game” to a greater majority of
neutral as time progressed. The Challenge Protocol was inverse
to this effect, where users eventually became unanimous in
disagreeing that they could notice the outside world during
gameplay. In essence, this suggests that users were much
more engaged in the game during the last two sessions of
the Challenge protocol.
Through analyzing the data from our two months study, we
observed the following phenomena:
PBF was able to elicit rehabilitative responses similar to
traditional therapy, including increases in muscle’s strength,
control and flexibility. The results suggest that across all users,
their resistance successfully increased throughout the study, as
evidenced by the weight increments that the users were able
to cope with. Heart rate increased for both protocols, which
we concluded were due to the increased weights that require
additional muscular efforts. Compliance improved more during
the Challenge Protocol than the Foundation Protocol, which
may suggest that users that were challenged with the complex
movements followed the protocol more carefully than asked to
perform simpler movements. During the exit interview, users
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TG.2021.3069445, IEEE
Transactions on Games
perceived that they gained significant strength and stability
through playing the game. They felt they would have been
unable to play the game at the beginning of the study using
their final session’s weights, and yet, users were able to
perform those exercises with those weights.
Users can remain engaged in physical therapy using PBF
and HTC Vive beyond the novelty effect period. One of the
dangers of long-term therapy is when users get bored and
lose interest in the exercises. We did not observe any decrease
of interest and engagement beyond the novelty effect (when
users were still new at iVR games). The greatest changes in all
the brainwave bands (which are often associated with levels of
stress [20], focus [21], [22], awareness [23], [24], motor [25],
[26], and cognition [27], [28]) were seen in the transition from
the Foundation Protocol to the Challenge Protocol. This sharp
increase in all bands suggests that the additional exercises were
able to engage the users considerably. Additionally, blinks
were the lowest for each protocol’s first session, indicating
the user may have been more focused during these sessions
[74]. This could mean that creating new types of movements
after the user has become accustomed to a set of exercises
can stimulate users to remain interested. When the Challenge
Protocol was introduced, jaw clenches were increased from
baseline, possibly indicating greater effort of the participant
[75]. The survey responses also showed that users felt engaged
by each protocol. In Q1 (Figure 7), users compared gameplay
exercises to their traditional therapy. GSR responses declined
over time in both protocols, and we speculate that this is
most likely due to users becoming more acclimated to the
game over time and thus causing drop in arousal. We should
note that at the end of each protocol, GSR’ level stabilizes,
indicating a steady state arousal. The survey results suggests
that PBF was more successful in enabling engagement of
physical therapy than traditional interventions. Questions Q2-4
also demonstrated that the user felt present in the game world,
possibly indicating a successful immersion. Additionally, users
stated that they enjoyed the first few sessions, but started
to lose enthusiasm as they felt the Foundation Protocol was
too repetitive and straightforward. With the introduction of
new and more complex movements that could not be easily
memorized in Challenge Protocol, the users were excited once
again to play the game. Some users stated that the more
complex movements kept them engaged rather than “zoning
out” as they did during the simple movements over time.
iVR games have the potential as a long-term physical
therapy tool that can be used at home. PBF was successful
in inducing rehabilitative response while maintaining extended
engagement. The basic version of PBF only requires the Vive
headset and none of the biosensors for remote usage. We argue
that this suggests a low-cost solution for rehabilitation exercise
compared to traditional long term physical therapy sessions
that require users to visit a clinic.
It appears that the differences in the difficulty levels and
goals between the two protocols induced noticeable change
in the different brainwave measures. The most substantial
changes were seen in the transition from Foundation Proto-
col to Challenge protocol, where new and more challenging
games were added to the already existing games. From the
context of Alpha band power (often associated with stress),
these results may indicate that Challenge Protocol has much
greater difficulty than Foundation Protocol, and induced a
higher amount of stress when transitioning to new and more
challenging exercises. It appears that Beta (often associated
with focus) spiked higher when difficulty was increased, which
was expected as users must focus harder when the game
became more difficult. One take away message from these two
findings is that, if we are to design games whose difficulty
levels adapt to users’ biometric changes, there should be a
careful consideration to design how sharply difficulty levels
increase, as increased levels of difficulty induces higher focus,
but also higher stress.
More work needs to be done in establishing dynamic pro-
gression of difficulty, utilizing biomarkers, and testing more
users. Progression during the game was an essential aspect
for all users. They related adding new movements to the
game to be like “unlocking a new level.” All participants
stated that they would like to have more levels to advance
through and clear goals for each level. This would help keep
things dynamic and avoid boredom due to repetition. Three of
the five participants would recommend this game to a friend
in its current version, while the last two stated they would
recommend the game if more levels, progression, and goals
were incorporated into the game.
It should be noted that future research should explore a
more significant number of users and VR experiences to
understand the long-term effects and user response of iVR
physical rehabilitation gaming. As more immersive virtual
environments are crafted for physical rehabilitation, there is
a need to establish how such a system can be tuned to the
user’s biometrics to induce a desirable range of activity and
understand how this will compare to conventional physical
rehabilitation. In this study, seven different movements and
one virtual environment were explored for upper extremity
physical rehabilitation. More motions and varied experiences
should be investigated to examine the game design, difficulty,
and adaptation to iVR stimuli. We are also mindful that
there were only five users that we followed for two months;
however, we believe that this study is an important step
towards gathering insights for future studies.
This study explored the effects of an immersive Virtual
Reality HMD gamified upper-extremity physical therapy that
record both physical and biometric responses over the course
of two months. To provide a more engaging experience, we
designed the study so that users completed their prescribed
therapeutic movements by protecting a virtual butterfly in a
dynamic and adaptive virtual environment. Two rehabilitative
goals were set in the study: recovery of foundational move-
ments and progressing with more complex motions. The study
results suggest that movement improvements over time can
be quantitatively assessed through game logs. The study also
concluded that the biometric responses can complement game
data and provide a richer insight on user engagement. These
findings may indicate that long term immersive Virtual Reality
physical rehabilitation is feasible.
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TG.2021.3069445, IEEE
Transactions on Games
In the future, we aim to expand PBF’s capabilities for
home health and to run larger trials for comparison with
conventional therapy methods. We would like to dive deeper
into the effects of immersive physio-rehabilitation through
controlled trials to understand how user-perceived confidence
and difficulty influences the recovery journey. Additionally,
virtual environments, such as PBF, provide an opportunity to
explore runtime biofeedback and adaptive difficulty with emo-
tion classification, which we also plan to investigate Motion
capture data with biomechanical simulation may be utilized to
estimate muscle forces for understanding biased movements
and how to best prescribe rehabilitation towards addressing
weaknesses. We plan to run biomechanical simulation for this
estimation The creation of an adaptive, personalized physical
therapy game that adjusts to the user’s mental and physical
state in runtime may yield immense potential. Subsequently,
there are far more butterflies to follow on the road ahead.
This material is based upon work supported by the National
Science Foundation under Grant No #1521532. Any opinions,
findings, and conclusions or recommendations expressed in
this material are those of the author(s) and do not necessarily
reflect the views of the National Science Foundation. The
authors thank the participants of this study, without their time
and effort, none of this would be possible.
[1] M. Beccue and C. Wheelock, “Research report: Virtual
reality for consumer markets,” Q4 2016. [Online]. Available:
[2] R. Skarbez, F. P. Brooks, Jr, and M. C. Whitton, “A survey of presence
and related concepts,” ACM Computing Surveys (CSUR), vol. 50, no. 6,
pp. 1–39, 2017.
[3] M. Slater, “Place illusion and plausibility can lead to realistic behaviour
in immersive virtual environments,Philosophical Transactions of the
Royal Society B: Biological Sciences, vol. 364, no. 1535, pp. 3549–3557,
[4] J. Diemer, G. W. Alpers, H. M. Peperkorn, Y. Shiban, and
A. M¨
uhlberger, “The impact of perception and presence on emotional
reactions: a review of research in virtual reality,” Frontiers in psychology,
vol. 6, 2015.
[5] K. R. Lohse, C. G. Hilderman, K. L. Cheung, S. Tatla, and H. M. Van der
Loos, “Virtual reality therapy for adults post-stroke: a systematic review
and meta-analysis exploring virtual environments and commercial games
in therapy,PloS one, vol. 9, no. 3, p. e93318, 2014.
[6] A. Elor and S. Kurniawan, “The Ultimate Display for Physical Reha-
bilitation: A Bridging Review on Immersive Virtual Reality,Frontiers
in Virtual Reality, vol. 1, p. 585993, Nov. 2020. [Online]. Available:
[7] S. C. Gobron, N. Zannini, N. Wenk, C. Schmitt, Y. Charrotton,
A. Fauquex, M. Lauria, F. Degache, and R. Frischknecht, “Serious games
for rehabilitation using head-mounted display and haptic devices,” in
International Conference on Augmented and Virtual Reality. Springer,
2015, pp. 199–219.
[8] E. P. Cherniack, “Not just fun and games: applications of virtual reality
in the identification and rehabilitation of cognitive disorders of the
elderly,Disability and rehabilitation: Assistive technology, vol. 6, no. 4,
pp. 283–289, 2011.
[9] S. Finkelstein, A. Nickel, Z. Lipps, T. Barnes, Z. Wartell, and E. A.
Suma, “Astrojumper: Motivating exercise with an immersive virtual
reality exergame,Presence: Teleoperators and Virtual Environments,
vol. 20, no. 1, pp. 78–92, 2011.
[10] A. Elor, S. Kurniawan, and M. Teodorescu, “Towards an Immersive
Virtual Reality Game for Smarter Post-Stroke Rehabilitation,” in 2018
IEEE International Conference on Smart Computing (SMARTCOMP).
Taormina: IEEE, Jun. 2018, pp. 219–225. [Online]. Available:
[11] A. Elor, M. Teodorescu, and S. Kurniawan, “Project Star
Catcher: A Novel Immersive Virtual Reality Experience for
Upper Limb Rehabilitation,” ACM Transactions on Accessible
Computing, vol. 11, no. 4, pp. 1–25, Nov. 2018. [Online]. Available:
[12] A. Elor, M. Powell, E. Mahmoodi, N. Hawthorne, M. Teodorescu,
and S. Kurniawan, “On Shooting Stars: Comparing CAVE and HMD
Immersive Virtual Reality Exergaming for Adults with Mixed Ability,”
ACM Transactions on Computing for Healthcare, vol. 1, no. 4, pp. 1–22,
Dec. 2020. [Online]. Available:
[13] N. Morina, H. Ijntema, K. Meyerbr ¨
oker, and P. M. Emmelkamp, “Can
virtual reality exposure therapy gains be generalized to real-life? a meta-
analysis of studies applying behavioral assessments,” Behaviour research
and therapy, vol. 74, pp. 18–24, 2015.
[14] M. V. Nararro-Haro, H. G. Hoffman, A. Garcia-Palacios, M. Sampaio,
W. Alhalabi, K. Hall, and M. Linehan, “The use of virtual reality to
facilitate mindfulness skills training in dialectical behavioral therapy for
borderline personality disorder: a case study,Frontiers in psychology,
vol. 7, 2016.
[15] Y. Shiban, I. Schelhorn, P. Pauli, and A. M¨
uhlberger, “Effect of com-
bined multiple contexts and multiple stimuli exposure in spider phobia:
a randomized clinical trial in virtual reality,Behaviour research and
therapy, vol. 71, pp. 45–53, 2015.
[16] M. Rus-Calafell, J. Guti´
errez-Maldonado, and J. Ribas-Sabat´
e, “A virtual
reality-integrated program for improving social skills in patients with
schizophrenia: a pilot study,Journal of behavior therapy and experi-
mental psychiatry, vol. 45, no. 1, pp. 81–89, 2014.
[17] D. Gromala, X. Tong, A. Choo, M. Karamnejad, and C. D. Shaw,
“The virtual meditative walk: virtual reality therapy for chronic pain
management,” in Proceedings of the 33rd Annual ACM Conference on
Human Factors in Computing Systems. ACM, 2015, pp. 521–524.
[18] P. J. Costello, Health and safety issues associated with virtual reality:
a review of current literature. Advisory Group on Computer Graphics,
[19] A. Al-Nafjan, M. Hosny, Y. Al-Ohali, and A. Al-Wabil, “Review
and classification of emotion recognition based on eeg brain-computer
interface system research: a systematic review,” Applied Sciences, vol. 7,
no. 12, p. 1239, 2017.
[20] J. J. Foster, D. W. Sutterer, J. T. Serences, E. K. Vogel, and E. Awh,
“Alpha-band oscillations enable spatially and temporally resolved track-
ing of covert spatial attention,Psychological science, vol. 28, no. 7,
pp. 929–941, 2017.
[21] M. Rangaswamy, B. Porjesz, D. B. Chorlian, K. Wang, K. A. Jones,
L. O. Bauer, J. Rohrbaugh, S. J. O’connor, S. Kuperman, T. Reich et al.,
“Beta power in the eeg of alcoholics,Biological psychiatry, vol. 52,
no. 8, pp. 831–842, 2002.
[22] J. Baumeister, T. Barthel, K.-R. Geiss, and M. Weiss, “Influence of
phosphatidylserine on cognitive performance and cortical activity after
induced stress,” Nutritional neuroscience, vol. 11, no. 3, pp. 103–110,
[23] J. A. Hobson and E. F. Pace-Schott, “The cognitive neuroscience of
sleep: neuronal systems, consciousness and learning,” Nature Reviews
Neuroscience, vol. 3, no. 9, p. 679, 2002.
[24] F. Brigo, “Intermittent rhythmic delta activity patterns,” Epilepsy &
Behavior, vol. 20, no. 2, pp. 254–256, 2011.
[25] J. O‘Keefe and N. Burgess, “Theta activity, virtual navigation and the
human hippocampus,” Trends in cognitive sciences, vol. 3, no. 11, pp.
403–406, 1999.
[26] M. E. Hasselmo and H. Eichenbaum, “Hippocampal mechanisms for
the context-dependent retrieval of episodes,Neural networks, vol. 18,
no. 9, pp. 1172–1190, 2005.
[27] J. R. Hughes, “Gamma, fast, and ultrafast waves of the brain: their
relationships with epilepsy and behavior,Epilepsy & Behavior, vol. 13,
no. 1, pp. 25–31, 2008.
[28] S. O‘Nuallain, “Zero power and selflessness: What meditation and
conscious perception have in common,Journal of Cognitive Sciences,
vol. 4, no. 2, pp. 46–64, 2009.
[29] W. Boucsein, Electrodermal activity. Springer Science & Business
Media, 2012.
[30] R. J. Rajae-Joordens, “Measuring experiences in gaming and tv appli-
cations,” in Probing experience. Springer, 2008, pp. 77–90.
[31] V. N. Salimpoor, M. Benovoy, G. Longo, J. R. Cooperstock, and R. J.
Zatorre, “The rewarding aspects of music listening are related to degree
of emotional arousal,” PloS one, vol. 4, no. 10, p. e7487, 2009.
[32] A. Goshvarpour, A. Abbasi, and A. Goshvarpour, “An accurate emotion
recognition system using ecg and gsr signals and matching pursuit
method,” biomedical journal, vol. 40, no. 6, pp. 355–368, 2017.
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TG.2021.3069445, IEEE
Transactions on Games
[33] M. Liu, D. Fan, X. Zhang, and X. Gong, “Human emotion recognition
based on galvanic skin response signal feature selection and svm,” in
2016 International Conference on Smart City and Systems Engineering
(ICSCSE). IEEE, 2016, pp. 157–160.
[34] M. S. Cameirao, I. B. S. Berm ´
udez, E. Duarte Oller, and P. F. Verschure,
“The rehabilitation gaming system: a review,” Stud Health Technol
Inform, vol. 145, no. 6, 2009.
[35] R. Soares, E. Siqueira, M. Miura, T. Silva, and C. Castanho, “Biofeed-
back sensors in game telemetry research,” Simp´
osio Brasileiro de Jogos
e Entretenimento Digital, pp. 81–89, 2016.
[36] D. Deutscher, S. D. Horn, R. Dickstein, D. L. Hart, R. J. Smout,
M. Gutvirtz, and I. Ariel, “Associations between treatment processes,
patient characteristics, and outcomes in outpatient physical therapy
practice,” Archives of physical medicine and rehabilitation, vol. 90,
no. 8, pp. 1349–1363, 2009.
[37] R. S. Calabr `
o, A. Naro, M. Russo, A. Leo, R. De Luca, T. Balletta,
A. Buda, G. La Rosa, A. Bramanti, and P. Bramanti, “The role of
virtual reality in improving motor performance as revealed by eeg: a
randomized clinical trial,” Journal of neuroengineering and rehabilita-
tion, vol. 14, no. 1, p. 53, 2017.
[38] M. D. Wiederhold, M. Crisci, V. Patel, M. Nonaka, and B. K. Wieder-
hold, “Physiological monitoring during augmented reality exercise con-
firms advantages to health and well-being,” Cyberpsychology, Behavior,
and Social Networking, vol. 22, no. 2, pp. 122–126, 2019.
[39] U. Qidwai, M. Ajimsha, and M. Shakir, “The role of eeg and emg
combined virtual reality gaming system in facial palsy rehabilitation-
a case report,” Journal of bodywork and movement therapies, vol. 23,
no. 2, pp. 425–431, 2019.
[40] S. Ahn and S. Hwang, “Virtual rehabilitation of upper extremity function
and independence for stoke: a meta-analysis,” Journal of exercise
rehabilitation, vol. 15, no. 3, p. 358, 2019.
[41] A. I. Corregidor-S´
anchez, A. Segura-Fragoso, M. Rodr´
J. J. Criado-Alvarez, G.-G. Jaime, and B. Polonio-L´
opez, “Can ex-
ergames contribute to improving walking capacity in older adults? a
systematic review and meta-analysis,Maturitas, 2019.
[42] M. C. Howard, “A meta-analysis and systematic literature review of
virtual reality rehabilitation programs,” Computers in Human Behavior,
vol. 70, pp. 317–327, 2017.
[43] I. Brunner, J. S. Skouen, H. Hofstad, J. Aßmus, F. Becker, A.-M.
Sanders, H. Pallesen, L. Q. Kristensen, M. Michielsen, L. Thijs et al.,
“Virtual reality training for upper extremity in subacute stroke (virtues):
a multicenter rct,” Neurology, vol. 89, no. 24, pp. 2413–2421, 2017.
[44] S. R. S. BPT and N. Giladi, “Improved mobility and reduced fall risk
in older adults after five weeks of virtual reality training,Journal of
Alternative Medicine Research, vol. 9, no. 2, p. 171, 2017.
[45] P. Standen, K. Threapleton, A. Richardson, L. Connell, D. Brown,
S. Battersby, F. Platts, and A. Burton, “A low cost virtual reality system
for home based rehabilitation of the arm following stroke: a randomised
controlled feasibility trial,” Clinical rehabilitation, vol. 31, no. 3, pp.
340–350, 2017.
[46] W. B. Kibler, J. McMULLEN, and T. Uhl, “Shoulder rehabilitation
strategies, guidelines, and practice,” Orthopedic Clinics, vol. 32, no. 3,
pp. 527–538, 2001.
[47] W. B. Kibler, “Shoulder rehabilitation: principles and practice.Medicine
and science in sports and exercise, vol. 30, no. 4 Suppl, pp. S40–50,
[48] P. Rego, P. M. Moreira, and L. P. Reis, “Serious games for rehabilitation:
A survey and a classification towards a taxonomy,” in 5th Iberian
conference on information systems and technologies. IEEE, 2010, pp.
[49] A. Elor, S. Lessard, M. Teodorescu, and S. Kurniawan, “Project
Butterfly: Synergizing Immersive Virtual Reality with Actuated
Soft Exosuit for Upper-Extremity Rehabilitation,” in 2019 IEEE
Conference on Virtual Reality and 3D User Interfaces (VR).
Osaka, Japan: IEEE, Mar. 2019, pp. 1448–1456. [Online]. Available:
[50] A. Elor and S. Kurniawan, “Deep Reinforcement Learning in Immersive
Virtual Reality Exergame for Agent Movement Guidance,” in 2020
IEEE 8th International Conference on Serious Games and Applications
for Health (SeGAH). Vancouver, BC, Canada: IEEE, Aug. 2020, pp.
1–7. [Online]. Available:
[51] A. Borrego, J. Latorre, M. Alca ˜
niz, and R. Llorens, “Comparison of
oculus rift and htc vive: feasibility for virtual reality-based exploration,
navigation, exergaming, and rehabilitation,Games for health journal,
vol. 7, no. 3, pp. 151–156, 2018.
[52] S. M. Palaniappan and B. S. Duerstock, “Developing rehabilitation
practices using virtual reality exergaming,” in 2018 IEEE International
Symposium on Signal Processing and Information Technology (ISSPIT).
IEEE, 2018, pp. 090–094.
[53] F. Soffel, M. Zank, and A. Kunz, “Postural stability analysis in virtual
reality using the htc vive,” in Proceedings of the 22nd ACM Conference
on Virtual Reality Software and Technology. ACM, 2016, pp. 351–352.
[54] D. C. Niehorster, L. Li, and M. Lappe, “The accuracy and precision of
position and orientation tracking in the htc vive virtual reality system for
scientific research,” i-Perception, vol. 8, no. 3, p. 2041669517708205,
[55] E. L. Deci and R. M. Ryan, “Self-determination theory.” 2012.
[56] L. A. Shaw, B. C. W¨
unsche, C. Lutteroth, S. Marks, J. Buckley, and
P. Corballis, “Development and evaluation of an exercycle game using
immersive technologies,” 2015.
[57] L. A. Shaw, J. Buckley, P. M. Corballis, C. Lutteroth, and B. C.
unsche, “Competition and cooperation with virtual players in an
exergame,PeerJ Computer Science, vol. 2, p. e92, 2016.
[58] A. Keesing, M. Ooi, O. Wu, X. Ye, L. Shaw, and B. C. W¨
unsche, “Hiit
with hits: Using music and gameplay to induce hiit in exergames,” in
Proceedings of the Australasian Computer Science Week Multiconfer-
ence, 2019, pp. 1–10.
[59] L. A. Shaw, B. C. Wuensche, C. Lutteroth, J. Buckley, and P. Corballis,
“Evaluating sensory feedback for immersion in exergames,” in Proceed-
ings of the Australasian Computer Science Week Multiconference, 2017,
pp. 1–6.
[60] J. C. Haller, Y. H. Jang, J. Haller, L. Shaw, and B. C. W¨
unsche, “Hiit the
road: Using virtual spectator feedback in hiit-based exergaming,” in Pro-
ceedings of the Australasian Computer Science Week Multiconference,
2019, pp. 1–9.
[61] C. D. Batson, P. A. Van Lange, N. Ahmad, and D. A. Lishner, “Altruism
and helping behavior,The Sage handbook of social psychology, vol. 1,
pp. 279–295, 2003.
[62] M. O. Wilhelm and R. Bekkers, “Helping behavior, dispositional em-
pathic concern, and the principle of care,” Social Psychology Quarterly,
vol. 73, no. 1, pp. 11–32, 2010.
[63] InteraXon, “Featured research with muse,” Tech.
Rep., 29.06.2019,
[64] N. Kovacevic, P. Ritter, W. Tays, S. Moreno, and A. R. McIntosh,
“‘my virtual dream’: Collective neurofeedback in an immersive art
environment,PloS one, vol. 10, no. 7, p. e0130129, 2015.
[65] O. E. Krigolson, C. C. Williams, A. Norton, C. D. Hassall, and F. L.
Colino, “Choosing muse: Validation of a low-cost, portable eeg system
for erp research,” Frontiers in neuroscience, vol. 11, p. 109, 2017.
[66] S. Bhayee, P. Tomaszewski, D. H. Lee, G. Moffat, L. Pino, S. Moreno,
and N. A. Farb, “Attentional and affective consequences of technology
supported mindfulness training: a randomised, active control, efficacy
trial,” BMC psychology, vol. 4, no. 1, p. 60, 2016.
[67] Neulog, “Gsr logger sensor nul-217,” Tech. Rep., 29.06.2019,
[68] Polar, “Polar oh1: optical heart rate sensor,” Tech. Rep., 29.06.2019,
[69] C. Jennett, A. L. Cox, P. Cairns, S. Dhoparee, A. Epps, T. Tijs, and
A. Walton, “Measuring and defining the experience of immersion in
games,” International journal of human-computer studies, vol. 66, no. 9,
pp. 641–661, 2008.
[70] K. Doherty and G. Doherty, “Engagement in hci: conception, theory
and measurement,” ACM Computing Surveys (CSUR), vol. 51, no. 5,
pp. 1–39, 2018.
[71] J. A. Russell, “A circumplex model of affect.Journal of personality
and social psychology, vol. 39, no. 6, p. 1161, 1980.
[72] P. Ekman, “Basic emotions,Handbook of cognition and emotion,
vol. 98, no. 45-60, p. 16, 1999.
[73] D. J. Higham and N. J. Higham, MATLAB guide. Siam, 2016, vol.
[74] M. Rosenfield, S. Jahan, K. Nunez, and K. Chan, “Cognitive demand,
digital screens and blink rate,” Computers in Human Behavior, vol. 51,
pp. 403–406, 2015.
[75] D.-H. Huang, S.-W. Chou, Y.-L. Chen, and W.-K. Chiou, “Frowning
and jaw clenching muscle activity reflects the perception of effort during
incremental workload cycling,Journal of sports science & medicine,
vol. 13, no. 4, p. 921, 2014.
... Unlike some questionable effects of immersive VR on the learning and cognitive thinking of healthy users, its effect on the mental treatment and cognitive training on patients seems more promising. Several cognitive training methods such as P300based social attention training for autism spectrum disorder [70], cognitive training for attention deficit hyperactivity disorder (ADHD) [71], [72], and a gamified exercise protocol for patients with upper-extremity injuries [73] were proposed and investigated, exhibiting the effect of the systems on patients with repeated training. Furthermore, EEG signals retrieved during training may also be utilized for screening cognitive impairment [74]. ...
Full-text available
Recent advancements in immersive virtual reality head-mounted displays allowed users to better engage with simulated graphical environments. Having the screen egocentrically stabilized in a way such that the users may freely rotate their heads to observe virtual surroundings, head-mounted displays present virtual scenarios with rich immersion. With such an enhanced degree of freedom, immersive virtual reality displays have also been integrated with electroencephalograms, which make it possible to study and utilize brain signals non-invasively, to analyze and apply their capabilities. In this review, we introduce recent progress that utilized immersive head-mounted displays along with electroencephalograms across various fields, focusing on the purposes and experimental designs of their studies. The paper also highlights the effects of using immersive virtual reality discovered through the electroencephalogram analysis and discusses existing limitations, current trends as well as future research opportunities that may hopefully act as a useful source of information for further improvement of electroencephalogram-based immersive virtual reality applications.
... However, with technologies that enable multisensory interactions with virtual environments, such as virtual reality (VR) and augmented reality (AR), the expressive and infectious power of the meta-universe is unparalleled by the techniques of the past. Some scholars have used immersive virtual reality (iVR) and head-mounted display (HMD) systems to help rehabilitate injured patients [53]. The concept of digital twins, which is related to the metaverse, has also had a profound impact on areas outside the Internet, such as industry [54]. ...
Full-text available
The metaverse provides us with an attractive virtual space in which the value of the virtual property has been increasingly recognized. However, the lack of effective cross-metaverse trading tools and the reputation guarantee makes it difficult to trade items among different metaverses. To this end, a decentralized reputation system for virtual items trading forum named DVIT is devised. To the best of our knowledge, DVIT is the first decentralized cross-metaverse item trading prototype inspired by the online-game trading system. We designed the corresponding transaction function and realized the autonomous governance of the community by introducing the reputation mechanism. An improved election mechanism is proposed to improve efficiency based on Delegated Proof-of-Stake (DPoS). Through token rewards associated with activity levels, users’ motivation can be stimulated. The experiments indicate that our proposed scheme could dynamically measure the trustworthiness degree of the users through the dynamic reputation value and thereby exclude malicious users from the blockchain within 20 epochs.
... Krol et al. [456] combined eye-tracking and brain-computer interface (BCI) and created a completely handsfree game Tetris clone where traditional actions (i.e., block manipulation) are performed using gaze control. Elor et al. [457] measure heart rate and galvanic skin response (GSR) with Immersive Virtual Reality (iVR) Head-Mounted Display (HMD) systems paired with exercise games to show how exercise games can positively affect physical rehabilitation. ...
Full-text available
Affective, emotional, and physiological states (AFFECT) detection and recognition by capturing human signals is a fast-growing area, which has been applied across numerous domains. The research aim is to review publications on how techniques that use brain and biometric sensors can be used for AFFECT recognition, consolidate the findings, provide a rationale for the current methods, compare the effectiveness of existing methods, and quantify how likely they are to address the issues/challenges in the field. In efforts to achieve the key goals of Society 5.0, Industry 5.0, and human-centered design better, the recognition of emotional, affective, and physiological states is progressively becoming an important matter and offers tremendous growth of knowledge and progress in these and other related fields. In this research, a review of AFFECT recognition brain and biometric sensors, methods, and applications was performed, based on Plutchik’s wheel of emotions. Due to the immense variety of existing sensors and sensing systems, this study aimed to provide an analysis of the available sensors that can be used to define human AFFECT, and to classify them based on the type of sensing area and their efficiency in real implementations. Based on statistical and multiple criteria analysis across 169 nations, our outcomes introduce a connection between a nation’s success, its number of Web of Science articles published, and its frequency of citation on AFFECT recognition. The principal conclusions present how this research contributes to the big picture in the field under analysis and explore forthcoming study trends.
... Until VR headsets are more ubiquitous among the general population of smokers, the potential for VR to deliver motivational messaging in smokers' own homes is limited. In addition, as most of our sample had at least some prior VR experience, this raises the question as to whether an immersive VR scenario focused on the health consequences of smoking may be more salient to those using VR for the first time (i.e., a potential novelty effect) [38,39]. Second, the effective point of randomisation occurred when participants booked a meeting to complete the online testing (with intervention and control participants receiving the same instructions), with many participants dropping out at this stage (i.e., not attending the testing). ...
Full-text available
Individual-level interventions for smokers unmotivated to quit remain scarce and have had limited success. Little is known about the potential of virtual reality (VR) for delivering messaging to smokers unmotivated to quit. This pilot trial aimed to assess the feasibility of recruitment and acceptability of a brief, theory-informed VR scenario and estimate proximal quitting outcomes. Unmotivated smokers (recruited between February-August 2021) aged 18+ years who had access to, or were willing to receive via post, a VR headset were randomly assigned (1:1) using block randomisation to view the intervention (i.e., a hospital-based scenario with motivational stop smoking messaging) or a ‘sham’ VR scenario (i.e., a scenario about the human body without any smoking-specific messaging) with a researcher present via teleconferencing software. The primary outcome was feasibility of recruitment (i.e., achieving the target sample size of 60 participants within 3 months of recruitment). Secondary outcomes included acceptability (i.e., positive affective and cognitive attitudes), quitting self-efficacy and intention to stop smoking (i.e., clicking on a weblink with additional stop smoking information). We report point estimates and 95% confidence intervals (CIs). The study protocol was pre-registered ( A total of 60 participants were randomised within 6 months (intervention: n = 30; control: n = 30), 37 of whom were recruited within a 2-month period of active recruitment following an amendment to gift inexpensive (£7) cardboard VR headsets via post. The mean (SD) age of participants was 34.4 (12.1) years, with 46.7% identifying as female. The mean (SD) cigarettes smoked per day was 9.8 (7.2). The intervention (86.7%, 95% CI = 69.3%-96.2%) and control (93.3%, 95% CI = 77.9%-99.2%) scenarios were rated as acceptable. Quitting self-efficacy and intention to stop smoking in the intervention (13.3%, 95% CI = 3.7%-30.7%; 3.3%, 95% CI = 0.1%-17.2%) and control (26.7%, 95% CI = 12.3%-45.9%; 0%, 95% CI = 0%-11.6%) arm were comparable. The target sample size was not achieved within the feasibility window; however, an amendment to gift inexpensive headsets via post appeared feasible. The brief VR scenario appeared acceptable to smokers unmotivated to quit.
... The relative novelty of VR as a media modality may intensify apprehension in ways not fully captured by traditional measures of technology apprehension that focus on computers and may have impacted learning outcomes. Some recent evidence noted that the effect of novelty is negligible using immersive VR experiences for learning (Huang, Roscoe, Johnson-Glenberg, & Craig, 2021) and physical rehabilitation training (Elor, Powell, Mahmoodi, Teodorescu, & Kurniawan, 2021), but further research is warranted. ...
Research on virtual reality (VR) in education and training has found that spatial presence, the perception that the body is inside a mediated environment, increases engagement. However, experiencing spatial presence requires the allocation of limited processing resources, potentially inhibiting the processing of other information. Guided by the frameworks of Limited Capacity Model of Motivated Mediated Message Processing (LC4MP), and the cognitive theory of multimedia learning, two experiments examined the effects of different modalities on spatial presence to test the prediction that spatial presence negatively impacts recall. Study 1 (N = 100) found that VR elicited higher spatial presence than video, but that high spatial presence reduced recall. Individual differences (technology apprehension) moderated spatial presence. Study 2 (N = 260) found that pre-existing interest in the learning content and aversive responses elicited by the learning content increased spatial presence. However, segmenting the VR content to reduce processing load for participants had little effect on spatial presence or information recall. In sum, modality features and individual differences drove user experiences of spatial presence, which negatively impacted recall, but segmentation of VR content had no effect on learning outcomes.
... For several decades, research on judgment and decision making has meandered between the rational (economic man/homo economicus) and irrational/emotional perspective (humanistic man). The main research stream tends to emphasize immersive, hedonic, and fantasy 'experientiality' in the virtual world such as enjoyment (Peukert, Pfeiffer, Meißner, Pfeiffer, & Weinhardt, 2019), novelty (Elor, Powell, Mahmoodi, Teodorescu, & Kurniawan, 2021), and vividness (Zeng, Cao, Lin, & Xiao, 2020), while the other stream focuses on the cognitive related aspects involving efficiency, and utilitarian performance, such as knowledge acquisition (Taçgın, 2020), information seeking (Lee, Lee, & Jeong, 2021), and cognitive functioning (Gamito et al., 2020). ...
Full-text available
Virtual technologies will change the way we consume in the digital environment in the future. Such technologies can provide consumers with a multi-sensory experience in contrast to the single-sensory stimulus in the conventional online environment. As human senses play a key role in consumption choices, we argue that virtual technologies provide greater opportunities to influence consumer decisions than the present digital environment. Consequently, we suggest that virtual technologies can potentially be used to nudge consumers towards sustainable consumption. We discuss technology-assisted sensory marketing, present the cognitive and emotive aspects of virtual reality, and propose applications of virtual reality technologies to encourage sustainable consumption. Our opinion paper concludes that virtual technologies are likely to change many aspects of human life and can have significant positive effects on the environment and climate change.
Virtual Reality Games offer highly immersive experiences that allow users to effectively dissociate themselves from reality. VR gaming leads to a strong sense of presence and can facilitate the experience of flow among its players. The current study examines how the balance between a player's skill and the difficulty of a VR rhythm game can influence the sense of flow. An experiment was conducted among 201 university students who played the rhythm game Beat Saber in VR. Difficulty settings were adjusted to each individual player's skill, making the game either too easy, matched, or too hard. Results indicated that the match between skill and difficulty led to higher levels of flow. If the game was too hard, this caused a decrease in flow due to frustration, but too easy did not lead to the expected decrease in flow due to boredom. A stronger sense of flow while playing this VR game was also related to better performance, higher physiological arousal, and more enjoyment.
Conference Paper
Full-text available
Robots are being taught by increasingly broader populations of people who provide training data for machine learning algorithms. Many studies over the past decade have begun demonstrating reproducible robot teaching methodologies and have highlighted benefits in human-robot interaction (HRI). However, there have been few investigations about what it is like for the people teaching these robots. In this study, we consider how teaching a skill to a robot arm, performing a reaching task (as opposed to observing the robot self-learning), influences a user's emotional experience and perceptions of the robot. In a 2x2 experiment (N=160), we varied the agent's learning technique (user reinforcement feedback or robot self-learning) and expressiveness (static agent face or performance-based valence expression with head following), using an online WebGL virtual environment to enable remote HRI. Our results demonstrate that users experience significantly more trust, believability, and emotional response when teaching the robot than when observing it learning, which can be amplified with agent expressiveness.
Full-text available
Modern technologies in virtual reality (VR) and augmented reality (AR) provide unique features that can be used to facilitate tasks in everyday life. Several courses can be built using augmented reality, such as engine maintenance, computer maintenance, chemistry lab, etc. Augmented reality technologies provide dynamic and interactive instructions to resolve a problem or present required concepts. Building an educational system based on augmented reality is not an easy task due to some difficulties and challenges, such as the cost of augmented reality tools and other hardware and software required. Also, training students with engineering concepts and precise parts involves a lot of analysis and practice to know problems and then design solutions. The paper aims to develop a virtual educational environment for training students in engineering sectors in practical laboratory sessions based on AR/VR techniques. The proposed system provides a safe and low-cost environment to train the student different concepts in engineering sector, such as basic concepts in electrical, mechanical and renewable energy engineering.
Full-text available
Physical rehabilitation is often an intensive process that presents many challenges, including a lack of engagement, accessibility, and personalization. Immersive media systems enhanced with physical and emotional intelligence can address these challenges. This review paper links immersive virtual reality with the concepts of therapy, human behavior, and biofeedback to provide a high-level overview of health applications with a particular emphasis on physical rehabilitation. We examine each of these crucial areas by reviewing some of the most influential published case studies and theories while also considering their limitations. Lastly, we bridge our review by proposing a theoretical framework for future systems that utilizes various synergies between each of these fields.
Conference Paper
Full-text available
Immersive Virtual Reality applied to exercise games has a unique potential to both guide and motivate users in performing physical exercise. Advances in modern machine learning open up new opportunities for more significant intelligence in such games. To this end, we investigate the following research question: What if we could train a virtual robot arm to guide us through physical exercises, compete with us, and test out various double-jointed movements? This paper presents a new game mechanic driven by artificial intelligence to visually assist users in their movements through the Unity Game Engine, Unity Ml-Agents, and the HTC Vive Head-Mounted Display. We discuss how deep reinforcement learning through Proximal Policy Optimization and Generative Adversarial Imitation Learning can be applied to complete physical exercises from the same immersive virtual reality game. We examine our mechanics with four users through protecting a virtual butterfly with an agent that visually helps users as a cooperative "ghost arm" and an independent competitor. Our results suggest that deep learning agents are effective at learning game exercises and may provide unique insights for users.
Full-text available
Inactivity and a lack of engagement with exercise is a pressing health problem in the United States and beyond. Immersive Virtual Reality (iVR) is a promising medium to motivate users through engaging virtual environments. Currently, modern iVR lacks a comparative analysis between research and consumer-grade systems for exercise and health. This paper examines two such iVR mediums: the Cave Automated Virtual Environment (CAVE) and the Head-Mounted Display (HMD). Specifically, we compare the room-scale Mechdyne CAVE and HTC Vive Pro HMD with a custom in-house exercise game that was designed such that user experiences were as consistent as possible between both systems. To ensure that our findings are generalizable for users of varying abilities, we recruited forty participants with and without cognitive disabilities concerning the fact that iVR environments and games can differ in their cognitive challenge between users. Our results show that across all abilities, the HMD excelled in-game performance, biofeedback response, and player engagement. We conclude with considerations in utilizing iVR systems for exergaming with users across cognitive abilities.
Conference Paper
Full-text available
Immersive Virtual Reality paired with soft robotics may be synergized to create personalized assistive therapy experiences. Virtual worlds hold power to stimulate the user with newly instigated low cost, high-performance commercial Virtual Reality (VR) devices to enable engaging and accurate physical therapy. Soft robotic wearables are a versatile tool in such stimulation. This preliminary study investigates a novel rehabilitative VR experience, Project Butterfly (PBF), that synergizes VR Mirror Visual Feedback Therapy with soft robotic exoskeletal support. Nine users of ranging ability explore an immersive gamified physio-therapy experience by following and protecting a virtual butterfly, completed with an actuated robotic wearable that motivates and assists the user to perform rehabilitative physical movement. Specifically, the goals of this study are to evaluate the feasibility, ease-of-use, and comfort of the proposed system. The study concludes with a set of design considerations for future immersive physio-rehab robotic-assisted games.
Full-text available
We aimed to conduct a systematic literature review with a meta-analysis to investigate whether virtual reality (VR) approaches have beneficial effects on the upper extremity function and independent activities of stroke survivors. Experimental studies published between 2007 and 2017 were searched from two databases (EBSCOhost and PubMed). This study reviewed abstracts and assessed full articles to obtain evidence on qualitative studies. For the meta-analysis, the studies that estimated the standardized mean between the two groups analyzed the statistical values necessary for calculating the effect size. The present study also evaluated the statistical heterogeneity. In total, 34 studies with 1,604 participants were included, and the number of participants in each study ranged from 10 to 376. Nine studies were assessed to evaluate the quantitative statistical analysis for 698 patients with hemiparetic stroke. The results of the meta-analysis were as follows: The overall effect size was moderate (0.41, P<0.001). The 95% confidence interval ranged from 0.25 to 0.57. However, no significant heterogeneity and publication bias were observed. The results of this study showed that VR approaches are effective in improving upper extremity function and independent activities in stroke survivors.
Full-text available
Augmented reality (AR) is a versatile tool that is changing the way we interact with the world. One way is through exergaming, integrating AR, and gaming with traditional exercise programs. This study aimed to test three different AR systems to determine their ability to elicit physiological responses similar to those experienced while exercising or performing rehabilitation exercises. The systems tested included a head-mounted display experience (Teapots), a Sony EyeToy game (Slap Stream), and a prototype of a PC-camera-based system (Drums). Physiological recordings show that Slap Stream and Drums produced significant increases (p < 0.05) in heart rate (HR) and skin conductance, and all three games produced nonsignificant increase in respiratory rate. Of the three games, Slap Stream produced the highest rates of low- (∼28 percent), moderate- (17 percent), and high-intensity (∼22 percent) HRs. This study successfully showed that inexpensive easy-to-use AR systems can effectively contribute to exercise programs. Future game design recommendations, including feedback, mixed reality, and stress tracking, are discussed.
Conference Paper
Full-text available
Exergames are an attractive way to promote physical activity in users, but rarely implement high intensity protocols. A popular example of high-intensity training is HIIT (high intensity interval training), which involves alternating periods of intense exercise with periods of light exercise. Few example of games implementing HIIT are available and their effectiveness is uncertain. In this paper we propose a method for improving user performance while maintaining intrinsic motivation without using a human instructor. Our VR exergaming solution uses virtual spectators, which produce a rhythmic clapping similar to that observed in sporting events. Our user study indicates that the virtual crowd feedback increases cycling speed and participants' heart rate.
Background: The accessibility, low cost and motivation generated by exergames has fostered its rapid expansion as a rehabilitation technique. Objective: To estimate the effectiveness of rehabilitation programs using IVGT in improving walking capacity of people aged 60 years and over. Materials and methods: The electronic data research following the PRISMA Statement (Scopus, Cochrane, Web of Science, OT Seeker, National Guideline Clearinghouse, Trip Database, CSIC Spanish National Research Council) was completed in September 2018. The results of randomized clinical trials using exergames for rehabilitation of walking capacity were combined. The calculations have followed the guidelines of the Cochrane Handbook for Systematic Reviews of Interventions. The Grading of Recommendations Assessment, Development and Evaluation system was used to evaluate the quality of the evidence. Results: We obtained data from 14 trials, including 11 meta-analysis studies. The size of exergames effects on walking capacity is moderate, but significant (SMD -0.56; 95 % CI: -0.90, -0.21; p = 0.002). Effectiveness was greater to recover the ability to transfer from one position or place to another (SMD -1.02; CI 95 %: -1.70, -0.35; P = 0.003). The intervention protocols, their duration and intensity varied considerably. The lack of masking, the allocation concealment, the absence of assessor blinding were the main causes of bias so the final grade of evidence has been low for walking and very low for transfers. Conclusions: Positive clinical effects of exergames have been found to improve walking capacity, but the quality of evidence to refute its effectiveness is weak with risk of bias. Further research is needed in order to know the actual magnitude of its effect.
Background: The recovery rates for facial palsy are usually excellent; however, regularly patients present with problems with their fine facial movements that affect their emotional expressions. Objective: To discover the viability and ease of using an Electroencephalogram (EEG) and Electromyography (EMG) combined Virtual Reality (VR) gaming system - the 'Oculus Rift' device in the evaluation and rehabilitation of facial palsy. Design: Single case study. Patient information: A young 23-year-old female with facial palsy. Clinical findings: Most of the patient's facial features were re-established within the recovery time frame, except for her right forehead and eyebrow movements. Intervention: A 10 day exercise program (Day 2-11) with an immersive virtual reality device, which randomly shoots virtually animated white balls in an unpredictable and testing pattern. Outcome measures: EEG and EMG patterns corresponding to the facial upper quadrant were taken at baseline, post-intervention, and at follow up. Results: EMG and EEG investigation revealed a progressive improvement in the muscle activation in response to the impulsive and unpredictable activities in the virtual environment provided through the immersive VR device. Conclusion: The case report found a positive relationship between VR, facial upper quadrant EMG activation and EEG pattern changes following the intervention.