Conference PaperPDF Available

Project Butterfly: Synergizing Immersive Virtual Reality with Actuated Soft Exosuit for Upper-Extremity Rehabilitation

Authors:

Abstract and Figures

Immersive Virtual Reality paired with soft robotics may be synergized to create personalized assistive therapy experiences. Virtual worlds hold power to stimulate the user with newly instigated low cost, high-performance commercial Virtual Reality (VR) devices to enable engaging and accurate physical therapy. Soft robotic wearables are a versatile tool in such stimulation. This preliminary study investigates a novel rehabilitative VR experience, Project Butterfly (PBF), that synergizes VR Mirror Visual Feedback Therapy with soft robotic exoskeletal support. Nine users of ranging ability explore an immersive gamified physio-therapy experience by following and protecting a virtual butterfly, completed with an actuated robotic wearable that motivates and assists the user to perform rehabilitative physical movement. Specifically, the goals of this study are to evaluate the feasibility, ease-of-use, and comfort of the proposed system. The study concludes with a set of design considerations for future immersive physio-rehab robotic-assisted games.
Content may be subject to copyright.
Project Butterfly: Synergizing Immersive Virtual Reality with Actuated
Soft Exosuit for Upper-Extremity Rehabilitation
Aviv Elor, Steven Lessard, Mircea Teodorescu, and Sri Kurniawan
University of California, Santa Cruz, Jack Baskin School of Engineering
{aelor, slessard, mteodore, skurnia}@ucsc.edu
ABSTRACT
Immersive Virtual Reality paired with soft robotics may be syner-
gized to create personalized assistive therapy experiences. Virtual
worlds hold power to stimulate the user with newly instigated low-
cost, high-performance commercial Virtual Reality (VR) devices to
enable engaging and accurate physical therapy. Soft robotic wear-
ables are a versatile tool in such stimulation. This preliminary study
investigates a novel rehabilitative VR experience, Project Butterfly
(PBF), that synergizes VR Mirror Visual Feedback Therapy with
soft robotic exoskeletal support. Nine users of ranging ability ex-
plore an immersive gamified physio-therapy experience by follow-
ing and protecting a virtual butterfly, completed with an actuated
robotic wearable that motivates and assists the user to perform re-
habilitative physical movement. Specifically, the goals of this study
are to evaluate the feasibility, ease-of-use, and comfort of the pro-
posed system. The study concludes with a set of design considera-
tions for future immersive physio-rehab robotic-assisted games.
Index Terms: Virtual Reality, Soft Robotics, Wearable Robotics,
Exosuit, Physio-Immersive Rehabilitation, Physical Therapy, Im-
mersive Experiences, Serious Games, Human Computer Interaction
1 INTRODUCTION
According to the latest US Census in 2010, there are more than 40
million older adults (defined as people aged 65 years old or older)
living in the US, comprising 13 percent of the US population. This
demographic represents a 15 percent growth compared to the 2000
US Census data [19] and is projected to continue to grow. Unfortu-
nately, studies have shown that motor functions decline with aging
[45]. The significant older population experiences an increasingly
prevalent issue of motor degeneration. Age-related motor perfor-
mance deficits include coordination difficulty, decreased variabil-
ity of motor ability, slowing of movement, and problems with bal-
ance and gait [37]. Movement slows with aging by as much as
15 to 30 percent. Research by Seidler-Dobrin et al. suggests that
older adults emphasize movement accuracy at the cost of movement
speed [38]. As a result, older adults show specific deficits in the
coordination of bimanual and multi-joint movements. For exam-
ple, movements become slower and less smooth when older adults
use their shoulder and elbow joints simultaneously as opposed to
performing single-joint actions[36]. Often postural stability is also
compromised with advancing age [40].
In addition to the decline in motor function, aging correlates to
the progressive loss of skeletal muscle mass and strength. Fre-
The authors are with the Departments of Computational Media & Elec-
trical Computer Engineering, University of California Santa Cruz, Jack
Baskin School of Engineering, 1156 High St, Santa Cruz, CA, USA. Ad-
ditionally, this work was supported by the Center for Information Technol-
ogy Research in the Interest of Society (CITRIS) and the National Science
Foundation (NSF).
quent exercise represents an effective therapeutic strategy to aug-
ment skeletal muscle mass and improve functional performance and
quality of life in older adults [22]. Many technological solutions
have been researched and developed over the past decade to reduce
motor loss, but there is still much to be done.
2 RE LATE D WORK
One modern approach to address muscular impairment is virtual-
reality (VR) therapy. Through the use of VR, stimulating immersive
environments can be programmed to increase therapy compliance,
accessibility, and data throughput [8, 9]. Psychological and physio-
logical research has featured increasing use of VR in the prior two
decades thanks to the ability to simulate realistic and complex situa-
tions critical to laboratory-based human behavior investigations [7].
Traditional forms of therapy and rehabilitation usually derive from
therapist observation and judgment. Drawbacks of this traditional
method are that they are often inaccurate, expensive, and timely
[30]. Virtual reality, however, addresses these concerns as a useful
tool for improving outcomes compared to conventional therapy by
enabling accurate motion capture, telepresence based sessions, and
low cost motivating experiences [26, 8]. The immersive visual ca-
pabilities of modern VR headsets, such as the HTC Vive and Ocu-
lus Rift, have had astounding promise and success with treatments
ranging from exposure in Post Traumatic Stress Disorder [34, 28],
Borderline Personality Disorder [31], various phobias [39, 14, 7],
schizophrenia [35], and many other psychological therapies. Re-
searchers are even reporting that integration VR into the clinical
setting can reduce pain similar to the effect of analgesic treatments
[15, 17].
Success in VR therapy often relates to the relationship between
presence and emotion with technology’s ability to bridge them [28].
Increasing the quantity and quality of stimuli in immersive VR is
key to influencing user behavior and experience [4, 28]. The past
five years have made strides in VR technology – VR is ever more
immersive, affordable, and accessible to the average consumer with
over 200 million immersive VR Head Mounted Displays projected
to be sold by the year 2020 [1, 10]. For these reasons, headset-
based VR systems like the HTC Vive and Oculus Rift could appeal
to low-income communities. VR as a therapeutic tool has, there-
fore, become the most effective and affordable it has ever been and
is projected by many researchers to continue along this forecasted
trajectory.
Many of these studies incorporate Mirror Visual Feedback Ther-
apy (MVFT), the visual or physical stimulation of a “pseudo“
movement on the damaged limb to promote recovery [6]. Patients
are given sensorimotor feedback by reflecting an abled arm in the
position of the impaired arm during exercise [27]. MVFT is sug-
gested to be a beneficial treatment for motor rehabilitation [33, 27],
where clinical studies have indicated that MVFT “can serve as a
versatile tool to promote motor recovery” in mobility and arm use
[6].
MVFT requires the superposition of a simulated arm on a phan-
tom limb which enables patients to relieve painful sensation and
increase movement [33]. A variety of other conditions were ex-
plored with MVFT, including with stroke survivors, where a simu-
lated limb is placed in the patients midsagittal plane, thus reflecting
movements of the nonparetic side as if it were the affected side to
stimulate brain plasticity [42]. However, users with severe motor
loss require increased physical assistance to perform MVFT, and
can often require therapist intervention, or in the case of this study:
a robotic wearable.
A large amount of physical therapy research has transpired on
the integration of rehabilitative robotic wearable devices. Several
upper-body robotic exoskeletons have been developed and explored
over the last ten years with many incorporating VR. Some of these
examples include the PERCRO (Perceptual Robotics laboratory) L-
EXOS system [11], Rutgers CyberGlove and Master II-ND (RMII)
force feedback glove [21], and Therapy Wilmington Robotic Ex-
oskeleton (T-WREX) [18]. Through combining VR with exoskele-
tons that provide arm gravity support, clinical testing showed a
range in improvement of mobility, strength, and satisfaction [5].
One attribute common to these exoskeletons is the use of rigid
structures. A significant flaw experienced by traditional rigid ex-
oskeletons is their inflexibility, and the burden users bear when
wearing them. Devices which have few degrees-of-freedom (DoF)
or heavy components inhibit some movements. This physical con-
straint can lead to imbalanced muscular growth and control, which
can injure users of these wearable robots [44]. As a result, softer
devices such as Lessard et al. exosuits have emerged as a flexible al-
ternative to traditional rigid exoskeletons [24, 25]. This study aims
to leverage such soft exosuits during VR therapy through the use of
Compliant Robotic Upper-Extremity Exosuit (CRUX), as shown in
Figure 1, to explore feasibility, ease-of-use, and comfort.
Figure 1: A demonstrator wears CRUX [25] (without IMUs). CRUX
is an augmentative wearable soft robot for upper-extremity rehabilita-
tion and can be combined with VR through Project Butterfly to enable
immersive rehabilitation.
3 SYSTEM DESIGN
The VR experience developed, Project Butterfly (PBF), is a game
that motivates users to perform upper body motion primitives by
having them protect and control a virtual butterfly in a meadow
while the system collects real-time data using the HTC Vive. Fig-
ure 3 displays an example of PBF gameplay. CRUX was integrated
accordingly to provide additional tactile feedback and physical as-
sistance.
3.1 The Soft Robotic Wearable
The purpose of using the CRUX exosuit was twofold: to physically
stimulate movement to achieve an ideal position, and to immerse
the user deeply into the VR environment. Since immersion is a
crucial factor in the influence of user behavior and compliance [4],
PBF paired with CRUX may significantly improve the user’s reha-
bilitation experience.
The designed exosuit, shown in Figure 1, is capable of lifting
the user’s arm in different directions to create smooth multi-jointed
movements. The concept of tensegrity for soft robotics inspired
the mechanics of CRUX. Tensegrity (a portmanteau of “tensile”
and “integrity”) defines structures as internally prestressed, free-
standing, pin-jointed networks in which the cables or tendons are
held in tension against a system of bars or struts [16].
A base layer of neoprene held CRUX together. Cables are routed
along the neoprene to integrate a network of “anchor points,” which
serve as the rigid components in an otherwise flexible system. The
exact placement of the cables was determined by recording arm
movement on people as they stretched out and expressed their full
range of motion [25]. Through examining extensive motion capture
of the arm, the area on the skin which sheared the least was deter-
mined to find the most stable places to plant anchor points [25].
Bicycle housing routes the cables onto anchor points to actuate
different parts of the arm just as how tendons pull limbs. Six mi-
cro DC motors were mounted on a modular backplate and connect
to the cables of the exosuit via 3d printed spool, and are manipu-
lated through a microcontroller powering the system with a 3-Cell
Lithium polymer battery. Each motor is capable of exerting 88 N
of force (125 oz-in). Figure 1 depicts CRUX being operated by
a demonstrator. The selected material of CRUX affords a compli-
ant and lightweight design. Like similar soft exosuits, CRUX is
lightweight. However, CRUX weighs 1.5 kg [24] compared to the
6.8 kg lower-limb gait-assisting exosuit developed by Wehner et al.
[46] or the 2.27 kg suit by Alvara et al. [3] for upper arm force
amplification
The suit’s controller was designed to have the weak arm follow
the movement of the healthy limb. This mirroring of limbs insti-
gates mimetic controller design. Mirroring the movement from one
side to the other side of the body was inferred from MVFT to in-
crease motor recovery and stimulate brain plasticity [42]. Figure 2
depicts the CRUX being fitted to a user by an evaluator for upper
arm force amplification.
To enable the mimetic control of the healthy arm onto the weak
limb, wireless connectivity capability was added to connect with
the Inertial Measurement Unit (IMU) networks. An IMU is an elec-
tronic device that measures and reports a body’s specific force and
angular rate using a combination of accelerometers, magnetome-
ters, and gyroscopes [2]. The IMU network added to the exosuit
consists of 4 IMU nodes where each node can measure 3-axis ori-
entation of itself and then send this data back to the microcontroller.
The IMU nodes on CRUX are enclosed in a 3D printed case with
adjustable Velcro straps to accommodate various body sizes.
A plunger button must be engaged by the user to allow for exo-
suit movement. This safeguard prevents accidental actuation when
the exosuit is in master-slave mode. If the user feels that the motor
is performing movements that are undesired, the user can release
the plunger button, effectively disengaging the motor.
In complementary locations, nodes are positioned on both arms
at the lateral forearm (midway between the wrist and the elbow) and
the lower medial triceps (slightly above the elbow) [25] as seen in
Figure 2. Each node transmits pose data to the central controller to
support the closed-loop function enabling the pose following from
the healthy arm onto the impaired arm. It should be noted that while
the VR device can perform motion capture in place of the IMUs,
creating this dependency would limit the flexibility of CRUX for
future use. For future example, the suit may be used beyond VR
MVFT to assist with active daily living activities, where the con-
trols and level of assistance are calibrated during the VR therapy
sessions.
Figure 2: A participant exploring CRUX [25]. Control is achieved
by using IMU Nodes and an internal controller for leader-follower
mimicry. A user can control their impaired arm using their healthy
arm to match the movement path.
Figure 3: A user playing Project Butterfly. a) is the study proctor’s
on-screen view b) is the in-person view of the study proctor and c) is
the in-game view from the user’s perspective.
3.2 The Immersive Virtual Reality Experience
A motion primitive is defined as a distinct movement achievable
by a single joint which creates a unique degree of freedom (DoF).
Thus, upper body motion primitives can be thought of as indivisible
building blocks that can be combined and permuted into a broader
range of potential movements. The HTC Vive, one of the highest
grossing VR Entertainment Systems [1] developed by the Valve and
HTC Corporation, can be used as a powerful tool to both track these
motion paths and motivate the user to achieve these motions. The
Vive is a VR Head-Mounted Display that implements room scale
4x4 m outside-in tracking technology by utilizing a “lighthouse”
system of lasers which enable the user to interact with the virtual
environment through accurate motion capture in a 3D virtual space
[20]. Complementary to the HTC Vive are two handheld controllers
that feature dynamic haptic feedback which enhances spatial orien-
tation [20]. The HMD provides a 110-degree field of view and 90
Hz refresh rate [41]. Motion capture is tracked at 120 Hz using in-
frared laser sweeping and photo-diodes that enable for recovery of
position and orientation [20]. The HTC Vive also features a safety
guidance system preventing users from potential injury in the real
world environment [41, 32]. The worst case tracking jitter of the
system has been reported to be under 2.1mm with an accuracy of
an absolute 2mm error [23]. Resultingly, the HTC Vive allows for
the ability to extract accurate gameplay data while providing an en-
veloping experience of touch, sound, and sight.
Paired with an HTC Vive controller, the exosuit assists the user
during VR gameplay. Testing of the system targeted two pairs
of motion primitives: elbow extension/flexion, and shoulder ab-
duction/adduction. Biceps received assistance by replacing the
user’s CRUX supported HTC Vive controller as a bubble shield and
having them protect the butterfly from incoming rain and projec-
tiles through a therapist-specified customized range of motion path.
Haptic feedback is enabled so that the user is indicated with strong
pulses whenever the motion primitive was not followed (failure to
encapsulate the butterfly inside the bubble). To increase the incen-
tive and track compliance, the user receives a scoring point per ev-
ery half second that they mirrored the required motion primitive.
This multi-sensory feedback guided users according to the objec-
tive of the game.
To generate the environment and the mechanics of PBF, the
Unity v2017.1.0b4 Game Engine along with the SteamVR Unity
plugin v1.2.0 became the chosen development tools. Both Unity
and SteamVR hold a large amount of open-access documenta-
tion, including flexibility with programming languages such as
Javascript and C# [43]. Using Unity’s built-in physics engine, the
Rigidbody class was used to model the butterfly along with spher-
ical colliders that detected contact with the butterfly. Assignment
of the moving projectiles and the butterfly with the rain were set
to a time-dependent spatial state, allowing for global physics-based
events to influence data capture. Runtime data collection was cap-
tured using C# and Microsoft .NET Framework at speeds of 90Hz
and higher.
To assist the user evaluation process, PBF includes a dynamic
evaluator GUI, seen in Figure 4, which automatically prompts the
evaluator or therapist to measure the length of each participant’s
arms through the motion capture (measuring the x-z plane maximal
distance between the VR HMD and the Vive Controller) or man-
ually entered ranges. This calibration stage is achieved entirely in
Unity during run-time with the HTC Vive so that user’s who may
not have access to CRUX in the future can still play PBF without
physical assistance. Figure 4 also features the option to change the
repetitions per minute of each motion primitive, and data exporta-
tion rate of each game session. In short, the evaluator GUI allowed
the evaluator to tailor the game to each unique individual and cus-
tomize data throughput.
The game-themed goals became protecting a butterfly from
heavy rain and projectiles using a bubble (as the avatar for the con-
troller of the weak arm), which focused on bicep curl and lateral
arm raise exercises. Users are instructed place to the bubble around
the butterfly to achieve a high score, and to “protect the butterfly”.
These mechanics required the users to smoothly follow the flight
of the butterfly within +/- 0.1m of the required motion path. As a
result, the biceps and lateral arm raise minigames became scripted
movements of high accuracy and scripted timing repetitions when
a user performed a motion primitive.
Lastly, a tutorial started before every game that the evaluator
could enable to adjust the user to the VR environment. In the tu-
torial, users were asked to identify three images placed on their
left, on their right, and behind them. Identification in the 360 VR
environment was implemented to familiarize the user with virtual
reality and show them that they can look in any direction. Addi-
tionally, a “How to Play” menu was added to teach evaluators how
to test users, and in-game instructions were added to clarify the goal
of each minigame. This pre-gameplay stage simultaneously intro-
duce users not only to virtual reality mechanics (such as 360-degree
views) but also to PBF specific mechanics like the arm movements
required to earn a higher score. These designs were intended to
tailor the game to each unique individual smoothly and intuitively,
all while increasing the incentives to perform the objective of the
game.
Figure 4: Dynamic Project Butterfly Evaluator Interface. This UI and
the data it gathered allowed for more balanced games for each sub-
sequent participant.
4 USABILITY ST UDY
The study investigated the usability of PBF synergized with CRUX
system design, which includes ease of use, comfort, and set base-
line feasibility with nine elderly users having motor dysfunction.
Users were observed playing the virtual reality game with the ex-
osuit as they performed tasks that identified the ability of targeted
muscles and muscle groups. The nine elderly participants in usabil-
ity evaluations represented three patient segments:
1. Three retirees from Elderday Adult Day Health Care Center,
Santa Cruz, CA, represented a mental disability use case be-
cause they were affected by memory loss or dementia. Evalu-
ations with these users were conducted at Elderday.
2. Three stroke survivors recruited from Cabrillo College’s
Stroke and Disability Learning Center (SDLC), Capitola, CA,
represented a physical disability use case because they were
affected by neglect syndrome, meaning the motor functions of
one side of their body was impaired by their stroke. Evalua-
tions with these users were conducted at SDLC.
3. Three retirees living independently within their community,
outside of daycare or hospice, which do not have a significant
physical or mental disability. Evaluations with these users
were conducted at the University of California, Santa Cruz.
All participants ranged between the ages of 60 to 80 years old
and were previously unfamiliar with both virtual reality and wear-
able robotics. These user groups represent three demographics:
physical disability, mental disability, and no disability-were se-
lected because the physical and mental disabilities are likely the
target demographics for PBF and CRUX. This preliminary study
serves as feasibility to justify further studies with larger user group
sizes and gain design insights.
For each user, the evaluation began by an proctor giving a de-
tailed explanation of what PBF and CRUX are, answering questions
as needed. Then the user was given a tutorial period where they
walked through the “How to Play” menu and played the game with-
out being recorded. The tutorial consisted of playing two rounds of
the biceps minigame and the lateral arm minigame for one minute
each. The first round with the minigames was played without wear-
ing the CRUX exosuit and the second was played while wearing it
to allow for MVFT. This allowed the user to try out the controller
and learn PBF’s basic game mechanics. The point score from the
tutorial period helped the evaluator calibrate each user (adjusting
arm length and speed).
After the tutorial period, the test commenced with the user play-
ing recorded sessions while wearing the CRUX exosuit and playing
1-minute minigames, shown in Figure 6. Sessions were recorded
using a webcam, collecting gameplay scores and positional data,
and conducting post-session interviews which allowed for the col-
lection of quantitative and qualitative data. User evaluation inter-
views were executed by asking users questions pertained to their
experience from a prepared form with questions such as:
Open-ended questions:
What day-to-day tasks do you struggle with?
How many years have you been doing physical rehabil-
itation therapy?
Do you enjoy video games?
How can the virtual reality game be improved?
How can the exosuit be improved?
Rate the statement on a 5-point Likert scale:
Q1: My current therapy is engaging.
Q2: The virtual reality game was enjoyable.
Q3: I became fatigued while playing the virtual reality
game.
– Q4: The virtual reality game distracted me from pain
when doing physical movement.
Q5: If I had access to virtual reality therapy games, I
would use it in the future.
After evaluations, proctors reviewed gameplay and interview data
and commented additional thoughts or observations. Note that re-
sulting answers from Q1-Q5 can be seen in Figure 5.
Figure 5: Nine user responses of 5-point Likert Scale Questions per-
taining to PBF
Figure 6: A user performs lateral arm raises during a Project Butterfly
game session with CRUX. To boost their scores, users had to mimic
the flight path of the butterfly, which in this case was an up-and-down
motion most easily copied by raising one’s arm up-and-down simi-
larly.
Figure 7: Position of the CRUX assisted controller during the lateral
arm raise mini-game. These graphs depict four users attempting to
mimic the movement of their targets (the butterfly avatar). Red is
the butterfly, green is users movements. Red arcs which are closely
matched by green trajectories (as shown in these graphs) mean that
users are successfully completing in-game objectives, which are tai-
lored to test their physical limits when wearing the exosuit.
5 RE SU LTS AN D DISCUSSION
The three primary objectives of the study were to evaluate the base-
line feasibility of the system, ease-of-use of the system, and comfort
of the system. These objectives are judged according to quantita-
tive metrics obtained from recorded metadata of users while play-
ing PBF. Qualitative data was acquired during post-gameplay inter-
views using mixed 5-point Likert Scale questions and open-ended
questions. The Likert scale questions are summarized in Figure 5.
5.1 System Feasibility
Most participants felt that CRUX augmented their upper limb
movements to some degree. Specifically, one participant (see User
1 in Figure 7) mentioned that their arm movements became more
“effortless” with the use of CRUX. Specifically in raising their
arm laterally, which had been challenging for them before donning
CRUX. They further elaborated that it was the first time in weeks
that they did not experience painful throbs when raising an arm
above their shoulders.
To observe an example of such movement, the position of the
CRUX assisted controller and controller target, the butterfly avatar,
were graphed against four scored users from SDLC (Users 2 and 4)
and Retirees (Users 1 and 3) for the lateral shoulder raise minigame
as seen in Figure 7. All four CRUX supported users were able to
achieve a score of 111/130 or higher, indicating that these users
were compliant to the motion primitive path for over 85% of game-
play. Additionally, the speed of the users controllers, headset, and
butterfly avatar (target to follow) is graphed as shown in Figure 8
as well as the acceleration of the user’s weak arm controller shown
in Figure 9. Each of these users demonstrated significantly differ-
ent gameplay movements even though they had close scores. When
considering Figure 7, User 1 had sharp changes, User 2 had shaky
movements, User 3 had smooth moves, and User 4 had smooth and
shaky movements. When looking at movement speed in Figure 8,
Users 1 and 3 maintained a constant speed, indicating a greater
control of their weaker arm. Whereas Users 2 and 4 had signifi-
cant spikes in movement speed indicating a lack of control in their
arms. It should be noted that while the exosuit assists in achiev-
ing position, it does not reduce the shaking of the limbs. Figure 9
reflects the findings in Figure 8 through acceleration, where User
1 maintains the most control nearing almost no acceleration, fol-
lowed by User 2 who spikes but nears zero, and Users 3 and 4 ex-
perienced occasional large shaking. Despite the difference in fine-
motor strength and precision amongst participants, each user was
able to achieve the desired goal of protecting the virtual butterfly.
This might suggest that PBF can be accommodating for people at
various stages of their physical therapy, thus making it more acces-
sible. Also, while User 1 and User 4 have differently lengthened
arms by about 0.2 meters, the compliance of over 85% can be seen
visible as their green arm paths overlay the object red path arch in
Figure 7. Although these results are promising, the sample size is
not statistically significant, which warrants further study as noted
in the Limitations Section 6.5.
On the subject of assistance, most participants felt that CRUX af-
fected augmenting their limbs. However, they all asked for stronger
motors on the exosuit. When asked for a potential reason to this,
most users indicated that they felt that the motors were not power-
ful enough to make the difference that they were expecting. One
user (see User 3 in Figure 7) commented that they understood the
function of the device but “[wanted] even more power behind it.
Additionally, none of the participants felt that the exosuit made it
more difficult for them to move.
5.2 Ease of Use
An interesting observation was that users said they “knew the goal
was always to protect the butterfly,” which was a gameplay theme
Figure 8: The speed of the user’s handheld controller and headset as
they attempt to catch the virtual butterfly avatar during the lateral arm
raise mini-game. (Green is the speed of the Weak Arm Controller,
Red is the speed of the Butterfly, Blue is the speed of the headset)
Figure 9: The acceleration of the user’s handheld controller as they
attempt to catch the virtual butterfly avatar during the lateral arm raise
mini-game. Users with high scores and large fluctuations in acceler-
ation are able to react quickly to in-game obstacles, suggesting a
higher level of control and strength than those who cannot. (Green is
the acceleration of the Weak Arm Controller)
added when brainstorming ideas with potential users. Creating in-
game goals which center around an archetypal emotional response
may have generated a quicker understanding of gameplay mechan-
ics in the evaluated users. When asked what they thought of the
difficulty, almost all participants thought it was appropriately chal-
lenging. This makes sense as the game‘s difficulty was adjusted
based on their first gameplay before and during the tutorial period.
By automating all control and dynamic elements of both CRUX
and PBF, there was an increase in compliance when performing mo-
tion primitives as well as reducing the contention between the suit
and virtual reality. Users tended to agree that they noticed the exo-
suit was assisting them smoothly as if part of the gameplay, due to
the mimetic control, suggesting that automating CRUX user-input
may have benefited players immersion.
5.3 Comfort
Users asked for greater ventilation in CRUX, due to the form-
fitting style of the neoprene. Additionally, since CRUX was proto-
typed using commercially available wetsuit neoprene, not all users
equally fit the base layer. The majority of the participants still re-
sponded that they enjoyed playing the minigames with CRUX as
seen in Figure 5.
Through observation and commentary, no one seemed dis-
pleased with the aesthetics of the minigames. Some users com-
plimented the aesthetics, citing the goal of butterfly protection as
“fun,” “engaging,” and even “meditative. This is particularly excit-
ing news since many people unfamiliar with video games are often
intimidated or dissuaded from immersive virtual environments, es-
pecially considering that five out of nine participants responded that
they dislike video games. The user who mentioned that the games
were “engaging” initially discussed their reservations towards video
games and how they preferred “real things. When users are psy-
chologically comfortable with a system, they are more likely to ben-
efit from it.
From these observations and recordings, we have found that
most users believe PBF was useful in helping them move their
arms as seen in Figure 5. Combining CRUX’s master-slave sys-
tem and PBF’s butterfly protection mechanics actively encouraged
players to perform the required movements while being distracted
from their physical therapy as agreed by six out of nine users (and
one additional neutral response). These users found PBF enjoyable
and agreed that the game did distract themselves from the exertion
of physical moving. Seven out of nine users stated that they would
regularly use PBF & CRUX in the future for exercises if it were
available to them.
5.4 Discussion
Both the quantitative and qualitative data generated suggested that
in the short term, the system can help users achieve in the moment
tasks by augmenting users upper body strength to enable them to
move their upper limb more easily. In the long term, the paired-
system serves as a boost to help them train their weaker upper arm
and make it stronger over time. In the case of stroke survivors from
SDLC who suffer from neglect syndrome, it additionally helped
them in recognizing what independently driven movement on the
neglected side feels like again. Qualitative lessons learning suggest
the following:
The exosuit must afford independence on behalf of the user.
In the gameplay sessions, the experimenters helped users to
don and off the suit. If this suit is indeed to be used as a
home-based exercise system, users must be able to put on and
take off the suit by themselves, which at the moment is still
challenging.
Using a neoprene top as the base to CRUX limited those who
could wear it and made it hot when worn over an extended pe-
riod and difficult to adhere. In future iterations, exosuits like
CRUX needs to be more easily customizable, for example by
replacing neoprene in the exosuit with elastane (e.g., Lycra).
Future iterations of robotic support should make the augmen-
tation slightly more powerful, so the aid from CRUX is more
apparent to users.
Given that the PBF and CRUX are new to players, they must
be acquainted with the technologies individually. When given
even just a few minutes to become acquainted, players made
much better use of the CRUX when playing PBF and were
less overwhelmed.
Through the study, a set of design guidelines was compiled for
other practitioners of wearables and VR games to augment upper
limb movements and motivate exercises, especially in older adults
and people with motor impairment:
Exosuit needs to be designed to be more flexible to fit a variety
of body sizes and shapes.
Exosuit augmentation needs to be noticeable, perhaps by
adding an in-game UI element such as the bubble lighting up
when the suit is activated. This will hopefully keep the user
immersed while also providing exosuit control indication to
the user.
Reduce various forms of stimulation to the minimum with-
out affecting the exercise goals without breaking immersion.
Feedback and color variations went unnoticed by some of the
users as they were not related directly to their goals.
Stay within the appropriate difficulty of the users in terms of
game speed, ranges of motion, etc.
The user feedback suggests that the users enjoyed protect-
ing the butterfly, as it caused an “emotional attachment.”
Emotion-driven immersion suggests a powerful tool in cre-
ating engaging experiences.
5.5 Limitations
One user responded “Disagree” for questions Q2 (Enjoyable
Game), Q4 (Distracted from Pain) and Q5 (Future Use) from Figure
5. The user commented that the novelty of VR was fun, but con-
tinuously protecting a butterfly would become boring if they used
it for regular therapy. The user requested that future games should
have a compelling storyline to keep them interested and motivated.
The intent of PBF was preliminary feasibility, and future games
should be developed and studied based on lessons learned, trying
out game mechanics, and the targeted therapy desired. Concerns
that this user poses should be investigated in subsequent games to
determine if specific game mechanics are more preferred by users
than others.
Only three motion primitives were explored and converted into
VR through PBF. Specifically, they were the Lateral Shoulder
Raise, Forward Arm Raise, and Horizontal Shoulder Rotation. Fu-
ture studies should incorporate a more variety of motion primitives
to catalyze potential benefits to the users’ potential mobility im-
provements. There may also be potential in incorporating com-
mon assessment tools into the VR environment for automated as-
sessment such as the Apley’s scratch test (Shoulder Mobility) [12],
Wolf Motor Function Test (Upper-Extremity Mobility) [29], Fugl-
Meyer Assessment (General Motor Ability) [13]; all of which as-
sess motion primitives for active daily living.
Furthermore, this study would benefit from a larger sample size
of the users, more therapists could be involved, and more stimuli
and testing must be done to determine feasibility, compliance, and
design further. There is a possible novelty effect of the VR, where
most of these users were exposed to the VR for their very first time.
The results suggest that the game design may account for the nov-
elty effect. However, a long term study must be done to address
these possibilities adequately. Subsequently, a long term study is
being planned with local hospitals in Santa Cruz, California, to ex-
plore the effects of Project Butterfly with CRUX further. An IRB
protocol for such is currently under review for approval.
6 CONCLUSIONS AND FUTURE WORK
Project Butterfly reports on the design and evaluation of a unique
VR experience paired with a soft body robotic wearable exosuit.
The pair of these technologies have been developed as a novel ex-
perience to rehabilitate upper-extremities. CRUX reduces the bur-
den and rigidity experienced by users of traditional wearable robots
through its softer, more structurally compliant constitution. How-
ever, a material other than neoprene should be used to make it
more comfortable and less likely to overheat the user. To tailor
to a more significant number of DoF, the designed VR game, PBF,
is aimed at focusing on motion primitives expressible in soft ex-
oskeletons – actions which the healthy arm can perform that can be
combined and permuted into all upper-extremity movement. PBF
thus serves as a motivator for the user to complete virtual objec-
tives and consequently, actual motions. The completion of these
objectives is assisted through CRUX’s augmentation of their up-
per body strength to perform the game-specific movements. When
evaluating the baseline feasibility of PBF and CRUX in augmenting
and promoting proper arm movement as defined by the established
motion primitives, most users were able to complete appropriately
challenging arm movements, suggesting that PBF and CRUX gave
users suitable strength of their augmented arm. Additionally, the
system’s ease-of-use and comfort were analyzed, and most users
felt that they were confident capitalizing on the therapy system.
Virtual reality paired exosuits could prove useful to make engag-
ing therapy for users with upper-extremity impairment. For more
significant impact, designing a new exosuit peripheral and increas-
ing more types of accompanying VR minigames which augment
various muscle groups (i.e., pectoral muscles and dorsal muscles)
can further improve the rehabilitation. These muscle groups sup-
port upper-extremity actions and strengthening them could bolster
arm muscles as a result. In a similar vein, using new materials for
a future iteration of a soft exosuit focused for VR could make this
technology more comfortable and accessible.
Furthermore, the real-time data produced from the HTC Vive and
Vive Tracking units can be integrated with the exosuit. Therapists
and users may also potentially benefit from further data extrapola-
tion with the HTC Vive. A complete view of a user’s body, achiev-
able through more precise motion tracking and inverse kinematics,
and could identify confounding postural issues, such as slouched
backs and other movement biases, which a physical therapist would
want to be aware of during gameplay. A long term study is being
planned with local hospitals to explore the effects of Project Butter-
fly with a next-generation CRUX design that accommodates more
body types, sizes, and weight. Finally, with the plethora of posi-
tional and behavioral data output produced from this VR experi-
ence, there is potential to integrate machine learning protocols and
AI to optimize suit controls and game difficulty to improve rehabil-
itation results.
The baseline feasibility, ease of use, and comfort created by syn-
ergizing an immersive physio-therapy VR game with an actuated
soft robotic exosuit had promising results in the potential future
of a more accessible, affordable, and personalized rehabilitation.
More research is needed to expand upon the preliminary work of
this study, discover best practices of soft exosuit integrated VR, and
validate clinical utility. Subsequently, there are more butterflies to
follow on the path ahead.
ACKNOWLEDGEMENTS
This material is based upon work supported by the National Science
Foundation under Grant No. #1521532. Any opinions, findings,
and conclusions or recommendations expressed in this material are
those of the author(s) and do not necessarily reflect the views of the
National Science Foundation. The authors would like to thank the
following UC Santa Cruz affiliates for their support: John McIn-
tyre, Pattawong Pansodtee, Ocean Hurd, Dustin Halsey, Evanjelin
Mahmoodi, Noah Brown, and Dr. Daniel Goodman Shapiro. The
authors additionally thank the participants of this study for their
time and effort.
REFERENCES
[1] Virtual reality for consumer markets - head-mounted displays, acces-
sory devices, and consumer virtual reality content: Global market
analysis and forecasts. report, Tractica Research, Jan 2016.
[2] N. Ahmad, R. A. R. Ghazilla, N. M. Khairi, and V. Kasi. Reviews on
various inertial measurement unit (imu) sensor applications. Interna-
tional Journal of Signal Processing Systems, 1(2):256–262, 2013.
[3] A. Alvara. Development of an exosuit for rehabilitative use and upper
arm force amplification. In 2017 AAAS Annual Meeting (February
16-20, 2017). aaas, 2017.
[4] R. M. Ba ˜
nos, C. Botella, M. Alca ˜
niz, V. Lia˜
no, B. Guerrero, and
B. Rey. Immersion and emotion: their impact on the sense of pres-
ence. CyberPsychology & Behavior, 7(6):734–741, 2004.
[5] M. S. Cameir˜
ao, S. Berm´
udez, and P. Verschure. Virtual reality based
upper extremity rehabilitation following stroke: a review. Journal of
CyberTherapy & Rehabilitation, 1(1):63–74, 2008.
[6] F. J. Deconinck, A. R. Smorenburg, A. Benham, A. Ledebt, M. G.
Feltham, and G. J. Savelsbergh. Reflections on mirror therapy: a sys-
tematic review of the effect of mirror visual feedback on the brain.
Neurorehabilitation and Neural Repair, 29(4):349–361, 2015.
[7] J. Diemer, G. W. Alpers, H. M. Peperkorn, Y. Shiban, and
A. M¨
uhlberger. The impact of perception and presence on emotional
reactions: a review of research in virtual reality. Frontiers in psychol-
ogy, 6, 2015.
[8] A. Elor, S. Kurniawan, and M. Teodorescu. Towards an immer-
sive virtual reality game for smarter post-stroke rehabilitation. In
2018 IEEE International Conference on Smart Computing (SMART-
COMP), pages 219–225. IEEE, 2018.
[9] A. Elor, M. Teodorescu, and S. Kurniawan. Project star catcher:
A novel immersive virtual reality experience for upper limb reha-
bilitation. ACM Transactions on Accessible Computing (TACCESS),
11(4):20, 2018.
[10] A. N. V. Follow. Virtual reality (vr) continuum - amp new ven-
tures, Dec 2016. https://www.slideshare.net/ampnewventures/virtual-
reality-vr-continuum-amp-new-ventures.
[11] A. Frisoli, F. Salsedo, M. Bergamasco, B. Rossi, and M. C. Car-
boncini. A force-feedback exoskeleton for upper-limb rehabilitation
in virtual reality. Applied Bionics and Biomechanics, 6(2):115–126,
2009.
[12] F. Garc´
ıa-Muro, ´
A. L. Rodr´
ıguez-Fern´
andez, and A. Herrero-de Lu-
cas. Treatment of myofascial pain in the shoulder with kinesio taping.
a case report. Manual therapy, 15(3):292–295, 2010.
[13] D. J. Gladstone, C. J. Danells, and S. E. Black. The fugl-meyer assess-
ment of motor recovery after stroke: a critical review of its measure-
ment properties. Neurorehabilitation and neural repair, 16(3):232–
240, 2002.
[14] H. Grillon, F. Riquier, B. Herbelin, and D. Thalmann. Virtual reality as
a therapeutic tool in the confines of social anxiety disorder treatment.
International journal on disability and human development, 5(3):243–
250, 2006.
[15] D. Gromala, X. Tong, A. Choo, M. Karamnejad, and C. D. Shaw.
The virtual meditative walk: virtual reality therapy for chronic pain
management. In Proceedings of the 33rd Annual ACM Conference on
Human Factors in Computing Systems, pages 521–524. ACM, 2015.
[16] A. Hanaor. Engineering properties of double-layer tensegrity grids.
SpatialStructures at the Tum oftheMillenium, pages 195–200, 1991.
[17] H. G. Hoffman, G. T. Chambers, W. J. Meyer, L. L. Arceneaux, W. J.
Russell, E. J. Seibel, T. L. Richards, S. R. Sharar, and D. R. Patterson.
Virtual reality as an adjunctive non-pharmacologic analgesic for acute
burn pain during medical procedures. Annals of Behavioral Medicine,
41(2):183–191, 2011.
[18] S. J. Housman, V. Le, T. Rahman, R. J. Sanchez, and D. J. Reinkens-
meyer. Arm-training with t-wrex after chronic stroke: preliminary
results of a randomized controlled trial. In Rehabilitation Robotics,
2007. ICORR 2007. IEEE 10th International Conference on, pages
562–568. IEEE, 2007.
[19] L. M. Howden and J. A. Meyer. Age and sex composition: 2010.
2010 Census Briefs, US Department of Commerce, Economics and
Statistics Administration. US CENSUS BUREAU, 2010.
[20] HTC-Corporation. Vive vr system. Vive, November 2018.
https://www.vive.com/us/product/vive-virtual-reality-system/.
[21] D. Jack, R. Boian, A. S. Merians, M. Tremaine, G. C. Burdea, S. V.
Adamovich, M. Recce, and H. Poizner. Virtual reality-enhanced
stroke rehabilitation. IEEE transactions on neural systems and re-
habilitation engineering, 9(3):308–318, 2001.
[22] R. Koopman and L. J. van Loon. Aging, exercise, and muscle protein
metabolism. Journal of Applied Physiology, 106(6):2040–2048, 2009.
[23] O. Kreylos. Lighthouse tracking examined. A developer’s perspec-
tive on immersive 3D computer graphics, May 2016. http://doc-
ok.org/?p=1478.
[24] S. Lessard, P. Pansodtee, A. Robbins, L. B. Baltaxe-Admony, J. M.
Trombadore, M. Teodorescu, A. Agogino, and S. Kurniawan. Crux:
A compliant robotic upper-extremity exosuit for lightweight, portable,
multi-joint muscular augmentation. In Rehabilitation Robotics
(ICORR), 2017 International Conference on, pages 1633–1638. IEEE,
2017.
[25] S. Lessard, P. Pansodtee, A. Robbins, J. M. Trombadore, S. Kurni-
awan, and M. Teodorescu. A soft exosuit for flexible upper-extremity
rehabilitation. IEEE Transactions on Neural Systems and Rehabilita-
tion Engineering, 2018.
[26] K. R. Lohse, C. G. Hilderman, K. L. Cheung, S. Tatla, and H. M.
Van der Loos. Virtual reality therapy for adults post-stroke: a sys-
tematic review and meta-analysis exploring virtual environments and
commercial games in therapy. PloS one, 9(3):e93318, 2014.
[27] M. E. Michielsen, M. Smits, G. M. Ribbers, H. J. Stam, J. N. Van
Der Geest, J. B. Bussmann, and R. W. Selles. The neuronal correlates
of mirror therapy: an fmri study on mirror induced visual illusions in
patients with stroke. Journal of neurology, neurosurgery & psychiatry,
82(4):393–398, 2011.
[28] N. Morina, H. Ijntema, K. Meyerbr ¨
oker, and P. M. Emmelkamp. Can
virtual reality exposure therapy gains be generalized to real-life? a
meta-analysis of studies applying behavioral assessments. Behaviour
research and therapy, 74:18–24, 2015.
[29] D. M. Morris, G. Uswatte, J. E. Crago, E. W. Cook III, and E. Taub.
The reliability of the wolf motor function test for assessing upper ex-
tremity function after stroke. Archives of physical medicine and reha-
bilitation, 82(6):750–755, 2001.
[30] H. Mousavi Hondori and M. Khademi. A review on technical and clin-
ical impact of microsoft kinect on physical therapy and rehabilitation.
Journal of Medical Engineering, 2014, 2014.
[31] M. V. Nararro-Haro, H. G. Hoffman, A. Garcia-Palacios, M. Sampaio,
W. Alhalabi, K. Hall, and M. Linehan. The use of virtual reality to fa-
cilitate mindfulness skills training in dialectical behavioral therapy for
borderline personality disorder: a case study. Frontiers in psychology,
7, 2016.
[32] D. C. Niehorster, L. Li, and M. Lappe. The accuracy and precision of
position and orientation tracking in the htc vive virtual reality system
for scientific research. i-Perception, 8(3):2041669517708205, 2017.
[33] V. S. Ramachandran and D. Rogers-Ramachandran. Synaesthesia in
phantom limbs induced with mirrors. Proceedings of the Royal Society
of London B: Biological Sciences, 263(1369):377–386, 1996.
[34] B. O. Rothbaum, M. Price, T. Jovanovic, S. D. Norrholm, M. Gerardi,
B. Dunlop, M. Davis, B. Bradley, E. J. Duncan, A. Rizzo, et al. A
randomized, double-blind evaluation of d-cycloserine or alprazolam
combined with virtual reality exposure therapy for posttraumatic stress
disorder in iraq and afghanistan war veterans. American Journal of
Psychiatry, 171(6):640–648, 2014.
[35] M. Rus-Calafell, J. Guti´
errez-Maldonado, and J. Ribas-Sabat´
e. A vir-
tual reality-integrated program for improving social skills in patients
with schizophrenia: a pilot study. Journal of behavior therapy and
experimental psychiatry, 45(1):81–89, 2014.
[36] R. D. Seidler, J. L. Alberts, and G. E. Stelmach. Changes in multi-joint
performance with age. Motor control, 6(1):19–31, 2002.
[37] R. D. Seidler, J. A. Bernard, T. B. Burutolu, B. W. Fling, M. T. Gor-
don, J. T. Gwin, Y. Kwak, and D. B. Lipps. Motor control and aging:
links to age-related brain structural, functional, and biochemical ef-
fects. Neuroscience & Biobehavioral Reviews, 34(5):721–733, 2010.
[38] R. D. Seidler-Dobrin, J. He, and G. E. Stelmach. Coactivation to re-
duce variability in the elderly. Motor control, 2(4):314–330, 1998.
[39] Y. Shiban, I. Schelhorn, P. Pauli, and A. M ¨
uhlberger. Effect of com-
bined multiple contexts and multiple stimuli exposure in spider pho-
bia: a randomized clinical trial in virtual reality. Behaviour research
and therapy, 71:45–53, 2015.
[40] N. Shkuratova, M. E. Morris, and F. Huxham. Effects of age on bal-
ance control during walking. Archives of physical medicine and reha-
bilitation, 85(4):582–588, 2004.
[41] F. Soffel, M. Zank, and A. Kunz. Postural stability analysis in virtual
reality using the htc vive. In Proceedings of the 22nd ACM Conference
on Virtual Reality Software and Technology, pages 351–352. ACM,
2016.
[42] H. Thieme, J. Mehrholz, M. Pohl, J. Behrens, and C. Dohle. Mirror
therapy for improving motor function after stroke. Stroke, 44(1):e1–
e2, 2013.
[43] ValveSoftware. Valvesoftware/openvr. GitHub, Jul.
https://github.com/ValveSoftware/openvr.
[44] H. Wang and T. Cochrane. Mobility impairment, muscle imbalance,
muscle weakness, scapular asymmetry and shoulder injury in elite
volleyball athletes. Journal of sports medicine and physical fitness,
41(3):403, 2001.
[45] N. S. Ward. Compensatory mechanisms in the aging motor system.
Ageing research reviews, 5(3):239–254, 2006.
[46] M. Wehner, B. Quinlivan, P. M. Aubin, E. Martinez-Villalpando,
M. Baumann, L. Stirling, K. Holt, R. Wood, and C. Walsh. A
lightweight soft exosuit for gait assistance. In Robotics and Automa-
tion (ICRA), 2013 IEEE International Conference on, pages 3362–
3369. IEEE, 2013.
... Another low-cost motion capture device is Microsoft's Kinect, which contains depth cameras allowing marker-less and full-body tracking [19,52,63]. One other approach that is low-cost is the use of inertial measurement unit (IMU) that can track users' joint angle in wrist movements [41,64] and can be attached to users' arms, suits or shoes when performing motor tasks [30,50]. ...
... In this rehabilitation game, the patients were encouraged to use their weak arm more by rewarding them with more scores as more stars were caught by the weaker arm. Another application by Elor et al. [30] allows users to control a butterfly using a virtual bubble (as shown in Fig. 4 (b)) for the upper extremity rehabilitation. ...
... Besides usability scales, other evaluation methods based on observations, interviews, focus groups, logging data, and user feedback, can also be adopted to gather supplementary usablity data [14]. In a study conducted by Elor et al. [30], the feasibility, ease of use, and comfort of their upper limb rehabilitation application was investigated via observations, interviews, and questionnaires. ...
Article
Full-text available
Virtual reality (VR) has emerged as a powerful tool for rehabilitation. Many effective VR applications have been developed to support motor rehabilitation of people affected by motor issues. Movement reproduction, which transfers users' movements from the physical world to the virtual environment, is commonly used in VR rehabilitation applications. Three major components are required for movement reproduction in VR: (1) movement input, (2) movement representation, and (3) movement modulation. Until now, movement reproduction in virtual rehabilitation has not yet been systematically studied. This paper aims to provide a state-of-the-art review on this subject by focusing on existing literature on immersive motor rehabilitation using VR. In this review, we provided in-depth discussions on the rehabilitation goals and outcomes, technology issues behind virtual rehabilitation, and user experience regarding movement reproduction. Similarly, we present good practices and highlight challenges and opportunities that can form constructive suggestions for the design and development of fit-for-purpose VR rehabilitation applications and can help frame future research directions for this emerging area that combines VR and health.
... II. SYSTEM DESIGN Our physical therapy system is based on a game called "Project Butterfly" (PBF) that was proposed by Elor et al [49], [50]. Previously, PBF explored the feasibility of a virtual reality enhanced exo-skeleton for post-stroke and elderly assistance through two exercises, but was not designed or tested for upper extremity physical therapy over a extended period of time with varying custom exercise movements as reported in this study. ...
... A. Data Collection Figure 1 shows the experimental setup. Through Vive and the Unity Game Engine, motion capture and behavior game data were recorded and stored during runtime at 90 Hz using a data exportation method developed in previous studies by Elor et al. [11], [49]. Motions were captured through utilizing the HTC Vive Pro 2018's outside-in tracking algorithm by a constellation of "lighthouse" laser systems [1]. ...
Article
Full-text available
Immersive Virtual Reality (iVR) Head-Mounted Display (HMD) systems paired with serious exercise games can positively augment physical rehabilitation process from both engagement and analytics perspectives. This paper presents a serious game for iVR HMD based long term upper-extremity exercise. We demonstrate the capabilities of our game through a case study with five users recovering from upper-extremity injuries. We examine how our program maintains engagement and motivation over eight weeks, where users completed bi-weekly prescribed movements framed as protecting a virtual butterfly. We assess user experiences through a mixture of biomarkers from brainwave, heart rate, and galvanic skin response recorded at runtime as well as motion capture and behavioral game data. Our results suggest that the iVR game was an effective medium in inducing high compliance, physical performance, and biometric changes even with increasing difficulty beyond the novelty effect period. We conclude with considerations of future work for iVR physical therapy games that adapt to biometric response.
... Project Star Catcher has demonstrated that iVR can beneit treatments such as Constraint-Induced Movement Therapy with stroke survivors by increasing compliance up to 40% when compared to traditional methods and providing an accessible medium for capturing patient success metrics with HMD motion capture [27,29]. Project Butterly, an iVR experience inspired by Mirror Visual Feedback Therapy, has explored the potential to engage patients for long-term treatment with immersive virtual environments, which is vital because stroke rehabilitation can span years [28,63]. REINVENT applied neurofeedback systems to iVR games with promising pilot results suggesting feasible and safe usage for severe stroke upper limb motor recovery [74]. ...
Article
Full-text available
Developing games is time-consuming and costly. Overly clinical therapy games run the risk of being boring, which defeats the purpose of using games to motivate healing in the first place [10, 23]. In this work, we adapt and repurpose an existing immersive virtual reality (iVR) game, Spellcasters, originally designed purely for entertainment for use as a stroke rehabilitation game—which is particularly relevant in the wake of COVID-19, where telehealth solutions are increasingly needed [4]. In preparation for participatory design sessions with stroke survivors, we collaborate with 14 medical professionals to ensure Spellcasters is safe and therapeutically valid for clinical adoption. We present our novel VR sandbox implementation that allows medical professionals to customize appropriate gestures and interactions for each patient's unique needs. Additionally, we share a co-designed companion app prototype based on clinicians' preferred data reporting mechanisms for telehealth. We discuss insights about adapting and repurposing entertainment games as serious games for health, features that clinicians value, and the potential broader impacts of applications like Spellcasters for stroke management.
... stroke survivors or cognitive disabilities). Our lab has worked with disability groups within our lab exploring various virtual reality mediums for users with cognitive disabilities, testing soft exo-suits meant for post-stroke rehabilitation, and physical rehabilitation games for users with cognitive disabilities [9], [56]- [58]. In the future we will collect motion data of these varying groups to develop more inclusive patient models. ...
Article
Full-text available
Objective: The adoption of telehealth rapidly accelerated due to the global COVID19 pandemic disrupting communities and in-person healthcare practices. While telehealth had initial benefits inenhancing accessibility for remote treatment, physical rehabilitation has been heavily limited due to the lossof hands-on evaluation tools. This paper presents an immersive virtual reality (iVR) pipeline for replicating physical therapy success metrics through applied machine learning of patient observation. Methods: Wedemonstrate a method of training gradient boosted decision-trees for kinematic estimation to replicate mobility and strength metrics with an off-the-shelf iVR system. During a two-month study, training datawas collected while a group of users completed physical rehabilitation exercises in an iVR game. Utilizing this data, we trained on iVR based motion capture data and OpenSim biomechanical simulations. Results:Our final model indicates that upper-extremity kinematics from OpenSim can be accurately predicted using the HTC Vive head-mounted display system with a Mean Absolute Error less than 0.78 degrees for jointangles and less than 2.34 Nm for joint torques. Additionally, these predictions are viable for run-time estimation, with approximately a 0.74 ms rate of prediction during exercise sessions. Conclusion: These findings suggest that iVR paired with machine learning can serve as an effective medium for collectingevidence-based patient success metrics in telehealth. Significance: Our approach can help increase the accessibility of physical rehabilitation with off-the-shelf iVR head-mounted display systems by providing therapists with metrics needed for remote evaluation.
... The study presented positive results for the virtual mirror task and proved that visual modulation is an effective form of therapy for UE rehabilitation in stroke patients. Another VR visual feedback therapy via HTC Vive HMD was explored in the butterfly project by Elor et al. (2019). The users experienced physiotherapy by following and guarding a virtual butterfly, with the help of a robot-based wearable device to assist the subject's UE movements. ...
Chapter
The idea of virtual reality (VR) and augmented reality (AR) is exploited, nowadays, in various industrial sectors such as automotive, healthcare, retail, tourism, architecture, and education to carry out human–computer interaction-based activities in an efficient manner. VR inculcates the idea of simulating 3D artificial environments and placing the users’ senses in the middle. On the other hand, AR is the result of superimposing perceptual information like sounds, images, and text on the actual environment to make the reality more interactive. These realities, as experienced by the user, can be familiar or distinctive from the actual world and can have varying degrees of immersiveness. In recent years, the advent of VR-based therapy is well appreciated in the healthcare sector for pain management, phobias treatment, cognitive and physical rehabilitation. In the present work, the efficacy of VR and AR systems is systematically reviewed for physical rehabilitation post-neurological disorders like Parkinson’s disease, brain injury, Stroke, and Multiple Sclerosis. The simulated environments using these digital technologies are found to be reconfigured according to the individual's level of motor impairment. At first, different types of VR and AR are presented based on the level of immersiveness and the markers used. Thereafter, these technologies are extensively reviewed for the upper extremity and lower extremity rehabilitation from the last 20 years. A PRISMA report is carried out to support the inclusion and exclusion of considered articles in the current study. Out of 533 articles, 72 articles are considered in this review process. The detailed findings in different cases of physical rehabilitation are presented in tabulated form based on disease, type of digital technology, training sessions, and outcome measures. The possible future opportunities and scope are discussed along with the concluding remarks. This review will help the research communities to understand the true potential of VR and AR systems for physical rehabilitation.
... Encontrados Aceitos Referências ACM 389 29 [22], [36], [52], [25], [41], [63], [61], [87], [48], [28], [16], [74], [34], [86], [1], [27], [46], [53], [23], [88], [12], [54], [44], [84], [75], [51], [4], [42], [24]. IEEE 164 38 [57], [73], [69], [6], [82], [29], [19], [72], [21], [37], [76], [30], [40], [79], [35], [71], [45], [65], [55], [7], [8], [59], [33], [93], [91], [2], [80], [77], [3], [43], [90], [5], [85], [67], [9], [70], [56], [78]. ...
Raw Data
Full-text available
A Tabela abaixo apresenta, de forma completa, a quantidade de artigos aceitos por base de dados, bem como suas respectivas referências bibliográficas.
... In 2019, seven million commercial HMDs were sold and were projected to reach 30 million sales per year by 2023 [4,30]. Such immersive input modalities with commercially available components may help us interpret user performance in serious games for player compliance, accessibility, and data throughput [10,12,13]. Researchers have reported considerable success in using virtual environments with serious games that explore psychological and physiological applications in various case studies and theories [9,11,29]. ...
Conference Paper
Full-text available
Haptic feedback vests afford a unique opportunity to enhance a user’s emotional engagement within the virtual world. In this paper, we present a two-stage study on user experience towards understanding emotional expression through utilizing the Pleasure-Arousal-Dominance model and the International Affective Picture System. We examine an authoring survey with 40 young adults, where users contextualized five emotion groups and designed patterns to express the feelings associated with each stimulus. Our resulting content analysis suggests design themes on a body-mind situation axis and an internal-external location axis. We found that vibrotactile actuation extends user emotion through phenomena which we call scene emulation, body function emulation, emotional resemblance,and emotional reflection. Lastly, we pilot these findings through an immersive virtual reality experience.
... As a result, there are more creative ways for post-stroke patients to do physical rehabilitation. Users may participate in immersive virtual reality games for post-stroke rehabilitation [2], [3], [4] and use portable exoskeleton suits in their homes to help with daily living. ...
Conference Paper
Full-text available
Wearable robots provide a unique opportunity for exosuits to stimulate enhanced stability and strength. The problem is creating an exoskeleton suit for users that is powerful, but lightweight, portable, and accessible for different body shapes. The actuators' position to control the suit's movement dramatically affects how much force an exoskeleton suit can exert. In this preliminary paper, we present a simulation approach for finding optimal actuation positions towards a bio-inspired exoskeleton. This modular approach leads towards custom design of actuator positioning based on bodily measurements. We discuss our approach through modeling a soft exosuit designed for upper-extremity movement.
... In addition to entertainment purposes, iVR holds vast potential for serious games, which has been on the rise due to the benefits of programmable iVR for physical and cognitive applications [2]. High-fidelity motion capture and telepresence capabilities allow these assitive experiences performed in VR environments to increase user game compliance, accessibility, and data throughput while using commercially available components [3], [4], [5], [6]. In this paper, we explore how iVR can be utilized to personalize an experience for emotion using the Unity3D Game Engine and the HTC Vive iVR System. ...
Conference Paper
Full-text available
Emotion, a crucial element of mental health, is not often explored in the field of immersive Virtual Reality (iVR). Enabling personalized affective iVR experiences may be incredibly useful for the expansion and evaluation of serious games. To further this direction of research, we present a playable iVR experience in which the user evaluates the emotion of images through an immersive Self-Assessment Manikin (iSAM). This game explores a pilot system for enabling efficient online fine-tuning of a user's Pleasure-Arousal-Dominance (PAD) emotional model using personalized deep-learning. We discuss adapting the International Affective Picture system (IAPs), in which our Artificial Intelligence (AI) model responds with a personalized image after learning from ten user supplied answers during an iVR session. Lastly, we evaluated our iVR experience with an initial pilot study of four users. Our preliminary results suggest that iSAM can successfully learn from user affect to better predict a 'happy' personalized image than the static base model.
Article
Full-text available
Introduction Soft robotic wearable devices, referred to as exosuits, can be a valid alternative to rigid exoskeletons when it comes to daily upper limb support. Indeed, their inherent flexibility improves comfort, usability, and portability while not constraining the user’s natural degrees of freedom. This review is meant to guide the reader in understanding the current approaches across all design and production steps that might be exploited when developing an upper limb robotic exosuit. Methods The literature research regarding such devices was conducted in PubMed, Scopus, and Web of Science. The investigated features are the intended scenario, type of actuation, supported degrees of freedom, low-level control, high-level control with a focus on intention detection, technology readiness level, and type of experiments conducted to evaluate the device. Results A total of 105 articles were collected, describing 69 different devices. Devices were grouped according to their actuation type. More than 80% of devices are meant either for rehabilitation, assistance, or both. The most exploited actuation types are pneumatic (52%) and DC motors with cable transmission (29%). Most devices actuate 1 (56%) or 2 (28%) degrees of freedom, and the most targeted joints are the elbow and the shoulder. Intention detection strategies are implemented in 33% of the suits and include the use of switches and buttons, IMUs, stretch and bending sensors, EMG and EEG measurements. Most devices (75%) score a technology readiness level of 4 or 5. Conclusion Although few devices can be considered ready to reach the market, exosuits show very high potential for the assistance of daily activities. Clinical trials exploiting shared evaluation metrics are needed to assess the effectiveness of upper limb exosuits on target users.
Article
Full-text available
Modern immersive virtual reality experiences have the unique potential to motivate patients undergoing physical therapy for performing intensive repetitive task-based treatment and can be utilized to collect real-time user data to track adherence and compliance rates. This article reports the design and evaluation of an immersive virtual reality game using the HTC Vive for upper limb rehabilitation, titled “Project Star Catcher” (PSC), aimed at users with hemiparesis. The game mechanics were adapted from modified Constraint Induced Therapy (mCIT), an established therapy method where users are asked to use the weaker arm by physically binding the stronger arm. Our adaptation changes the physical to psychological binding by providing various types of immersive stimulation to influence the use of the weaker arm. PSC was evaluated by users with combined developmental and physical impairments as well as stroke survivors. The results suggest that we were successful in providing a motivating experience for performing mCIT as well as a cost-effective solution for real-time data capture during therapy. We conclude the article with a set of considerations for immersive virtual reality therapy game design.
Conference Paper
Full-text available
Traditional forms of physical therapy and rehabilitation are often based on therapist observation and judgment, coincidentally this process oftentimes can be inaccurate, expensive, and non-timely. Modern immersive Virtual Reality systems provide a unique opportunity to make the therapy process smarter. In this paper, we present an immersive virtual reality stroke rehabilitation game based on a widely accepted therapy method, Constraint-Induced Therapy, that was evaluated by nine poststroke participants. We implement our game as a dynamically adapting system that can account for the user’s motor abilities while recording real-time motion capture and behavioral data. The game also can be used for tele-rehabilitation, effectively allowing therapists to connect with the participant remotely while also having access to +90Hz real-time biofeedback data. Our quantitative and qualitative results suggest that our system is useful in increasing affordability, accuracy, and accessibility of post-stroke motor treatment.
Article
Full-text available
The advent of inexpensive consumer virtual reality equipment enables many more researchers to study perception with naturally moving observers. One such system, the HTC Vive, offers a large field-of-view, high-resolution head mounted display together with a room-scale tracking system for less than a thousand U.S. dollars. If the position and orientation tracking of this system is of sufficient accuracy and precision, it could be suitable for much research that is currently done with far more expensive systems. Here we present a quantitative test of the HTC Vive’s position and orientation tracking as well as its end-to-end system latency. We report that while the precision of the Vive’s tracking measurements is high and its system latency (22 ms) is low, its position and orientation measurements are provided in a coordinate system that is tilted with respect to the physical ground plane. Because large changes in offset were found whenever tracking was briefly lost, it cannot be corrected for with a one-time calibration procedure. We conclude that the varying offset between the virtual and the physical tracking space makes the HTC Vive at present unsuitable for scientific experiments that require accurate visual stimulation of self-motion through a virtual world. It may however be suited for other experiments that do not have this requirement.
Article
Full-text available
Borderline personality disorder (BPD) is a severe mental disorder characterized by a dysfunctional pattern of affective instability, impulsivity, and disturbed interpersonal relationships. Dialectical Behavior Therapy (DBT®) is the most effective treatment for Borderline Personality Disorder, but demand for DBT® far exceeds existing clinical resources. Most patients with BPD never receive DBT®. Incorporating computer technology into the DBT® could help increase dissemination. Immersive Virtual Reality technology (VR) is becoming widely available to mainstream consumers. This case study explored the feasibility/clinical potential of using immersive virtual reality technology to enhance DBT® mindfulness skills training of a 32 year old female diagnosed with BPD. Prior to using VR, the patient experienced difficulty practicing DBT® mindfulness due to her emotional reactivity, and difficulty concentrating. To help the patient focus her attention, and to facilitate DBT® mindfulness skills learning, the patient looked into virtual reality goggles, and had the illusion of slowly “floating down” a 3D computer-generated river while listening to DBT® mindfulness training audios. Urges to commit suicide, urges to self harm, urges to quit therapy, urges to use substances, and negative emotions were all reduced after each VR mindfulness session and VR mindfulness was well accepted/liked by the patient. Although case studies are scientifically inconclusive by nature, results from this feasibility study were encouraging. Future controlled studies are needed to quantify whether VR-enhanced mindfulness training has long term benefits e.g., increasing patient acceptance and/or improving therapeutic outcome. Computerizing some of the DBT® skills treatment modules would reduce cost and increase dissemination.
Article
For stroke survivors and many other people with upper-extremity impairment, daily life can be difficult without properly functioning arms. Some modern physical therapy exercises focus on rehabilitating people with these troubles by correcting patients’ perceptions of their own body to eventually regain complete control and strength over their arms again. Augmentative wearable robots, such as upper-extremity exoskeletons and exosuits, may be able to assist this endeavor. A common drawback in many of these exoskeletons however is their inability to conform to the natural flexibility of the human body without a rigid base. We have built one such exosuit to address this challenge: CRUX (Compliant Robotic Upper-extremity eXosuit). This robot is a compliant, lightweight, multi-DoF, portable exosuit that affords its wearer the ability to augment themselves in many unconventional settings (i.e. outside of a clinic). These attributes are largely achieved by using a modified tensegrity design situated according to measured lines of minimal-extension where a network of tension members provide a foundation to apply augmentative forces via precisely placed power-lines. In this paper, we detail the design process of CRUX, report on CRUX’s prototypical composition, and describe the mimetic control algorithm used.We also discuss the results of three studies that illustrate the efficacy of CRUX’s mimetic controller, CRUX’s flexibility and compliance, and the metabolic cost reduction when users exercise with assistance from CRUX as opposed to without. We conclude the paper with a summary of our findings, potential use cases for this technology, and the direction of future related work.
Conference Paper
Wearable robots can potentially offer their users enhanced stability and strength. These augmentations are ideally designed to actuate harmoniously with the user's movements and provide extra force as needed. The creation of such robots, however, is particularly challenging due to the underlying complexity of the human body. In this paper, we present a compliant, robotic exosuit for upper extremities called CRUX. This exosuit, inspired by tensegrity models of the human arm, features a lightweight (1.3 kg), flexible multi-joint design for portable augmentation. We also illustrate how CRUX maintains the full range of motion of the upper-extremities for its users while providing multi-DoF strength amplification to the major muscles of the arm, as evident by tracking the heart rate of an individual exercising said arm. Exosuits such as CRUX may be useful in physical therapy and in extreme environments where users are expected to exert their bodies to the fullest extent.
Conference Paper
Postural stability is an important measure for many medical diseases such as Parkinson. In the last years, research focused on using inexpensive and portable devices to measure postural stability, while the visual targets were physical objects in the environment. Sensing balancing boards were used to measure stance forces, while movements of the upper body were not taken into account. Within this paper, postural stability was measured using the HTC Vive. A variation of a virtual fixation point’s distance was analyzed and compared to a reference condition with closed eyes. It is shown that body sway in the VR conditions is increased in the anterior-posterior and decreased in the medial-lateral direction.
Article
This paper presents the design and the clinical validation of an upper-limb force-feedback exoskeleton, the L-EXOS, for robotic-assisted rehabilitation in virtual reality (VR). The L-EXOS is a five degrees of freedom exoskeleton with a wearable structure and anthropomorphic workspace that can cover the full range of motion of human arm. A specific VR application focused on the reaching task was developed and evaluated on a group of eight post-stroke patients, to assess the efficacy of the system for the rehabilitation of upper limb. The evaluation showed a significant reduction of the performance error in the reaching task (paired t -test, p < 0.02)