Conference PaperPDF Available

Project Butterfly: Synergizing Immersive Virtual Reality with Actuated Soft Exosuit for Upper-Extremity Rehabilitation


Abstract and Figures

Immersive Virtual Reality paired with soft robotics may be synergized to create personalized assistive therapy experiences. Virtual worlds hold power to stimulate the user with newly instigated low cost, high-performance commercial Virtual Reality (VR) devices to enable engaging and accurate physical therapy. Soft robotic wearables are a versatile tool in such stimulation. This preliminary study investigates a novel rehabilitative VR experience, Project Butterfly (PBF), that synergizes VR Mirror Visual Feedback Therapy with soft robotic exoskeletal support. Nine users of ranging ability explore an immersive gamified physio-therapy experience by following and protecting a virtual butterfly, completed with an actuated robotic wearable that motivates and assists the user to perform rehabilitative physical movement. Specifically, the goals of this study are to evaluate the feasibility, ease-of-use, and comfort of the proposed system. The study concludes with a set of design considerations for future immersive physio-rehab robotic-assisted games.
Content may be subject to copyright.
Project Butterfly: Synergizing Immersive Virtual Reality with Actuated
Soft Exosuit for Upper-Extremity Rehabilitation
Aviv Elor, Steven Lessard, Mircea Teodorescu, and Sri Kurniawan
University of California, Santa Cruz, Jack Baskin School of Engineering
{aelor, slessard, mteodore, skurnia}
Immersive Virtual Reality paired with soft robotics may be syner-
gized to create personalized assistive therapy experiences. Virtual
worlds hold power to stimulate the user with newly instigated low-
cost, high-performance commercial Virtual Reality (VR) devices to
enable engaging and accurate physical therapy. Soft robotic wear-
ables are a versatile tool in such stimulation. This preliminary study
investigates a novel rehabilitative VR experience, Project Butterfly
(PBF), that synergizes VR Mirror Visual Feedback Therapy with
soft robotic exoskeletal support. Nine users of ranging ability ex-
plore an immersive gamified physio-therapy experience by follow-
ing and protecting a virtual butterfly, completed with an actuated
robotic wearable that motivates and assists the user to perform re-
habilitative physical movement. Specifically, the goals of this study
are to evaluate the feasibility, ease-of-use, and comfort of the pro-
posed system. The study concludes with a set of design considera-
tions for future immersive physio-rehab robotic-assisted games.
Index Terms: Virtual Reality, Soft Robotics, Wearable Robotics,
Exosuit, Physio-Immersive Rehabilitation, Physical Therapy, Im-
mersive Experiences, Serious Games, Human Computer Interaction
According to the latest US Census in 2010, there are more than 40
million older adults (defined as people aged 65 years old or older)
living in the US, comprising 13 percent of the US population. This
demographic represents a 15 percent growth compared to the 2000
US Census data [19] and is projected to continue to grow. Unfortu-
nately, studies have shown that motor functions decline with aging
[45]. The significant older population experiences an increasingly
prevalent issue of motor degeneration. Age-related motor perfor-
mance deficits include coordination difficulty, decreased variabil-
ity of motor ability, slowing of movement, and problems with bal-
ance and gait [37]. Movement slows with aging by as much as
15 to 30 percent. Research by Seidler-Dobrin et al. suggests that
older adults emphasize movement accuracy at the cost of movement
speed [38]. As a result, older adults show specific deficits in the
coordination of bimanual and multi-joint movements. For exam-
ple, movements become slower and less smooth when older adults
use their shoulder and elbow joints simultaneously as opposed to
performing single-joint actions[36]. Often postural stability is also
compromised with advancing age [40].
In addition to the decline in motor function, aging correlates to
the progressive loss of skeletal muscle mass and strength. Fre-
The authors are with the Departments of Computational Media & Elec-
trical Computer Engineering, University of California Santa Cruz, Jack
Baskin School of Engineering, 1156 High St, Santa Cruz, CA, USA. Ad-
ditionally, this work was supported by the Center for Information Technol-
ogy Research in the Interest of Society (CITRIS) and the National Science
Foundation (NSF).
quent exercise represents an effective therapeutic strategy to aug-
ment skeletal muscle mass and improve functional performance and
quality of life in older adults [22]. Many technological solutions
have been researched and developed over the past decade to reduce
motor loss, but there is still much to be done.
One modern approach to address muscular impairment is virtual-
reality (VR) therapy. Through the use of VR, stimulating immersive
environments can be programmed to increase therapy compliance,
accessibility, and data throughput [8, 9]. Psychological and physio-
logical research has featured increasing use of VR in the prior two
decades thanks to the ability to simulate realistic and complex situa-
tions critical to laboratory-based human behavior investigations [7].
Traditional forms of therapy and rehabilitation usually derive from
therapist observation and judgment. Drawbacks of this traditional
method are that they are often inaccurate, expensive, and timely
[30]. Virtual reality, however, addresses these concerns as a useful
tool for improving outcomes compared to conventional therapy by
enabling accurate motion capture, telepresence based sessions, and
low cost motivating experiences [26, 8]. The immersive visual ca-
pabilities of modern VR headsets, such as the HTC Vive and Ocu-
lus Rift, have had astounding promise and success with treatments
ranging from exposure in Post Traumatic Stress Disorder [34, 28],
Borderline Personality Disorder [31], various phobias [39, 14, 7],
schizophrenia [35], and many other psychological therapies. Re-
searchers are even reporting that integration VR into the clinical
setting can reduce pain similar to the effect of analgesic treatments
[15, 17].
Success in VR therapy often relates to the relationship between
presence and emotion with technology’s ability to bridge them [28].
Increasing the quantity and quality of stimuli in immersive VR is
key to influencing user behavior and experience [4, 28]. The past
five years have made strides in VR technology – VR is ever more
immersive, affordable, and accessible to the average consumer with
over 200 million immersive VR Head Mounted Displays projected
to be sold by the year 2020 [1, 10]. For these reasons, headset-
based VR systems like the HTC Vive and Oculus Rift could appeal
to low-income communities. VR as a therapeutic tool has, there-
fore, become the most effective and affordable it has ever been and
is projected by many researchers to continue along this forecasted
Many of these studies incorporate Mirror Visual Feedback Ther-
apy (MVFT), the visual or physical stimulation of a “pseudo“
movement on the damaged limb to promote recovery [6]. Patients
are given sensorimotor feedback by reflecting an abled arm in the
position of the impaired arm during exercise [27]. MVFT is sug-
gested to be a beneficial treatment for motor rehabilitation [33, 27],
where clinical studies have indicated that MVFT “can serve as a
versatile tool to promote motor recovery” in mobility and arm use
MVFT requires the superposition of a simulated arm on a phan-
tom limb which enables patients to relieve painful sensation and
increase movement [33]. A variety of other conditions were ex-
plored with MVFT, including with stroke survivors, where a simu-
lated limb is placed in the patients midsagittal plane, thus reflecting
movements of the nonparetic side as if it were the affected side to
stimulate brain plasticity [42]. However, users with severe motor
loss require increased physical assistance to perform MVFT, and
can often require therapist intervention, or in the case of this study:
a robotic wearable.
A large amount of physical therapy research has transpired on
the integration of rehabilitative robotic wearable devices. Several
upper-body robotic exoskeletons have been developed and explored
over the last ten years with many incorporating VR. Some of these
examples include the PERCRO (Perceptual Robotics laboratory) L-
EXOS system [11], Rutgers CyberGlove and Master II-ND (RMII)
force feedback glove [21], and Therapy Wilmington Robotic Ex-
oskeleton (T-WREX) [18]. Through combining VR with exoskele-
tons that provide arm gravity support, clinical testing showed a
range in improvement of mobility, strength, and satisfaction [5].
One attribute common to these exoskeletons is the use of rigid
structures. A significant flaw experienced by traditional rigid ex-
oskeletons is their inflexibility, and the burden users bear when
wearing them. Devices which have few degrees-of-freedom (DoF)
or heavy components inhibit some movements. This physical con-
straint can lead to imbalanced muscular growth and control, which
can injure users of these wearable robots [44]. As a result, softer
devices such as Lessard et al. exosuits have emerged as a flexible al-
ternative to traditional rigid exoskeletons [24, 25]. This study aims
to leverage such soft exosuits during VR therapy through the use of
Compliant Robotic Upper-Extremity Exosuit (CRUX), as shown in
Figure 1, to explore feasibility, ease-of-use, and comfort.
Figure 1: A demonstrator wears CRUX [25] (without IMUs). CRUX
is an augmentative wearable soft robot for upper-extremity rehabilita-
tion and can be combined with VR through Project Butterfly to enable
immersive rehabilitation.
The VR experience developed, Project Butterfly (PBF), is a game
that motivates users to perform upper body motion primitives by
having them protect and control a virtual butterfly in a meadow
while the system collects real-time data using the HTC Vive. Fig-
ure 3 displays an example of PBF gameplay. CRUX was integrated
accordingly to provide additional tactile feedback and physical as-
3.1 The Soft Robotic Wearable
The purpose of using the CRUX exosuit was twofold: to physically
stimulate movement to achieve an ideal position, and to immerse
the user deeply into the VR environment. Since immersion is a
crucial factor in the influence of user behavior and compliance [4],
PBF paired with CRUX may significantly improve the user’s reha-
bilitation experience.
The designed exosuit, shown in Figure 1, is capable of lifting
the user’s arm in different directions to create smooth multi-jointed
movements. The concept of tensegrity for soft robotics inspired
the mechanics of CRUX. Tensegrity (a portmanteau of “tensile”
and “integrity”) defines structures as internally prestressed, free-
standing, pin-jointed networks in which the cables or tendons are
held in tension against a system of bars or struts [16].
A base layer of neoprene held CRUX together. Cables are routed
along the neoprene to integrate a network of “anchor points,” which
serve as the rigid components in an otherwise flexible system. The
exact placement of the cables was determined by recording arm
movement on people as they stretched out and expressed their full
range of motion [25]. Through examining extensive motion capture
of the arm, the area on the skin which sheared the least was deter-
mined to find the most stable places to plant anchor points [25].
Bicycle housing routes the cables onto anchor points to actuate
different parts of the arm just as how tendons pull limbs. Six mi-
cro DC motors were mounted on a modular backplate and connect
to the cables of the exosuit via 3d printed spool, and are manipu-
lated through a microcontroller powering the system with a 3-Cell
Lithium polymer battery. Each motor is capable of exerting 88 N
of force (125 oz-in). Figure 1 depicts CRUX being operated by
a demonstrator. The selected material of CRUX affords a compli-
ant and lightweight design. Like similar soft exosuits, CRUX is
lightweight. However, CRUX weighs 1.5 kg [24] compared to the
6.8 kg lower-limb gait-assisting exosuit developed by Wehner et al.
[46] or the 2.27 kg suit by Alvara et al. [3] for upper arm force
The suit’s controller was designed to have the weak arm follow
the movement of the healthy limb. This mirroring of limbs insti-
gates mimetic controller design. Mirroring the movement from one
side to the other side of the body was inferred from MVFT to in-
crease motor recovery and stimulate brain plasticity [42]. Figure 2
depicts the CRUX being fitted to a user by an evaluator for upper
arm force amplification.
To enable the mimetic control of the healthy arm onto the weak
limb, wireless connectivity capability was added to connect with
the Inertial Measurement Unit (IMU) networks. An IMU is an elec-
tronic device that measures and reports a body’s specific force and
angular rate using a combination of accelerometers, magnetome-
ters, and gyroscopes [2]. The IMU network added to the exosuit
consists of 4 IMU nodes where each node can measure 3-axis ori-
entation of itself and then send this data back to the microcontroller.
The IMU nodes on CRUX are enclosed in a 3D printed case with
adjustable Velcro straps to accommodate various body sizes.
A plunger button must be engaged by the user to allow for exo-
suit movement. This safeguard prevents accidental actuation when
the exosuit is in master-slave mode. If the user feels that the motor
is performing movements that are undesired, the user can release
the plunger button, effectively disengaging the motor.
In complementary locations, nodes are positioned on both arms
at the lateral forearm (midway between the wrist and the elbow) and
the lower medial triceps (slightly above the elbow) [25] as seen in
Figure 2. Each node transmits pose data to the central controller to
support the closed-loop function enabling the pose following from
the healthy arm onto the impaired arm. It should be noted that while
the VR device can perform motion capture in place of the IMUs,
creating this dependency would limit the flexibility of CRUX for
future use. For future example, the suit may be used beyond VR
MVFT to assist with active daily living activities, where the con-
trols and level of assistance are calibrated during the VR therapy
Figure 2: A participant exploring CRUX [25]. Control is achieved
by using IMU Nodes and an internal controller for leader-follower
mimicry. A user can control their impaired arm using their healthy
arm to match the movement path.
Figure 3: A user playing Project Butterfly. a) is the study proctor’s
on-screen view b) is the in-person view of the study proctor and c) is
the in-game view from the user’s perspective.
3.2 The Immersive Virtual Reality Experience
A motion primitive is defined as a distinct movement achievable
by a single joint which creates a unique degree of freedom (DoF).
Thus, upper body motion primitives can be thought of as indivisible
building blocks that can be combined and permuted into a broader
range of potential movements. The HTC Vive, one of the highest
grossing VR Entertainment Systems [1] developed by the Valve and
HTC Corporation, can be used as a powerful tool to both track these
motion paths and motivate the user to achieve these motions. The
Vive is a VR Head-Mounted Display that implements room scale
4x4 m outside-in tracking technology by utilizing a “lighthouse”
system of lasers which enable the user to interact with the virtual
environment through accurate motion capture in a 3D virtual space
[20]. Complementary to the HTC Vive are two handheld controllers
that feature dynamic haptic feedback which enhances spatial orien-
tation [20]. The HMD provides a 110-degree field of view and 90
Hz refresh rate [41]. Motion capture is tracked at 120 Hz using in-
frared laser sweeping and photo-diodes that enable for recovery of
position and orientation [20]. The HTC Vive also features a safety
guidance system preventing users from potential injury in the real
world environment [41, 32]. The worst case tracking jitter of the
system has been reported to be under 2.1mm with an accuracy of
an absolute 2mm error [23]. Resultingly, the HTC Vive allows for
the ability to extract accurate gameplay data while providing an en-
veloping experience of touch, sound, and sight.
Paired with an HTC Vive controller, the exosuit assists the user
during VR gameplay. Testing of the system targeted two pairs
of motion primitives: elbow extension/flexion, and shoulder ab-
duction/adduction. Biceps received assistance by replacing the
user’s CRUX supported HTC Vive controller as a bubble shield and
having them protect the butterfly from incoming rain and projec-
tiles through a therapist-specified customized range of motion path.
Haptic feedback is enabled so that the user is indicated with strong
pulses whenever the motion primitive was not followed (failure to
encapsulate the butterfly inside the bubble). To increase the incen-
tive and track compliance, the user receives a scoring point per ev-
ery half second that they mirrored the required motion primitive.
This multi-sensory feedback guided users according to the objec-
tive of the game.
To generate the environment and the mechanics of PBF, the
Unity v2017.1.0b4 Game Engine along with the SteamVR Unity
plugin v1.2.0 became the chosen development tools. Both Unity
and SteamVR hold a large amount of open-access documenta-
tion, including flexibility with programming languages such as
Javascript and C# [43]. Using Unity’s built-in physics engine, the
Rigidbody class was used to model the butterfly along with spher-
ical colliders that detected contact with the butterfly. Assignment
of the moving projectiles and the butterfly with the rain were set
to a time-dependent spatial state, allowing for global physics-based
events to influence data capture. Runtime data collection was cap-
tured using C# and Microsoft .NET Framework at speeds of 90Hz
and higher.
To assist the user evaluation process, PBF includes a dynamic
evaluator GUI, seen in Figure 4, which automatically prompts the
evaluator or therapist to measure the length of each participant’s
arms through the motion capture (measuring the x-z plane maximal
distance between the VR HMD and the Vive Controller) or man-
ually entered ranges. This calibration stage is achieved entirely in
Unity during run-time with the HTC Vive so that user’s who may
not have access to CRUX in the future can still play PBF without
physical assistance. Figure 4 also features the option to change the
repetitions per minute of each motion primitive, and data exporta-
tion rate of each game session. In short, the evaluator GUI allowed
the evaluator to tailor the game to each unique individual and cus-
tomize data throughput.
The game-themed goals became protecting a butterfly from
heavy rain and projectiles using a bubble (as the avatar for the con-
troller of the weak arm), which focused on bicep curl and lateral
arm raise exercises. Users are instructed place to the bubble around
the butterfly to achieve a high score, and to “protect the butterfly”.
These mechanics required the users to smoothly follow the flight
of the butterfly within +/- 0.1m of the required motion path. As a
result, the biceps and lateral arm raise minigames became scripted
movements of high accuracy and scripted timing repetitions when
a user performed a motion primitive.
Lastly, a tutorial started before every game that the evaluator
could enable to adjust the user to the VR environment. In the tu-
torial, users were asked to identify three images placed on their
left, on their right, and behind them. Identification in the 360 VR
environment was implemented to familiarize the user with virtual
reality and show them that they can look in any direction. Addi-
tionally, a “How to Play” menu was added to teach evaluators how
to test users, and in-game instructions were added to clarify the goal
of each minigame. This pre-gameplay stage simultaneously intro-
duce users not only to virtual reality mechanics (such as 360-degree
views) but also to PBF specific mechanics like the arm movements
required to earn a higher score. These designs were intended to
tailor the game to each unique individual smoothly and intuitively,
all while increasing the incentives to perform the objective of the
Figure 4: Dynamic Project Butterfly Evaluator Interface. This UI and
the data it gathered allowed for more balanced games for each sub-
sequent participant.
The study investigated the usability of PBF synergized with CRUX
system design, which includes ease of use, comfort, and set base-
line feasibility with nine elderly users having motor dysfunction.
Users were observed playing the virtual reality game with the ex-
osuit as they performed tasks that identified the ability of targeted
muscles and muscle groups. The nine elderly participants in usabil-
ity evaluations represented three patient segments:
1. Three retirees from Elderday Adult Day Health Care Center,
Santa Cruz, CA, represented a mental disability use case be-
cause they were affected by memory loss or dementia. Evalu-
ations with these users were conducted at Elderday.
2. Three stroke survivors recruited from Cabrillo College’s
Stroke and Disability Learning Center (SDLC), Capitola, CA,
represented a physical disability use case because they were
affected by neglect syndrome, meaning the motor functions of
one side of their body was impaired by their stroke. Evalua-
tions with these users were conducted at SDLC.
3. Three retirees living independently within their community,
outside of daycare or hospice, which do not have a significant
physical or mental disability. Evaluations with these users
were conducted at the University of California, Santa Cruz.
All participants ranged between the ages of 60 to 80 years old
and were previously unfamiliar with both virtual reality and wear-
able robotics. These user groups represent three demographics:
physical disability, mental disability, and no disability-were se-
lected because the physical and mental disabilities are likely the
target demographics for PBF and CRUX. This preliminary study
serves as feasibility to justify further studies with larger user group
sizes and gain design insights.
For each user, the evaluation began by an proctor giving a de-
tailed explanation of what PBF and CRUX are, answering questions
as needed. Then the user was given a tutorial period where they
walked through the “How to Play” menu and played the game with-
out being recorded. The tutorial consisted of playing two rounds of
the biceps minigame and the lateral arm minigame for one minute
each. The first round with the minigames was played without wear-
ing the CRUX exosuit and the second was played while wearing it
to allow for MVFT. This allowed the user to try out the controller
and learn PBF’s basic game mechanics. The point score from the
tutorial period helped the evaluator calibrate each user (adjusting
arm length and speed).
After the tutorial period, the test commenced with the user play-
ing recorded sessions while wearing the CRUX exosuit and playing
1-minute minigames, shown in Figure 6. Sessions were recorded
using a webcam, collecting gameplay scores and positional data,
and conducting post-session interviews which allowed for the col-
lection of quantitative and qualitative data. User evaluation inter-
views were executed by asking users questions pertained to their
experience from a prepared form with questions such as:
Open-ended questions:
What day-to-day tasks do you struggle with?
How many years have you been doing physical rehabil-
itation therapy?
Do you enjoy video games?
How can the virtual reality game be improved?
How can the exosuit be improved?
Rate the statement on a 5-point Likert scale:
Q1: My current therapy is engaging.
Q2: The virtual reality game was enjoyable.
Q3: I became fatigued while playing the virtual reality
– Q4: The virtual reality game distracted me from pain
when doing physical movement.
Q5: If I had access to virtual reality therapy games, I
would use it in the future.
After evaluations, proctors reviewed gameplay and interview data
and commented additional thoughts or observations. Note that re-
sulting answers from Q1-Q5 can be seen in Figure 5.
Figure 5: Nine user responses of 5-point Likert Scale Questions per-
taining to PBF
Figure 6: A user performs lateral arm raises during a Project Butterfly
game session with CRUX. To boost their scores, users had to mimic
the flight path of the butterfly, which in this case was an up-and-down
motion most easily copied by raising one’s arm up-and-down simi-
Figure 7: Position of the CRUX assisted controller during the lateral
arm raise mini-game. These graphs depict four users attempting to
mimic the movement of their targets (the butterfly avatar). Red is
the butterfly, green is users movements. Red arcs which are closely
matched by green trajectories (as shown in these graphs) mean that
users are successfully completing in-game objectives, which are tai-
lored to test their physical limits when wearing the exosuit.
The three primary objectives of the study were to evaluate the base-
line feasibility of the system, ease-of-use of the system, and comfort
of the system. These objectives are judged according to quantita-
tive metrics obtained from recorded metadata of users while play-
ing PBF. Qualitative data was acquired during post-gameplay inter-
views using mixed 5-point Likert Scale questions and open-ended
questions. The Likert scale questions are summarized in Figure 5.
5.1 System Feasibility
Most participants felt that CRUX augmented their upper limb
movements to some degree. Specifically, one participant (see User
1 in Figure 7) mentioned that their arm movements became more
“effortless” with the use of CRUX. Specifically in raising their
arm laterally, which had been challenging for them before donning
CRUX. They further elaborated that it was the first time in weeks
that they did not experience painful throbs when raising an arm
above their shoulders.
To observe an example of such movement, the position of the
CRUX assisted controller and controller target, the butterfly avatar,
were graphed against four scored users from SDLC (Users 2 and 4)
and Retirees (Users 1 and 3) for the lateral shoulder raise minigame
as seen in Figure 7. All four CRUX supported users were able to
achieve a score of 111/130 or higher, indicating that these users
were compliant to the motion primitive path for over 85% of game-
play. Additionally, the speed of the users controllers, headset, and
butterfly avatar (target to follow) is graphed as shown in Figure 8
as well as the acceleration of the user’s weak arm controller shown
in Figure 9. Each of these users demonstrated significantly differ-
ent gameplay movements even though they had close scores. When
considering Figure 7, User 1 had sharp changes, User 2 had shaky
movements, User 3 had smooth moves, and User 4 had smooth and
shaky movements. When looking at movement speed in Figure 8,
Users 1 and 3 maintained a constant speed, indicating a greater
control of their weaker arm. Whereas Users 2 and 4 had signifi-
cant spikes in movement speed indicating a lack of control in their
arms. It should be noted that while the exosuit assists in achiev-
ing position, it does not reduce the shaking of the limbs. Figure 9
reflects the findings in Figure 8 through acceleration, where User
1 maintains the most control nearing almost no acceleration, fol-
lowed by User 2 who spikes but nears zero, and Users 3 and 4 ex-
perienced occasional large shaking. Despite the difference in fine-
motor strength and precision amongst participants, each user was
able to achieve the desired goal of protecting the virtual butterfly.
This might suggest that PBF can be accommodating for people at
various stages of their physical therapy, thus making it more acces-
sible. Also, while User 1 and User 4 have differently lengthened
arms by about 0.2 meters, the compliance of over 85% can be seen
visible as their green arm paths overlay the object red path arch in
Figure 7. Although these results are promising, the sample size is
not statistically significant, which warrants further study as noted
in the Limitations Section 6.5.
On the subject of assistance, most participants felt that CRUX af-
fected augmenting their limbs. However, they all asked for stronger
motors on the exosuit. When asked for a potential reason to this,
most users indicated that they felt that the motors were not power-
ful enough to make the difference that they were expecting. One
user (see User 3 in Figure 7) commented that they understood the
function of the device but “[wanted] even more power behind it.
Additionally, none of the participants felt that the exosuit made it
more difficult for them to move.
5.2 Ease of Use
An interesting observation was that users said they “knew the goal
was always to protect the butterfly,” which was a gameplay theme
Figure 8: The speed of the user’s handheld controller and headset as
they attempt to catch the virtual butterfly avatar during the lateral arm
raise mini-game. (Green is the speed of the Weak Arm Controller,
Red is the speed of the Butterfly, Blue is the speed of the headset)
Figure 9: The acceleration of the user’s handheld controller as they
attempt to catch the virtual butterfly avatar during the lateral arm raise
mini-game. Users with high scores and large fluctuations in acceler-
ation are able to react quickly to in-game obstacles, suggesting a
higher level of control and strength than those who cannot. (Green is
the acceleration of the Weak Arm Controller)
added when brainstorming ideas with potential users. Creating in-
game goals which center around an archetypal emotional response
may have generated a quicker understanding of gameplay mechan-
ics in the evaluated users. When asked what they thought of the
difficulty, almost all participants thought it was appropriately chal-
lenging. This makes sense as the game‘s difficulty was adjusted
based on their first gameplay before and during the tutorial period.
By automating all control and dynamic elements of both CRUX
and PBF, there was an increase in compliance when performing mo-
tion primitives as well as reducing the contention between the suit
and virtual reality. Users tended to agree that they noticed the exo-
suit was assisting them smoothly as if part of the gameplay, due to
the mimetic control, suggesting that automating CRUX user-input
may have benefited players immersion.
5.3 Comfort
Users asked for greater ventilation in CRUX, due to the form-
fitting style of the neoprene. Additionally, since CRUX was proto-
typed using commercially available wetsuit neoprene, not all users
equally fit the base layer. The majority of the participants still re-
sponded that they enjoyed playing the minigames with CRUX as
seen in Figure 5.
Through observation and commentary, no one seemed dis-
pleased with the aesthetics of the minigames. Some users com-
plimented the aesthetics, citing the goal of butterfly protection as
“fun,” “engaging,” and even “meditative. This is particularly excit-
ing news since many people unfamiliar with video games are often
intimidated or dissuaded from immersive virtual environments, es-
pecially considering that five out of nine participants responded that
they dislike video games. The user who mentioned that the games
were “engaging” initially discussed their reservations towards video
games and how they preferred “real things. When users are psy-
chologically comfortable with a system, they are more likely to ben-
efit from it.
From these observations and recordings, we have found that
most users believe PBF was useful in helping them move their
arms as seen in Figure 5. Combining CRUX’s master-slave sys-
tem and PBF’s butterfly protection mechanics actively encouraged
players to perform the required movements while being distracted
from their physical therapy as agreed by six out of nine users (and
one additional neutral response). These users found PBF enjoyable
and agreed that the game did distract themselves from the exertion
of physical moving. Seven out of nine users stated that they would
regularly use PBF & CRUX in the future for exercises if it were
available to them.
5.4 Discussion
Both the quantitative and qualitative data generated suggested that
in the short term, the system can help users achieve in the moment
tasks by augmenting users upper body strength to enable them to
move their upper limb more easily. In the long term, the paired-
system serves as a boost to help them train their weaker upper arm
and make it stronger over time. In the case of stroke survivors from
SDLC who suffer from neglect syndrome, it additionally helped
them in recognizing what independently driven movement on the
neglected side feels like again. Qualitative lessons learning suggest
the following:
The exosuit must afford independence on behalf of the user.
In the gameplay sessions, the experimenters helped users to
don and off the suit. If this suit is indeed to be used as a
home-based exercise system, users must be able to put on and
take off the suit by themselves, which at the moment is still
Using a neoprene top as the base to CRUX limited those who
could wear it and made it hot when worn over an extended pe-
riod and difficult to adhere. In future iterations, exosuits like
CRUX needs to be more easily customizable, for example by
replacing neoprene in the exosuit with elastane (e.g., Lycra).
Future iterations of robotic support should make the augmen-
tation slightly more powerful, so the aid from CRUX is more
apparent to users.
Given that the PBF and CRUX are new to players, they must
be acquainted with the technologies individually. When given
even just a few minutes to become acquainted, players made
much better use of the CRUX when playing PBF and were
less overwhelmed.
Through the study, a set of design guidelines was compiled for
other practitioners of wearables and VR games to augment upper
limb movements and motivate exercises, especially in older adults
and people with motor impairment:
Exosuit needs to be designed to be more flexible to fit a variety
of body sizes and shapes.
Exosuit augmentation needs to be noticeable, perhaps by
adding an in-game UI element such as the bubble lighting up
when the suit is activated. This will hopefully keep the user
immersed while also providing exosuit control indication to
the user.
Reduce various forms of stimulation to the minimum with-
out affecting the exercise goals without breaking immersion.
Feedback and color variations went unnoticed by some of the
users as they were not related directly to their goals.
Stay within the appropriate difficulty of the users in terms of
game speed, ranges of motion, etc.
The user feedback suggests that the users enjoyed protect-
ing the butterfly, as it caused an “emotional attachment.”
Emotion-driven immersion suggests a powerful tool in cre-
ating engaging experiences.
5.5 Limitations
One user responded “Disagree” for questions Q2 (Enjoyable
Game), Q4 (Distracted from Pain) and Q5 (Future Use) from Figure
5. The user commented that the novelty of VR was fun, but con-
tinuously protecting a butterfly would become boring if they used
it for regular therapy. The user requested that future games should
have a compelling storyline to keep them interested and motivated.
The intent of PBF was preliminary feasibility, and future games
should be developed and studied based on lessons learned, trying
out game mechanics, and the targeted therapy desired. Concerns
that this user poses should be investigated in subsequent games to
determine if specific game mechanics are more preferred by users
than others.
Only three motion primitives were explored and converted into
VR through PBF. Specifically, they were the Lateral Shoulder
Raise, Forward Arm Raise, and Horizontal Shoulder Rotation. Fu-
ture studies should incorporate a more variety of motion primitives
to catalyze potential benefits to the users’ potential mobility im-
provements. There may also be potential in incorporating com-
mon assessment tools into the VR environment for automated as-
sessment such as the Apley’s scratch test (Shoulder Mobility) [12],
Wolf Motor Function Test (Upper-Extremity Mobility) [29], Fugl-
Meyer Assessment (General Motor Ability) [13]; all of which as-
sess motion primitives for active daily living.
Furthermore, this study would benefit from a larger sample size
of the users, more therapists could be involved, and more stimuli
and testing must be done to determine feasibility, compliance, and
design further. There is a possible novelty effect of the VR, where
most of these users were exposed to the VR for their very first time.
The results suggest that the game design may account for the nov-
elty effect. However, a long term study must be done to address
these possibilities adequately. Subsequently, a long term study is
being planned with local hospitals in Santa Cruz, California, to ex-
plore the effects of Project Butterfly with CRUX further. An IRB
protocol for such is currently under review for approval.
Project Butterfly reports on the design and evaluation of a unique
VR experience paired with a soft body robotic wearable exosuit.
The pair of these technologies have been developed as a novel ex-
perience to rehabilitate upper-extremities. CRUX reduces the bur-
den and rigidity experienced by users of traditional wearable robots
through its softer, more structurally compliant constitution. How-
ever, a material other than neoprene should be used to make it
more comfortable and less likely to overheat the user. To tailor
to a more significant number of DoF, the designed VR game, PBF,
is aimed at focusing on motion primitives expressible in soft ex-
oskeletons – actions which the healthy arm can perform that can be
combined and permuted into all upper-extremity movement. PBF
thus serves as a motivator for the user to complete virtual objec-
tives and consequently, actual motions. The completion of these
objectives is assisted through CRUX’s augmentation of their up-
per body strength to perform the game-specific movements. When
evaluating the baseline feasibility of PBF and CRUX in augmenting
and promoting proper arm movement as defined by the established
motion primitives, most users were able to complete appropriately
challenging arm movements, suggesting that PBF and CRUX gave
users suitable strength of their augmented arm. Additionally, the
system’s ease-of-use and comfort were analyzed, and most users
felt that they were confident capitalizing on the therapy system.
Virtual reality paired exosuits could prove useful to make engag-
ing therapy for users with upper-extremity impairment. For more
significant impact, designing a new exosuit peripheral and increas-
ing more types of accompanying VR minigames which augment
various muscle groups (i.e., pectoral muscles and dorsal muscles)
can further improve the rehabilitation. These muscle groups sup-
port upper-extremity actions and strengthening them could bolster
arm muscles as a result. In a similar vein, using new materials for
a future iteration of a soft exosuit focused for VR could make this
technology more comfortable and accessible.
Furthermore, the real-time data produced from the HTC Vive and
Vive Tracking units can be integrated with the exosuit. Therapists
and users may also potentially benefit from further data extrapola-
tion with the HTC Vive. A complete view of a user’s body, achiev-
able through more precise motion tracking and inverse kinematics,
and could identify confounding postural issues, such as slouched
backs and other movement biases, which a physical therapist would
want to be aware of during gameplay. A long term study is being
planned with local hospitals to explore the effects of Project Butter-
fly with a next-generation CRUX design that accommodates more
body types, sizes, and weight. Finally, with the plethora of posi-
tional and behavioral data output produced from this VR experi-
ence, there is potential to integrate machine learning protocols and
AI to optimize suit controls and game difficulty to improve rehabil-
itation results.
The baseline feasibility, ease of use, and comfort created by syn-
ergizing an immersive physio-therapy VR game with an actuated
soft robotic exosuit had promising results in the potential future
of a more accessible, affordable, and personalized rehabilitation.
More research is needed to expand upon the preliminary work of
this study, discover best practices of soft exosuit integrated VR, and
validate clinical utility. Subsequently, there are more butterflies to
follow on the path ahead.
This material is based upon work supported by the National Science
Foundation under Grant No. #1521532. Any opinions, findings,
and conclusions or recommendations expressed in this material are
those of the author(s) and do not necessarily reflect the views of the
National Science Foundation. The authors would like to thank the
following UC Santa Cruz affiliates for their support: John McIn-
tyre, Pattawong Pansodtee, Ocean Hurd, Dustin Halsey, Evanjelin
Mahmoodi, Noah Brown, and Dr. Daniel Goodman Shapiro. The
authors additionally thank the participants of this study for their
time and effort.
[1] Virtual reality for consumer markets - head-mounted displays, acces-
sory devices, and consumer virtual reality content: Global market
analysis and forecasts. report, Tractica Research, Jan 2016.
[2] N. Ahmad, R. A. R. Ghazilla, N. M. Khairi, and V. Kasi. Reviews on
various inertial measurement unit (imu) sensor applications. Interna-
tional Journal of Signal Processing Systems, 1(2):256–262, 2013.
[3] A. Alvara. Development of an exosuit for rehabilitative use and upper
arm force amplification. In 2017 AAAS Annual Meeting (February
16-20, 2017). aaas, 2017.
[4] R. M. Ba ˜
nos, C. Botella, M. Alca ˜
niz, V. Lia˜
no, B. Guerrero, and
B. Rey. Immersion and emotion: their impact on the sense of pres-
ence. CyberPsychology & Behavior, 7(6):734–741, 2004.
[5] M. S. Cameir˜
ao, S. Berm´
udez, and P. Verschure. Virtual reality based
upper extremity rehabilitation following stroke: a review. Journal of
CyberTherapy & Rehabilitation, 1(1):63–74, 2008.
[6] F. J. Deconinck, A. R. Smorenburg, A. Benham, A. Ledebt, M. G.
Feltham, and G. J. Savelsbergh. Reflections on mirror therapy: a sys-
tematic review of the effect of mirror visual feedback on the brain.
Neurorehabilitation and Neural Repair, 29(4):349–361, 2015.
[7] J. Diemer, G. W. Alpers, H. M. Peperkorn, Y. Shiban, and
A. M¨
uhlberger. The impact of perception and presence on emotional
reactions: a review of research in virtual reality. Frontiers in psychol-
ogy, 6, 2015.
[8] A. Elor, S. Kurniawan, and M. Teodorescu. Towards an immer-
sive virtual reality game for smarter post-stroke rehabilitation. In
2018 IEEE International Conference on Smart Computing (SMART-
COMP), pages 219–225. IEEE, 2018.
[9] A. Elor, M. Teodorescu, and S. Kurniawan. Project star catcher:
A novel immersive virtual reality experience for upper limb reha-
bilitation. ACM Transactions on Accessible Computing (TACCESS),
11(4):20, 2018.
[10] A. N. V. Follow. Virtual reality (vr) continuum - amp new ven-
tures, Dec 2016.
[11] A. Frisoli, F. Salsedo, M. Bergamasco, B. Rossi, and M. C. Car-
boncini. A force-feedback exoskeleton for upper-limb rehabilitation
in virtual reality. Applied Bionics and Biomechanics, 6(2):115–126,
[12] F. Garc´
ıa-Muro, ´
A. L. Rodr´
andez, and A. Herrero-de Lu-
cas. Treatment of myofascial pain in the shoulder with kinesio taping.
a case report. Manual therapy, 15(3):292–295, 2010.
[13] D. J. Gladstone, C. J. Danells, and S. E. Black. The fugl-meyer assess-
ment of motor recovery after stroke: a critical review of its measure-
ment properties. Neurorehabilitation and neural repair, 16(3):232–
240, 2002.
[14] H. Grillon, F. Riquier, B. Herbelin, and D. Thalmann. Virtual reality as
a therapeutic tool in the confines of social anxiety disorder treatment.
International journal on disability and human development, 5(3):243–
250, 2006.
[15] D. Gromala, X. Tong, A. Choo, M. Karamnejad, and C. D. Shaw.
The virtual meditative walk: virtual reality therapy for chronic pain
management. In Proceedings of the 33rd Annual ACM Conference on
Human Factors in Computing Systems, pages 521–524. ACM, 2015.
[16] A. Hanaor. Engineering properties of double-layer tensegrity grids.
SpatialStructures at the Tum oftheMillenium, pages 195–200, 1991.
[17] H. G. Hoffman, G. T. Chambers, W. J. Meyer, L. L. Arceneaux, W. J.
Russell, E. J. Seibel, T. L. Richards, S. R. Sharar, and D. R. Patterson.
Virtual reality as an adjunctive non-pharmacologic analgesic for acute
burn pain during medical procedures. Annals of Behavioral Medicine,
41(2):183–191, 2011.
[18] S. J. Housman, V. Le, T. Rahman, R. J. Sanchez, and D. J. Reinkens-
meyer. Arm-training with t-wrex after chronic stroke: preliminary
results of a randomized controlled trial. In Rehabilitation Robotics,
2007. ICORR 2007. IEEE 10th International Conference on, pages
562–568. IEEE, 2007.
[19] L. M. Howden and J. A. Meyer. Age and sex composition: 2010.
2010 Census Briefs, US Department of Commerce, Economics and
Statistics Administration. US CENSUS BUREAU, 2010.
[20] HTC-Corporation. Vive vr system. Vive, November 2018.
[21] D. Jack, R. Boian, A. S. Merians, M. Tremaine, G. C. Burdea, S. V.
Adamovich, M. Recce, and H. Poizner. Virtual reality-enhanced
stroke rehabilitation. IEEE transactions on neural systems and re-
habilitation engineering, 9(3):308–318, 2001.
[22] R. Koopman and L. J. van Loon. Aging, exercise, and muscle protein
metabolism. Journal of Applied Physiology, 106(6):2040–2048, 2009.
[23] O. Kreylos. Lighthouse tracking examined. A developer’s perspec-
tive on immersive 3D computer graphics, May 2016. http://doc-
[24] S. Lessard, P. Pansodtee, A. Robbins, L. B. Baltaxe-Admony, J. M.
Trombadore, M. Teodorescu, A. Agogino, and S. Kurniawan. Crux:
A compliant robotic upper-extremity exosuit for lightweight, portable,
multi-joint muscular augmentation. In Rehabilitation Robotics
(ICORR), 2017 International Conference on, pages 1633–1638. IEEE,
[25] S. Lessard, P. Pansodtee, A. Robbins, J. M. Trombadore, S. Kurni-
awan, and M. Teodorescu. A soft exosuit for flexible upper-extremity
rehabilitation. IEEE Transactions on Neural Systems and Rehabilita-
tion Engineering, 2018.
[26] K. R. Lohse, C. G. Hilderman, K. L. Cheung, S. Tatla, and H. M.
Van der Loos. Virtual reality therapy for adults post-stroke: a sys-
tematic review and meta-analysis exploring virtual environments and
commercial games in therapy. PloS one, 9(3):e93318, 2014.
[27] M. E. Michielsen, M. Smits, G. M. Ribbers, H. J. Stam, J. N. Van
Der Geest, J. B. Bussmann, and R. W. Selles. The neuronal correlates
of mirror therapy: an fmri study on mirror induced visual illusions in
patients with stroke. Journal of neurology, neurosurgery & psychiatry,
82(4):393–398, 2011.
[28] N. Morina, H. Ijntema, K. Meyerbr ¨
oker, and P. M. Emmelkamp. Can
virtual reality exposure therapy gains be generalized to real-life? a
meta-analysis of studies applying behavioral assessments. Behaviour
research and therapy, 74:18–24, 2015.
[29] D. M. Morris, G. Uswatte, J. E. Crago, E. W. Cook III, and E. Taub.
The reliability of the wolf motor function test for assessing upper ex-
tremity function after stroke. Archives of physical medicine and reha-
bilitation, 82(6):750–755, 2001.
[30] H. Mousavi Hondori and M. Khademi. A review on technical and clin-
ical impact of microsoft kinect on physical therapy and rehabilitation.
Journal of Medical Engineering, 2014, 2014.
[31] M. V. Nararro-Haro, H. G. Hoffman, A. Garcia-Palacios, M. Sampaio,
W. Alhalabi, K. Hall, and M. Linehan. The use of virtual reality to fa-
cilitate mindfulness skills training in dialectical behavioral therapy for
borderline personality disorder: a case study. Frontiers in psychology,
7, 2016.
[32] D. C. Niehorster, L. Li, and M. Lappe. The accuracy and precision of
position and orientation tracking in the htc vive virtual reality system
for scientific research. i-Perception, 8(3):2041669517708205, 2017.
[33] V. S. Ramachandran and D. Rogers-Ramachandran. Synaesthesia in
phantom limbs induced with mirrors. Proceedings of the Royal Society
of London B: Biological Sciences, 263(1369):377–386, 1996.
[34] B. O. Rothbaum, M. Price, T. Jovanovic, S. D. Norrholm, M. Gerardi,
B. Dunlop, M. Davis, B. Bradley, E. J. Duncan, A. Rizzo, et al. A
randomized, double-blind evaluation of d-cycloserine or alprazolam
combined with virtual reality exposure therapy for posttraumatic stress
disorder in iraq and afghanistan war veterans. American Journal of
Psychiatry, 171(6):640–648, 2014.
[35] M. Rus-Calafell, J. Guti´
errez-Maldonado, and J. Ribas-Sabat´
e. A vir-
tual reality-integrated program for improving social skills in patients
with schizophrenia: a pilot study. Journal of behavior therapy and
experimental psychiatry, 45(1):81–89, 2014.
[36] R. D. Seidler, J. L. Alberts, and G. E. Stelmach. Changes in multi-joint
performance with age. Motor control, 6(1):19–31, 2002.
[37] R. D. Seidler, J. A. Bernard, T. B. Burutolu, B. W. Fling, M. T. Gor-
don, J. T. Gwin, Y. Kwak, and D. B. Lipps. Motor control and aging:
links to age-related brain structural, functional, and biochemical ef-
fects. Neuroscience & Biobehavioral Reviews, 34(5):721–733, 2010.
[38] R. D. Seidler-Dobrin, J. He, and G. E. Stelmach. Coactivation to re-
duce variability in the elderly. Motor control, 2(4):314–330, 1998.
[39] Y. Shiban, I. Schelhorn, P. Pauli, and A. M ¨
uhlberger. Effect of com-
bined multiple contexts and multiple stimuli exposure in spider pho-
bia: a randomized clinical trial in virtual reality. Behaviour research
and therapy, 71:45–53, 2015.
[40] N. Shkuratova, M. E. Morris, and F. Huxham. Effects of age on bal-
ance control during walking. Archives of physical medicine and reha-
bilitation, 85(4):582–588, 2004.
[41] F. Soffel, M. Zank, and A. Kunz. Postural stability analysis in virtual
reality using the htc vive. In Proceedings of the 22nd ACM Conference
on Virtual Reality Software and Technology, pages 351–352. ACM,
[42] H. Thieme, J. Mehrholz, M. Pohl, J. Behrens, and C. Dohle. Mirror
therapy for improving motor function after stroke. Stroke, 44(1):e1–
e2, 2013.
[43] ValveSoftware. Valvesoftware/openvr. GitHub, Jul.
[44] H. Wang and T. Cochrane. Mobility impairment, muscle imbalance,
muscle weakness, scapular asymmetry and shoulder injury in elite
volleyball athletes. Journal of sports medicine and physical fitness,
41(3):403, 2001.
[45] N. S. Ward. Compensatory mechanisms in the aging motor system.
Ageing research reviews, 5(3):239–254, 2006.
[46] M. Wehner, B. Quinlivan, P. M. Aubin, E. Martinez-Villalpando,
M. Baumann, L. Stirling, K. Holt, R. Wood, and C. Walsh. A
lightweight soft exosuit for gait assistance. In Robotics and Automa-
tion (ICRA), 2013 IEEE International Conference on, pages 3362–
3369. IEEE, 2013.
... This can be attributed to the consumer proliferation of VR devices over AR headsets and the availability of entry level consumer VR devices to support recent advances in the VR gaming industry. Therefore, many current solutions presented in the literature focus heavily on the VR spectrum of devices with solutions being proposed for rehabilitation (Elor et al., 2019), training (Oates et al., 2008) and monitoring/assessment (Yeh et al., 2012), whilst fewer studies have been conducted in relation to AR experiences. ...
Augmented and virtual reality experiences present significant barriers for disabled people, making it challenging to fully engage with immersive platforms. Whilst researchers have started to explore potential solutions addressing these accessibility issues, we currently lack a comprehensive understanding of research areas requiring further investigation to support the development of inclusive AR/VR systems. To address current gaps in knowledge, we led a series of multidisciplinary sandpits with relevant stakeholders (i.e., academic researchers, industry specialists, people with lived experience of disability, assistive technologists, and representatives from disability organisations, charities, and special needs educational institutions) to collaboratively explore research challenges, opportunities, and solutions. Based on insights shared by participants, we present a research agenda identifying key areas where further work is required in relation to specific forms of disability (i.e., across the spectrum of physical, visual, cognitive, and hearing impairments), including wider considerations associated with the development of more accessible immersive platforms.
... ACM 389 29 [40], [76], [104], [52], [85], [125], [122], [173], [96], [64], [31], [145], [73], [171], [1], [61], [91], [108], [46], [174], [26], [110], [89], [167], [148], [102], [11], [86], [51]. IEEE 164 38 [116], [144], [139], [16], [163], [65], [36], [142], [39], [79], [150], [66], [84], [154], [75], [141], [90], [131], [111], [17], [19], [120], [72], [183], [177], [2], [155], [124], [4], [88], [176], [12], [168], [137], [20], [140], [114], [151]. ...
Full-text available
There are several factors involved in functional rehabilitation, some of which are crucial for the effectiveness of the treatment performed, such as the patient's engagement in the activity and the physical therapist's active participation. Difficulties in accessing the therapy, because of many reasons, have given rise to initiatives of telerehabilitation, that is, holding sessions via telecommunication technologies. In this scenario, the therapist's active participation is compromised and, on the other hand, the exercises performed are often repetitive and boring, decreasing the patient's engagement and motivation, and that impacts on the sessions' results. Considering these issues, the present work aims to validate the hypothesis that an exergame (a serious game developed with the objective of motivating the performance of physical exercises) built from a distributed architecture is effective in carrying out telerehabilitation sessions that involve a cycle ergometer as a device - a bedside bicycle used in rehabilitation sessions for patients with motor disorders (affected by Cerebral Vascular Disorders, Cystic Fibrosis or who were hospitalized with COVID-19, for example). In order to validate this hypothesis, the construction of a prototype of the exergame was undertaken, followed by two validation steps done by 16 physical therapists from Hospital das Clínicas of UFG (based on the Delphi Method) and an experiment involving 12 volunteers (Player Experience assessment). About 88,8% of the specialists, after the two rounds of evaluation, considered that the exergame is appropriate for telerehabilitation sessions. Furthermore, 75% of the volunteers rated the experience as “very satisfying”, while the remaining 25% characterized it as “satisfactory”. Therefore, the results allowed to conclude the effectiveness and satisfiability of the game, proving the validity of the research hypothesis.
... It determined the optimal cable paths of minimal extension by conducting measurement experiments and could be operated using either a two-axis analog joystick or a closed-loop controller. Paired with virtual reality technology, it was applied to a novel rehabilitative experience called Project Butterfly [45]. Relevant experiments demonstrated its performance including feasibility, ease of use and comfort. ...
Full-text available
In recent years, more and more research has begun to focus on the flexible and lightweight design of wearable robots. During this process, many novel concepts and achievements have been continuously made and shown to the public, while new problems have emerged at the same time, which need to be solved. In this paper, we give an overview of the development status of soft wearable robots for human movement assistance. On the basis of a clear definition, we perform a system classification according to the target assisted joint and attempt to describe the overall prototype design level in related fields. Additionally, it is necessary to sort out the latest research progress of key technologies such as structure, actuation, control and evaluation, thereby analyzing the design ideas and basic characteristics of them. Finally, we discuss the possible application fields, and propose the main challenges of this valuable research direction.
... Project Star Catcher has demonstrated that iVR can beneit treatments such as Constraint-Induced Movement Therapy with stroke survivors by increasing compliance up to 40% when compared to traditional methods and providing an accessible medium for capturing patient success metrics with HMD motion capture [27,29]. Project Butterly, an iVR experience inspired by Mirror Visual Feedback Therapy, has explored the potential to engage patients for long-term treatment with immersive virtual environments, which is vital because stroke rehabilitation can span years [28,63]. REINVENT applied neurofeedback systems to iVR games with promising pilot results suggesting feasible and safe usage for severe stroke upper limb motor recovery [74]. ...
Full-text available
Developing games is time-consuming and costly. Overly clinical therapy games run the risk of being boring, which defeats the purpose of using games to motivate healing in the first place [10, 23]. In this work, we adapt and repurpose an existing immersive virtual reality (iVR) game, Spellcasters, originally designed purely for entertainment for use as a stroke rehabilitation game—which is particularly relevant in the wake of COVID-19, where telehealth solutions are increasingly needed [4]. In preparation for participatory design sessions with stroke survivors, we collaborate with 14 medical professionals to ensure Spellcasters is safe and therapeutically valid for clinical adoption. We present our novel VR sandbox implementation that allows medical professionals to customize appropriate gestures and interactions for each patient's unique needs. Additionally, we share a co-designed companion app prototype based on clinicians' preferred data reporting mechanisms for telehealth. We discuss insights about adapting and repurposing entertainment games as serious games for health, features that clinicians value, and the potential broader impacts of applications like Spellcasters for stroke management.
... stroke survivors or cognitive disabilities). Our lab has worked with disability groups within our lab exploring various virtual reality mediums for users with cognitive disabilities, testing soft exo-suits meant for post-stroke rehabilitation, and physical rehabilitation games for users with cognitive disabilities [9], [56]- [58]. In the future we will collect motion data of these varying groups to develop more inclusive patient models. ...
Full-text available
Objective: The adoption of telehealth rapidly accelerated due to the global COVID19 pandemic disrupting communities and in-person healthcare practices. While telehealth had initial benefits inenhancing accessibility for remote treatment, physical rehabilitation has been heavily limited due to the lossof hands-on evaluation tools. This paper presents an immersive virtual reality (iVR) pipeline for replicating physical therapy success metrics through applied machine learning of patient observation. Methods: Wedemonstrate a method of training gradient boosted decision-trees for kinematic estimation to replicate mobility and strength metrics with an off-the-shelf iVR system. During a two-month study, training datawas collected while a group of users completed physical rehabilitation exercises in an iVR game. Utilizing this data, we trained on iVR based motion capture data and OpenSim biomechanical simulations. Results:Our final model indicates that upper-extremity kinematics from OpenSim can be accurately predicted using the HTC Vive head-mounted display system with a Mean Absolute Error less than 0.78 degrees for jointangles and less than 2.34 Nm for joint torques. Additionally, these predictions are viable for run-time estimation, with approximately a 0.74 ms rate of prediction during exercise sessions. Conclusion: These findings suggest that iVR paired with machine learning can serve as an effective medium for collectingevidence-based patient success metrics in telehealth. Significance: Our approach can help increase the accessibility of physical rehabilitation with off-the-shelf iVR head-mounted display systems by providing therapists with metrics needed for remote evaluation.
Full-text available
Acculturative stress disproportionately impacts first-generation Latine-Americans, leading to significant mental health risks stemming from intergenerational cultural norms around gender identity and sexuality. Facilitating communication is critical in reducing this stress, yet it can be challenging for Latine individuals to take the first step in expanding their views due to limited resources, cultural pressure, and motivational needs. On the other hand, serious games provide a unique opportunity to address this challenge by introducing novel experiences to encourage the growth of perspectives in acculturative norms. This article presents a narrative review that bridges three key concepts: (1) acculturative stress in Latine-American communities, (2) modern behavior change theory and model, and (3) the design of serious games. We conclude by proposing a framework for Acculturative Game Design (AGD) and discuss considerations for fostering the support of intergenerational relationships around Latine identity.
Full-text available
Wearable robotics, also called exoskeletons, have been engineered for human-centered assistance for decades. They provide assistive technologies for maintaining and improving patients’ natural capabilities towards self-independence and also enable new therapy solutions for rehabilitation towards pervasive health. Upper limb exoskeletons can significantly enhance human manipulation with environments, which is crucial to patients’ independence, self-esteem, and quality of life. For long-term use in both in-hospital and at-home settings, there are still needs for new technologies with high comfort, biocompatibility, and operability. The recent progress in soft robotics has initiated soft exoskeletons (also called exosuits), which are based on controllable and compliant materials and structures. Remarkable literature reviews have been performed for rigid exoskeletons ranging from robot design to different practical applications. Due to the emerging state, few have been focused on soft upper limb exoskeletons. This paper aims to provide a systematic review of the recent progress in wearable upper limb robotics including both rigid and soft exoskeletons with a focus on their designs and applications in various pervasive healthcare settings. The technical needs for wearable robots are carefully reviewed and the assistance and rehabilitation that can be enhanced by wearable robotics are particularly discussed. The knowledge from rigid wearable robots may provide practical experience and inspire new ideas for soft exoskeleton designs. We also discuss the challenges and opportunities of wearable assistive robotics for pervasive health.
Full-text available
Introduction Soft robotic wearable devices, referred to as exosuits, can be a valid alternative to rigid exoskeletons when it comes to daily upper limb support. Indeed, their inherent flexibility improves comfort, usability, and portability while not constraining the user’s natural degrees of freedom. This review is meant to guide the reader in understanding the current approaches across all design and production steps that might be exploited when developing an upper limb robotic exosuit. Methods The literature research regarding such devices was conducted in PubMed, Scopus, and Web of Science. The investigated features are the intended scenario, type of actuation, supported degrees of freedom, low-level control, high-level control with a focus on intention detection, technology readiness level, and type of experiments conducted to evaluate the device. Results A total of 105 articles were collected, describing 69 different devices. Devices were grouped according to their actuation type. More than 80% of devices are meant either for rehabilitation, assistance, or both. The most exploited actuation types are pneumatic (52%) and DC motors with cable transmission (29%). Most devices actuate 1 (56%) or 2 (28%) degrees of freedom, and the most targeted joints are the elbow and the shoulder. Intention detection strategies are implemented in 33% of the suits and include the use of switches and buttons, IMUs, stretch and bending sensors, EMG and EEG measurements. Most devices (75%) score a technology readiness level of 4 or 5. Conclusion Although few devices can be considered ready to reach the market, exosuits show very high potential for the assistance of daily activities. Clinical trials exploiting shared evaluation metrics are needed to assess the effectiveness of upper limb exosuits on target users.
Conference Paper
Full-text available
Robots are being taught by increasingly broader populations of people who provide training data for machine learning algorithms. Many studies over the past decade have begun demonstrating reproducible robot teaching methodologies and have highlighted benefits in human-robot interaction (HRI). However, there have been few investigations about what it is like for the people teaching these robots. In this study, we consider how teaching a skill to a robot arm, performing a reaching task (as opposed to observing the robot self-learning), influences a user's emotional experience and perceptions of the robot. In a 2x2 experiment (N=160), we varied the agent's learning technique (user reinforcement feedback or robot self-learning) and expressiveness (static agent face or performance-based valence expression with head following), using an online WebGL virtual environment to enable remote HRI. Our results demonstrate that users experience significantly more trust, believability, and emotional response when teaching the robot than when observing it learning, which can be amplified with agent expressiveness.
Full-text available
Modern immersive virtual reality experiences have the unique potential to motivate patients undergoing physical therapy for performing intensive repetitive task-based treatment and can be utilized to collect real-time user data to track adherence and compliance rates. This article reports the design and evaluation of an immersive virtual reality game using the HTC Vive for upper limb rehabilitation, titled “Project Star Catcher” (PSC), aimed at users with hemiparesis. The game mechanics were adapted from modified Constraint Induced Therapy (mCIT), an established therapy method where users are asked to use the weaker arm by physically binding the stronger arm. Our adaptation changes the physical to psychological binding by providing various types of immersive stimulation to influence the use of the weaker arm. PSC was evaluated by users with combined developmental and physical impairments as well as stroke survivors. The results suggest that we were successful in providing a motivating experience for performing mCIT as well as a cost-effective solution for real-time data capture during therapy. We conclude the article with a set of considerations for immersive virtual reality therapy game design.
Conference Paper
Full-text available
Traditional forms of physical therapy and rehabilitation are often based on therapist observation and judgment, coincidentally this process oftentimes can be inaccurate, expensive, and non-timely. Modern immersive Virtual Reality systems provide a unique opportunity to make the therapy process smarter. In this paper, we present an immersive virtual reality stroke rehabilitation game based on a widely accepted therapy method, Constraint-Induced Therapy, that was evaluated by nine poststroke participants. We implement our game as a dynamically adapting system that can account for the user’s motor abilities while recording real-time motion capture and behavioral data. The game also can be used for tele-rehabilitation, effectively allowing therapists to connect with the participant remotely while also having access to +90Hz real-time biofeedback data. Our quantitative and qualitative results suggest that our system is useful in increasing affordability, accuracy, and accessibility of post-stroke motor treatment.
Full-text available
The advent of inexpensive consumer virtual reality equipment enables many more researchers to study perception with naturally moving observers. One such system, the HTC Vive, offers a large field-of-view, high-resolution head mounted display together with a room-scale tracking system for less than a thousand U.S. dollars. If the position and orientation tracking of this system is of sufficient accuracy and precision, it could be suitable for much research that is currently done with far more expensive systems. Here we present a quantitative test of the HTC Vive’s position and orientation tracking as well as its end-to-end system latency. We report that while the precision of the Vive’s tracking measurements is high and its system latency (22 ms) is low, its position and orientation measurements are provided in a coordinate system that is tilted with respect to the physical ground plane. Because large changes in offset were found whenever tracking was briefly lost, it cannot be corrected for with a one-time calibration procedure. We conclude that the varying offset between the virtual and the physical tracking space makes the HTC Vive at present unsuitable for scientific experiments that require accurate visual stimulation of self-motion through a virtual world. It may however be suited for other experiments that do not have this requirement.
Full-text available
Borderline personality disorder (BPD) is a severe mental disorder characterized by a dysfunctional pattern of affective instability, impulsivity, and disturbed interpersonal relationships. Dialectical Behavior Therapy (DBT®) is the most effective treatment for Borderline Personality Disorder, but demand for DBT® far exceeds existing clinical resources. Most patients with BPD never receive DBT®. Incorporating computer technology into the DBT® could help increase dissemination. Immersive Virtual Reality technology (VR) is becoming widely available to mainstream consumers. This case study explored the feasibility/clinical potential of using immersive virtual reality technology to enhance DBT® mindfulness skills training of a 32 year old female diagnosed with BPD. Prior to using VR, the patient experienced difficulty practicing DBT® mindfulness due to her emotional reactivity, and difficulty concentrating. To help the patient focus her attention, and to facilitate DBT® mindfulness skills learning, the patient looked into virtual reality goggles, and had the illusion of slowly “floating down” a 3D computer-generated river while listening to DBT® mindfulness training audios. Urges to commit suicide, urges to self harm, urges to quit therapy, urges to use substances, and negative emotions were all reduced after each VR mindfulness session and VR mindfulness was well accepted/liked by the patient. Although case studies are scientifically inconclusive by nature, results from this feasibility study were encouraging. Future controlled studies are needed to quantify whether VR-enhanced mindfulness training has long term benefits e.g., increasing patient acceptance and/or improving therapeutic outcome. Computerizing some of the DBT® skills treatment modules would reduce cost and increase dissemination.
Full-text available
This paper presents the design and the clinical validation of an upper-limb force-feedback exoskeleton, the L-EXOS, for robotic-assisted rehabilitation in virtual reality (VR). The L-EXOS is a five degrees of freedom exoskeleton with a wearable structure and anthropomorphic workspace that can cover the full range of motion of human arm. A specific VR application focused on the reaching task was developed and evaluated on a group of eight post-stroke patients, to assess the efficacy of the system for the rehabilitation of upper limb. The evaluation showed a significant reduction of the performance error in the reaching task (paired t -test, p < 0.02)
For stroke survivors and many other people with upper-extremity impairment, daily life can be difficult without properly functioning arms. Some modern physical therapy exercises focus on rehabilitating people with these troubles by correcting patients’ perceptions of their own body to eventually regain complete control and strength over their arms again. Augmentative wearable robots, such as upper-extremity exoskeletons and exosuits, may be able to assist this endeavor. A common drawback in many of these exoskeletons however is their inability to conform to the natural flexibility of the human body without a rigid base. We have built one such exosuit to address this challenge: CRUX (Compliant Robotic Upper-extremity eXosuit). This robot is a compliant, lightweight, multi-DoF, portable exosuit that affords its wearer the ability to augment themselves in many unconventional settings (i.e. outside of a clinic). These attributes are largely achieved by using a modified tensegrity design situated according to measured lines of minimal-extension where a network of tension members provide a foundation to apply augmentative forces via precisely placed power-lines. In this paper, we detail the design process of CRUX, report on CRUX’s prototypical composition, and describe the mimetic control algorithm used.We also discuss the results of three studies that illustrate the efficacy of CRUX’s mimetic controller, CRUX’s flexibility and compliance, and the metabolic cost reduction when users exercise with assistance from CRUX as opposed to without. We conclude the paper with a summary of our findings, potential use cases for this technology, and the direction of future related work.
Conference Paper
Wearable robots can potentially offer their users enhanced stability and strength. These augmentations are ideally designed to actuate harmoniously with the user's movements and provide extra force as needed. The creation of such robots, however, is particularly challenging due to the underlying complexity of the human body. In this paper, we present a compliant, robotic exosuit for upper extremities called CRUX. This exosuit, inspired by tensegrity models of the human arm, features a lightweight (1.3 kg), flexible multi-joint design for portable augmentation. We also illustrate how CRUX maintains the full range of motion of the upper-extremities for its users while providing multi-DoF strength amplification to the major muscles of the arm, as evident by tracking the heart rate of an individual exercising said arm. Exosuits such as CRUX may be useful in physical therapy and in extreme environments where users are expected to exert their bodies to the fullest extent.
Conference Paper
Postural stability is an important measure for many medical diseases such as Parkinson. In the last years, research focused on using inexpensive and portable devices to measure postural stability, while the visual targets were physical objects in the environment. Sensing balancing boards were used to measure stance forces, while movements of the upper body were not taken into account. Within this paper, postural stability was measured using the HTC Vive. A variation of a virtual fixation point’s distance was analyzed and compared to a reference condition with closed eyes. It is shown that body sway in the VR conditions is increased in the anterior-posterior and decreased in the medial-lateral direction.