Project Butterfly: Synergizing Immersive Virtual Reality with Actuated Soft Exosuit for Upper-Extremity Rehabilitation

Abstract

Immersive Virtual Reality paired with soft robotics may be synergized to create personalized assistive therapy experiences. Virtual
worlds hold power to stimulate the user with newly instigated low cost, high-performance commercial Virtual Reality (VR) devices to
enable engaging and accurate physical therapy. Soft robotic wearables are a versatile tool in such stimulation. This preliminary study
investigates a novel rehabilitative VR experience, Project Butterfly
(PBF), that synergizes VR Mirror Visual Feedback Therapy with
soft robotic exoskeletal support. Nine users of ranging ability explore an immersive gamified physio-therapy experience by following and protecting a virtual butterfly, completed with an actuated
robotic wearable that motivates and assists the user to perform rehabilitative physical movement. Specifically, the goals of this study
are to evaluate the feasibility, ease-of-use, and comfort of the proposed system. The study concludes with a set of design considerations for future immersive physio-rehab robotic-assisted games.

Full text

Project Butterfly: Synergizing Immersive Virtual Reality with Actuated
Soft Exosuit for Upper-Extremity Rehabilitation
Aviv Elor, Steven Lessard, Mircea Teodorescu, and Sri Kurniawan
University of California, Santa Cruz, Jack Baskin School of Engineering ∗
{aelor, slessard, mteodore, skurnia}@ucsc.edu
ABSTRACT
Immersive Virtual Reality paired with soft robotics may be syner-
gized to create personalized assistive therapy experiences. Virtual
worlds hold power to stimulate the user with newly instigated low-
cost, high-performance commercial Virtual Reality (VR) devices to
enable engaging and accurate physical therapy. Soft robotic wear-
ables are a versatile tool in such stimulation. This preliminary study
investigates a novel rehabilitative VR experience, Project Butterfly
(PBF), that synergizes VR Mirror Visual Feedback Therapy with
soft robotic exoskeletal support. Nine users of ranging ability ex-
plore an immersive gamified physio-therapy experience by follow-
ing and protecting a virtual butterfly, completed with an actuated
robotic wearable that motivates and assists the user to perform re-
habilitative physical movement. Specifically, the goals of this study
are to evaluate the feasibility, ease-of-use, and comfort of the pro-
posed system. The study concludes with a set of design considera-
tions for future immersive physio-rehab robotic-assisted games.
Index Terms: Virtual Reality, Soft Robotics, Wearable Robotics,
Exosuit, Physio-Immersive Rehabilitation, Physical Therapy, Im-
mersive Experiences, Serious Games, Human Computer Interaction
1 INTRODUCTION
According to the latest US Census in 2010, there are more than 40
million older adults (defined as people aged 65 years old or older)
living in the US, comprising 13 percent of the US population. This
demographic represents a 15 percent growth compared to the 2000
US Census data [19] and is projected to continue to grow. Unfortu-
nately, studies have shown that motor functions decline with aging
[45]. The significant older population experiences an increasingly
prevalent issue of motor degeneration. Age-related motor perfor-
mance deficits include coordination difficulty, decreased variabil-
ity of motor ability, slowing of movement, and problems with bal-
ance and gait [37]. Movement slows with aging by as much as
15 to 30 percent. Research by Seidler-Dobrin et al. suggests that
older adults emphasize movement accuracy at the cost of movement
speed [38]. As a result, older adults show specific deficits in the
coordination of bimanual and multi-joint movements. For exam-
ple, movements become slower and less smooth when older adults
use their shoulder and elbow joints simultaneously as opposed to
performing single-joint actions[36]. Often postural stability is also
compromised with advancing age [40].
In addition to the decline in motor function, aging correlates to
the progressive loss of skeletal muscle mass and strength. Fre-
∗The authors are with the Departments of Computational Media & Elec-
trical Computer Engineering, University of California Santa Cruz, Jack
Baskin School of Engineering, 1156 High St, Santa Cruz, CA, USA. Ad-
ditionally, this work was supported by the Center for Information Technol-
ogy Research in the Interest of Society (CITRIS) and the National Science
Foundation (NSF).
quent exercise represents an effective therapeutic strategy to aug-
ment skeletal muscle mass and improve functional performance and
quality of life in older adults [22]. Many technological solutions
have been researched and developed over the past decade to reduce
motor loss, but there is still much to be done.
2 RE LATE D WORK
One modern approach to address muscular impairment is virtual-
reality (VR) therapy. Through the use of VR, stimulating immersive
environments can be programmed to increase therapy compliance,
accessibility, and data throughput [8, 9]. Psychological and physio-
logical research has featured increasing use of VR in the prior two
decades thanks to the ability to simulate realistic and complex situa-
tions critical to laboratory-based human behavior investigations [7].
Traditional forms of therapy and rehabilitation usually derive from
therapist observation and judgment. Drawbacks of this traditional
method are that they are often inaccurate, expensive, and timely
[30]. Virtual reality, however, addresses these concerns as a useful
tool for improving outcomes compared to conventional therapy by
enabling accurate motion capture, telepresence based sessions, and
low cost motivating experiences [26, 8]. The immersive visual ca-
pabilities of modern VR headsets, such as the HTC Vive and Ocu-
lus Rift, have had astounding promise and success with treatments
ranging from exposure in Post Traumatic Stress Disorder [34, 28],
Borderline Personality Disorder [31], various phobias [39, 14, 7],
schizophrenia [35], and many other psychological therapies. Re-
searchers are even reporting that integration VR into the clinical
setting can reduce pain similar to the effect of analgesic treatments
[15, 17].
Success in VR therapy often relates to the relationship between
presence and emotion with technology’s ability to bridge them [28].
Increasing the quantity and quality of stimuli in immersive VR is
key to influencing user behavior and experience [4, 28]. The past
five years have made strides in VR technology – VR is ever more
immersive, affordable, and accessible to the average consumer with
over 200 million immersive VR Head Mounted Displays projected
to be sold by the year 2020 [1, 10]. For these reasons, headset-
based VR systems like the HTC Vive and Oculus Rift could appeal
to low-income communities. VR as a therapeutic tool has, there-
fore, become the most effective and affordable it has ever been and
is projected by many researchers to continue along this forecasted
trajectory.
Many of these studies incorporate Mirror Visual Feedback Ther-
apy (MVFT), the visual or physical stimulation of a “pseudo“
movement on the damaged limb to promote recovery [6]. Patients
are given sensorimotor feedback by reflecting an abled arm in the
position of the impaired arm during exercise [27]. MVFT is sug-
gested to be a beneficial treatment for motor rehabilitation [33, 27],
where clinical studies have indicated that MVFT “can serve as a
versatile tool to promote motor recovery” in mobility and arm use
[6].
MVFT requires the superposition of a simulated arm on a phan-
tom limb which enables patients to relieve painful sensation and
increase movement [33]. A variety of other conditions were ex-
plored with MVFT, including with stroke survivors, where a simu-

lated limb is placed in the patients midsagittal plane, thus reflecting
movements of the nonparetic side as if it were the affected side to
stimulate brain plasticity [42]. However, users with severe motor
loss require increased physical assistance to perform MVFT, and
can often require therapist intervention, or in the case of this study:
a robotic wearable.
A large amount of physical therapy research has transpired on
the integration of rehabilitative robotic wearable devices. Several
upper-body robotic exoskeletons have been developed and explored
over the last ten years with many incorporating VR. Some of these
examples include the PERCRO (Perceptual Robotics laboratory) L-
EXOS system [11], Rutgers CyberGlove and Master II-ND (RMII)
force feedback glove [21], and Therapy Wilmington Robotic Ex-
oskeleton (T-WREX) [18]. Through combining VR with exoskele-
tons that provide arm gravity support, clinical testing showed a
range in improvement of mobility, strength, and satisfaction [5].
One attribute common to these exoskeletons is the use of rigid
structures. A significant flaw experienced by traditional rigid ex-
oskeletons is their inflexibility, and the burden users bear when
wearing them. Devices which have few degrees-of-freedom (DoF)
or heavy components inhibit some movements. This physical con-
straint can lead to imbalanced muscular growth and control, which
can injure users of these wearable robots [44]. As a result, softer
devices such as Lessard et al. exosuits have emerged as a flexible al-
ternative to traditional rigid exoskeletons [24, 25]. This study aims
to leverage such soft exosuits during VR therapy through the use of
Compliant Robotic Upper-Extremity Exosuit (CRUX), as shown in
Figure 1, to explore feasibility, ease-of-use, and comfort.
Figure 1: A demonstrator wears CRUX [25] (without IMUs). CRUX
is an augmentative wearable soft robot for upper-extremity rehabilita-
tion and can be combined with VR through Project Butterfly to enable
immersive rehabilitation.
3 SYSTEM DESIGN
The VR experience developed, Project Butterfly (PBF), is a game
that motivates users to perform upper body motion primitives by
having them protect and control a virtual butterfly in a meadow
while the system collects real-time data using the HTC Vive. Fig-
ure 3 displays an example of PBF gameplay. CRUX was integrated
accordingly to provide additional tactile feedback and physical as-
sistance.
3.1 The Soft Robotic Wearable
The purpose of using the CRUX exosuit was twofold: to physically
stimulate movement to achieve an ideal position, and to immerse
the user deeply into the VR environment. Since immersion is a
crucial factor in the influence of user behavior and compliance [4],
PBF paired with CRUX may significantly improve the user’s reha-
bilitation experience.
The designed exosuit, shown in Figure 1, is capable of lifting
the user’s arm in different directions to create smooth multi-jointed
movements. The concept of tensegrity for soft robotics inspired
the mechanics of CRUX. Tensegrity (a portmanteau of “tensile”
and “integrity”) defines structures as internally prestressed, free-
standing, pin-jointed networks in which the cables or tendons are
held in tension against a system of bars or struts [16].
A base layer of neoprene held CRUX together. Cables are routed
along the neoprene to integrate a network of “anchor points,” which
serve as the rigid components in an otherwise flexible system. The
exact placement of the cables was determined by recording arm
movement on people as they stretched out and expressed their full
range of motion [25]. Through examining extensive motion capture
of the arm, the area on the skin which sheared the least was deter-
mined to find the most stable places to plant anchor points [25].
Bicycle housing routes the cables onto anchor points to actuate
different parts of the arm just as how tendons pull limbs. Six mi-
cro DC motors were mounted on a modular backplate and connect
to the cables of the exosuit via 3d printed spool, and are manipu-
lated through a microcontroller powering the system with a 3-Cell
Lithium polymer battery. Each motor is capable of exerting 88 N
of force (125 oz-in). Figure 1 depicts CRUX being operated by
a demonstrator. The selected material of CRUX affords a compli-
ant and lightweight design. Like similar soft exosuits, CRUX is
lightweight. However, CRUX weighs 1.5 kg [24] compared to the
6.8 kg lower-limb gait-assisting exosuit developed by Wehner et al.
[46] or the 2.27 kg suit by Alvara et al. [3] for upper arm force
amplification
The suit’s controller was designed to have the weak arm follow
the movement of the healthy limb. This mirroring of limbs insti-
gates mimetic controller design. Mirroring the movement from one
side to the other side of the body was inferred from MVFT to in-
crease motor recovery and stimulate brain plasticity [42]. Figure 2
depicts the CRUX being fitted to a user by an evaluator for upper
arm force amplification.
To enable the mimetic control of the healthy arm onto the weak
limb, wireless connectivity capability was added to connect with
the Inertial Measurement Unit (IMU) networks. An IMU is an elec-
tronic device that measures and reports a body’s specific force and
angular rate using a combination of accelerometers, magnetome-
ters, and gyroscopes [2]. The IMU network added to the exosuit
consists of 4 IMU nodes where each node can measure 3-axis ori-
entation of itself and then send this data back to the microcontroller.
The IMU nodes on CRUX are enclosed in a 3D printed case with
adjustable Velcro straps to accommodate various body sizes.
A plunger button must be engaged by the user to allow for exo-
suit movement. This safeguard prevents accidental actuation when
the exosuit is in master-slave mode. If the user feels that the motor
is performing movements that are undesired, the user can release
the plunger button, effectively disengaging the motor.
In complementary locations, nodes are positioned on both arms
at the lateral forearm (midway between the wrist and the elbow) and
the lower medial triceps (slightly above the elbow) [25] as seen in
Figure 2. Each node transmits pose data to the central controller to
support the closed-loop function enabling the pose following from
the healthy arm onto the impaired arm. It should be noted that while

the VR device can perform motion capture in place of the IMUs,
creating this dependency would limit the flexibility of CRUX for
future use. For future example, the suit may be used beyond VR
MVFT to assist with active daily living activities, where the con-
trols and level of assistance are calibrated during the VR therapy
sessions.
Figure 2: A participant exploring CRUX [25]. Control is achieved
by using IMU Nodes and an internal controller for leader-follower
mimicry. A user can control their impaired arm using their healthy
arm to match the movement path.
Figure 3: A user playing Project Butterfly. a) is the study proctor’s
on-screen view b) is the in-person view of the study proctor and c) is
the in-game view from the user’s perspective.
3.2 The Immersive Virtual Reality Experience
A motion primitive is defined as a distinct movement achievable
by a single joint which creates a unique degree of freedom (DoF).
Thus, upper body motion primitives can be thought of as indivisible
building blocks that can be combined and permuted into a broader
range of potential movements. The HTC Vive, one of the highest
grossing VR Entertainment Systems [1] developed by the Valve and
HTC Corporation, can be used as a powerful tool to both track these
motion paths and motivate the user to achieve these motions. The
Vive is a VR Head-Mounted Display that implements room scale
4x4 m outside-in tracking technology by utilizing a “lighthouse”
system of lasers which enable the user to interact with the virtual
environment through accurate motion capture in a 3D virtual space
[20]. Complementary to the HTC Vive are two handheld controllers
that feature dynamic haptic feedback which enhances spatial orien-
tation [20]. The HMD provides a 110-degree field of view and 90
Hz refresh rate [41]. Motion capture is tracked at 120 Hz using in-
frared laser sweeping and photo-diodes that enable for recovery of
position and orientation [20]. The HTC Vive also features a safety
guidance system preventing users from potential injury in the real
world environment [41, 32]. The worst case tracking jitter of the
system has been reported to be under 2.1mm with an accuracy of
an absolute 2mm error [23]. Resultingly, the HTC Vive allows for
the ability to extract accurate gameplay data while providing an en-
veloping experience of touch, sound, and sight.
Paired with an HTC Vive controller, the exosuit assists the user
during VR gameplay. Testing of the system targeted two pairs
of motion primitives: elbow extension/flexion, and shoulder ab-
duction/adduction. Biceps received assistance by replacing the
user’s CRUX supported HTC Vive controller as a bubble shield and
having them protect the butterfly from incoming rain and projec-
tiles through a therapist-specified customized range of motion path.
Haptic feedback is enabled so that the user is indicated with strong
pulses whenever the motion primitive was not followed (failure to
encapsulate the butterfly inside the bubble). To increase the incen-
tive and track compliance, the user receives a scoring point per ev-
ery half second that they mirrored the required motion primitive.
This multi-sensory feedback guided users according to the objec-
tive of the game.
To generate the environment and the mechanics of PBF, the
Unity v2017.1.0b4 Game Engine along with the SteamVR Unity
plugin v1.2.0 became the chosen development tools. Both Unity
and SteamVR hold a large amount of open-access documenta-
tion, including flexibility with programming languages such as
Javascript and C# [43]. Using Unity’s built-in physics engine, the
Rigidbody class was used to model the butterfly along with spher-
ical colliders that detected contact with the butterfly. Assignment
of the moving projectiles and the butterfly with the rain were set
to a time-dependent spatial state, allowing for global physics-based
events to influence data capture. Runtime data collection was cap-
tured using C# and Microsoft .NET Framework at speeds of 90Hz
and higher.
To assist the user evaluation process, PBF includes a dynamic
evaluator GUI, seen in Figure 4, which automatically prompts the
evaluator or therapist to measure the length of each participant’s
arms through the motion capture (measuring the x-z plane maximal
distance between the VR HMD and the Vive Controller) or man-
ually entered ranges. This calibration stage is achieved entirely in
Unity during run-time with the HTC Vive so that user’s who may
not have access to CRUX in the future can still play PBF without
physical assistance. Figure 4 also features the option to change the
repetitions per minute of each motion primitive, and data exporta-
tion rate of each game session. In short, the evaluator GUI allowed
the evaluator to tailor the game to each unique individual and cus-
tomize data throughput.
The game-themed goals became protecting a butterfly from
heavy rain and projectiles using a bubble (as the avatar for the con-
troller of the weak arm), which focused on bicep curl and lateral
arm raise exercises. Users are instructed place to the bubble around
the butterfly to achieve a high score, and to “protect the butterfly”.
These mechanics required the users to smoothly follow the flight
of the butterfly within +/- 0.1m of the required motion path. As a
result, the biceps and lateral arm raise minigames became scripted
movements of high accuracy and scripted timing repetitions when
a user performed a motion primitive.
Lastly, a tutorial started before every game that the evaluator
could enable to adjust the user to the VR environment. In the tu-
torial, users were asked to identify three images placed on their

left, on their right, and behind them. Identification in the 360 VR
environment was implemented to familiarize the user with virtual
reality and show them that they can look in any direction. Addi-
tionally, a “How to Play” menu was added to teach evaluators how
to test users, and in-game instructions were added to clarify the goal
of each minigame. This pre-gameplay stage simultaneously intro-
duce users not only to virtual reality mechanics (such as 360-degree
views) but also to PBF specific mechanics like the arm movements
required to earn a higher score. These designs were intended to
tailor the game to each unique individual smoothly and intuitively,
all while increasing the incentives to perform the objective of the
game.
Figure 4: Dynamic Project Butterfly Evaluator Interface. This UI and
the data it gathered allowed for more balanced games for each sub-
sequent participant.
4 USABILITY ST UDY
The study investigated the usability of PBF synergized with CRUX
system design, which includes ease of use, comfort, and set base-
line feasibility with nine elderly users having motor dysfunction.
Users were observed playing the virtual reality game with the ex-
osuit as they performed tasks that identified the ability of targeted
muscles and muscle groups. The nine elderly participants in usabil-
ity evaluations represented three patient segments:
1. Three retirees from Elderday Adult Day Health Care Center,
Santa Cruz, CA, represented a mental disability use case be-
cause they were affected by memory loss or dementia. Evalu-
ations with these users were conducted at Elderday.
2. Three stroke survivors recruited from Cabrillo College’s
Stroke and Disability Learning Center (SDLC), Capitola, CA,
represented a physical disability use case because they were
affected by neglect syndrome, meaning the motor functions of
one side of their body was impaired by their stroke. Evalua-
tions with these users were conducted at SDLC.
3. Three retirees living independently within their community,
outside of daycare or hospice, which do not have a significant
physical or mental disability. Evaluations with these users
were conducted at the University of California, Santa Cruz.
All participants ranged between the ages of 60 to 80 years old
and were previously unfamiliar with both virtual reality and wear-
able robotics. These user groups represent three demographics:
physical disability, mental disability, and no disability-were se-
lected because the physical and mental disabilities are likely the
target demographics for PBF and CRUX. This preliminary study
serves as feasibility to justify further studies with larger user group
sizes and gain design insights.
For each user, the evaluation began by an proctor giving a de-
tailed explanation of what PBF and CRUX are, answering questions
as needed. Then the user was given a tutorial period where they
walked through the “How to Play” menu and played the game with-
out being recorded. The tutorial consisted of playing two rounds of
the biceps minigame and the lateral arm minigame for one minute
each. The first round with the minigames was played without wear-
ing the CRUX exosuit and the second was played while wearing it
to allow for MVFT. This allowed the user to try out the controller
and learn PBF’s basic game mechanics. The point score from the
tutorial period helped the evaluator calibrate each user (adjusting
arm length and speed).
After the tutorial period, the test commenced with the user play-
ing recorded sessions while wearing the CRUX exosuit and playing
1-minute minigames, shown in Figure 6. Sessions were recorded
using a webcam, collecting gameplay scores and positional data,
and conducting post-session interviews which allowed for the col-
lection of quantitative and qualitative data. User evaluation inter-
views were executed by asking users questions pertained to their
experience from a prepared form with questions such as:
•Open-ended questions:
–What day-to-day tasks do you struggle with?
–How many years have you been doing physical rehabil-
itation therapy?
–Do you enjoy video games?
–How can the virtual reality game be improved?
–How can the exosuit be improved?
•Rate the statement on a 5-point Likert scale:
– Q1: My current therapy is engaging.
– Q2: The virtual reality game was enjoyable.
– Q3: I became fatigued while playing the virtual reality
game.
– Q4: The virtual reality game distracted me from pain
when doing physical movement.
– Q5: If I had access to virtual reality therapy games, I
would use it in the future.
After evaluations, proctors reviewed gameplay and interview data
and commented additional thoughts or observations. Note that re-
sulting answers from Q1-Q5 can be seen in Figure 5.
Figure 5: Nine user responses of 5-point Likert Scale Questions per-
taining to PBF

Figure 6: A user performs lateral arm raises during a Project Butterfly
game session with CRUX. To boost their scores, users had to mimic
the flight path of the butterfly, which in this case was an up-and-down
motion most easily copied by raising one’s arm up-and-down simi-
larly.
Figure 7: Position of the CRUX assisted controller during the lateral
arm raise mini-game. These graphs depict four users attempting to
mimic the movement of their targets (the butterfly avatar). Red is
the butterfly, green is users movements. Red arcs which are closely
matched by green trajectories (as shown in these graphs) mean that
users are successfully completing in-game objectives, which are tai-
lored to test their physical limits when wearing the exosuit.
5 RE SU LTS AN D DISCUSSION
The three primary objectives of the study were to evaluate the base-
line feasibility of the system, ease-of-use of the system, and comfort
of the system. These objectives are judged according to quantita-
tive metrics obtained from recorded metadata of users while play-
ing PBF. Qualitative data was acquired during post-gameplay inter-
views using mixed 5-point Likert Scale questions and open-ended
questions. The Likert scale questions are summarized in Figure 5.
5.1 System Feasibility
Most participants felt that CRUX augmented their upper limb
movements to some degree. Specifically, one participant (see User
1 in Figure 7) mentioned that their arm movements became more
“effortless” with the use of CRUX. Specifically in raising their
arm laterally, which had been challenging for them before donning
CRUX. They further elaborated that it was the first time in weeks
that they did not experience painful throbs when raising an arm
above their shoulders.
To observe an example of such movement, the position of the
CRUX assisted controller and controller target, the butterfly avatar,
were graphed against four scored users from SDLC (Users 2 and 4)
and Retirees (Users 1 and 3) for the lateral shoulder raise minigame
as seen in Figure 7. All four CRUX supported users were able to
achieve a score of 111/130 or higher, indicating that these users
were compliant to the motion primitive path for over 85% of game-
play. Additionally, the speed of the users controllers, headset, and
butterfly avatar (target to follow) is graphed as shown in Figure 8
as well as the acceleration of the user’s weak arm controller shown
in Figure 9. Each of these users demonstrated significantly differ-
ent gameplay movements even though they had close scores. When
considering Figure 7, User 1 had sharp changes, User 2 had shaky
movements, User 3 had smooth moves, and User 4 had smooth and
shaky movements. When looking at movement speed in Figure 8,
Users 1 and 3 maintained a constant speed, indicating a greater
control of their weaker arm. Whereas Users 2 and 4 had signifi-
cant spikes in movement speed indicating a lack of control in their
arms. It should be noted that while the exosuit assists in achiev-
ing position, it does not reduce the shaking of the limbs. Figure 9
reflects the findings in Figure 8 through acceleration, where User
1 maintains the most control nearing almost no acceleration, fol-
lowed by User 2 who spikes but nears zero, and Users 3 and 4 ex-
perienced occasional large shaking. Despite the difference in fine-
motor strength and precision amongst participants, each user was
able to achieve the desired goal of protecting the virtual butterfly.
This might suggest that PBF can be accommodating for people at
various stages of their physical therapy, thus making it more acces-
sible. Also, while User 1 and User 4 have differently lengthened
arms by about 0.2 meters, the compliance of over 85% can be seen
visible as their green arm paths overlay the object red path arch in
Figure 7. Although these results are promising, the sample size is
not statistically significant, which warrants further study as noted
in the Limitations Section 6.5.
On the subject of assistance, most participants felt that CRUX af-
fected augmenting their limbs. However, they all asked for stronger
motors on the exosuit. When asked for a potential reason to this,
most users indicated that they felt that the motors were not power-
ful enough to make the difference that they were expecting. One
user (see User 3 in Figure 7) commented that they understood the
function of the device but “[wanted] even more power behind it.”
Additionally, none of the participants felt that the exosuit made it
more difficult for them to move.
5.2 Ease of Use
An interesting observation was that users said they “knew the goal
was always to protect the butterfly,” which was a gameplay theme

Figure 8: The speed of the user’s handheld controller and headset as
they attempt to catch the virtual butterfly avatar during the lateral arm
raise mini-game. (Green is the speed of the Weak Arm Controller,
Red is the speed of the Butterfly, Blue is the speed of the headset)
Figure 9: The acceleration of the user’s handheld controller as they
attempt to catch the virtual butterfly avatar during the lateral arm raise
mini-game. Users with high scores and large fluctuations in acceler-
ation are able to react quickly to in-game obstacles, suggesting a
higher level of control and strength than those who cannot. (Green is
the acceleration of the Weak Arm Controller)
added when brainstorming ideas with potential users. Creating in-
game goals which center around an archetypal emotional response
may have generated a quicker understanding of gameplay mechan-
ics in the evaluated users. When asked what they thought of the
difficulty, almost all participants thought it was appropriately chal-
lenging. This makes sense as the game‘s difficulty was adjusted
based on their first gameplay before and during the tutorial period.
By automating all control and dynamic elements of both CRUX
and PBF, there was an increase in compliance when performing mo-
tion primitives as well as reducing the contention between the suit
and virtual reality. Users tended to agree that they noticed the exo-
suit was assisting them smoothly as if part of the gameplay, due to
the mimetic control, suggesting that automating CRUX user-input
may have benefited players immersion.
5.3 Comfort
Users asked for greater ventilation in CRUX, due to the form-
fitting style of the neoprene. Additionally, since CRUX was proto-
typed using commercially available wetsuit neoprene, not all users
equally fit the base layer. The majority of the participants still re-
sponded that they enjoyed playing the minigames with CRUX as
seen in Figure 5.
Through observation and commentary, no one seemed dis-
pleased with the aesthetics of the minigames. Some users com-
plimented the aesthetics, citing the goal of butterfly protection as
“fun,” “engaging,” and even “meditative.” This is particularly excit-
ing news since many people unfamiliar with video games are often
intimidated or dissuaded from immersive virtual environments, es-
pecially considering that five out of nine participants responded that
they dislike video games. The user who mentioned that the games
were “engaging” initially discussed their reservations towards video
games and how they preferred “real things.” When users are psy-
chologically comfortable with a system, they are more likely to ben-
efit from it.
From these observations and recordings, we have found that
most users believe PBF was useful in helping them move their
arms as seen in Figure 5. Combining CRUX’s master-slave sys-
tem and PBF’s butterfly protection mechanics actively encouraged
players to perform the required movements while being distracted
from their physical therapy as agreed by six out of nine users (and
one additional neutral response). These users found PBF enjoyable
and agreed that the game did distract themselves from the exertion
of physical moving. Seven out of nine users stated that they would
regularly use PBF & CRUX in the future for exercises if it were
available to them.
5.4 Discussion
Both the quantitative and qualitative data generated suggested that
in the short term, the system can help users achieve in the moment
tasks by augmenting users upper body strength to enable them to
move their upper limb more easily. In the long term, the paired-
system serves as a boost to help them train their weaker upper arm
and make it stronger over time. In the case of stroke survivors from
SDLC who suffer from neglect syndrome, it additionally helped
them in recognizing what independently driven movement on the
neglected side feels like again. Qualitative lessons learning suggest
the following:
•The exosuit must afford independence on behalf of the user.
In the gameplay sessions, the experimenters helped users to
don and off the suit. If this suit is indeed to be used as a
home-based exercise system, users must be able to put on and
take off the suit by themselves, which at the moment is still
challenging.
•Using a neoprene top as the base to CRUX limited those who
could wear it and made it hot when worn over an extended pe-

riod and difficult to adhere. In future iterations, exosuits like
CRUX needs to be more easily customizable, for example by
replacing neoprene in the exosuit with elastane (e.g., Lycra).
•Future iterations of robotic support should make the augmen-
tation slightly more powerful, so the aid from CRUX is more
apparent to users.
•Given that the PBF and CRUX are new to players, they must
be acquainted with the technologies individually. When given
even just a few minutes to become acquainted, players made
much better use of the CRUX when playing PBF and were
less overwhelmed.
Through the study, a set of design guidelines was compiled for
other practitioners of wearables and VR games to augment upper
limb movements and motivate exercises, especially in older adults
and people with motor impairment:
•Exosuit needs to be designed to be more flexible to fit a variety
of body sizes and shapes.
•Exosuit augmentation needs to be noticeable, perhaps by
adding an in-game UI element such as the bubble lighting up
when the suit is activated. This will hopefully keep the user
immersed while also providing exosuit control indication to
the user.
•Reduce various forms of stimulation to the minimum with-
out affecting the exercise goals without breaking immersion.
Feedback and color variations went unnoticed by some of the
users as they were not related directly to their goals.
•Stay within the appropriate difficulty of the users in terms of
game speed, ranges of motion, etc.
•The user feedback suggests that the users enjoyed protect-
ing the butterfly, as it caused an “emotional attachment.”
Emotion-driven immersion suggests a powerful tool in cre-
ating engaging experiences.
5.5 Limitations
One user responded “Disagree” for questions Q2 (Enjoyable
Game), Q4 (Distracted from Pain) and Q5 (Future Use) from Figure
5. The user commented that the novelty of VR was fun, but con-
tinuously protecting a butterfly would become boring if they used
it for regular therapy. The user requested that future games should
have a compelling storyline to keep them interested and motivated.
The intent of PBF was preliminary feasibility, and future games
should be developed and studied based on lessons learned, trying
out game mechanics, and the targeted therapy desired. Concerns
that this user poses should be investigated in subsequent games to
determine if specific game mechanics are more preferred by users
than others.
Only three motion primitives were explored and converted into
VR through PBF. Specifically, they were the Lateral Shoulder
Raise, Forward Arm Raise, and Horizontal Shoulder Rotation. Fu-
ture studies should incorporate a more variety of motion primitives
to catalyze potential benefits to the users’ potential mobility im-
provements. There may also be potential in incorporating com-
mon assessment tools into the VR environment for automated as-
sessment such as the Apley’s scratch test (Shoulder Mobility) [12],
Wolf Motor Function Test (Upper-Extremity Mobility) [29], Fugl-
Meyer Assessment (General Motor Ability) [13]; all of which as-
sess motion primitives for active daily living.
Furthermore, this study would benefit from a larger sample size
of the users, more therapists could be involved, and more stimuli
and testing must be done to determine feasibility, compliance, and
design further. There is a possible novelty effect of the VR, where
most of these users were exposed to the VR for their very first time.
The results suggest that the game design may account for the nov-
elty effect. However, a long term study must be done to address
these possibilities adequately. Subsequently, a long term study is
being planned with local hospitals in Santa Cruz, California, to ex-
plore the effects of Project Butterfly with CRUX further. An IRB
protocol for such is currently under review for approval.
6 CONCLUSIONS AND FUTURE WORK
Project Butterfly reports on the design and evaluation of a unique
VR experience paired with a soft body robotic wearable exosuit.
The pair of these technologies have been developed as a novel ex-
perience to rehabilitate upper-extremities. CRUX reduces the bur-
den and rigidity experienced by users of traditional wearable robots
through its softer, more structurally compliant constitution. How-
ever, a material other than neoprene should be used to make it
more comfortable and less likely to overheat the user. To tailor
to a more significant number of DoF, the designed VR game, PBF,
is aimed at focusing on motion primitives expressible in soft ex-
oskeletons – actions which the healthy arm can perform that can be
combined and permuted into all upper-extremity movement. PBF
thus serves as a motivator for the user to complete virtual objec-
tives and consequently, actual motions. The completion of these
objectives is assisted through CRUX’s augmentation of their up-
per body strength to perform the game-specific movements. When
evaluating the baseline feasibility of PBF and CRUX in augmenting
and promoting proper arm movement as defined by the established
motion primitives, most users were able to complete appropriately
challenging arm movements, suggesting that PBF and CRUX gave
users suitable strength of their augmented arm. Additionally, the
system’s ease-of-use and comfort were analyzed, and most users
felt that they were confident capitalizing on the therapy system.
Virtual reality paired exosuits could prove useful to make engag-
ing therapy for users with upper-extremity impairment. For more
significant impact, designing a new exosuit peripheral and increas-
ing more types of accompanying VR minigames which augment
various muscle groups (i.e., pectoral muscles and dorsal muscles)
can further improve the rehabilitation. These muscle groups sup-
port upper-extremity actions and strengthening them could bolster
arm muscles as a result. In a similar vein, using new materials for
a future iteration of a soft exosuit focused for VR could make this
technology more comfortable and accessible.
Furthermore, the real-time data produced from the HTC Vive and
Vive Tracking units can be integrated with the exosuit. Therapists
and users may also potentially benefit from further data extrapola-
tion with the HTC Vive. A complete view of a user’s body, achiev-
able through more precise motion tracking and inverse kinematics,
and could identify confounding postural issues, such as slouched
backs and other movement biases, which a physical therapist would
want to be aware of during gameplay. A long term study is being
planned with local hospitals to explore the effects of Project Butter-
fly with a next-generation CRUX design that accommodates more
body types, sizes, and weight. Finally, with the plethora of posi-
tional and behavioral data output produced from this VR experi-
ence, there is potential to integrate machine learning protocols and
AI to optimize suit controls and game difficulty to improve rehabil-
itation results.
The baseline feasibility, ease of use, and comfort created by syn-
ergizing an immersive physio-therapy VR game with an actuated
soft robotic exosuit had promising results in the potential future
of a more accessible, affordable, and personalized rehabilitation.
More research is needed to expand upon the preliminary work of
this study, discover best practices of soft exosuit integrated VR, and
validate clinical utility. Subsequently, there are more butterflies to
follow on the path ahead.

ACKNOWLEDGEMENTS
This material is based upon work supported by the National Science
Foundation under Grant No. #1521532. Any opinions, findings,
and conclusions or recommendations expressed in this material are
those of the author(s) and do not necessarily reflect the views of the
National Science Foundation. The authors would like to thank the
following UC Santa Cruz affiliates for their support: John McIn-
tyre, Pattawong Pansodtee, Ocean Hurd, Dustin Halsey, Evanjelin
Mahmoodi, Noah Brown, and Dr. Daniel Goodman Shapiro. The
authors additionally thank the participants of this study for their
time and effort.
REFERENCES
[1] Virtual reality for consumer markets - head-mounted displays, acces-
sory devices, and consumer virtual reality content: Global market
analysis and forecasts. report, Tractica Research, Jan 2016.
[2] N. Ahmad, R. A. R. Ghazilla, N. M. Khairi, and V. Kasi. Reviews on
various inertial measurement unit (imu) sensor applications. Interna-
tional Journal of Signal Processing Systems, 1(2):256–262, 2013.
[3] A. Alvara. Development of an exosuit for rehabilitative use and upper
arm force amplification. In 2017 AAAS Annual Meeting (February
16-20, 2017). aaas, 2017.
[4] R. M. Ba ˜
nos, C. Botella, M. Alca ˜
niz, V. Lia˜
no, B. Guerrero, and
B. Rey. Immersion and emotion: their impact on the sense of pres-
ence. CyberPsychology & Behavior, 7(6):734–741, 2004.
[5] M. S. Cameir˜
ao, S. Berm´
udez, and P. Verschure. Virtual reality based
upper extremity rehabilitation following stroke: a review. Journal of
CyberTherapy & Rehabilitation, 1(1):63–74, 2008.
[6] F. J. Deconinck, A. R. Smorenburg, A. Benham, A. Ledebt, M. G.
Feltham, and G. J. Savelsbergh. Reflections on mirror therapy: a sys-
tematic review of the effect of mirror visual feedback on the brain.
Neurorehabilitation and Neural Repair, 29(4):349–361, 2015.
[7] J. Diemer, G. W. Alpers, H. M. Peperkorn, Y. Shiban, and
A. M¨
uhlberger. The impact of perception and presence on emotional
reactions: a review of research in virtual reality. Frontiers in psychol-
ogy, 6, 2015.
[8] A. Elor, S. Kurniawan, and M. Teodorescu. Towards an immer-
sive virtual reality game for smarter post-stroke rehabilitation. In
2018 IEEE International Conference on Smart Computing (SMART-
COMP), pages 219–225. IEEE, 2018.
[9] A. Elor, M. Teodorescu, and S. Kurniawan. Project star catcher:
A novel immersive virtual reality experience for upper limb reha-
bilitation. ACM Transactions on Accessible Computing (TACCESS),
11(4):20, 2018.
[10] A. N. V. Follow. Virtual reality (vr) continuum - amp new ven-
tures, Dec 2016. https://www.slideshare.net/ampnewventures/virtual-
reality-vr-continuum-amp-new-ventures.
[11] A. Frisoli, F. Salsedo, M. Bergamasco, B. Rossi, and M. C. Car-
boncini. A force-feedback exoskeleton for upper-limb rehabilitation
in virtual reality. Applied Bionics and Biomechanics, 6(2):115–126,
2009.
[12] F. Garc´
ıa-Muro, ´
A. L. Rodr´
ıguez-Fern´
andez, and A. Herrero-de Lu-
cas. Treatment of myofascial pain in the shoulder with kinesio taping.
a case report. Manual therapy, 15(3):292–295, 2010.
[13] D. J. Gladstone, C. J. Danells, and S. E. Black. The fugl-meyer assess-
ment of motor recovery after stroke: a critical review of its measure-
ment properties. Neurorehabilitation and neural repair, 16(3):232–
240, 2002.
[14] H. Grillon, F. Riquier, B. Herbelin, and D. Thalmann. Virtual reality as
a therapeutic tool in the confines of social anxiety disorder treatment.
International journal on disability and human development, 5(3):243–
250, 2006.
[15] D. Gromala, X. Tong, A. Choo, M. Karamnejad, and C. D. Shaw.
The virtual meditative walk: virtual reality therapy for chronic pain
management. In Proceedings of the 33rd Annual ACM Conference on
Human Factors in Computing Systems, pages 521–524. ACM, 2015.
[16] A. Hanaor. Engineering properties of double-layer tensegrity grids.
SpatialStructures at the Tum oftheMillenium, pages 195–200, 1991.
[17] H. G. Hoffman, G. T. Chambers, W. J. Meyer, L. L. Arceneaux, W. J.
Russell, E. J. Seibel, T. L. Richards, S. R. Sharar, and D. R. Patterson.
Virtual reality as an adjunctive non-pharmacologic analgesic for acute
burn pain during medical procedures. Annals of Behavioral Medicine,
41(2):183–191, 2011.
[18] S. J. Housman, V. Le, T. Rahman, R. J. Sanchez, and D. J. Reinkens-
meyer. Arm-training with t-wrex after chronic stroke: preliminary
results of a randomized controlled trial. In Rehabilitation Robotics,
2007. ICORR 2007. IEEE 10th International Conference on, pages
562–568. IEEE, 2007.
[19] L. M. Howden and J. A. Meyer. Age and sex composition: 2010.
2010 Census Briefs, US Department of Commerce, Economics and
Statistics Administration. US CENSUS BUREAU, 2010.
[20] HTC-Corporation. Vive vr system. Vive, November 2018.
https://www.vive.com/us/product/vive-virtual-reality-system/.
[21] D. Jack, R. Boian, A. S. Merians, M. Tremaine, G. C. Burdea, S. V.
Adamovich, M. Recce, and H. Poizner. Virtual reality-enhanced
stroke rehabilitation. IEEE transactions on neural systems and re-
habilitation engineering, 9(3):308–318, 2001.
[22] R. Koopman and L. J. van Loon. Aging, exercise, and muscle protein
metabolism. Journal of Applied Physiology, 106(6):2040–2048, 2009.
[23] O. Kreylos. Lighthouse tracking examined. A developer’s perspec-
tive on immersive 3D computer graphics, May 2016. http://doc-
ok.org/?p=1478.
[24] S. Lessard, P. Pansodtee, A. Robbins, L. B. Baltaxe-Admony, J. M.
Trombadore, M. Teodorescu, A. Agogino, and S. Kurniawan. Crux:
A compliant robotic upper-extremity exosuit for lightweight, portable,
multi-joint muscular augmentation. In Rehabilitation Robotics
(ICORR), 2017 International Conference on, pages 1633–1638. IEEE,
2017.
[25] S. Lessard, P. Pansodtee, A. Robbins, J. M. Trombadore, S. Kurni-
awan, and M. Teodorescu. A soft exosuit for flexible upper-extremity
rehabilitation. IEEE Transactions on Neural Systems and Rehabilita-
tion Engineering, 2018.
[26] K. R. Lohse, C. G. Hilderman, K. L. Cheung, S. Tatla, and H. M.
Van der Loos. Virtual reality therapy for adults post-stroke: a sys-
tematic review and meta-analysis exploring virtual environments and
commercial games in therapy. PloS one, 9(3):e93318, 2014.
[27] M. E. Michielsen, M. Smits, G. M. Ribbers, H. J. Stam, J. N. Van
Der Geest, J. B. Bussmann, and R. W. Selles. The neuronal correlates
of mirror therapy: an fmri study on mirror induced visual illusions in
patients with stroke. Journal of neurology, neurosurgery & psychiatry,
82(4):393–398, 2011.
[28] N. Morina, H. Ijntema, K. Meyerbr ¨
oker, and P. M. Emmelkamp. Can
virtual reality exposure therapy gains be generalized to real-life? a
meta-analysis of studies applying behavioral assessments. Behaviour
research and therapy, 74:18–24, 2015.
[29] D. M. Morris, G. Uswatte, J. E. Crago, E. W. Cook III, and E. Taub.
The reliability of the wolf motor function test for assessing upper ex-
tremity function after stroke. Archives of physical medicine and reha-
bilitation, 82(6):750–755, 2001.
[30] H. Mousavi Hondori and M. Khademi. A review on technical and clin-
ical impact of microsoft kinect on physical therapy and rehabilitation.
Journal of Medical Engineering, 2014, 2014.
[31] M. V. Nararro-Haro, H. G. Hoffman, A. Garcia-Palacios, M. Sampaio,
W. Alhalabi, K. Hall, and M. Linehan. The use of virtual reality to fa-
cilitate mindfulness skills training in dialectical behavioral therapy for
borderline personality disorder: a case study. Frontiers in psychology,
7, 2016.
[32] D. C. Niehorster, L. Li, and M. Lappe. The accuracy and precision of
position and orientation tracking in the htc vive virtual reality system
for scientific research. i-Perception, 8(3):2041669517708205, 2017.
[33] V. S. Ramachandran and D. Rogers-Ramachandran. Synaesthesia in
phantom limbs induced with mirrors. Proceedings of the Royal Society
of London B: Biological Sciences, 263(1369):377–386, 1996.
[34] B. O. Rothbaum, M. Price, T. Jovanovic, S. D. Norrholm, M. Gerardi,
B. Dunlop, M. Davis, B. Bradley, E. J. Duncan, A. Rizzo, et al. A
randomized, double-blind evaluation of d-cycloserine or alprazolam
combined with virtual reality exposure therapy for posttraumatic stress
disorder in iraq and afghanistan war veterans. American Journal of

Psychiatry, 171(6):640–648, 2014.
[35] M. Rus-Calafell, J. Guti´
errez-Maldonado, and J. Ribas-Sabat´
e. A vir-
tual reality-integrated program for improving social skills in patients
with schizophrenia: a pilot study. Journal of behavior therapy and
experimental psychiatry, 45(1):81–89, 2014.
[36] R. D. Seidler, J. L. Alberts, and G. E. Stelmach. Changes in multi-joint
performance with age. Motor control, 6(1):19–31, 2002.
[37] R. D. Seidler, J. A. Bernard, T. B. Burutolu, B. W. Fling, M. T. Gor-
don, J. T. Gwin, Y. Kwak, and D. B. Lipps. Motor control and aging:
links to age-related brain structural, functional, and biochemical ef-
fects. Neuroscience & Biobehavioral Reviews, 34(5):721–733, 2010.
[38] R. D. Seidler-Dobrin, J. He, and G. E. Stelmach. Coactivation to re-
duce variability in the elderly. Motor control, 2(4):314–330, 1998.
[39] Y. Shiban, I. Schelhorn, P. Pauli, and A. M ¨
uhlberger. Effect of com-
bined multiple contexts and multiple stimuli exposure in spider pho-
bia: a randomized clinical trial in virtual reality. Behaviour research
and therapy, 71:45–53, 2015.
[40] N. Shkuratova, M. E. Morris, and F. Huxham. Effects of age on bal-
ance control during walking. Archives of physical medicine and reha-
bilitation, 85(4):582–588, 2004.
[41] F. Soffel, M. Zank, and A. Kunz. Postural stability analysis in virtual
reality using the htc vive. In Proceedings of the 22nd ACM Conference
on Virtual Reality Software and Technology, pages 351–352. ACM,
2016.
[42] H. Thieme, J. Mehrholz, M. Pohl, J. Behrens, and C. Dohle. Mirror
therapy for improving motor function after stroke. Stroke, 44(1):e1–
e2, 2013.
[43] ValveSoftware. Valvesoftware/openvr. GitHub, Jul.
https://github.com/ValveSoftware/openvr.
[44] H. Wang and T. Cochrane. Mobility impairment, muscle imbalance,
muscle weakness, scapular asymmetry and shoulder injury in elite
volleyball athletes. Journal of sports medicine and physical fitness,
41(3):403, 2001.
[45] N. S. Ward. Compensatory mechanisms in the aging motor system.
Ageing research reviews, 5(3):239–254, 2006.
[46] M. Wehner, B. Quinlivan, P. M. Aubin, E. Martinez-Villalpando,
M. Baumann, L. Stirling, K. Holt, R. Wood, and C. Walsh. A
lightweight soft exosuit for gait assistance. In Robotics and Automa-
tion (ICRA), 2013 IEEE International Conference on, pages 3362–
3369. IEEE, 2013.