Content uploaded by Sebastian Koenig
Author content
All content in this area was uploaded by Sebastian Koenig on Aug 18, 2017
Content may be subject to copyright.
Clinical Virtual Reality
1
In Press: Neuropsychology
Is Clinical Virtual Reality Ready for Primetime?
Albert “Skip” Rizzo, Ph.D., Director of the Medical Virtual Reality Lab
University of Southern California Institute for Creative Technologies
Sebastian Thomas Koenig, Ph.D., Director, Human Interface Technology Engineer
Katana Simulations Pty Ltd, Adelaide, Australia
Abstract
Objective: Since the mid-1990s, a significant scientific literature has evolved regarding the
outcomes from the use of what we now refer to as Clinical Virtual Reality (VR). This use of
VR simulation technology has produced encouraging results when applied to address
cognitive, psychological, motor, and functional impairments across a wide range of clinical
health conditions. This article addresses the question, “Is Clinical VR Ready for Primetime?”
Method: After a brief description of the various forms of VR technology, we discuss the
trajectory of Clinical VR over the last 20 years and summarize the basic assets that VR
offers for creating clinical applications. The discussion then addresses the question of
readiness in terms of the theoretical basis for Clinical VR assets, the research to date, the
pragmatic factors regarding availability, usability, and costs of Clinical VR content/systems,
and the ethical issues for the safe use of VR with clinical populations. Results: Our review of
the theoretical underpinnings and research findings to date leads to the prediction that
Clinical VR will have a significant impact on future research and practice. Pragmatic issues
that can influence adoption across many areas of psychology also appear favorable, but
professional guidelines will be needed to promote its safe and ethical use.
Conclusions: While there is still much research needed to advance the science in this area,
we strongly believe that Clinical VR applications will become indispensable tools in the
toolbox of psychological researchers and practitioners and will only grow in relevance and
popularity in the future. Keywords: Clinical Virtual Reality, Psychology, Rehabilitation,
Neuropsychology
Public Significance Statement: Virtual Reality (VR) technology offers new opportunities for
clinical research, assessment, and intervention. Advances in the underlying VR-enabling
technologies and methods can now support the creation of low-cost, yet sophisticated,
immersive and interactive VR systems, capable of running on consumer-level computing
devices. It is predicted that the clinical use of VR will have a significant impact on mental
healthcare in areas where the research demonstrates added value.
Acknowledgments: The efforts described here have been variously sponsored by the Army
Research Lab, U.S. Army, U.S. Air Force, Defense Advanced Research Projects Agency,
Defense Center of Excellence, Infinite Hero Foundation, Office of Naval Research, and the
Telemedicine and Advanced Technology Center. Any opinions, content or information
presented does not necessarily reflect the position or the policy of the United States
Government or Foundations, and no official endorsement should be inferred.
© 2017, American Psychological Association. This paper is not the copy of record and
may not exactly replicate the final, authoritative version of the article. Please do not
copy or cite without authors permission. The final article will be available, upon
publication, via its DOI: 10.1037/neu0000405
Clinical Virtual Reality
2
Introduction
Virtual Reality (VR) technology offers new opportunities for clinical research, assessment,
and intervention. Since the mid-1990s, VR-based testing, training, and treatment
approaches have been developed by clinicians and researchers that would be difficult, if not
impossible, to deliver using traditional methods. During this time, a large (but still maturing)
scientific literature has evolved regarding the outcomes and effects from the use of what we
now refer to as Clinical VR applications targeting cognitive, psychological, motor, and
functional impairments across a wide range of clinical health conditions. Moreover,
continuing advances in the underlying enabling technologies for creating and delivering VR
applications have resulted in its widespread availability as a consumer product, sometimes
at a very low cost. So, when one studies the scientific literature, examines the evolving state
of the technology, and observes the growing enthusiasm for VR in the popular culture, a big
question emerges for psychology, neuropsychology, and rehabilitation: “Is Clinical VR ready
for Primetime?”. While many well-thought-out VR-based research prototypes have
generated a provocative scientific literature and a fair share of excitement, how far are we
away from mainstream availability, adoption, and implementation? To address this question,
the current article will briefly describe VR technology, discuss the trajectory of Clinical VR
over the last 20 years, and summarize the assets that VR offers for creating clinical
applications. The discussion section addresses the question of readiness based on an
assessment of the theoretical basis for VR relevant to clinical applications, the science to
date in specific areas of use, the pragmatic factors regarding availability, usability, and costs
of Clinical VR content/systems, and the ethical issues for the safe use of VR with clinical
populations. Some of the discussion in the current paper includes topics that have been
discussed in previous papers, which may be consulted for additional reading (Lange,
Koenig, Chang, McConnell, Suma, Bolas, & Rizzo, 2012; Rizzo, Buckwalter, & Neumann,
1997; Rizzo, Schultheis, Kerns, & Mateer, 2004).
What is Virtual Reality?
The concept and definition of Virtual Reality has been subject to debate by scientists and
clinicians over the years. VR has been very generally defined as a way for humans to
visualize, manipulate, and interact with computers and extremely complex data
(Aukstakalnis & Blatner, 1992). From this baseline perspective, VR can be seen as an
advanced form of human-computer interaction (Rizzo et al., 1997) that allows a user to more
naturally interact with computers beyond what is typically afforded with standard mouse and
keyboard interface devices. Moreover, some VR formats enable users to become immersed
within synthetic computer-generated virtual environments. However, VR is not defined or
limited by any one technological approach or hardware set up. The creation of an engaged
VR user experience can be accomplished using combinations of a wide variety of interaction
devices, sensory display systems, and content presented in the virtual environment. Thus,
there are three common variations for how VR can be created and used.
Non-immersive VR is the most basic format and is similar to the experience of
someone playing a modern computer or console videogame. Content is delivered on a
standard flat-screen computer monitor or TV with no occlusion of the outside world. Users
interact with three-dimensional (3D) computer graphics using a gamepad, a joystick,
specialized interface devices (from a treadmill to a handheld Nintendo Wii remote), as well
as basic mouse or keyboard. Modern computer games that support user interaction and
navigation within such 3D worlds, even though presented on a flat-screen display, can
technically be referred to as VR environments.
Immersive VR can be produced by the integration of computers, head-mounted
displays (HMDs), body-tracking sensors, specialized interface devices, and 3D graphics.
Clinical Virtual Reality
3
These set-ups allow users to operate in a computer-generated simulated world that changes
in a natural or intuitive way with head and body motion. Using an HMD that occludes the
user’s view of the outside world, an engaged immersive virtual experience employs head
and body-tracking technology that senses the user’s position and movement and sends that
information to a computing system that can update the sensory stimuli presented to the user
in near real-time, contingent on user activity. This serves to create the illusion of being
immersed “in” a virtual space, within which users can interact. When immersed within
computer-generated visual imagery and sounds of a simulated virtual scene, user interaction
produces an experience that corresponds to what the individual would see and hear if the
scene were real. Another less common method for producing immersive VR experiences
uses stereoscopic projection screens arrayed around a user in various configurations.
Sometimes six-walled projection rooms known as cave automatic virtual
environments (CAVEs) (Cruz-Neira et al., 1993; DeFanti et al., 2011) are used that allow for
interaction in a less encumbered, wide field of view simulated environment for multiple
concurrent users. However, such CAVE systems are more costly and complex, and are
typically beyond the practical resources of most clinical service providers and/or basic
researchers.
Regardless of the technical approach, the key aim of these immersive systems is to
perceptually replace the outside world with the virtual world to psychologically engage users
with simulated digital content designed to create a specific user experience. Immersive VR
(most commonly delivered in an HMD) is typically the choice for applications where a
controlled stimulus environment is desirable for constraining a user’s perceptual experience
within a specific synthetic world. This format has been often used in Clinical VR applications
for anxiety disorder exposure therapy, analgesic distraction for patients undergoing acutely
painful medical procedures, and in the cognitive assessment of users to measure
performance under a range of systematically delivered challenges and distractions.
A Very Brief History of Clinical Virtual Reality
VR has recently captured the public’s imagination as the next big thing in media. However,
the technology for creating VR experiences and its clinical use has existed for at least two
decades. During the 1990s the growing availability and rapid evolution of personal
computing drove the global adoption of innovative digital technologies for the purposes of
productivity enhancement, communication, and social interaction. At the same time, the
advances in modern computing power required to automate processes and store/analyze
vast quantities of data did not go unnoticed by clinical researchers and providers, who
imagined and prototyped novel behavioral healthcare applications. Primordial efforts from
this period can be seen in developments that aimed to use personal computers to enhance
productivity in patient documentation and record-keeping, automate the administration and
scoring of psychometric tests, and in the computer-delivery of cognitive training/rehabilitation
activities (Robertson, 1990). As well, with the rapid improvements in internet connectivity
seen during the 1990s, the idea of enhancing access to care via internet-based teletherapy
(Cuijpers, van Straten, & Andersson, 2008; Putrino, 2014; Rizzo, Strickland, and Bouchard,
2004; Stamm, 1998) and self-help cognitive behavioral programs (Carlbring et al., 2001
Spek et al., 2007) was explored. Since that time, the impact of computer and information
technology on society has grown dramatically. This can be seen in the current adoption and
growing demand for mobile devices, high speed network access, smart televisions, social
media sites, photorealistic digital games, wearable behavioral sensing devices, and now, the
2nd coming of Virtual Reality. Such consumer-driven technologies, thought of as visionary
just 10 years ago, have now become increasingly common and essential fixtures in the
digital landscape of a global society.
Clinical Virtual Reality
4
The idea of using VR for clinical purposes was first recognized in the early-to-mid 90s
with initial efforts to design VR simulations to deliver exposure therapy for specific phobias
(e.g., fear of heights, flying, spiders, and public speaking) (Lamson, 1994; Rothbaum et al.,
1995) and for cognitive rehabilitation (Brown et al., 1998; Cromby et al.,1996; Pugnetti et al.
1995; Rizzo, 1994). The compelling feature that drove this innovation was that VR could
leverage computing beyond its cardinal purpose - the automation of processes - to instead
use computers to produce and deliver sensory stimuli for the creation of embodied,
interactive, and immersive user experiences. This was recognized early on in the visionary
article “The Experience Society” by VR pioneer, Myron Krueger (1993), in his prophetic
statement that, “…Virtual Reality arrives at a moment when computer technology in general
is moving from automating the paradigms of the past, to creating new ones for the future.”
(p. 163). Viewed from this perspective, VR afforded the opportunity to create highly realistic,
interactive, and systematically controllable stimulus environments that users could be
immersed in, and interact with, for human performance measurement and training, as well
as clinical assessment and intervention. Clinicians and scientists who were drawn to the idea
of VR during this time were often guided by the belief that its core features and assets could
support the development of innovative clinical approaches that were not possible with
existing traditional methodologies.
The added value for such VR systems can be seen in the technology’s capacity to
create systematic human testing, training, teaching, and treatment environments that allow
for the precise control of complex, multi-sensory, dynamic 3D stimulus presentations. Within
such simulations, sophisticated behavioral interaction is possible and such physical activity
can be precisely tracked, recorded, and analyzed to study human performance and
behavior. Much like an aircraft simulator serves to test and train piloting ability under a wide
variety of controlled conditions, VR can be used to create relevant simulated environments
where the assessment and treatment of cognitive, emotional, and sensorimotor processes
can take place under stimulus conditions that are not easily deliverable and controllable in
the physical world. When combining VR’s stimulus control features with the ability to
immerse users in functional and ecologically relevant virtual environments, early Clinical VR
scientists envisioned a fundamental advancement in how human assessment and
intervention could be addressed. It could be conjectured that this “Ultimate Skinner Box”
perspective was what human experimental researchers and clinicians had always strived for,
but were limited by the constraints imposed by the laws of physics that govern physical
reality. This “vision” drove the enthusiasm and innovative efforts seen in the fledgling area of
Clinical VR in the 1990s.
Unfortunately, many technical challenges needed to be overcome before this vision
of Clinical VR could be achieved. When discussion of the potential use of VR for human
research and clinical intervention first emerged in the 90s, the technology needed to deliver
on this vision was not sufficiently mature. Consequently, during these early years VR
suffered from a somewhat imbalanced “expectation-to-delivery” ratio, as most who explored
VR systems during that time will attest. Computers were too slow, 3D graphics were
primitive, and user interface devices were awkward, requiring more effort than users were
willing to expend to learn how to operate them effectively. Moreover, VR HMDs were costly,
bulky, and had limited tracking speed, resolution, and field of view. As a consequence, VR
commenced its “nuclear winter” period in 1995 as the public became disenchanted with the
quality of a typical VR experience and generally viewed it as a failed technology. Thus, VR
languished for many years in what the Gartner Group has termed “the trough of
disillusionment”, the stage in technology adoption that often follows the “peak of inflated
expectations” period described in their regularly updated “Hype Cycle for Emerging
Technologies” (Gartner, 2016).
In spite of these technical challenges, the core vision of Clinical VR was sound and
VR “enthusiasts” continued to pursue the research and development needed to advance the
Clinical Virtual Reality
5
technology and document its added clinical value. And, over the last 22 years, the
technology for creating VR systems gradually caught up with the vision of creating
compelling, usable, and effective Clinical VR applications. This has been possible in large
part due to the gradual, but continuous, advances in the underlying VR-enabling
technologies and methods (e.g., computational speed, computer graphics, panoramic video,
audio/visual/haptic displays, natural user interfaces, tracking sensors, speech and language
processing, artificial intelligence, virtual human agents, authoring software, etc.). Such
advances have resulted in the technical capability needed to support the creation of low-
cost, yet sophisticated, immersive, and interactive VR systems, capable of running on
commodity-level computing devices. In part driven by the digital gaming and entertainment
sectors, and a near insatiable global demand for mobile and interactive networked consumer
products, these advances in technological “prowess” and accessibility have provided the
hardware and software platforms needed to produce more adaptable and high-fidelity
Clinical VR scenarios. This has created a state of affairs where Clinical VR applications can
now usefully leverage the interactive and immersive assets that VR affords as the
technology continues to get faster, better, and cheaper moving forward toward the third
decade of the 21st Century! Moreover, since the 1990s a significant scientific literature has
evolved, almost under the radar, reporting many positive outcomes across a range of clinical
applications that have leveraged the assets provided by VR (Botella et al., 2015; Dascal et
al., 2017; Freeman et al., 2017; Hoffman et al., 2011; Howard, 2017; Maples-Keller et al.,
2017; Morina et al., 2015; Rizzo et al., 2015ab; Rose et al., 2005; Slater & Sanchez-Vives,
2016).
A short list of the areas where Clinical VR has been usefully applied includes fear
reduction in persons with specific phobias (Morina et al., 2015; Opris et al., 2012; Parsons
and Rizzo, 2008a; Powers and Emmelkamp, 2008), treatment for posttraumatic stress
disorder (PTSD), depression, and paranoid delusions (Beidel et al., 2017; Botella et al.,
2015; Difede et al., 2007, 2013; Falconer, et al., 2016; Freeman et al., 2016; Rizzo et al.,
2010, 2013, 2015a; Rothbaum et al., 2001, 2014), discomfort reduction in cancer patients
undergoing chemotherapy (Chirico et al., 2016; Schneider et al., 2010), acute pain reduction
during wound care and physical therapy with burn patients (Hoffman et al., 2011) and in
other painful procedures (Gold et al., 2006; Mosadeghi et al., 2016; Tashjian, et al., 2017;
Trost et al., 2015), body image disturbances in patients with eating disorders (Riva, 2011),
navigation and spatial training in children and adults with motor impairments (John et al.,
2017; Rizzo et al., 2004; Stanton et al., 1998), functional skill training and motor
rehabilitation in patients with central nervous system dysfunction (e.g., stroke, traumatic
brain injury (TBI), spinal cord injury (SCI), cerebral palsy, multiple sclerosis, etc.) (Deutsch &
McCoy, 2017; Holden, 2005; Howard, 2017; Klamroth-Marganska et al., 2014; Lange et al.,
2012; Merians et al., 2002, 2010), and for the assessment and rehabilitation of attention,
memory, spatial skills, and other cognitive functions in both clinical and unimpaired
populations (Bogdanova, Yee, Ho, & Cicerone, 2016; Brooks et al., 1999; Brown et al., 1998;
Matheis et al., 2007; Ogourtsova, Silva, Archambault, & Lamontagne; 2015; Parsons, Rizzo,
Rogers, and York, 2009; Passig, Tzuriel, & Eshel-Kedmi, 2016; Pugnetti et al., 1995; Rizzo,
1994; Rizzo et al., 2006; Rose et al., 2005; Valladares-Rodriguez et al., 2016). To do this,
Clinical VR scientists have constructed virtual airplanes, skyscrapers, spiders, battlefields,
social settings, beaches, fantasy worlds, and the mundane (but highly relevant) functional
environments of the schoolroom, office, home, street, and supermarket. In essence, VR
environments mimicking real or imagined worlds can be applied to engage users in
simulations that support the aims and mechanics of a specific clinical assessment or
therapeutic approach. As a result, there is a growing consensus that VR has now emerged
as a promising tool in many domains of research (Bohil et al., 2011; Larson, Feigon,
Gagiardo, & Dvorkin, 2014) and clinical care (Goldman-Sachs, 2016; Freeman et al., 2016;
Lange et al., 2012; Norcross et al., 2013).
Clinical Virtual Reality
6
Analysis of Clinical VR Assets
What makes Clinical VR so distinctively innovative is that it represents more than a simple
linear extension of existing computer technology for human use. By way of VRs capacity to
immerse a user within an interactive computer-generated simulation, new possibilities exist
that can go beyond the simple automation of previous clinical assessment and intervention
approaches. Nevertheless, in deciding as to whether Clinical VR is ready for primetime, one
needs to consider what features VR offers that may make it especially suited for clinical and
research usage.
On a very general level VR can be seen to foster core processes that are relevant
across a variety of clinical domains. These processes can be briefly summarized as expose
(e.g., exposure therapy for anxiety disorders, PTSD, or addiction treatment), distract (e.g.,
distracting attention away from painful medical procedures to reduce pain perception or
promote discomfort reduction), motivate (e.g., motivating clients in cognitive/physical
rehabilitation to perform repetitive and sometimes boring tasks by embedding them within
game-like contexts), measure (e.g., measuring performance on physical/cognitive
assessment activities), and engage (e.g., generally seen as the captivation of
attention/action that is useful for engaging participation with clinical applications). To
effectively drive these processes in a thoughtful fashion, it is helpful to be aware of the
features and assets that are available for clinical use of VR technology. These assets have
been specifically detailed as they relate to the predecessor field of aviation simulation
technology (Jentsch & Curtis, 2017) and an earlier detailing of these assets for
neuropsychology appeared in Rizzo et al. (2004). However, in view of the rapidly advancing
state of VR technology, a revisiting of its current status is warranted, especially as it pertains
to general clinical applications.
Ecological Relevance
Clinical VR scenarios can be modeled after relevant contexts that exist in everyday life.
Within such simulated environments, it is possible to create activities that mimic challenges
faced by clinical populations, and implement them as part of assessment and intervention
strategies. This has been a guiding feature in Clinical VR development since the 1990s,
leading to the creation of many standard archetypic testing and treatment spaces (e.g.,
homes, offices, classrooms, stores, tall buildings, cars, battlefields, hospital settings, social
gatherings, public speaking auditoriums, etc.). The primary driver for these efforts is the view
that we can better predict or enhance human functioning (e.g., behavioral outcomes,
emotional coping, cognitive/motor task performance) in the real world by providing
systematic and highly controllable assessments and interventions within functionally similar
virtual worlds.
This is particularly relevant in view of the underlying concepts of generalization and
transfer of training that have been “big” issues across all domains of psychology and
rehabilitation. For example, traditional neuropsychological assessment and rehabilitation has
been criticized by some authors (Parsons, Carlew, Magtoto, & Stonecipher, 2015; Rizzo,
Buckwalter, & Neumann , 1997; Sbordone, & Long, 1996; Wilson, 1997) as limited in the
area of ecological validity, that is, the degree of relevance or similarity that a test or training
system has relative to the “real” world (Neisser, 1978). A number of examples illustrate
efforts to enhance the ecological validity of assessment and rehabilitation by designing VR
scenarios that are “replicas” of relevant archetypic functional environments. This has
included the creation of virtual cities (Brown et al., 1998; Costas, Carvalho & de Aragon,
2000; Gamito et al., 2016), supermarkets (Cromby et al., 1996; Josman et al., 2014; Levy et
al., 2015); homes (Koenig, 2012; Rose et al., 2001); kitchens (Christiansen et al., 1998;
Davies et al., 1998; Foloppe et al., 2015; Wall et al., 2017), school environments (Rizzo et
Clinical Virtual Reality
7
al., 2000, 2006; Stanton et al., 1998), workspaces/offices (Koenig et al., 2012; Krch et al.,
2013; Matheis et al., 2007; McGeorge et al., 2001); rehabilitation wards (Brooks et al., 1999)
and even a virtual beach (Elkind et al., 2001). From these efforts, recent reviews have
provided support for the impact of ecologically-relevant Clinical VR applications on real world
treatment outcomes in both clinical psychology (Morina et al., 2015) and in rehabilitation
(Howard, 2017).
While early attempts at the creation of these environments varied significantly in their
level of pictorial or graphical realism, this fidelity factor may be secondary in importance,
relative to the actual activities that are carried out in the environment for determining their
value from an ecological relevance standpoint (cf. Parsons, 2015; Rizzo et al., 2006).
Interestingly, when in a virtual environment, humans often times display a high capacity to
“suspend disbelief” and respond as if the scenario was real. It could be conjectured that the
“ecological value” of a VR task that needs to be performed may be well supported in spite of
limited graphical realism. In essence, as long as the VR scenario “resembles” the real world,
possesses design elements that replicate key real-life challenges, and the system responds
well to user interaction, then the graphical realism can be less important for activating
behavior and emotion. This has especially been observed by clinicians using VR to conduct
exposure-based therapies for anxiety disorders, PTSD, and addiction (Bordnick et al., 2013).
Clients commonly report significant emotional activation in spite of the “cartoonish” nature of
the visual content seen in some VR scenarios. Thus, while a number of the successful VR
scenarios designed for exposure-based therapy of specific phobias would never be mistaken
for the real world, clients experiencing these VR worlds still manifest physiological responses
and report subjective units of discomfort levels that suggest they are responding “as if” they
are in the presence of the feared stimuli (Costanzo et al., 2014; Norrholm et al., 2016;
Wiederhold & Wiederhold, 1998).
The recent advances in computer graphics as seen in modern computer games have
now made the “fidelity” issue less of a concern. As well, the growing popularity of panoramic
360-degree cameras and photogrammetry has provided an affordable means to create
photorealistic content for VR applications. While expectations of computer graphics have
also increased steadily, especially with a younger generation that has grown up with
computer and console games and may be put off by low-quality graphics, perceptually
convincing VR scenarios are now more the norm than the exception in current VR
development. Although it is yet to be documented that increased realism has had an impact
on improving clinical outcomes, the ability to create more compelling visual VR content may,
at the very least, improve face validity and increase user buy-in from patients and clinical
end-users.
Systematic delivery and control of sensory stimuli
One of the cardinal assets of any advanced form of simulation technology involves the
capacity for systematic delivery and control of stimuli. This asset provides significant
opportunities for developing Clinical VR methods. In fact, one could conjecture that the
systematic delivery and control of stimuli in a testing or treatment environment provides the
basic foundation of all human research and clinical methodologies along with the
subsequent capture and analysis of the behavior that occurs in response to those conditions.
In this regard, an ideal match exists between the stimulus delivery assets of VR simulation
systems and the requirements of any clinical assessment and intervention procedure. This
can be seen as a core asset whether one is testing construct-specific cognitive processes
(e.g., selective attention performance contingent on varying levels of stimulus intensity and
distraction) (Rizzo et al., 2006; Mühlberger, 2016), to the complex targeting of more molar
functional behaviors (e.g., planning and initiating the steps to function within a complex office
or home setting) (Keefe et al., 2016; Krch et al., 2013), to the precise titration of anxiety
Clinical Virtual Reality
8
activating content in the service of pacing exposure therapy for the treatment of phobias or
PTSD (Rizzo et al. 2015a; Rothbaum et al., 1995,1999).
Moreover, the precise control over multiple concurrent tasks and presentation of
realistic distractions during these tasks presents a unique opportunity to simulate complex,
lifelike scenarios that is only starting to receive attention in Clinical VR research and
development. This approach stands in stark contrast to the traditional single-construct
exposure to cognitive tasks in distraction-free environments such as a clinician’s office.
This capacity for systematic stimulus control within the context of ecologically
relevant simulations of everyday life (i.e., The Ultimate Skinner Box) for assessment and
intervention purposes is one of the key areas that differentiate Clinical VR from all previous
methodologies. Thus, VR’s stimulus delivery capability has been recognized as a significant
asset for supporting the integration of VR with brain imaging and psychophysiology studies
(Bohil et al., 2011; Chou et al., 2012; Costanzo et al., 2014; Norrholm et al., 2016; Tarr &
Warren, 2002). This is especially relevant for the field of neuropsychology which has been
increasingly integrating advanced neural imaging technologies (e.g., fMRI, DTI, SPECT,
QUEEG, CT, etc.) in its quest for a better accounting of the structure and processes
underlying brain/behavior relationships. In fact, the use of VR in imaging studies has nearly
as long a history as the direct use of the technology for clinical interventions (Astur et al.,
1998).
For example, a VR simulation of the Morris Water Maze test of spatial navigation and
place learning, commonly used with rodents, has generated significant research examining
the role of the hippocampus in human learning (Astur et al., 1998, 2002, 2004). In this
elegant and well-matched use of VR, a human user must navigate a space to find a hidden
platform using visual cues in the surrounding environment. Used in conjunction with fMRI,
the test has been applied to assess place learning performance while concurrently
measuring hippocampal activity. Research with this VR system has reported poorer
performance and decreased activation in health conditions where the hippocampus is
implicated such as with Alzheimer’s disease (Shipman & Astur, 2008), PTSD (Astur et al.,
2006), and schizophrenia (Folley et al., 2010). Other researchers have similarly integrated
VR and brain imaging and have reported, reduced activation of pain-related regions of
interest using VR as a distractor from experimentally induced pain (Hoffman et al., 2006,
2011); changes in brain activation (i.e., amygdala and 3 frontal areas) to VR stimuli following
exposure therapy for PTSD (Roy et al., 2010, 2014); neural predictors of change in emotion
recognition in persons on the autism spectrum using VR social cognition training (Yang et
al., 2017); and cortical reorganization and associated locomotor recovery in chronic stroke
patients with VR game-based rehabilitation (You et al., 2005). In a recent effort to combine a
virtual classroom scenario with Near-Infrared Spectroscopy (NIRS), Blume et al., (2017) in
collaboration with Katana Simulations, created an immersive virtual classroom
neurofeedback training to treat deficits associated with attention-deficit hyperactivity disorder
(ADHD; Blume et al., 2017). A clinical trial is currently evaluating the efficacy of the training,
that utilizes the NIRS signal to control the classroom’s lighting intensity as a feedback
mechanism. It is hypothesized that a training protocol of 15 sessions, containing activation
and deactivation trials, will facilitate self-regulation skills, and improve ADHD symptoms and
motor activity in the participating 90 children with ADHD. Participants are randomly assigned
to the NIRS-based training in the VR classroom, a NIRS-based training in a 2D classroom,
or an electromyogram-based training in VR. This clinical trial is ongoing.
Although head movement is restricted in most brain imaging systems (excluding
NIRS), specialized “magnet-friendly” interaction devices and displays, can still allow users to
engage with dynamic virtual content, albeit the experience is different than a typical
unrestricted VR application. With that limitation acknowledged, the integration of VR as a
tool for delivering complex, interactive stimuli with advanced brain imaging techniques may
Clinical Virtual Reality
9
support neuropsychology in reaching its stated purpose, that of determining unequivocal
brain-behavior relationships, in addition to advancing the state of the science in other clinical
disciplines.
Delivery of strategic real-time performance feedback
VR simulations can be designed to provide users with feedback as to the state of their
performance during task practice (knowledge of performance) and after task completion
(knowledge of results) (Levin, Weiss, & Keshner, 2015). A primary aim is to promote
behavioral calibration of the clients’ actions using clear signals that indicate their status
towards achieving performance outcomes. However, careful consideration needs to be
placed on the use of positive and negative feedback during and after correct and incorrect
performance to balance short-term and long-term goals as they relate to user motivation and
task performance (Burgers et al., 2015). Delivery of feedback stimuli can appear in graded
(degree) or absolute (correct/incorrect) forms and can be presented via auditory, visual, or
tactile sensory modalities depending on the goals of the application and the needs of the
user. Moreover, feedback can be inherent to the task and the way the user’s actions are
represented in the VR environment. For example, representing the user’s hands through
virtual models is also a form of feedback, providing real-time information about the user’s
movements. This feedback can be modulated, such as exaggerating, dampening, slowing
down, speeding up, or even mirroring movement (e.g. Regenbrecht et al., 2014), depending
on the user’s therapeutic goals.
Feedback delivery is an intuitively essential component for rehabilitation efforts as it
is generally accepted to be necessary for most forms of learning or skill acquisition (Levin,
Weiss, & Keshner, 2015; Sohlberg & Mateer, 1989; 2001). While VR-based feedback can be
presented to signal performance status in a form that wouldn’t naturally occur in the real
world (e.g., a soft tone indicating a correct response), more relevant or naturalistic sounds
can also be creatively applied to enhance both ecological relevance and the believability of
the scenario. For example, in a VR application designed to help children with learning
disabilities practice escape from a house fire (Strickland, 2001), the sound of a smoke
detector alarm raises in volume as the child gets near to the fire’s location. As the child
successfully navigates to safety, the alarm fades contingent on her choosing the correct
escape route. More recently, Jin et al. (2016) have implemented a biofeedback methodology
for users aiming to reduce chronic pain via treadmill interaction within a virtual forest walking
task. As users lower their level of skin conductance level (as part of an effort to teach
relaxation and mindfulness strategies), the “fog” within the forest gradually lifts to reveal an
engaging and idyllic wilderness setting. Physical rehabilitation applications have also
leveraged the strategic delivery of performance feedback to enhance relearning of upper
extremity abilities following stroke or traumatic brain injury (Adamovich et al., 2009; Badia et
al., 2016; Deutsch, Latonio, Burdea, & Boian, 2001; Jack et al., 2001; Klamroth-Marganska,
et al., 2014).
Delivery of cueing stimuli to guide successful performance and impact behavior
The capacity for dynamic stimulus delivery and control within a virtual environment also
supports the presentation of cueing stimuli that can be used to guide user performance. This
is especially relevant for cognitive rehabilitation applications that implement “error-free”
learning strategies. Error-free training in contrast to trial and error learning has been shown
to be successful in a number of non-VR investigations with diverse test populations including
persons with developmental disabilities, schizophrenia, and CNS disorders (Fish et al., 2015;
Wilson et al., 1994, 1996). This asset underscores the idea that for some clinical approaches
it may not be desirable for VR to simply mimic reality with all its opportunities for error.
Instead, cueing stimuli features that are not easily deliverable in the real world can be
Clinical Virtual Reality
10
presented in the virtual world to help guide and train successful performance. In this special
case of stimulus delivery, cues are given to the user prior to a response in order to help
guide successful error-free performance.
While the use of cueing to support errorless learning is compelling and can now be
easily programmed as a feature within VR simulations, it has rarely been applied and tested
in VR contexts. In the only VR-based head-to-head comparison of this type, Connor et al.
(2002), reported on a series of case studies of users with TBI operating a haptic joystick-
mediated “Trails B” type training task. In the error-free condition, the haptic joystick restricted
movement on the non-immersive Trails task such that the user was not allowed to make
navigation errors. Mixed findings were reported, but error-free training resulted in significant
response speed improvements compared to trial and error training in some cases. In a case
report, Brooks et al. (1999) used error-free training for wayfinding in a rehabilitation ward as
one component in a VR training system that produced positive transfer to the real ward.
Harrison et al. (2002) also reported the use of cueing stimuli in a VR system designed to
train maneuverability and route-finding in novice motorized wheelchair users. Arrows were
presented to trainees with the caption “Go this way” to guide successful route navigation
whenever the user would stray into areas where invisible “collision boxes” were programmed
in the environment. Two patients with severe memory impairments took part in route finding
training over the course of seven days with the patients successfully learning two
subsections of the test routes while failing to eradicate errors on two further subsections of
the routes. Cueing was also incorporated into a VR system designed for executive function
assessment and training in the context of a series of food preparation tasks within a virtual
kitchen scenario (Christiansen et al., 1998). This VR scenario assessed the ability to perform
30 discrete steps required to prepare a can of soup and make a sandwich using both visual
and auditory cues to prompt successful performance. However, the specific effect of this
cueing was not isolated, nor was a system in place to prevent errors from actually occurring.
Finally, a more recent case report has shown positive gains in a user with Alzheimer’s
disease using a similar virtual kitchen (Foloppe et al., 2015). Generally, it appears that the
use of cueing stimuli to support error-free VR rehabilitation is promising in concept but there
is currently only limited research support with its use in VR. However, while empirical
support is still lacking, the ease for programming these components within VR make it an
appealing feature to test more rigorously in future research.
Beyond errorless learning for cognitive rehabilitation, perhaps the use of verbal
cueing could be applied for cognitive behavioral approaches that address self-talk within
provocative VR settings. For example, if key prompting statements could be specified in
advance, users could pre-record supportive self-talk cues in their own voice. These cues
could then be played back to the user in a modulated “dreamlike” vocal tone during strategic
points within a socially stressful VR scenario designed to help users deal with anger
management, social phobia, or shyness issues. This form of natural “inner-voice” guidance
might be useful for self-talk methods within virtual social scenarios with the aim to improve
generalization of the user’s self-generated sub vocal cognitions that could facilitate more
optimal social interaction in the real world.
A dramatic extension of this type of proposed self-cueing feature worth mentioning
involves recent innovative VR efforts to address the cognitive distortions of persons with
depression (Falconer et al., 2016). With the goal of improving “self-compassion”, clinically
depressed users were invited to enter a virtual world for 8 minutes where they were
requested to “console” a distressed virtual child using tactics on which they had received
prior coaching. After a short period of time, the user was switched into the role (and virtual
body) of the child and presented with a replay of their own attempts at consoling the child.
The replay was delivered by an adult virtual representation of themselves that expressed
their own consoling words back to them in their own voice captured from their previous
verbalization and behavioral activity with the virtual child. In a small initial trial (n=15), after
Clinical Virtual Reality
11
three repetitions of this body-swapping scenario, significant reductions were measured in
depression severity and self-criticism, along with a significant increase in self-compassion,
from baseline to 4-week follow-up. Four patients showed clinically significant improvement.
Although this effect should still be considered preliminary, it does underscore the potential
for Clinical VR to present sophisticated cueing content, in this case a fully naturalistic
rendition of self-delivered, self-compassion, that produced significant emotional impact on
users in a fashion that would be near impossible to deliver with previously existing methods.
Behavioral performance capture and retrospective and intuitive after action reviews
The review of a client’s behavioral performance in any assessment and training activity
typically involves examination of numeric data and subsequent translation of that information
into graphic representations in the form of tables and graphs. Sometimes videotaping of the
actual event is used for a more naturalistic review and for behavior rating purposes. These
methods, while of value, are typically quite labor intensive to produce, and sometimes result
in a less than intuitive method for visualizing and understanding a complex performance
record. These challenges are compounded when the goal of the review is to provide
feedback and insight to clients whose impairments may preclude a useful understanding of
traditional forms of data presentation. VR offers the capability to capture and review a
complete digital record of performance in a virtual environment from many perspectives. For
example, performance in a virtual environment can be later observed from the perspective of
the user, from the view of a third party or position within the scenario, and from what is
sometimes termed, a “God’s eye view”, from above the scene with options to adjust the
position and scale of the view. This can allow a client to observe and repeatedly review their
performance from multiple perspectives. Options for this review also include the modulation
of presentation as in allowing the client to slow down rate of activity and observe each
behavioral step in the sequence in “slow motion” (Rizzo et al., 2004).
Advanced programs that incorporate such methods have been in steady use by the
military to conduct what is termed After Action Reviews (AAR) (Morrison & Meliza, 1999). In
military VR applications that often include multiple participants in a shared virtual space, a
computerized AAR tool can allow the behavior of any participant to be reviewed from
multiple vantage points at any temporal point in the digital training exercise. This is now
standard procedure for military simulation training, but has had limited application in Clinical
VR approaches. With the exception of less naturalistic review of paper and pencil results and
the occasional review of a client’s videotaped performance from fixed camera positions, the
capacity to provide more intuitive “first-person” perspective views to clients has not been
feasible with existing technology, and thus VR now provides a powerful asset in this area
(Rizzo et al., 2004).
Early efforts to leverage this VR asset appeared as a feature for reviewing
navigational performance in a number of wayfinding and place learning applications (Astur,
Oriz & Sutherland, 1998; Jacobs, Laurance & Thomas, 1997; Kober et al., 2013; Koenig,
Crucian, Dalrymple-Alford, & Dünser, 2010; Skelton et al., 2000). This has primarily been
used in applications where a tracked movement record is vital for measuring and visualizing
the dependent variable of exploratory behavior. A review method was also developed for
replaying a child’s head movements while they are tracking stimuli within a virtual classroom
in a VR assessment of attention (Rizzo et al., 2006). This application took data from a
tracking device positioned on top of the VR HMD and represented the captured movement
via a virtual representation of a person’s head. The virtual head is rendered to face outward
from the screen and a “straightforward” head position represents the attentive gaze at the
virtual blackboard where target hit stimuli are displayed to the child. During video playback
after a test session, it is possible to observe the child’s head movements during discrete
periods when distracting stimuli are presented around the classroom (see:
Clinical Virtual Reality
12
https://youtu.be/BQyO3oDMKbI). Head movements away from the center of the screen
represent the child’s actual movements to follow the distracting stimuli on each side of the
classroom instead of the face forward position required to view the target stimuli. This
playback format delivers an extremely intuitive understanding of the distractibility of children
diagnosed with Attention Deficit Hyperactivity Disorder (ADHD) during VR classroom
performance testing that were revealed from the complex statistical analyses of this
movement data. The provision of this type of intuitive behavioral visualization could serve to
improve the understanding of the behavior of an ADHD child by professionals, parents, and
perhaps even the tested child in a manner not possible with graphs and data tables (Rizzo et
al., 2004; 2006). Systematic studies of the clinical use of this form of performance record
review have yet to appear in the literature, although this form of visualization asset illustrates
how VR may add value for assessment and intervention that is not readily available with
existing traditional tools.
The “Pause Button” for mid-session review and analysis with the clinician
In any intervention that activates cognitive, behavioral, and emotional processes for a clinical
purpose, clinician review/feedback is an essential component for building a therapeutic
alliance and fostering clients’ self-awareness. While feedback can be delivered digitally
within a simulation for guiding real time performance, and retrospectively for past
performance review (see previous three assets), Clinical VR interactions can be paused and
restarted at precisely the next moment in the digital sequence or replayed from an earlier
juncture for the purposes of face-to-face therapist engagement/support as needed. It is easy
to think of VR as an all-encompassing computerized environment that delivers all the
ingredients for good intervention, but that would be naïve. Rather, the use of such potent
and emotionally evocative simulations should be viewed simply as tools for extending the
skills of a well-trained clinician and as a method that may amplify client engagement with a
therapeutic process that is known to have efficacy in a real-world delivery context. From this
view, the capacity to pause a simulation to engage in clinical dialog at strategic junctures is a
distinction that is often overlooked due to its simplicity. This functionality has relevance
across all areas of clinical intervention and needs to be specifically designed during the VR
development process to augment the therapist-client relationship instead of hindering it.
Specifically, immediate therapist response to client performance is one form of
feedback that is commonly seen in the rehabilitation of clinical populations. This may be of
particular value for clinical populations who have memory difficulties that require more
frequent review and feedback during a training session. While pausing is of course possible
with any assessment or intervention approach, VRs unique assets offer the opportunity to
pause or “freeze time” in the middle of a functional “real world” simulated task. This can
result in additive learning benefits, whereby you can “stop and evaluate” not only individual
performance, but also by examining what environmental elements may be affecting
performance. For example, during activities in a VR kitchen for the completion of a simple
task (i.e., making a can of soup), performance may be paused for the correction of errors
(missed procedure steps), evaluation of safety elements of the task (where are the sharp
objects) or discussion of perceptual difficulties (inappropriate visual scanning) (Rizzo et al.,
2004). The simulation can then be restarted or backed up to an earlier point to allow for a
“redo”. Similarly, for psychological treatment, when a user is immersed within a provocative
simulation where they are confronting a digital recreation of a traumatic event or an
environment designed to deliver anger or addictive behavior cues, the simulation can be
paused for direct coping strategy coaching with the clinician.
Thus, the ability to pause performance “mid-digital stream” allows a clinician to
intervene strategically to enhance client processing and discussion of decision-making,
memory strategies, coping behaviors, assertive language, cognitive restructuring, or any of
Clinical Virtual Reality
13
the myriad clinical tactics that are commonly applied as the elements of quality evidence-
based (and empathy-based) therapeutic care. Contrary to some of the negative concerns we
have heard expressed over the years regarding the use of technology in clinical practice (“it
puts a barrier between the therapist and the client”), the ability to pause (and later restart) a
client’s simulated experience for a direct clinical intervention may actually serve to remove a
key barrier—the lack of an immediate shared experience. Therapy can involve a lot of
discussion of abstract concepts that sometimes don’t lend themselves to an easily shared
understanding of the client’s experience of everyday life. A clinician who has the opportunity
for close observation of the client’s behavior within an emotionally or cognitively challenging
VR simulation, and who then can pause it to provide strategic support or reflection, may
have an edge for developing a closer understanding of the client. This edge may reside in
the clinician’s newfound ability to observe the client as they address a challenge that would
have previously remained unseen by the clinician due to its exclusive occurrence outside of
the therapy office.
Safe testing and training environments that minimize the risks due to errors.
This is an area where Clinical VR provides an obvious asset by creating options for users
with cognitive or sensorimotor impairments to be tested and trained in the safety of a
simulated digital environment. The value of this has already been amply demonstrated in the
predecessor field of aviation simulator research where actual flying accidents dropped
precipitously following the early introduction of even very crude aircraft simulation training
(Johnston, 1995). Early on in the Clinical VR domain, this asset served as a driving force for
VR system design and research with both clinical and unimpaired populations. For example,
the simple (but potentially dangerous) act of street crossing has been tested and trained in
VR with unimpaired children (McComas, MacKay, & Pivak, 2002; Morrongiello, Corbett,
Switzer, & Hall, 2015; Schwebel, McClure, & Severson, 2014), populations with learning and
developmental disabilities (Brown et al., 1998; Josman, Ben-Chaim, Friedrich, & Weiss,
2008; Strickland, 2001), and adult TBI and stroke groups with neglect (Navarro et al., 2013;
Naveh, Katz, & Weiss, 2000). Other relevant application areas include kitchen safety (Rose,
Brooks, & Attree, 2000), escape from a burning house with children on the autism spectrum
(Strickland, 2001); preventing falls with at risk elderly (Jaffe, 2004; Neri et al., 2017), use of
public transportation (Mowafty et al., 1995), and driving with a range of clinical populations
(Akinwuntan, Wachtel, & Rosen, 2012; Liu, Miyazaki & Watson, 1999; Pietrzak, Pullman, &
McGuire, 2014; Rizzo, Reinach, McGehee, & Dawson, 1997; Schultheis & Mourant, 2001).
And, more recently, there has been an increased interest in VR driving applications to
reduce risk in both novice and aged populations (Casutt, Martin, Keller, & Jäncke, 2014; Cox
et al., 2015). In addition to the goal of promoting safe performance in the real world, some
researchers have reported positive results for building a more rational client self-awareness
of deficits using a VR approach. For example, Davis and Wachtel, (2000), have reported a
number of instances where older adults, post-stroke, had decided not to continue making a
return to driving a primary immediate goal after they had spent time in a challenging VR
driving system.
Finally, one concern that may exist with this asset involves the potential that practice
of activities that are dangerous in real life, within the safety of a VE, might create a false
sense of security or omnipotence that would put the client at risk upon subsequent action in
the real world. In essence, can safe transfer of training occur in the real world when the
consequences of errors are prevented from occurring in VR? This is a very challenging
concern that needs careful consideration. Perhaps, one option would be to provide a noxious
sound cue, contingent on the occurrence of dangerous errors in VR, as a means to condition
a proper attitude of caution in clients. This concern further underscores the need for a
professional to closely monitor client activity in order to recognize possible patterns of risk
taking behavior that could emerge when using VR (Rizzo et al., 2004).
Clinical Virtual Reality
14
Independent practice of therapeutic activities outside of the clinic
Independent home-based physical therapy or cognitive training by clients following a stroke
or TBI is a common and highly recommended component for most approaches to
rehabilitation. Similarly, with standard cognitive behavioral therapy (CBT) for psychological
disorders, it is generally accepted that by having clients do between-session “homework”,
that generalization of skills learned in therapy session will be promoted in everyday life.
Thus, clients are routinely encouraged to engage in clinician-recommended therapeutic
activities independently as part of a general approach to clinical care. Up until the last few
years, access to VR technology for supporting clinical care outside of the clinic was a
hopeful vision, but very limited by the immature state of the technology. Consequently, there
is very little research on the additive value of home-based VR for bolstering clinic-based
interventions on clinical outcomes.
Researchers over the last 20 years have proposed and tested various configurations
for pushing VR game-based physical rehabilitation into home-based systems (Piron et al.,
2001; Proffitt & Lange, 2015; Standen et al., 2014). However, as compelling as this idea
sounds in concept, limitations due to the cost of equipment and complexity of set up and use
limited the general adoption of this approach. One challenge for physical rehabilitation early
on was seen in the need for specialized interface devices and body tracking systems
required to foster interaction with virtual rehabilitation task content. This has been somewhat
minimized in recent years with various commercially available camera-based 3D tracking
systems like the Microsoft Kinect or the Leap Motion sensor. There are a number of
commercial and non-commercial entities that develop such VR systems based on low-cost
sensors, but they have been primarily focused on clinic-based use (Faria, Andrade, Soares,
& Bermudez i Badia, 2016; MindMaze, 2017; SilverFit, 2017). Movement of these systems
into the homes of users for independent practice and online tracking of use/performance by
a supervising clinician is only starting to become technically feasible and future effort in this
area is expected to accelerate, especially in view of the positive findings that have emerged
from studies of in-clinic use (Howard, 2017; Klamroth-Marganska et al., 2014).
Efforts to use immersive VR for CBT home-based activities have been similarly
hampered by cost and complexity issues. That is also expected to change in the near future
as low-cost VR HMDs that are easy to operate are now coming into the marketplace. This is
in part due to the widespread access to personal computing, previously limited to standalone
computers, but now bolstered by the ubiquitous presence of mobile phones/devices. Thus,
access to computing power is no longer a significant bottleneck for supporting independent
Clinical VR practice. Moreover, access to a VR HMD for personal use is no longer a limiting
factor as new technology has accelerated the availability and adoption of low-cost consumer
level HMDs. This can be seen in the rapid developments in mobile phone enabled HMDs.
Such products as the Samsung Gear VR or the Google Daydream, offer fairly good fidelity at
the price of a mobile phone and a HMD housing costing less than $100, into which the
phone can be inserted to create a working VR headset. These systems are easy to use and
there is content that is becoming readily available that can be applied for clinical purposes.
For example, low-cost “fear of public speaking” VR software is readily downloadable
(Hypergrid Business, 2016) for these systems. The software allows users to practice their
speaking skills in front of a wide range of virtual audiences along with the presentation of
public speaking coaching content. However, while self-treatment for this form of anxiety
when viewed as a skill training intervention appears on the surface to be relatively benign, it
does open the door to other types of self-help VR anxiety disorder applications. This state of
affairs will require a deeper analysis as to the ethical use of such emotionally evocative
software and the issues surrounding VR self-help will be discussed later in this article.
Clinical Virtual Reality
15
Adaptable user interfaces and sensory displays to promote access
The emerging human computer interaction research area referred to as “3D User
Interaction” (LaViola et al., 2017) recognizes that interaction with VR content requires
thoughtful attention to both design principles and the needs of the targeted user groups. This
is especially relevant for clinical users with sensory or motor impairments where their
capacity to receive value from a VR assessment or rehabilitation approach is always
governed by their ability to interact with the VR content (Rizzo et al., 2004). While an
extensive literature exists in the area of interface design for persons with disabilities (Barrett,
McCrindle, Cook & Booy, 2002; Darejeh & Singh, 2013; Lanyi et al., 2012), a full discussion
of that area is beyond the scope of this article. However, since VR content can be interacted
with using a wide variety of adaptive interface devices, we will briefly address how that
capability can be leveraged as an asset for Clinical VR. This is particularly relevant as
sensory and motor impairments are commonly seen in persons with central nervous system
(CNS) dysfunction. A question that often arises in assessment and rehabilitation, concerns
the degree to which a client’s performance reflects CNS-based cognitive dysfunction vs.
artifacts due to sensorimotor impairments. VR offers two ways in which this challenge may
be addressed in the testing and training of cognitive and everyday functional abilities in
persons with sensorimotor impairments.
One approach places emphasis on the use of adapted human computer interface
devices for VR interaction. Such devices can allow a user with significant motor impairments
to interact with VR assessment and training content, beyond what is possible for similar
clinical activities in the physical world. Interface adaptations can support interaction by
leveraging alternative or augmented movement, speech, expired air, tracked eye movement,
and by way of recent advances in brain computer interfaces (Kaplan et al., 2013; Millan et
al., 2010; Remsik et al., 2016). One very basic example involves the use of a gaming
joystick to navigate in a VR scenario modeled after an amnestic client’s rehabilitation unit
that was found to be effective for teaching wayfinding around the real unit (Brooks et al.,
1999). The authors partially attributed the observed positive training effects to the client’s
capability for quicker traversing of the VR training world using a joystick compared to what
her ambulatory impairments (using a walker) would allow in the real environment. This
strategy supported efficient use of training time by increasing the number of training trials
that were possible (i.e., 10 trials in VR in the time it would take to complete one with the
walker). Quite simply, by minimizing the impact of the user’s ambulatory impairments, CNS
wayfinding functions could be more efficiently trained.
A second approach can be seen in efforts to tailor the sensory modality of the stimuli
presented in the VR world around the needs of persons with visual impairments. The few
efforts in this area have mainly attempted to build simulated structures around the use of
enhanced immersive 3D audio (Lumbreras & Sanchez, 2000) and tactile stimuli (Connor,
2002). For example, Lumbreras et al. (2000), aiming to design computer games for blind
children, created a 3D audio VR system referred to as “AudioDOOM”. In this application,
blind children used a specialized joystick to navigate the mazelike game environment
exclusively on the basis of 3D audio cues (e.g., footstep sounds, doors that “creak” open,
echoes, etc.) while chasing “monsters” around the environment. Following varied periods of
time in the VE, the children are then given “Legos” to construct their impression of the
structure of the layout. The resulting Lego constructions were noteworthy in their striking
resemblance to the actual structure of the audio-based layout of the maze. Children using
this system (who never actually have “seen” the physical visual world) were able to use the
3D sound cues to create a spatial-cognitive map of the space and then accurately represent
this space with physical objects (i.e., Legos, Clay, Sand). Examples of some of these
constructions are available on the Internet
(http://www.dcc.uchile.cl/~mlumbrer/audiodoom/audiodoom.html). Such adaptive interaction
Clinical Virtual Reality
16
approaches in VR offer the potential for factoring out sensorimotor impairments that can
confound clear assessment or rehabilitation of functioning in a way that might not be feasible
or valid within the constraints of the physical world.
Virtual humans for addressing social interaction and training.
The feasibility for creating Clinical VR applications has advanced in part due to substantial
progress in 3D computer graphics rendering that now support the creation of ever more
believable context-relevant “structural” VR environments (e.g. combat scenes, homes,
classrooms, offices, markets) for clinical purposes. However, the next stage in the evolution
of Clinical VR will involve populating these environments with Virtual Human (VH)
representations that can engage real human users in credible and useful interactions. This
capability has been around since the 1990s, but the previous limitations in graphical
rendering, natural language processing, speech recognition, and face and gesture animation
made the creation of credible VHs for interaction a costly and labor-intensive process. Thus,
until recently VHs existed primarily in the domain of high-end special effect studios that
catered to the film or game industry, far from the reach of those who thought to employ them
in clinical health applications.
This is not to say that representations of human forms have not previously appeared
in Clinical VR scenarios. In fact, since the mid-1990s, VR applications have routinely
employed “primitive” VHs (e.g., low fidelity graphics, non-language interactive, limited face
and gesture expression) to serve as stimulus elements to enhance the realism of a virtual
world simply by their static presence. For example, VR exposure therapy applications for the
treatment of specific phobias (e.g., fear of public speaking, social phobia) were successfully
deployed using immersive simulations that were inhabited by “still-life” rendered characters
or 2D photographic sprites (i.e., static full body green screen captured photo images of a
person) (Anderson et al., 2005; Klinger, 2005; Pertaub et al., 2002). By simply adjusting the
number and location of such VH representations, the intensity of these anxiety-provoking VR
contexts could be systematically modulated with the aim to gradually habituate phobic
patients to what they feared, leading to improved functioning in the real world with real
people. In spite of the primitive nature of these VHs, phobic clients appeared to be especially
primed to react to such representations and thus, they provided the necessary stimulus
elements to be effective in these types of exposure-based cognitive behavioral treatment
scenarios.
Other clinical applications have also used animated graphic VHs as stimulus entities
to support and train social and safety skills in persons with high functioning autism (Padget
et al., 2006; Parsons et al., 2012; Rutten et al., 2003) and as distracter stimuli for attention
assessments conducted in a virtual classroom (Rizzo et al., 2006). VHs have also been used
effectively for the conduct of social psychology experiments, essentially replicating and
extending findings from studies conducted with real humans on social influence, conformity,
racial bias, and social proxemics (Bailenson & Beall, 2006; Blascovich et al., 2002; McCall et
al., 2009).
As the technology has evolved, VH agents can now be created that control computer
generated bodies and can interact with users through natural language speech and gesture
in virtual environments (Gratch et al., 2002; Rizzo, Kenny, & Parsons, 2011; Rizzo & Talbot,
2016a). Moreover, with advances in artificial intelligence, VHs can engage in rich
conversations (Morbini et al., 2014), recognize nonverbal cues (Rizzo et al., 2015b, 2016b;
Scherer et al., 2014), improve interactional rapport with users (Park et al., 2013), reason
about social and emotional factors (Gratch & Marsella, 2004), and synthesize human
communication and nonverbal expressions (Thiebaux et al., 2008). This has resulted in VH
agent systems that serve as: virtual patients for training novice clinicians (Rizzo et al., 2011,
Clinical Virtual Reality
17
2016a; Talbot et al., 2012), job interviewers for training young adults on the autism spectrum
to perform better in that context (Bresnahan et al., 2016); clinical interviewers to reduce
stigma (resulting in higher endorsement of clinical symptoms) (Rizzo et al., 2015b, 2016b),
and as health care guides and clinical support agents (Rizzo et al., 2015b). For example,
results from of sample of military service members (SMs) who were interviewed by a VH
clinical interviewer before and after a deployment to Afghanistan indicated that SMs revealed
more PTSD symptoms to the VH than they reported on the Post Deployment Health
Assessment (Rizzo et al., 2016b). In another study using the same VH agent system, civilian
users reported less concern about being evaluated, disclosed more personal information,
and displayed more sadness in an interview with a VH agent compared to interacting with a
VH avatar that they believed was being operated by a human-in-the-loop “Wizard of Oz”
controller (Lucas et al., 2014).
Thus, VHs now are capable of fostering interactions with real people that can
address a wide variety of clinical concerns. There is a growing literature in this area and it is
not hard to see the power of VH applications to foster roleplay training targeting social
interaction, anger management, relapse prevention for addiction, and in many other areas
where clinical populations could benefit from low social risk interaction with a non-judgmental
VH (Albright, 2016; Bickmore et al., 2016; Rizzo et al., 2016b; Tegos et al., 2016; Zhang et
al., 2017). Although some authors have expressed legitimate concerns about the role of VH
“automation” supplanting the role of clinicians (Innes & Morrison, 2017), VHs applications
developed thus far, serve more to fill gaps where a clinical provider is not available, than to
aim at replacement of human providers.
Game-Based interaction to enhance motivation and engagement
Plato was reputed to have said, “You can discover more about a person in an hour of play
than in a year of conversation." (cited in Moncur & Moncur, 2002). This ancient quote may
have particular relevance for future applications of Clinical VR. Observing and/or quantifying
a person’s approach or strategy when participating in a gaming activity may provide insight
into cognitive and psychological functioning similar to the types of challenges found in
traditional performance assessments. However, a more compelling clinical direction may
involve leveraging gaming features and incentives for the challenging task of enhancing
motivation and engagement levels in clients participating in rehabilitation, or any other
clinical activity for that matter. For example, one possible factor that may contribute to the
mixed outcomes reported in cognitive or physical rehabilitation research may be in part due
to the inability to maintain a client’s motivation and engagement when confronting them with
a repetitive series of retraining challenges, whether using wordlist exercises, range of motion
exercises, or real-life functional activities (Rizzo et al., 2004). The benefits of gamification for
enhancing psychological interventions have also been detailed in Granic et al. (2014)
specifically citing research support of its value for improving cognition (e.g., attention),
motivation (e.g., resilience in the face of failure), emotion (e.g., mood management), and
social interaction (e.g., prosocial behavior). In this regard, an understanding of gaming
features and their integration into VR-based rehabilitation systems to enhance client
motivation and subsequent clinical outcomes may be a useful direction to explore.
Rehabilitation, whether cognitive or physical, provides a clear use case for how the
integration of gaming features with VR is well-matched to the various requirements for
creating effective rehabilitation tasks (Lange et al., 2012; Rizzo et al., 1994, 2004). This can
be illustrated by first detailing the general requirements for good rehabilitation tasks and then
examining how they match up with the features that game-based VR provides. To do this,
we conjecture seven core requirements for a good rehabilitation task.
Clinical Virtual Reality
18
The rehabilitation task must be:
1. grounded in data-based assessment to specify the target activity to be precisely
rehabilitated.
2. adjustable in terms of difficulty level from something that is possible for the user to
perform, to a level representing the desired end-goal performance.
3. capable of repetitive and hierarchical administration to the user.
4. quantifiable in order to measure performance and progress.
5. capable of providing the user with strategic feedback as to the outcome of
performance.
6. relevant to real world functional activity.
7. capable of motivating user engagement and interaction with the task.
Clinical VR assets are well-matched to meet these requirements, once a
rehabilitation objective is specified by state-of-the-art data-based assessment
methodologies. VR’s capacity for stimulus control (specified earlier in the article) can support
the setting of a baseline challenge level that the user is capable of accomplishing. The
stimulus control asset can also leverage the tireless capacity of the computer to generate the
repetitive and hierarchical delivery of stimulus challenges across a range of programmable
difficulty levels. In this way, an individual's rehabilitation activity can be customized to begin
at a stimulus challenge level attainable and comfortable for them, with gradual titration to
higher or lower difficulty levels based on user performance. The interaction between the
user’s behavior and task demands can be automatically scored by the VR software to
measure performance, and provide real-time strategic feedback that can be automatically
administered as needed to shape and modulate performance toward a successful goal. All of
this can occur within simulated VR contexts that embody the complex functional challenges
that exist in everyday ecologically-relevant settings. Thus, the experimental control required
for rigorous scientific measurement, analysis, and replication can still be maintained while
the user is presented with challenges that require real-world functional behaviors.
At each step in this process, computer game development principles and evidence-
based rehabilitation task design (Lange et al., 2009, 2010; 2012), can be combined to
promote user motivation and engagement. The VR assets described here follow the same
structure for good computer game design. For example, to maintain motivation, game
designers develop content that provides challenges within what is called the “flow channel”.
Schell (2014) details the flow channel, derived from Csikszentmihályi (1990), as, “…the
narrow margin of challenge that lies between boredom and frustration, for both of these
unpleasant extremes cause our mind to change its focus to a new activity.” (pp. 119). By
integrating such game development principles with Clinical VRs capacity to deliver
systematically controllable simulations, it is now possible to create compelling rehabilitation
tasks to enhance client motivation and engagement beyond what may be possible with other
existing methodologies. The feasibility of translating traditional evidence-based interventions
into computer gaming formats is increasingly being recognized by clinicians and scientists as
a methodology for exploiting the features of games for therapeutic change (Fleming et al.,
2016). Moreover, the growing recognition of the potential value of gamification (and the need
for more research) in healthcare and the field of “Games for Health” is evidenced by the
appearance of scientific journals and conferences focused on this topic, in addition to an
evolving scientific literature. Since a full review of this area is beyond the scope of this
Clinical Virtual Reality
19
article, the reader is directed to other detailed reviews (Baranowski et al., 2016; Fleming et
al., 2016; Granic et al., 2014; Kato, 2010; Papastergiou, 2009).
Beyond efficacy: VR as a tool for breaking down barriers to care
This final asset is really a more speculative discussion of how VR at the current time may
have value beyond improving the efficacy of a clinical process and rather, is more concerned
with how VR could serve to break down some barriers to care. It is included here since some
of these factors may serve to inform later judgments as to Clinical VR’s readiness for
improving clinical practice and research. The main premise here is that the best evidence-
based approach for assessing or treating a clinical health condition serves little value if
clients do not seek it out and participate in it. There are many reasons why these barriers
limit client access to care and more detail can be found in (Andrade et al., 2015; Clement et
al., 2015). To more readily consider these barriers, we have constructed an intuitive model
for detailing them, called the 7A’s. The 7A’s stand for: Awareness, Anticipated Benefit,
Access, Availability of well-trained providers, Acceptability for seeking treatment, Adherence,
and Affordability.
Clinical VR may be strategically well-placed to break down some (but not all) of the
barriers that keep people from receiving the benefits of clinical care. To start, client
awareness of the range of available evidence-based treatment options may be limited.
Perhaps some remedy for this exists in the media exposure that is currently at an all-time
high for VR. In addition to the media excitement and interest in novel efforts to use VR for
gaming and entertainment purposes, there has also been significant coverage of VR
healthcare applications. This may be in part due to a desire in some quarters of popular
culture to promote VR’s image as useful for pro-social purposes, beyond first person shooter
games. Thus, a quick search of the internet will uncover a large volume of “heartstring
tugging” media reports on VR’s application with clinical conditions, especially those that are
at the forefront of the public consciousness (e.g., PTSD, Autism, Stroke, Alzheimer’s,
Depression, Opiate Addiction, etc.). For better or worse, and in spite of the occasional
scientific and factual errors in the popular press, there is no doubt that Clinical VR
applications have received significant media visibility. Whether this builds public awareness
of treatment options that leads to actual help-seeking is still an open question in need of
more research.
As well, the double-edged sword of media claims about anticipated benefit can be
problematic. The balance between over-wrought claims of clinical success and actual data
points can sometimes err on the side of higher-than-warranted expectations. However, when
a Clinical VR research study does provide positive evidence, the popular media’s focus on
covering that finding is fairly certain, thus reaching the eyes and ears of people who will
hopefully seek help, either for themselves or a loved one. For example, our PTSD VR
exposure work has garnered significant popular media reporting that is typically followed by
an uptick in client or family member queries as to where treatment can be accessed. The
perception of the “sexiness” of the use of “exotic” VR technology in the popular culture may
also build expectations of success that in the end may drive a stronger placebo effect in
those who undergo VR-based services.
Making treatment more accessible, is a factor for people who live in remote
locations or who face transportation challenges, and has served to drive efforts at using
teletherapy or online self-help CBT programming. However, as stated in the “independent
practice” section, VR as a tool for pushing care outside of the clinic is still limited by cost and
complexity issues, as well as by ethical concerns. This may be less limited in the future with
the growing availability of low-cost VR technology in the home, but for now, Clinical VR is not
seen to reduce the impact of this barrier. Similarly, the availability of well-trained providers
Clinical Virtual Reality
20
who are properly trained in Clinical VR procedures is still limited. While many VR
approaches follow the procedures and mechanics of traditional forms of therapy (e.g., VR
exposure therapy for anxiety disorders uses the same treatment protocol endorsed for
imaginal exposure approaches), the operation of VR equipment does require some
specialized training. This training is becoming more available either from standalone
workshops or CME offerings at respected conferences, but it is not commonplace at the
current time. However, the use of VH patients for training novice clinicians (Talbot et al.,
2012; Rizzo et al., 2011, 2016a) is an emerging area of focus, and this may have a direct
impact on improving clinical use and supporting the greater availability of well-trained
providers.
The acceptability of seeking care can be improved by reducing the internal or
external perceptions of stigma that a potential client may feel when admitting that they have
a problem. Although this may be less relevant for those seeking help to address a CNS-
related condition, it is often a factor that limits help-seeking for those with psychological
health conditions. This is an area where Clinical VR has some early research support. In a
survey study to assess openness to seeking care in 325 active duty Army SMs (Wilson et
al., 2008), results indicated that 83% of the participants reported that they were neutral-to-
very-willing to use some technology as part of a treatment; 71% were equally willing or more
willing to use a treatment based on technology than to talk to a therapist in a traditional
treatment setting. Moreover 20% of SMs, who stated they were not willing to seek traditional
psychotherapy, rated their willingness to use a VR-based treatment as neutral to very willing.
One possible interpretation of this finding is that a subgroup of this sample of SMs with a
significant disinterest in traditional mental health treatment would be willing to pursue
treatment with a VR-based approach. Thus, VR exposure therapy may offer an appealing
treatment option for “digital generation” SMs and Veterans who may be reluctant to seek out
what they perceive as traditional talk therapies. Other research using VR exposure for PTSD
and phobias with civilian groups has shown high levels of treatment satisfaction with VR
(Banos et al., 2009; Beck et al., 2007;) and in some reports, participants reported that it was
easier to take the first step in confronting fears with VR compared imaginal exposure.
Certainly, more research is needed to determine whether Clinical VR approaches reduce
stigma and promote help-seeking. However, one can speculate that younger groups who
have grown up in this “digital age” may actually be more attracted to and comfortable with
participation in a Clinical VR approach and this could be a factor for reducing stigma and
increasing the acceptability of VR-based care.
Finally, more research is needed to investigate the impact of Clinical VR for
promoting adherence to a full course of treatment. Although a number of small studies have
suggested a higher positive interest in continuing treatment with VR (cf. Bryanton et al.,
2006), most research examining treatment adherence as a specific variable has been
underpowered. While the motivating factors of Clinical VR tools are frequently referred to in
the literature, we are not aware of any systematic evaluations of VR treatment
characteristics and their impact on patient attrition for prolonged, repetitive treatment
protocols. We expect factors such as multiplayer and competitive training content, level of
immersion, story-driven/narrated treatment content, or relevance of treatment content to the
patient’s everyday life to be important factors for sustained patient motivation. The relevance
of the aforementioned “flow channel” (Schell, 2014) and its impact on user motivation and
engagement cannot be overstated. Thus, bridging the gap between scientific construction of
evidence-based treatment tasks and artistic design of game-based content seems a
worthwhile target for further investigation.
Affordability has also been an issue that has limited VR treatment access in the
past. This is expected to be less of a limiting factor, now that higher fidelity, yet low-cost
systems have come onto the market. As a point of comparison, it is now possible to
purchase a high-fidelity VR HMD for $800 (HTC Corporation, 2017) that has superior
Clinical Virtual Reality
21
specifications compared to a system that would have cost $20,000 (NVIS Inc, 2017) to
purchase 5 years ago. In addition, low-cost smartphone-based VR HMDs are likely to
achieve parity with computer-tethered systems for some Clinical VR applications and this is
predicted to dramatically reduce hardware costs and improve affordability. With large
technology companies such as Facebook, Google, Apple, and Samsung invested in the VR
market, we anticipate new and affordable hardware and software to be released more
frequently over the next few years. Moreover, successful companies in the Clinical VR space
(e.g. MindMaze, SilverFit, Gesturetek Health) are paving the way for a competitive
landscape of VR tools for clinical assessment and treatment that will inevitably result in more
affordable options for researchers and clinical providers. As these companies continue their
R&D work on innovative VR applications, we hope to see diversity and accessibility in this
growing market, not unlike Google’s Play Store or Apple’s App Store, again with the result of
more affordable prices for clinical end-users and eventually for home-based use by patients.
Discussion – Is Clinical Virtual Reality Ready for Primetime?
The question of Clinical VR’s readiness for widespread clinical use can be considered
across the criteria of theory, research, pragmatics, and ethics. On the basis of the clear
assets and features that are available with simulation technology, there is a sound
theoretical basis for the development and implementation of informed Clinical VR
applications. General simulation technology has a long history of adding value in aviation
simulation, military planning, automotive/aircraft design, and surgical planning (Virtual
Reality Society, 2017). By leveraging these same assets, but in a form factor that can deliver
VR experiences within a clinicians’ office or research laboratory, a new set of virtual tools
become possible for psychology and rehabilitation. While any given Clinical VR application
will likely not leverage all of the VR assets described in this article, a clear specification of
what features can be brought to bear on a clinical target is recommended to guide design,
implementation, and evaluation in a systematic fashion.
A guiding principle in our work is to first look at known processes operating in
physical reality that are believed to contribute to the creation of an evidence-based approach
to assessment and intervention. With that as a starting point, one can specify the VR assets
that can underlie and guide the creation of a VR application to: provide more reliable and
valid assessments, amplify treatment effects, break down barriers to care, or simply reduce
costs by automating processes. For example, we know that the use of imaginal exposure
approaches for anxiety disorders are evidence-based in the physical world. From that, one
can see a direct case for using VR to deliver ecologically relevant simulations, within which
we can precisely control and titrate the delivery of progressively more provocative stimuli to
pace exposure for the end goal of promoting extinction learning. Similarly, we know that the
sheer amount of physical rehabilitation activity that a stroke survivor engages in (all other
factors being equal) is related to improved outcomes. From that, it is logical to hypothesize
that if compelling game-based VR rehabilitation tasks are developed, it may be possible to
motivate users to do more repetitions, leading to improved outcomes. These thumbnail
examples simply present one or two of the assets that can inform the rationale for Clinical
VR use cases, but in reality, there may be any number of additional features that can be
specified and marshalled (e.g., strategic feedback, cueing stimuli, safety, etc.) for adding
value over existing traditional methods. Thus, it is our perspective that the theoretical basis
for using Clinical VR is sound and supportive of its “primetime” application.
The research support for the use of Clinical VR is promising, albeit not fully mature.
There seems to be a consensus in the literature, that VR can produce equivalent or better
outcomes for exposure-based approaches for anxiety disorder treatment (e.g., Bouchard et
al., 2017; Maples-Keller et al., 2017; Morina et al., 2015; Rizzo et al., 2015a). Consistent
findings have also been produced in support of VR as an effective distraction tool for
Clinical Virtual Reality
22
reducing the perception of pain in patients undergoing acutely painful medical procedures
(e.g. Hoffman et al., 2011; Trost et al., 2015). A growing body of research is indicating that
VR can increase participation in physical rehabilitation, with patients reporting more
motivation to engage in rehab tasks within a game-based VR context compared to
standalone training (e.g. Granic et al., 2014). Cognitive assessment methods using VR have
produced promising results in construct validation studies, and for distinguishing between
clinical groups and healthy controls (e.g. Man et al., 2016; Nir-Hadad et al., 2015; Parsons &
Rizzo, 2008b; Rizzo et al., 2006). And finally, the use of Virtual Humans in Clinical VR
applications has produced promising results indicating that they can foster credible
interactions with real people for training, as healthcare guides, and in the role of clinical
assessors, but this area is still in a very early state of maturity (Rizzo et al., 2015b, 2016ab;
Scherer et al., 2014; Talbot et al., 2012). By contrast, whether due to the complexity of the
problem space or the lack of standards in VR research methodology, cognitive rehabilitation
studies using VR interventions have provided more mixed outcomes. Again, there is
consensus about the promise of VR cognitive rehabilitation tools (e.g. Bogdanova, Yee, Ho,
& Cicerone, 2016; Ogourtsova, Silva, Archambault, & Lamontagne; 2015; Valladares-
Rodriguez et al., 2016), but the majority of conducted studies are pilot trials without sufficient
power or the study design needed to draw decisive conclusions about efficacy, transfer of
gained skills to the daily life of clients, long-term outcomes, and cost-effectiveness.
A continued focus on research methodology, selection of outcome measures,
quantification of training transfer to daily life, and the identification of “active ingredients” of
Clinical VR tools is needed to advance its thoughtful and scientifically valid use. This
includes answering questions about: the frequency and modality of feedback and cues;
treatment doses and frequencies; complexity of VR tasks and environments; importance of
graphical realism and fidelity; selection and usability of interface devices; relevance of
gamification and multiplayer/competitive elements; and many other factors that inform VR
system design. Importantly, these questions need to be posed for each of the diverse patient
populations that stand to benefit from Clinical VR tools. In sum, the research is generally
supportive for the “primetime” use of Clinical VR in some areas, but there should be no
illusion as to the need for more research investigating the boundary conditions for its safe
and effective application.
The positive outcomes seen in the Clinical VR literature thus far are actually quite
encouraging when viewed in the context of the challenges that researchers faced in these
areas. First, the general availability of the technology has only existed for about 25 years
and for the first 10-15 years of that, the maturity of the hardware and software was quite
variable. During those early years, with the notable exception of exposure therapy
applications, Clinical VR R&D was essentially exploratory, primarily characterized by one-off,
proof-of-concept, prototype systems. While these systems produced interesting results in
uncontrolled, small sample size studies, only a few applications were subjected to rigorous
parametric tests by independent researchers. As a result, most Clinical VR review articles
include the staple recommendation that, “while current VR findings are promising, more
controlled research with larger sample sizes are needed.”. This is not a slight on innovative
researchers who had to bear the double burden of acquiring funding for both system
development and clinical tests, with a technology that was sometimes perceived by grant
reviewers as being too “science-fiction-y” to support good science! Rather, it is just an
observation on the challenges that have slowed the progression of tightly controlled research
in some Clinical VR areas. Thus, when one considers that psychology as a science has
been around for about 125 years with a focus on studying human behavior and interaction in
the physical world, it makes sense that we may need a few more years to evolve the science
for how humans behave and interact in the virtual world.
By contrast, the pragmatics for developing and using Clinical VR systems are quite
favorable. Over the last 10 years, the technology has gradually advanced enough to support
Clinical Virtual Reality
23
widespread VR system development beyond what was only possible within very specialized
research institutes. This has now been recognized by the Gartner Group (2016) with their
elevation of VR from the “trough of disillusionment” to the “slope of enlightenment” in the
Hype Cycle for Emerging Technologies. A key factor for VR’s recent expansion is a growing
VR development community that thrives on access to affordable design tools and VR
hardware. Development software (e.g. game engines Unity3D, Unreal Engine, Amazon
Lumberyard) has seen a large boost in popularity over the past five years and has even
found its way into high school and college computer science curriculums. Any interested
student, educator, hobbyist, or entrepreneur can pick up these tools for free and begin
developing VR applications without much upfront investment or any of the barriers that VR
R&D teams faced in the past. We expect this momentum and growth of the VR developer
community to translate to a surge of new VR applications, including Clinical VR tools. The
online PC distribution platform and community Steam (Valve Corporation, 2017) is currently
listing more than 1900 VR-enabled PC games. We anticipate similar distribution platforms to
emerge for Clinical VR content that will provide greater access to affordable libraries of
archetypic treatment and assessment scenarios for healthcare providers and researchers.
As we look to the future, we see Clinical VR as one of the larger domains of general
VR usage. In the recent Goldman Sachs (2016) market analysis looking at the future of VR
in 2025, we of course see that Gaming and Entertainment garners the largest market share.
While this is to be expected with the public’s chronic demand for new and better ways to
consume media, the little noticed item in that market analysis is that “healthcare” comes in
second for the VR market share. This is not a surprise to researchers and clinicians who
have worked in this area over the years, especially as we see healthcare costs becoming
one of the largest line items in the US Govt. budget, after Defense. Interest in Clinical VR by
actual therapists also seems to be substantial. Norcross et al. (2013) surveyed 70
psychotherapy experts regarding interventions they predicted to increase in the next decade
and VR was ranked 4th out of 45 options with other computer-supported methods occupying
4 out of the top 5 positions.
The ethical use of VR needs to be considered thoughtfully in any assessment of its
future primetime impact on psychological practice or science. Current VR technology now
allows for the creation of emotionally evocative virtual experiences. With Clinical VR, we
often aim to leverage that capability for a positive impact in client care. But if we accept that
it is possible to create experiences that can evoke strong emotions for a positive clinical
purpose, we must also accept the probability of some risks for the occurrence of unforeseen
negative emotional reactions. Thus, the question of safe and ethical use of VR has been
addressed in detail at various junctures (Madary & Metzinger, 2016; Rizzo, Schultheis, &
Rothbaum, 2003; Yellowlees, Holloway, & Parish, 2012; Tart, 1993). While there are a
variety of ethical issues for the general application of VR beyond its clinical use (e.g. motion
sickness side effects, overuse, violent content, etc.), our focus here is limited to the use of
VR as a tool for clinical diagnosis and treatment.
Thus far, a significant literature has emerged in support of the positive impact of well-
designed, theory-informed VR applications on mental health and physical functioning. These
applications are typically administered within the controlled and safe context of the therapy
setting, supervised by a well-trained clinician. However, what happens if these types of VR
experiences become commodity products that are readily accessible to anyone who self-
diagnoses their clinical condition and then uses VR treatment content as a “self-help”
therapy? While some might say this is not much different than purchasing a self-help book
and following the instructions and recommendations therein, VR experiences may have
more impact on a user than what may occur from reading a book. Similar to most areas of
mental health care, there is also a risk that this form of self-diagnosis and treatment is based
on inaccurate or counterproductive information. Another kind of ethical challenge can also
emerge if a clinician decides that VR would be great for generating a buzz for their practice
Clinical Virtual Reality
24
and result in more business, but the clinician hasn’t had sufficient training in its use and safe
application. Thus, there are issues of concern here from the perspective of patients and
providers. Consequently, there is a need for ethical guidelines regarding the safe and
informed use of Clinical VR applications, much like the way that pharmaceutical treatments
are managed by a well-trained and qualified physician.
In the area of clinical practice, the American Psychological Association’s ethical code
provides a clear and well-endorsed set of guidelines that can serve as a good starting point
for understanding and proactively addressing some of the basic issues for the creation and
use of VR applications in clinical practice (APA, 2003). Three core areas of clinical practice
concerns and recommendations can be derived from these guidelines (two of which come
directly from the APA code):
1. “2.01 Boundaries of Competence
(a) Psychologists provide services, teach and conduct research with populations and in
areas only within the boundaries of their competence, based on their education, training,
supervised experience, consultation, study or professional experience.”
Recommendation: VR-delivered mental health assessment/treatment may require
fundamentally different skill sets than what is needed for traditional “talk therapy”
approaches. Clinicians need to have specialized training, and possibly in the future, some
level of certification in the safe and ethical use of VR for therapy.
2. “2.04 Bases for Scientific and Professional Judgments
Psychologists' work is based upon established scientific and professional knowledge of the
discipline.”
Recommendation: VR applications that are developed for clinical assessment and treatment
must be based on a theoretical framework and documented with some level of research
before they can be endorsed as evidence-based and marketed as such. In an emerging area
like VR where unique and specific guidelines have yet to be established, the practitioner
must be fully transparent about the evidence base for the approach and take precautions to
preserve the safety and integrity of the patient.
3. Self-Diagnosis / Self-Treatment
While not cited as an APA standard, the issues regarding patient self-diagnosis and self-
treatment deserve further mention. Mental health conditions can be extremely complex and
in some instances the self-awareness of the patient may be compromised. This can
oftentimes lead to a faulty self-diagnosis as well as the problems that arise when the patient
searches for symptom information on the internet where reliable and valid content can be
questionable. The same issues come into play with self-treatment. The problems that can
ensue are two-fold.
a. The patient makes errors in either or both areas and achieves no clinical benefit,
or worse, aggravates the existing condition with an ineffective or inappropriate VR
approach that actually does more harm.
b. By pursuing a “seductive” VR self-help approach that is misaligned with their
actual needs or has no evidence for its efficacy, the patient could miss the
opportunity to receive quality evidence-based care that is designed and delivered
based on the informed judgment of a trained expert diagnostician or clinical care
provider.
Clinical Virtual Reality
25
These two negative impacts could occur if a company produces a VR approach
without sufficient validation and markets it to the public as a valid test or cure. This has been
seen over the years with many forms of quack medicine and there needs to be some
principle about the promotion of a VR application that has the consumer’s protection in mind.
This issue is particularly important at the current time in view of all the public exposure,
hype, and genuine excitement surrounding VR. There are many new companies emerging in
the healthcare space, essentially being driven by venture capitalists and game developers,
without any credible expert clinical and/or research guidance. Such companies could not
only do harm to users, but the uninformed development and over-hype of the benefits to be
derived from a VR clinical application leading to negative effects could serve to create the
general impression that VR is a “snake oil” approach and lead to people not seeking (or
benefiting from) an otherwise well-validated VR approach.
An example of a “grey area” in this domain concerns one of the most common fears
that people report - public speaking. Technically, in an extreme form where it significantly
impairs social and occupational functioning, public speaking anxiety would qualify as a
phobia and be diagnosed as an anxiety disorder. However, since most people do have some
level of sub-clinical fear of public speaking (that they eventually get over with practice), this
has been one of the first areas where widespread consumer access to Public Speaking VR
exposure therapy software has occurred (Hypergrid Business, 2016). Users can practice
their presentation “skills” on a low-cost smartphone-based VR HMD (e.g. Google
Cardboard/Daydream, Samsung Gear VR) in front of various types of audiences and
settings. In this case, most clinicians would not show much concern for this type of self-help
skills training approach and the potential for damaging effects to a user appears to be fairly
minimal. But, from this example, can we now expect that applications will be made readily
available for other and perhaps more complex anxiety disorder-based phobias (Fear of
Flying, Social phobia, Driving, Spiders, Intimacy, etc.), or even for PTSD treatment?
Consequently, it appears that ethical guidelines may be needed to support the safe use of
Clinical VR.
In conclusion, interest in the clinical uses of VR technology has accelerated and will
likely continue to be fueled by a societal zeitgeist in which this form of immersive and
interactive technology inspires the public’s attention and imagination. While previously
hamstrung by costs, complexity, and clinician unfamiliarity with VR equipment, the
technology has evolved dramatically in the consumer marketplace with new low-cost, hi-
fidelity, product offerings that are poised to drive wider scale adoption. This will result in a
probable future scenario where VR devices will become like toasters—although you may not
use it every day, every household will have one. When such market penetration occurs, the
general public will have more access to a range of VR experiences. This may serve to
accelerate the uptake of Clinical VR as users, more familiar with the technology, begin to
imagine its value beyond the world of digital games.
The momentum generated by the growing public awareness of VR coupled with
advances in the technology has created a unique opportunity for psychology and
rehabilitation. Our analysis of the theoretical underpinnings and research findings to date
leads us to predict that the application of Clinical VR will have a significant impact on future
research and practice. The pragmatic issues that may influence its adoption as a tool across
many areas of psychology also appear favorable, but professional guidelines will be needed
to promote its safe and ethical use. Such guidelines should inform the development of
principles for Clinical VR application design, distribution, practice, and training. While there is
still much work to be done to advance the science in this area, we strongly believe that
Clinical VR applications will become indispensable tools in the toolbox of psychological
researchers and practitioners and will only grow in relevance and popularity in the future.
Thus, it is our assessment that Clinical VR is indeed ready for primetime!
Clinical Virtual Reality
26
For access to a large library of online videos demonstrating many of the applications
discussed here, go to: https://www.youtube.com/user/albertskiprizzo
References
Adamovich, S. V., Fluet, G. G., Tunik, E., & Merians, A. S. (2009). Sensorimotor training in
virtual reality: a review. NeuroRehabilitation, 25(1), 29-44.
Akinwuntan, A.E., Wachtel, J., & Rosen, P.N. (2012). Driving Simulation for Evaluation and
Rehabilitation of Driving After Stroke. Journal of Stroke and Cerebrovascular Diseases,
21(6), 478-486.
Albright, G., Adam, C., Serri, D., Bleeker, S., & Goldman, R. (2016). Harnessing the power
of conversations with virtual humans to change health behaviors. mHealth, 2(11).
American Psychological Association (2003). Ethical Principles of Psychologists and Code of
Conduct. Retrieved from http://www.apa.org/ethics/code/.
Anderson, P.L., Zimand, E., Hodges, L.F., & Rothbaum, B.O. (2005). Cognitive behavioral
therapy for public-speaking anxiety using virtual reality for exposure. Depression and
Anxiety. 22(3), 156-158.
Andrade, L.H., Alonso, J., Mneimneh, Z., Wells, J.E., Al-Hamzawi, A., et al. (2015). Barriers
to Mental Health Treatment: Results from the WHO World Mental Health (WMH) Surveys.
Psychol Med, 44(6), 1303–1317. doi: 10.1017/S0033291713001943
Astur, R.S., Oriz, M.L. & Sutherland, R.J., (1998). The characterization of performance by
men and women in a virtual Morris water task: A large and reliable sex difference.
Behavioral Brain Research, 93, 185-190.
Astur, R. S., St. Germain, S. A., Tolin, D., Ford, J., Russell, D., & Stevens, M. (2006).
Hippocampus function predicts severity of post-traumatic stress disorder. Cyberpsychology
& Behavior, 9(2), 234-240.
Astur, R.S., Taylor, L.B., Mamelak, A.N., Philpott, L. and Sutherland, R.J. (2002). Humans
with hippocampus damage display severe spatial memory impairments in a virtual Morris
water task. Behavioural Brain Research, 132, 77-84.
Astur, R.S., Tropp, J., Sava, S., Constable, R.T., and Markus, E.T. (2004). Sex differences in
a virtual Morris water task and a virtual eight-arm maze. Behavioural Brain Research. 151(1-
2):103-15.
Aukstakalnis, & Blatner, D. (1992). Silicon Mirage: The Art and Science of Virtual Reality.
Peachpit Press, Berkeley, CA, USA.
Badia, S. B., Fluet, G. G., Llorens, R., & Deutsch, J. E. (2016). Virtual reality for
sensorimotor rehabilitation post stroke: Design principles and evidence.
In Neurorehabilitation Technology (pp. 573-603). Springer International Publishing. Cham,
Switzerland.
Bailenson, J.N. & Beall, A.C. (2006). Transformed social interaction: Exploring the digital
plasticity of avatars. In Schroeder, R. & Axelsson, A.'s (Eds.), Avatars at Work and Play:
Collaboration and Interaction in Shared Virtual Environments, 1-16, Springer-Verlag, Berlin,
Germany,
Clinical Virtual Reality
27
Baños, R.M., Botella, C., Guillen, V., García-Palacios, A., Quero, S., Bretón-López, J., &
Alcañiz, M. (2009). An adaptive display to treat stress-related disorders: EMMA’s world.
British Journal of Guidance and Counseling, 37(3), 347–356.
Baños, R.M., Guillen, V., Quero, S., García-Palacios, A., Alcaniz, M. & Botella, C. (2011). A
virtual reality system for the treatment of stress-related disorders: A preliminary analysis of
efficacy compared to a standard cognitive behavioral program. International Journal of
Human Computer Studies, 69(9), 602–613.
Baranowski, T., Blumberg, F., Buday, R., DeSmet, A., Fiellin, L. E., Green, C. S., ... & Morrill,
B. A. (2016). Games for health for children—Current status and needed research. Games for
health journal, 5(1), 1-12. Downloaded from:
http://www.icdvrat.org/2016/papers/ICDVRAT2016_S02N2_Bresnahan_etal.pdf
Barrett, J., McCrindle, R.J., Cook, G.K., Booy, D.A. (2002). Accessible interface design: A
review of access to information and communication technology for older and disabled
people. Technical Report, available from Department of Computer Science, The University
of Reading.
Beck, A.T., Epstein, N., Brown, G. and Steer, R.A. (1988). An inventory for measuring
clinical anxiety: psychometric properties Journal of Consulting and Clinical Psychology, 56,
893-897.
Beck, J.G., Palyo, S.A., Winer, E.H., Schwagler, B.E., & Ang, E.J. (2007). Virtual reality
exposure therapy for PTSD symptoms after a road accident: an uncontrolled case series.
Behavior Therapy, 38(1), 39–48.
Beidel, D. C., Frueh, B. C., Neer, S. M., & Lejuez, C. W. (2017). The efficacy of Trauma
Management Therapy: A controlled pilot investigation of a three-week intensive outpatient
program for combat-related PTSD. Journal of Anxiety Disorders, 50, 23-32.
Bickmore, T. W., Utami, D., Matsuyama, R., & Paasche-Orlow, M. K. (2016). Improving
access to online health information with conversational agents: a randomized controlled
experiment. Journal of medical Internet research, 18(1).
Blascovich, J., Loomis, J., Beall, A., Swinth, K., Hoyt, C., & Bailenson, J. (2002). Immersive
virtual environment technology: Not just another research tool for social psychology.
Psychological Inquiry, 13, 103-124.
Blume, F., Hudak, J., Dresler, T., Ehlis, A. C., Kuhnhausen, J., Renner, T. J., & Gawrilow, C.
(2017). NIRS-based neurofeedback training in a virtual reality classroom for children with
attention-deficit/hyperactivity disorder: study protocol for a randomized controlled trial. Trials,
18(1), 41. doi:10.1186/s13063-016-1769-3
Bogdanova, Y., Yee, M.K., Ho, V.T., & Cicerone, K.D. (2016). Computerized Cognitive
Rehabilitation of Attention and Executive Function in Acquired Brain Injury: A Systematic
Review. J Head Trauma Rehabil 31(6), 419-433.
Bohil, C. J., Alicea, B., & Biocca, F. A. (2011). Virtual reality in neuroscience research and
therapy. Nature Reviews Neuroscience, 12(12), 752-762.
Bordnick, P.S., Yoon, J.H., Kaganoff, E., & Carter B. (2013). Virtual Reality Cue Reactivity
Assessment: A Comparison of Treatment-vs. Nontreatment-Seeking Smokers. Research on
Social Work Practice, 23(4), 419-425.
Clinical Virtual Reality
28
Botella, C., Serrano, B., Baños, R. M. & Garcia–Palacios, A. (2015). Virtual reality exposure-
based therapy for the treatment of post-traumatic stress disorder: a review of its efficacy, the
adequacy of the treatment protocol, and its acceptability. Neuropsychiatric Disease and
Treatment, 11, 2533–2545.
Bouchard, S., Dumoulin, S., Robillard, G., Guitard, T., Klinger, E., Forget, H., Loranger, C., &
Roucaut, F.X. (2017). Virtual reality compared with in vivo exposure in the treatment of
social anxiety disorder: a three-arm randomised controlled trial. The British Journal of
Psychiatry, 210(4), 276-283. DOI: 10.1192/bjp.bp.116.184234.
Bresnahan, T., Rizzo, A.A., Burke, S.L., Partin, M., Ahlness, R.M., & Trimmer, M. (2016).
Using Virtual Interactive Training Agents (VITA) with Adults with Autism and other
Developmental Disabilities. In The 2016 Proceedings of the International Conference on
Disability, Virtual Reality and Associated Technologies. Available at:
Brooks, B.M., McNeil, J.E., Rose, F.D., Greenwood, R.J., Attree, E.A. and Leadbetter, A.G.
(1999). Route learning in a case of amnesia: A preliminary investigation into the efficacy of
training in a virtual environment. Neuropsychological Rehabilitation, 9, 63-76.
Brown, D. J., Kerr, S. J., and Bayon, V. (1998). The development of the Virtual City: A user
centred approach. In Sharkey, P., Rose, D. and Lindstrom, J. (Eds.), Proceedings of the 2nd
European Conference on Disability, Virtual Reality and Associated Technologies
(ECDVRAT). Reading UK: University of Reading. 11-16.
Burgers, C., Eden, A., van Engelenburg, M., & Buningh, S. (2015). How feedback boosts
motivation and play in a brain-training game. Computers in Human Behavior, 48, 94-103.
Bryanton, C., Bosse, J., Brien, M., Mclean, J., McCormick, A., & Sveistrup, H. (2006).
Feasibility, motivation, and selective motor control: virtual reality compared to conventional
home exercise in children with cerebral palsy. Cyberpsychology & behavior, 9(2), 123-128.
Carlbring, P., Westling, B.E., Ljungstrand, P., Ekselius, L., & Andersson, G. (2001).
Treatment of Panic Disorder Via the Internet: A Randomized Trial of a Self-Help Program.
Behavior Therapy, 32, 751-764.
Casutt, G., Martin, M., Keller, M., & Jäncke, L. (2014). The relation between performance in
on-road driving, cognitive screening and driving simulator in older healthy drivers.
Transportation Research Part F, 22, 232-244.
Chirico, A., Lucidi, F., de Laurentiis, M., Milanese, C., Napoli, A., & Giordano, A. Virtual
Reality in Health System: Beyond Entertainment. A Mini-Review on the Efficacy of VR
During Cancer Treatment. J Cell Physiol, 231, 275-287. DOI: 10.1002/jcp.25117.
Chou, Y. H., Weingarten, C. P., Madden, D. J., Song, A. W., & Chen, N. K. (2012).
Applications of virtual reality technology in brain imaging studies. In Eichenberg, C. (Ed)
Virtual Reality in Psychological, Medical and Pedagogical Applications. InTech, DOI:
10.5772/48445.
Clement, S., Schauman, O., Graham, T., Maggioni, F., Evans-Lacko, S., et al. (2015). What
is the impact of mental health-related stigma on help-seeking? A systematic review of
quantitative and qualitative studies. Psychol Med, 45(1), 11–27.
https://doi.org/10.1017/S0033291714000129
Clinical Virtual Reality
29
Christiansen, C., Abreu, B., Ottenbacher, K., Huffman, K., Massel, B., & Culpepper, R.
(1998) Task performance in virtual environments used for cognitive rehabilitation after
traumatic brain injury. Archives of Physical Medicine and Rehabilitation, 79, 888-892.
Connor, B. Wing, A.M., Humphreys, G.W., Bracewell, R.M. & Harvey, D.A. (2002). Errorless
Learning Using Haptic Guidance: Research in Cognitive Rehabilitation Following Stroke. In
Sharkey, P. Lanyi, C.S. & Standen, P. (Eds.), Proceedings of the 4th International
Conference on Disability, Virtual Reality, and Associated Technology. Reading, UK:
University of Reading, 77-86.
Costanzo, M.E, Leaman, S., Jovanovic, T., Norrholm, S.D., Rizzo, A.A., Taylor, P., & Roy,
M.J. (2014). Psychophysiological Response to Virtual Reality and Sub-threshold PTSD
Symptoms in Recently Deployed Military. Psychosomatic Medicine, 76(9), 670-677.
Costas, R., Carvalho, L. & de Aragon, D. (2000). Virtual city for cognitive rehabilitation. In
Sharkey, P., Cesarani, A., Pugnetti, L. & Rizzo, A. (Eds.), Proceedings of the 3rd
International Conference on Disability, Virtual Reality, and Associated Technology (pp. 305-
313). Reading, UK: University of Reading.
Cox, S.M., Cox, D.J., Kofler, M.J., Moncrief, M.A., Johnson, R.J., et. al. (2015). Driving
Simulator Performance in Novice Drivers with Autism Spectrum Disorder: The Role of
Executive Functions and Basic Motor Skills. J Autism Dev Disord, DOI 10.1007/s10803-015-
2677-1.
Cromby, J., Standen, P., Newman, J., Tasker, H. (1996). Successful transfer to the real
world of skills practiced in a virtual environment by student with severe learning disabilities.
In: Sharkey, P.M. (Ed.) Proceedings of the 1st European Conference on Disability, Virtual
Reality and Associated Technologies. Reading, UK: University of Reading, 305-313.
Cruz-Neira C., Sandin D. J., DeFanti T. A. (1993). Surround-screen projection-based virtual
reality: the design and implementation of the CAVE. In Whitton, M.C. (Ed.) Proc. 20th Annual
Conf. on Computer Graphics and Interactive Techniques, 135–142, New York, NY: ACM.
Csikszentmihályi, M. (1990). Flow: The Psychology of Optimal Experience. Harper & Row.
Cuijpers, P., van Straten, A., & Andersson, G. (2008). Internet-administered cognitive
behavior therapy for health problems: a systematic review. J Behav Med, 31, 169–177.
Darejeh, A. & Singh, D. (2013). A review on user interface design principles to increase
software usability for users with less computer literacy. Journal of Computer Science, 9(11),
1443-1450. doi:10.3844/jcssp.2013.1443.1450.
Dascal, J., Reid, M., IsHak, W. W., Spiegel, B., Recacho, J., Rosen, B., & Danovitch, I.
(2017). Virtual reality and medical inpatients: A systematic review of randomized, controlled
trials. Innovations in clinical neuroscience, 14(1-2), 14.
Davies, R. C., Johansson, G., Boschian, K, Lindén, A., Minör, U., & Sonesson, B. (1998). A
practical example using virtual reality in the assessment of brain injury. In Sharkey, P., Rose,
D. & Lindstrom, J. (Eds.), Proceedings of the 2nd European Conference on Disability, Virtual
Reality and Associated Techniques. Reading UK: University of Reading, 61-68.
Davis, E.S. & Wachtel, J, (2000). Interactive Driving Simulation as a Tool for Insight
Development and Motivation in a Rehabilitation Setting. Paper presented at the 8th Annual
Medicine Meets Virtual Reality Conference, Newport Beach, CA. January 22, 2000.
Clinical Virtual Reality
30
DeFanti, T.A., Acevedo, D., Ainsworth, R.A., Brown, M.D., Cutchin S., et al. (2011). The
Future of the CAVE. Cent. Eur. J. Eng.,1(1), 16-37. doi: 10.2478/s13531-010-0002-5.
Deutsch, J.E., Latonio, J., Burdea, G. & Boian, R. (2001) Post-Stroke rehabilitation with the
Rutgers Ankle System: A Case Study. Presence: Teleoperators and Virtual Environments,
10, Presence: Teleoperators and Virtual Environments, 10, 416-430.
Deutsch, J. E., & McCoy, S. W. (2017). Virtual Reality and Serious Games in
Neurorehabilitation of Children and Adults: Prevention, Plasticity, and Participation. Pediatric
Physical Therapy, 29, S23-S36.
DeVault, D., Mell, J., & Gratch, J. (2015, March). Toward natural turn-taking in a virtual
human negotiation agent. In S. Andrist, D. Bohus, E. Horvitz, B. Mutlu, & D. Schlangen
(Eds.), AAAI Spring Symposium on Turn-taking and Coordination in Human-Machine
Interaction, (pp. 2-9), Stanford, CA: AAAI Press. Available at:
https://www.aaai.org/ocs/index.php/SSS/SSS15/paper/view/10335/10100
Difede, J., Cukor, J., Jayasinghe, N., Patt, I., Jedel, S., Spielman, L., et al. (2007). Virtual
Reality exposure therapy for the treatment of posttraumatic stress disorder following
September 11, 2001. Journal of Clinical Psychiatry, 68, 1639-1647.
Difede, J., Cukor, J., Wyka, K., Olden, M., Hoffman, H., Lee, F. S., & Altemus, M. (2014). D-
cycloserine augmentation of exposure therapy for post-traumatic stress disorder: a pilot
randomized clinical trial. Neuropsychopharmacology, 39(5), 1052-1058.
Elkind, J.S., Rubin, E., Rosenthal, S., Skoff, B. & Prather, P. (2001). A simulated reality
scenario compared with the computerized Wisconsin Card Sorting test: An analysis of
preliminary results. Cyberpsychology and Behavior, 4, 489-496.
Falconer, C.J., Rovira, A., King, J.A., Gilbert, P., Antley, A., Fearon, P., Ralph, N., Slater, M.,
& Brewin, C.R. (2016). Embodying self-compassion within virtual reality and its effects on
patients with depression. British Journal of Psychiatry Open, 2, 74–80.
Faria, A.L., Andreia, A., Soares, L., & Bermudez i Badia (2016). Benefits of virtual reality
based cognitive rehabilitation through simulated activities of daily living: a randomized
controlled trial with stroke patients. Journal of NeuroEngineering and Rehabilitation, 13:96.
DOI: 10.1186/s12984-016-0204-z
Fish, J. E., Manly, T., Kopelman, M. D., & Morris, R. G. (2015). Errorless learning of
prospective memory tasks: An experimental investigation in people with memory disorders.
Neuropsychological rehabilitation, 25(2), 159-188.
Fleming, T. M., Bavin, L., Stasiak, K., Hermansson-Webb, E., Merry, S. N., Cheek, C., ... &
Hetrick, S. (2016). Computer-games for mental health: Current status and promising
directions. Frontiers in Psychiatry, 7, 215.
Folley. B.S., Astur, R.S., Jagannathan, K., Calhoun, V.D., Pearlson, G.D. (2010). Anomalous
neural circuit function in schizophrenia during a virtual Morris water task. Neuroimage, 49(4),
3373-84.
Foloppe, D. A., Richard, P., Yamaguchi, T., Etcharry-Bouyx, F., & Allain, P. (2015). The
potential of virtual reality-based training to enhance the functional autonomy of Alzheimer’s
disease patients in cooking activities: A single case study. Neuropsychological
Rehabilitation, 25(1), 1-25.
Clinical Virtual Reality
31
Freeman, D., Bradley, J., Antley, A., Bourke, E., DeWeevers, N., & Evans, N., et al. (2016).
Virtual reality in the treatment of persecutory delusions: randomised controlled experimental
study testing how to reduce delusional conviction. British Journal of Psychiatry, 209(1):62–
67. doi:10.1192/bjp.bp.115.176438
Freeman, D., Reeve, S., Robinson, A., Ehlers, A., Clark, D., Spanlang, B., & Slater, M.
(2017). Virtual reality in the assessment, understanding, and treatment of mental health
disorders. Psychological Medicine, 47, 1-8. Published online: 22 March 2017
(https://www.cambridge.org/core/journals/psychological-
medicine/listing?q=virtual+reality&searchWithinIds=289CD5D69C062986BB9145015E56F0
E1) .
Gartner Inc. (2016, July 19). Hype Cycle for Emerging Technologies, 2016. Retrieved from
http://www.gartner.com/newsroom/id/3412017.
Gamito, P., Oliveira, J., Brito, R., Lopes, P., Rodelo, L., Pinto, L., & Morais, D. (2016).
Evaluation of Cognitive Functions through the Systemic Lisbon Battery: Normative Data.
Methods Inf Med, http://dx.doi.org/10.3414/ME14-02-0021.
Gamito, P., Oliveira, J., Coelho, C., Morais, D., Lopes, P., Pacheco, J., ... & Barata, A. F.
(2015). Cognitive training on stroke patients via virtual reality-based serious
games. Disability and Rehabilitation, 39 (4), 385-388.
Gold, J.I., Kim, S.H., Kant, A.J., Joseph, M.H. and Rizzo, A.A. (2006). Effectiveness of
Virtual Reality for Pediatric Pain Distraction during IV Placement. CyberPsychology and
Behavior, 9(2), 207-213.
Goldman-Sachs (January 13, 2016). Virtual and Augmented Reality Understanding the Race
for the Next Computing Platform. Available at: http://www.goldmansachs.com/our-
thinking/pages/technology-driving-innovation-folder/virtual-and-augmented-reality/report.pdf
Granic, I., Lobel, A., & Engels, R. C. (2014). The benefits of playing video games. American
Psychologist, 69(1), 66.
Gratch, J. & Marsella, S. (2004). A domain independent framework for modeling emotion.
Journal of Cognitive Systems Research. 5(4), 269-306.
Gratch, J., Rickel, J., Andre, E., Cassell, J., Petajan, E., Badler, N. (2002). Creating
Interactive Virtual Humans: Some Assembly Required. IEEE Intelligent Systems.
July/August: 54-61.
Harrison, A., Derwent, G., Enticknap, A., Rose, F.D. & Attree, E.A. (2002). The role of virtual
reality technology in the assessment and training of inexperienced powered wheelchair
users. Disability and Rehabilitation, 24, 599-606.
Hoffman, H. G., Richards, T. L., Bills, A. R., Van Oostrom, T., Magula, J., Seibel, E. J., &
Sharar, S. R. (2006). Using fMRI to study the neural correlates of virtual reality
analgesia. CNS spectrums, 11(1), 45-51.
Hoffman, H.G., Chambers, G.T., Meyer, W.J., Araceneaux, L.L., Russell, W.J., Seibel, E.J.,
Richards, T.L. and Sharar, S.R. (2011). Virtual Reality as an Adjunctive Non-pharmacologic
Analgesic for Acute Burn Pain During Medical Procedures, Annals of Behavioral Medicine.
41(2), 183-191.
Clinical Virtual Reality
32
Holden, M.K. (2005). Virtual Environments for Motor Rehabilitation: Review. CyberPsych.
and Behav. 8(3).187-211.
Howard, M. C. (2017). A Meta-Analysis and Systematic Literature Review of Virtual Reality
Rehabilitation Programs. Computers in Human Behavior. 70, 317-327.
HTC Corporation (2017). HTC Vive. Retrieved from https://www.vive.com.
Hypergrid Business (2016, August 13). 5 VR apps that help you be a better public speaker.
Retrieved from http://www.hypergridbusiness.com/2016/08/5-vr-apps-that-help-you-be-a-
better-public-speaker/
Innes, M. and Morrison, B. (2017). Projecting the future impact of advanced technologies:
Will a robot take my job? InPsych, 39(2), Downloaded on June 6, 2017, from:
https://www.psychology.org.au/inpsych/2017/april/innes/
Jack, D., Boian, R., Merians, A., Tremaine, M., Burdea, G., Adamovich, S., Recce, M. &
Poizner, H. (2001). Virtual Reality-Enhanced Stroke Rehabilitation. IEEE Transactions on
Neurological Systems and Rehabilitation Engineering, 9, 308-318.
Jacobs, W.J., Laurance, H.E., & Thomas, K. (1997). Place learning in virtual space I:
Acquisition, overshadowing and transfer. Learning and Motivation. 28, 521-541.
Jaffe, D. L., Brown, D. A., Pierson-Carey, C. D., Buckley, E. L., & Lew, H. L. (2004).
Stepping over obstacles to improve walking in individuals with poststroke
hemiplegia. Journal of rehabilitation research and development, 41(3), 283.
Jentsch, F., & Curtis, M. (2017). Simulation in aviation training. Routledge.
Jin, W., Choo, A., Gromala, D., Shaw, C., & Squire, P. (2016). A Virtual Reality Game for
Chronic Pain Management: A Randomized, Controlled Clinical Study. Stud Health Technol
Inform, 220, 154-60.
John, N. W., Pop, S. R., Day, T. W., Ritsos, P. D., & Headleand, C. J. (2017). The
Implementation and Validation of a Virtual Environment for Training Powered Wheelchair
Manoeuvres. IEEE Transactions on Visualization and Computer Graphics.
http://dx.doi.org/10.1109/TVCG.2017.2700273
Johnston, R. (1995). Is it live or is it memorized? Virtual Reality Special Report. 2, 53-56.
Josman, N., Ben-Chaim, H. M., Friedrich, S., & Weiss, P. L. (2008). Effectiveness of virtual
reality for teaching street-crossing skills to children and adolescents with
autism. International Journal on Disability and Human Development, 7(1), 49-56.
Josman, N., Kizony, R., Hof, E., Goldenberg, K., Weiss, P.L., & Klinger, E. (2014). Using the
virtual action planning-supermarket for evaluating executive functions in people with stroke.
Journal of Stroke and Cerebrovascular Diseases, 23(5), 879-887.
Kaplan, A.Y., Shishkin, S.L., Ganin, I.P., Basyul, I.A., & Zhigalov, A.Y. (2013). Adapting the
P300-based brain-computer interface for gaming: a review. IEEE Transactions on
Computational Intelligence and AI in Games. DOI: 10.1109/TCIAIG.2012.2237517.
Kato, P. M. (2010). Video games in health care: Closing the gap. Review of General
Psychology, 14(2), 113.
Clinical Virtual Reality
33
Keefe, R.S.E., Davis, V.G., Atkins, A.S., Vaughan, A., Patterson, T., Narasimhan, M., &
Harvey, P.D. (2016). Validation of a Computerized Test of Functional Capacity.
Schizophrenia Research, 175(1-3), 90-96.
Klamroth-Marganska, V., Blanco, J., Campen, K., Curt, A., Dietz, V., Ettlin, T., Felder, M.,
Fellinghauer, B., Guidali, M., Kollmar, A., Luft, A., Nef, T., Schuster-Amft, C., Stahel, W., &
Riener, R. (2014). Three-dimensional, task-specific robot therapy of the arm after stroke: a
multicentre, parallel-group randomised trial. The Lancet Neurology, 13(2): 159-166.
Klinger, E. (2005). Virtual reality therapy for social phobia: Its efficacy through a control
study. Paper presented at Cybertherapy 2005 (July 7-10), Basal, Switzerland.
Kober, S.E., Wood, G., Hofer, D., Kreuzig, W., Kiefer, M., & Neuper, C. (2013). Virtual reality
in neurologic rehabilitation of spatial disorientation. Journal of NeuroEngineering and
Rehabilitation. doi:10.1186/1743-0003-10-17.
Koenig, S. T. (2012). Individualized virtual reality rehabilitation after brain injuries.
(Unpublished doctoral dissertation). Human Interface Technology Lab New Zealand, College
of Engineering, University of Canterbury, Christchurch, New Zealand.
Koenig, S.T., Crucian, G.P., Dalrymple-Alford, J.C., & Dünser, A. (2010, August 31-
September 2). Assessing navigation in real and virtual environments: a validation study. In
Sharkey, P. & Sanchez, J. (Eds.) Proceedings of the International Conference Series on
Disability, Virtual Reality and Associated Technologies (ICDVRAT), Vina del Mar, Chile.
Koenig, S.T., Krch, D., Chiaravalloti, N., Lengenfelder, J., Nikelshpur, O., Lange, B.S.,
DeLuca, J., & Rizzo, A.A. (2012, September 10-12). User-centered development of a virtual
reality cognitive assessment. In Sharkey, P., & Klinger, E. (Eds.) Proc. 9th Intl Conf.
Disability, Virtual Reality & Associated Technologies, Laval, France.
Krch, D., Koenig, S., Lavrador, S., Rizzo, A., & Chiaravalloti, N. D. (2013, August 26-29).
Pilot results from a virtual reality executive function task. Paper presented at the proceedings
of the International Conference on Virtual Rehabilitation, Philadelphia, PA.
Krueger, M.W. (1993). The experience society. Presence, 2, 162-168.
Lamson, R.J. (1994). Virtual therapy of anxiety disorders. Cyberedge Journal, 4(1), 6-8.
Lange, B.S., Flynn, S.M., Proffitt, R., Chang, C.Y., & Rizzo, A.A. (2010). Development of an
interactive game-based rehabilitation tool for dynamic balance training. Topics in Stroke
Rehabilitation, 17(5), 345–352.
Lange, B.S., Flynn, S.M., & Rizzo, A.A. (2009). Initial usability assessment of off-the-shelf
video game consoles for clinical game-based motor rehabilitation. Physical Therapy
Reviews, 14(5), 355-363.
Lange, B., Koenig, S., Chang, C-Y., McConnell, E., Suma, E., Bolas M. & Rizzo, A.A. (2012).
Designing Informed Game-Based Rehabilitation Tasks Leveraging Advances in Virtual
Reality. Disability and Rehabilitation. 34(22): 1863-1870.
Lanyi, C.S., Brown, D.J., Standen, P., Lewis, J., & Butkute, V. (2012). Results of User
Interface Evaluation of Serious Games for Students with Intellectual Disability. Acta
Polytechnica Hungarica, 9(1), 225-245.
Clinical Virtual Reality
34
Larson, E.B., Feigon, M., Gagliardo, P., & Dvorkin, A.Y. (2014). Virtual reality and cognitive
rehabilitation: A review of current outcome research. NeuroRehabilitation, 34, 759-772.
DOI:10.3233/NRE-141078.
LaViola, J. J., Jr., Kruijff, E., McMahan, R. P., Bowman, D., & Poupyrev, I. P. (2017). 3D user
interfaces: Theory and practice (2nd ed.). Boston: Addison Wesley.
Levin, M., Weiss, P.L., & Keshner, E.A. (2015). Emergence of Virtual Reality as a Tool for
Upper Limb Rehabilitation. Physical Therapy, 95, 415-425.
Levy, C. E., Halan, S., Silverman, E. P., Marsiske, M., Lehman, L., Omura, D., & Lok, B. C.
(2015). Virtual environments and virtual humans for military mild traumatic brain injury and
posttraumatic stress disorder: an emerging concept. American Journal of Physical Medicine
& Rehabilitation, 94(4), e31-e32.
Liu, L., Miyazaki, M. & Watson, B. (1999). Norms and validity of the DriVR: A virtual reality
driving assessment for persons with head injuries. Cyberpsychology & Behavior, 2, 53-67.
Lucas, G.M., Gratch, J., King, A., and Morency, L.-P. (2014). It’s only a computer: Virtual
humans increase willingness to disclose. Computers in Human Behavior, 37, 94–100.
Lumbreras, M. & Sanchez, J. (2000). Usability and cognitive impact of the interaction with
3D virtual interactive acoustic environments by blind children. In Sharkey, P., Cesarani, A.,
Pugnetti, L. & Rizzo, A. (Eds.), Proceedings of the 3rd International Conference on Disability,
Virtual Reality, and Associated Technology. Reading, UK: University of Reading, 129-136.
Madary, M., & Metzinger, T. K. (2016). Real virtuality: a code of ethical conduct.
recommendations for good scientific practice and the consumers of vr-technology. Frontiers
in Robotics and AI, 3, 3.
Man, D.W.K., Ganesan, B., Yip, C.C.K., Lee, C.O.P., Tsang, S.Y.L., Yu, P.W.P., et al.
(2016). Validation of the virtual-reality prospective memory test (Hong Kong Chinese
version) for individuals with first-episode schizophrenia. Neuropsychological Rehabilitation,
http://dx.doi.org/10.1080/09602011.2016.1251949.
Maples-Keller, J. L., Bunnell, B. E., Kim, S. J., & Rothbaum, B. O. (2017). The Use of Virtual
Reality Technology in the Treatment of Anxiety and Other Psychiatric Disorders. Harvard
review of psychiatry, 25(3), 103-113.
Matheis, R., Schultheis, M.T., Tiersky, L.A., DeLuca, J., Mills, S.R. and Rizzo, A.A. (2007). Is
learning and memory different in a virtual environment? The Clin Neuropsych. 21, 146-161.
McCall, C., Blascovich, J., Young, A., & Persky, S. (2009). Proxemic behaviors as predictors
of aggression towards Black (but not White) males in an immersive virtual
environment. Social Influence, 4(2), 138-154.
McComas, J., MacKay, M. & Pivak, J. (2002). Effectiveness of virtual reality for teaching
pedestrian safety. CyberPsychology and Behavior, 5, 185-190.
McGeorge, P., Phillips, L.H., Crawford, J.R., Garden, S.E., Della Sala, S., Milne, A.B.,
Hamilton, S. & Callander, J. (2001). Using virtual environments in the assessment of
executive dysfunction. Presence-Teleoperators and Virtual Environments. 10, 375-383.
Clinical Virtual Reality
35
Merians, A. S., Jack, D., Boian, R., Tremaine, M., Burdea, G. C., Adamovich, S. V., Poizner,
H. (2002). Virtual reality–augmented rehabilitation for patients following stroke. Physical
therapy, 82(9), 898-915.
Merians, A. S., Fluet, G. G., Qiu, Q., Saleh, S., Lafond, I., & Adamovich, S. V. (2010).
Integrated arm and hand training using adaptive robotics and virtual reality simulations. In P.
Sharkey & J. Sánchez (Eds.) Proceedings of the 2010 International Conference on
Disability, Virtual Reality and Associated Technology (pp. 213-222). Reading UK: University
of Reading.
Millan, J.d.R., Rupp, R., Müller-Putz, G.R., Murray-Smith, R., Giugliemma, C., et al. (2010).
Combining brain-computer interfaces and assistive technologies: state-of-the-art and
challenges. Frontiers in Neuroscience, https://doi.org/10.3389/fnins.2010.00161
MindMaze (2017). MindMaze MindMotion, Retrieved from
https://www.mindmaze.com/mindmotion/
Moncur, M. & Moncur, L. (2002). QuotationsPage.com. At:
http://www.quotationspage.com/quotes/Plato/
Morbini, F., DeVault, D., Sagae, K., Gerten, J., Nazarian, A., & Traum, D. (2014). FLoReS: A
forward looking, reward seeking, dialogue manager. In J. Mariani, S. Rosset, M. Garnier-
Rizet, L. Devillers (Eds.). Natural interaction with robots, knowbots and smartphones (pp.
313–325). New York, NY: Springer. http://dx.doi.org/10.1007/978-1-4614-8280-2_28
Morina, N., Ijntema, H., Meyerbr€oker, K., & Emmelkamp, P. M. G. (2015). Can virtual reality
exposure therapy gains be generalized to real-life? A meta-analysis of studies applying
behavioral assessments. Behaviour Research and Therapy, 74, 18e24.
Morrison, J.E. & Meliza, L.L. (1999). Foundations of the after action review process. U.S.
Army Research Institute for the Behavioral and Social Sciences Special Report #42.
Morrongiello, B.A., Corbett, M., Switzer, J., & Hall, T. (2015). Using a Virtual Environment to
Study Pedestrian Behaviors: How Does Time Pressure Affect Children’s and Adults’ Street
Crossing Behaviors? Journal of Pediatric Psychology, 40(7), 697-703.
Mosadeghi, S., Reid, M.W., Martinez, B., Rosen, B.T., & Spiegel, B.M.R. (2016). Feasibility
of an Immersive Virtual Reality Intervention for Hospitalized Patients: An Observational
Cohort Study. JMIR Ment Health, 3(2), e28. doi: 10.2196/mental.5801.
Mowafty L. & Pollack, J. (1995). Train to travel. Ability, 15, 18-20.
Mühlberger, A., Jekel, K., Probst, T., Schecklmann, M., Conzelmann, A., Andreatta, M.,
Rizzo, A.A., Pauli, P., & Romanos, M. (2016). The Influence of Methylphenidate on
Hyperactivity and Attention Deficits in ADHD: A Virtual Classroom Test. Journal of Attention
Disorders.
Navarro, MD.; Lloréns Rodríguez, R.; Noé, E.; Ferri, J.; Alcañiz Raya, ML. (2013). Validation
of a low-cost virtual reality system for training street-crossing. A comparative study in
healthy, neglected and non-neglected stroke individuals. Neuropsychological Rehabilitation.
23(4):597-618. doi:10.1080/09602011.2013.806269.
Naveh, Y., Katz, N. & Weiss, P. (2000). The effect of interactive virtual environment training
on independent safe street crossing of right CVA patients with unilateral spatial neglect. In
Sharkey, P., Cesarani, A., Pugnetti, L. & Rizzo, A. (Eds.), Proceedings of the 3rd
Clinical Virtual Reality
36
International Conference on Disability, Virtual Reality, and Associated Technology. Reading,
UK: University of Reading, 243-248.
Neisser, U. (1978). Memory: what are the important questions? In M. M. Gruneberg, P. E.
Morris, & R. N. Sykes (Eds.), Practical Aspects of Memory. London: Academic Press. 3-24.
Neri, S.G.R., Cardoso, J.R., Cruz, L., Lima, R.M., de Oliveira, R.J., Iversen, M.D., &
Carregaro, R.L. (2017). Do virtual reality games improve mobility skills and balance
measurements in community-dwelling older adults? Systematic review and meta-analysis.
Clinical Rehabilitation. http://dx.doi.org/10.1177%2F0269215517694677
Nir-Hadad, S. Y., Weiss, P. L., Waizman, A., Schwartz, N., & Kizony, R. (2017). A virtual
shopping task for the assessment of executive functions: Validity for people with
stroke. Neuropsychological rehabilitation, 27(5), 808-833.
Norcross, J.C., Pfund, R.A., & Prochaska, J.O. (2013). Psychotherapy in 2022. A Delphi poll
on its future. Professional Psychology: Research & Practice. 44(5): 363–70.
Norrholm, S. D., Jovanovic, T., Gerardi, M., Breazeale, K. G., Price, M., Davis, M., Duncan,
E., Ressler, K.J., Bradley, B., Rizzo, A., & Tuerk, P. W. (2016). Baseline psychophysiological
and cortisol reactivity as a predictor of PTSD treatment outcome in virtual reality exposure
therapy. Behaviour research and therapy, 82, 28-37.
NVIS Inc (2017). NVIS Products, Retrieved from
http://www.nvisinc.com/product/standard.html
Ogourtsova, T., Souza Silva, W., Archambault, P. S., & Lamontagne, A. (2017). Virtual
reality treatment and assessments for post-stroke unilateral spatial neglect: a systematic
literature review. Neuropsychological rehabilitation, 27(3), 409-454.
Opris, D., et al., (2012). Virtual reality exposure therapy in anxiety disorders: A quantitative
meta-analysis. Depression and Anxiety, 29(2): p. 85-93.
Padgett, L., Strickland, D., & Coles, C. (2006). Case study: Using a virtual reality computer
game to teach fire safety skills to children diagnosed with Fetal Alcohol Syndrome (FAS),
Journal of Pediatric Psychology, 31(1), 65-70.
Papastergiou, M. (2009). Exploring the potential of computer and video games for health and
physical education: A literature review. Computers & Education, 53(3), 603-622.
Park, S., Scherer, S., Gratch, J., Carnevale, P., & Morency, L. P. (2013, September). Mutual
behaviors during dyadic negotiation: Automatic prediction of respondent reactions. In T. Pun,
C. Pelachaud, N. Sebe (Eds). 2013 Humaine Association Conference on Affective
Computing and Intelligent Interaction (ACII), (pp. 423-428). Los Alomitos, CA: IEEE.
Parsons, T.D. (2015). Virtual Reality for Enhanced Ecological Validity and Experimental
Control in the Clinical, Affective and Social Neurosciences. Front. Hum. Neurosci., 9(660),
https://doi.org/10.3389/fnhum.2015.00660
Parsons, S. Bauminger, N., Cobb, S., Gal, E., Glover, T., Weiss, P.L. Zancanaro, M., Eden,
S., Garib-Penna, S., Hawkins, T., Millen, L., & Rietdijk, W. (2012). Collaborative
Technologies for supporting social conversation and collaboration in the classroom:
overview and findings of the COSPATIAL project. 1st International Conference on Innovative
Technologies for Autism Spectrum Disorders. ASD: Tools, Trends and Testimonials in
Valencia, Spain. July 6-8, 2012.
Clinical Virtual Reality
37
Parsons, T. D., Carlew, A. R., Magtoto, J., & Stonecipher, K. (2017). The potential of
function-led virtual environments for ecologically valid measures of executive function in
experimental and clinical neuropsychology. Neuropsychological rehabilitation, 27(5), 777-
807.
Parsons, T. D. & Rizzo, A. A. (2008a). Affective Outcomes of Virtual Reality Exposure
Therapy for Anxiety and Specific Phobias: A Meta-Analysis. Journal of Behavior Therapy
and Experimental Psychiatry, 39, 250-261.
Parsons, T.D. & Rizzo, A.A. (2008b). Initial Validation of a Virtual Environment for
Assessment of Memory Functioning: Virtual Reality Cognitive Performance Assessment
Test. Cyberpsychology and Behavior, 11(1), 17-25.
Parsons, T.D., Rizzo, A.A., Rogers, S. and York, P. (2009). Virtual reality in paediatric
rehabilitation: A review. Developmental Neurorehabilitation, 12:4 224-238.
Parsons, T.D., Rizzo, A.A., Rogers, S. and York, P. (2009). Virtual reality in paediatric
rehabilitation: A review. Developmental Neurorehabilitation, 12:4 224-238.
Passig, D., Tzuriel, D., & Eshel-Kedmi, G. (2016). Improving children’s cognitive modifiability
by dynamic assessment in 3D Immersive Virtual Reality environments. Computers &
Education, 95, 296-308.
Pertaub, D-P., Slater, M., & Barker, C. (2002). An Experiment on Public Speaking Anxiety in
Response to Three Different Types of Virtual Audience. Presence. 11(1), 68-78.
Pietrzak, E., Pullman, S., & McGuire, A. (2014). Using Virtual Reality and Videogames for
Traumatic Brain Injury Rehabilitation: A Structured Literature Review. Games For Health
Journal: Research, Development, and Clinical Applications, 3(4), 202-215.
Piron, L., Tonin, P., Atzori, A. M., Zanotti, E., Massaro, C., Trivello, E., & Dam, M. (2001).
Virtual environment system for motor tele-rehabilitation. Studies in health technology and
informatics, 85, 355-361.
Powers, M. and Emmelkamp, P. M. G. (2008). Virtual reality exposure therapy for anxiety
disorders: A meta-analysis. Journal of Anxiety Disorders. 22, 561-569.
Proffitt, R., & Lange, B. (2015). Feasibility of a Customized, In-Home, Game-Based Stroke
Exercise Program Using the Microsoft Kinect® Sensor. Int J Telerehabil, 7(2), 23–34.
Pugnetti, L., Mendozzi, L., Motta, A., Cattaneo, A., Barbieri, E., and Brancotti, S. (1995).
Evaluation and retraining of adults’ cognitive impairments: Which role for virtual reality
technology? Computers in Biology and Medicine, 25, 213-227.
Putrino, D. (2014). Telerehabilitation and emerging virtual reality approaches to stroke
rehabilitation. Current Opinion in Neurology, 27(6), 631-636.
Regenbrecht, H., Hoermann, S., Ott, C., Müller, L., & Franz, E. (2014). Manipulating the
Experience of Reality for Rehabilitation Applications. Proceedings of the IEEE, 102(2), 170-
184. DOI: 10.1109/JPROC.2013.2294178.
Remsik, A., Young, B., Vermilyea, R., Kiekoefer, L., Abrams, J., et al. (2016). A review of the
progression and future implications of brain-computer interface therapies for restoration of
distal upper extremity motor function after stroke. Expert Review of Medical Devices.
http://dx.doi.org/10.1080/17434440.2016.1174572
Clinical Virtual Reality
38
Riva, G. (2011). The Key to Unlocking the Virtual Body: Virtual Reality in the Treatment of
Obesity and Eating Disorders. Journal of Diabetes Science and Technology, 5(2), 283-292.
Rizzo, A.A. (1994). Virtual Reality applications for the cognitive rehabilitation of persons with
traumatic head injuries. In Murphy, H.J. (ed.), Proceedings of the 2nd International
Conference on Virtual Reality And Persons With Disabilities. CSUN: Northridge.
(available at: http://www.csun.edu/cod/conf/1994/proceedings/Table94.htm)
Rizzo, A.A., Buckwalter, J.G., and Neumann, U. (1997). Virtual reality and cognitive
Rehabilitation: A brief review of the future. The Jour. of Head Trauma Rehab.,12(6),1-15.
Rizzo, M., Reinach, S., McGehee, D., & Dawson, J. (1997). Simulated car crashes and
crash predictors in drivers with Alzheimer disease. Archives of neurology, 54(5), 545-551.
Rizzo, A.A., Buckwalter, J.G., Humphrey, L., van der Zaag, C., Bowerly, T., Chua, C.,
Neumann, U., Kyriakakis, C., van Rooyen, A. & Sisemore, D. (2000). The Virtual Classroom:
A Virtual Environment for the Assessment and Rehabilitation Of Attention Deficits.
CyberPsychology and Behavior, 3(3), 483-499.
Rizzo, A., Schultheis, M. T., & Rothbaum, B. O. (2003). Ethical issues for the use of virtual
reality in the psychological sciences. In S. S. Bush, & M. L. Drexler (Eds.), Ethical issues in
clinical neuropsychology, (pp. 243–280), Amsterdam NL: Swets & Zeitlinger.
Rizzo, A.A., Schultheis, M.T., Kerns, K. and Mateer, C. (2004). Analysis of Assets for Virtual
Reality Applications in Neuropsychology. Neuropsychological Rehabilitation, 14(1/2), 207-
239.
Rizzo, A.A., Strickland, D. & Bouchard, S. (2004). Issues and Challenges for Using Virtual
Environments in Telerehabilitation. Telemedicine Journal and e-Health, 10(2), 184-195.
Rizzo, A.A., Bowerly, T., Buckwater, J.G., Klimchuk, D., Mitura, R., & Parsons, R.D. (2006).
A virtual reality scenario for all seasons: the virtual classroom. CNS Spectums, 11(1). 35-44.
Rizzo, A., Difede, J., Rothbaum, B.O., and Reger, G. (2010). Virtual Iraq/Afghanistan:
Development and early evaluation of a Virtual Reality Exposure Therapy System for combat-
related PTSD. Annals of the New York Academy of Sciences (NYAS), 1208, 114-125.
Rizzo, A.A., Kenny, P. & Parsons, T., (2011). Intelligent Virtual Humans for Clinical Training.
International Journal of Virtual Reality and Broadcasting, 8(3), Available:
http://www.jvrb.org/8.2011/
Rizzo, A.A., Buckwalter, J.G., Forbell, E., Reist, C., Difede, J., Rothbaum, B.O., Lange, B.,
Koenig, S. & Talbot, B. (2013). Virtual Reality Applications to Address the Wounds of War.
Psychiatric Annals, 43 (3), 123-138.
Rizzo, A.A., Cukor, J., Gerardi, M., Alley, S., Reist, C., Roy, M., Rothbaum, B.O., & Difede,
J. (2015a). Virtual Reality Exposure Therapy for PTSD due to Military Combat and Terrorist
Attacks. Journal of Contemporary Psychotherapy, 45(4), 255-264.
Rizzo, A.A., Shilling, R., Forbell, E., D., Scherer, S., Gratch, J., & Morency, L-P. (2015b).
Autonomous Virtual Human Agents for Healthcare Information Support and Clinical
Interviewing. In: Luxton, D.D. (Ed). Artificial Intelligence in Mental Healthcare Practice.
Academic Press: Oxford. 53-80.
Clinical Virtual Reality
39
Rizzo, A.A. & Talbot, T. (2016a). Virtual Reality Standardized Patients for Clinical Training.
In: Combs, C.D., Sokolowski, J.A. & Banks, C.M. (Eds). The Digital Patient: Advancing
Medical Research, Education, and Practice. Wiley: New York. 257-272.
Rizzo, A. A., Lucas, G., Gratch, J., Stratou, G., Morency, L.-P., Shilling, R., & Scherer, S.
(2016b). Clinical interviewing by a virtual human agent with automatic behavior analysis. In
P. Sharkey and A.A. Rizzo (Eds.) 2016 Proceedings of the International Conference on
Disability, Virtual Reality and Associated Technologies, (p. 57-64) Reading UK: University of
Reading. Available at:
http://www.icdvrat.org/2014/papers/ICDVRAT2014_S03N3_Rizzo_etal.pdf
Robertson, I. (1990). Does computerized cognitive rehabilitation work? A review.
Aphasiology, 4(4), 381-405.
Rose, F.D., Attree. E.A., Brooks, B.M. & Andrews, T.K. (2001). Learning and memory in
virtual environments - a role in neurorehabilitation? Questions (and occasional answers)
from UEL. Presence-Teleoperators and Virtual Environments. 10, 345-358.
Rose, D., Brooks, B.M. & Attree, E.A. (2000). Virtual reality in vocational training of people
with learning disabilities. In Sharkey, P., Cesarani, A., Pugnetti, L. & Rizzo, A. (Eds.),
Proceedings of the 3rd International Conference on Disability, Virtual Reality, and Associated
Technology. Reading, UK: University of Reading. 129-136.
Rose, F.D., Brooks, B.M. and Rizzo, A.A. (2005). Virtual Reality in Brain Damage
Rehabilitation: Review. CyberPsychology and Behavior. 8(3), 241-262.
Rothbaum, B.O. & Hodges, L.F. (1999). The Use of Virtual Reality Exposure in the
Treatment of Anxiety Disorders. Behavior Modification, 23(4), 507-525.
Rothbaum, B. O., Hodges, L. F., Kooper, R., Opdyke, D., Williford, J. S., & North, M. (1995).
Virtual reality graded exposure in the treatment of acrophobia: A case report. Behavior
Therapy, 26(3), 547-554.
Rothbaum, B.O., Hodges, L., Ready, D., Graap, K. and Alarcon, R. (2001). Virtual reality
exposure therapy for Vietnam veterans with posttraumatic stress disorder. Journal of Clinical
Psychiatry, 62, 617-622.
Rothbaum, B.O., Price, M., Jovanovic, T., Norrholm, S., Gerardi, M., Dunlop, B., Davis, M.,
Bradley, B., Duncan, E.J., Rizzo, A., Ressler, K. (2014). A Randomized, Double-blind
Evaluation of D-Cycloserine or Alprazolam Combined with Virtual Reality Exposure Therapy
for Posttraumatic Stress Disorder (PTSD) in OEF/OIF War Veterans. American Journal of
Psychiatry, 171, 640-648.
Roy, M.J., Costanzo, M.E., Blair, J.R. & Rizzo, A.A. (2014). Compelling Evidence that
Exposure Therapy for PTSD Normalizes Brain Function. Studies in HealthTechnology and
Informatics,199, 61-65.
Roy, M.J., Francis, J., Friedlander, J., Banks‐Williams, L., Lande, R.G., Taylor, P., Blair, J.,
McLellan, J., Law, W., Tarpley, V., & Patt, I. (2010). Improvement in cerebral function with
treatment of posttraumatic stress disorder. Annals of the New York Academy of
Sciences, 1208(1), 142-149.
Rutten, A., Cobb, S., Neale, H., Kerr, S. Leonard, A., Parsons, S., & Mitchell, P. (2003). The
AS interactive project: single-user and collaborative virtual environments for people with
Clinical Virtual Reality
40
high-functioning autistic spectrum disorders. Journal of Visualization and Computer
Animation. 14(5), 233-241.
Sbordone, R. J., & Long, C. (1996). Ecological validity of neuropsychological testing. Boca
Raton, FL: CRC Press.
Schell, J. (2014). The art of game design: A book of lenses. Boca Raton, FL: CRC Press.
http://dx.doi.org/10.1201/b17723
Scherer, S., Stratou, G., Lucas, G., Mahmoud, M., Boberg, J., Gratch, J., Rizzo, A.A. &
Morency, LP. (2014). Automatic Audiovisual Behavior Descriptors for Psychological Disorder
Analysis. Image and Vision Computing. 32. 648–658.
Schneider, S.M., Kisby, C.K. and Flint, E.P. (2010, December 10, 2010). Effect of virtual
reality on time perception in patients receiving chemotherapy. Supportive Care in Cancer,
19(4), 555-564. DOI: 10.1007/s00520-010-0852-7.
Schultheis M.T. & Mourant, R.R. (2001). Virtual Reality and Driving: The Road to Better
Assessment of Cognitively Impaired Populations. Presence: Teleoperators and Virtual
Environments, 10, 436-444.
Schwebel, D.C., McClure, L.A., & Severson, J. (2014). Teaching Children to Cross Streets
Safely: A Randomized Controlled Trial, Health Psychol, 33(7), 628–638.
doi:10.1037/hea0000032.
Shipman, S., and Astur, R. (2008). Factors affecting the hippocampal BOLD response during
spatial memory. Behavioural Brain Research. 187(2). 433-441.
SilverFit (2017). SilverFit Applications, Retrieved from http://silverfit.com/en/applications.
Skelton, RW, Bukach, CM, Laurance, HE, Thomas, KG, Jacobs, J. (2000). Humans with
traumatic brain injuries show place learning deficits in computer generated virtual space. J
Clin Exp Neuropsychol. 22; 157-175.
Slater, M. & Sanchez-Vives, M.V. (2016). Enhancing Our Lives with Immersive Virtual
Reality. Frontiers in Robotics and Artificial Intelligence. Published: 19 December 2016. doi:
10.3389/frobt.2016.00074.
Sohlberg, M.K.M. & Mateer, C.A. (1989). Introduction to cognitive rehabilitation: Theory and
practice. The Guilford Press, New York, NY, USA.
Sohlberg, M.K.M. & Mateer, C.A. (2001). Cognitive Rehabilitation: An Integrative
Neuropsychological Approach. The Guilford Press, New York, NY, USA.
Spek, V., Cuijpers, P., Nyklicek, I., Riper, H., Keyzer, J., & Pop., V. (2007). Internet-based
cognitive behaviour therapy for symptoms of depression and anxiety: a meta-analysis.
Psychological Medicine, 37, 319-328.
Stamm, B. H. (1998). Clinical applications of telehealth in mental health care. Professional
Psychology: Research and Practice, 29(6), 536.
Standen, P. J., Threapleton, K., Connell, L., Richardson, A., Brown, D. J., Battersby, S., . . .
Platts, F. (2014). Patients’ use of home-based virtual reality system to provide rehabilitation
of the upper limb following stroke. Physical therapy, 95(3), 350-359.
http://irep.ntu.ac.uk/id/eprint/25917/1/221608_PubSub3351_Brown.pdf
Clinical Virtual Reality
41
Stanton, D., Foreman, N. and Wilson, P. (1998). Uses of virtual reality in clinical training:
Developing the spatial skills of children with mobility impairments. In Riva, G., Wiederhold, B.
and Molinari, E. (Eds.), Virtual reality in clinical psychology and neuroscience. Amsterdam:
IOS Press. 219-232.
Strickland, D. (2001, August 16). Virtual reality and multimedia applications for mental health
assessment and treatment. Paper presented at the ACM SIGGRAPH 2001 Conference. Los
Angeles, CA. Aug. 12-17, 2001.
Talbot, T.B., Sagae, K., John, B., Rizzo, A.A. (2012). Sorting out the Virtual Patient: How to
exploit artificial intelligence, game technology and sound educational practices to create
engaging role-playing simulations. International Journal of Gaming and Computer-Mediated
Simulations, 4(3). 1-19.
Tarr, M. J., & Warren, W. H. (2002). Virtual reality in behavioral neuroscience and
beyond. Nature Neuroscience, 5, 1089-1092.
Tart, C. T. (1993). Mind embodied: Computer generated virtual reality as a new, interactive
dualism. In K. Ramakrishna Rao (Ed.), Cultivating consciousness (pp. 123–137). Santa
Barbara, CA: Praeger.
Tashjian, V.C., Mosadeghi, S., Howard, A.R., Lopez, M., Dupuy, T., et al. (2017). Virtual
Reality for Management of Pain in Hospitalized Patients: Results of a Controlled Trial. JMIR
Ment Health, 4(1), e9. doi: 10.2196/mental.7387.
Tegos, S., Demetriadis, S., Papadopoulos, P. M., & Weinberger, A. (2016). Conversational
agents for academically productive talk: a comparison of directed and undirected agent
interventions. International Journal of Computer-Supported Collaborative Learning, 11(4),
417-440.
Thiebaux, M., Marsella, S., Marshall, A. N., & Kallmann, M. (2008, May). Smartbody:
Behavior realization for embodied conversational agents. In L. Padgham & D. Parkes (Eds.),
Proceedings of the 7th international joint conference on Autonomous agents and multiagent
systems-Volume 1 (pp. 151-158). Estoril, Portugal — May 12-16, 2008 . New York, NY:
ACM.
Trost, Z., Zielke, M., Guck, A., Nowlin, L., Zakhidov, Z., France, C.R., & Keefe F. (2015). The
promise and challenge of virtual gaming technologies for chronic pain: the case of graded
exposure for low back pain. Pain Management, DOI: 10.2217/pmt.15.6.
Valladares-Rodriguez, S., Perez-Rodriguez, R., Anido-Rifon, L., & Fernandez-Iglesias, M.
(2016). Trends on the application of serious games to neuropsychological evaluation: A
scoping review. Journal of Biomedical Informatics, 64, 296-319.
http://dx.doi.org/10.1016/j.jbi.2016.10.019
Valve Corporation (2017). Steam. Retrieved from http://store.steampowered.com/
Virtual Reality Society. (2017). Applications of Virtual Reality. Retrieved from:
https://www.vrs.org.uk/virtual-reality-applications/
Wall, K.J., Cumming T.B., Koenig, S.T., Pelecanos, A.M., & Copland, D.A. (2017). Using
technology to overcome the language barrier: The Cognitive Assessment for Aphasia App.
Disability and Rehabilitation. http://dx.doi.org/10.1080/09638288.2017.1294210
Clinical Virtual Reality
42
Wiederhold, B.K. & Wiederhold, M.D. (1998). A review of virtual reality as a
psychotherapeutic tool. CyberPsychology and Behavior, 1, 45-52.
Wilson, B.A. (1997). Cognitive rehabilitation: How it is and how it might be. Journal of the
International Neuropsychological Society, 3. 487 - 496.
Wilson, B.A., Baddeley, A. Evans, J., and Shiel, A. (1994). Errorless learning in the
rehabilitation of memory impaired people. Neuropsychological Rehabilitation, 4(3), 307-326.
Wilson, B.A. & Evans, J.J. (1996). Error-free learning in the rehabilitation of people with
memory impairments. Journal of Head Trauma Rehabilitation, 11, 54-64.
Wilson, J, Onorati, K., Mishkind, M., Reger, M., & Gahm, G.A. (2008). Soldier attitudes about
technology-based approaches to mental healthcare. Cyberpsychol Behavior, 11, 767–769.
Yang, Y. D., Allen, T., Abdullahi, S. M., Pelphrey, K. A., Volkmar, F. R., & Chapman, S. B.
(2017). Brain responses to biological motion predict treatment outcome in young adults with
autism receiving Virtual Reality Social Cognition Training: Preliminary findings. Behaviour
research and therapy, 93, 55-66.
Yellowlees, P.M., Holloway, K.M., & Parish, M.B. (2012). Therapy in Virtual Environments –
Clinical and Ethical Issues. Telemedicine and e-Health, 18(7), 558-
564. https://doi.org/10.1089/tmj.2011.0195.
You, S.H., Jang, S.H., Kim, Y.H., Hallett, M., Ahn, S.H., Kwon, Y.H., Kim, J.H. and Lee, M.Y.
(2005). Virtual reality–induced cortical reorganization and associated locomotor recovery in
chronic stroke. Stroke, 36(6), 1166-1171.
Zhang, Z., Bickmore, T. W., & Paasche-Orlow, M. K. (2017). Perceived organizational
affiliation and its effects on patient trust: Role modeling with embodied conversational
agents. Patient Education and Counseling, 11, 417.
A preview of this full-text is provided by American Psychological Association.
Content available from Neuropsychology
This content is subject to copyright. Terms and conditions apply.
