ArticlePDF Available

Interactive Scenario Immersion: Health Emergency Decision Training in JUST Project

Authors:

Abstract and Figures

The paper discusses the main requirements, constraints and challenges involved in practical realization of an immersive VR situation training system that would support simulation of interactive scenarios of various type. A special attention is paid to the demanding health emergency decision training domain. As an example an immersive JUST VR health emergency training system built in frame of EU IST JUST project is presented in more detail.
Content may be subject to copyright.
Interactive Scenario Immersion:
Health Emergency Decision Training in JUST Project
Michal Ponder (*), Bruno Herbelin (*), Tom Molet (**),
Sebastien Schertenleib (*), Branislav Ulicny (*), George Papagiannakis (**),
Nadia Magnenat-Thalmann (**), Daniel Thalmann (*)
(*) Virtual Reality Lab (VRlab)
Swiss Federal Institute of Technology (EPFL)
(**) MIRALab, University of Geneva
Abstract
The paper discusses the main requirements, constraints and challenges involved in
practical realization of an immersive VR situation training system that would support
simulation of interactive scenarios of various type. A special attention is paid to the
demanding health emergency decision training domain. As an example an immersive
JUST VR health emergency training system built in frame of EU IST JUST project is
presented in more detail.
Keywords
VR situation training system, health emergency, VR immersion, interactive scenario
1 Introduction
One of the effects of the rapid technological progress that we witness nowadays is a
quickly rising need for more effective training methodologies concerning broad and
continuously growing spectrum of human activities. Using those novel methodologies
individuals should be able to learn and then continuously maintain and upgrade their
knowledge and skills.
Nowadays common training methods include formal classes, books, multimedia
applications, interactive simulations, on-job training, etc. The latter, on-job training
method is particularly effective in complex tasks where a great deal of independence is
granted to the task performer. Unfortunately, on-job training is the most expensive.
Furthermore it is limited by the need of an experienced personnel that would be able to
conduct such training as well as by frequent unavailability of the training context itself.
Due to the reasons given above, in the recent years, many Virtual Reality (VR)
applications has been implemented that try to address the challenge of effective training
addressing nowadays requirements. Successful use stories as well as new attempts can
be found in a very broad range of domains such as space and aviation (e.g. flight and
mission simulators), medicine (e.g. surgery), industry (e.g. machine operation, mining),
psychology (e.g. public speech training), emergency (e.g. fire fighting, mining, healt
emergency), etc. An overview of many techniques used to implement VR applications in
different domains is given in [1][2].
Use of VR is particularly useful when the training domain is complex and difficult to
master and when the audio-visual (possibly assisted by haptic feedback) features of the
training environment are crucial to the overall training success. This makes virtual
environments the solution of choice for practicing and learning domains where the
context of the training is not easily available, such as health emergency situations.
Most of the VR-based medical trainers developed up to date target training of the
specific manual procedures and skills involving advanced medical instruments (e.g.
laparoscopy, neurosurgery, orthopedics, etc.) [3][4][5][6]. They fall into the category of
task training. In contrast, health emergency trainers should allow the trainees to develop
their psychological skills required to face reality of an emergency site. In effect the
training should lower the psychological barriers by focusing on rapid situation
assessment and decision-making under highly stressful conditions. This type of
application falls into the situation training category.
In the following sections of this paper we will discuss the main issues involved in a
practical realization of a VR situation training system targeting health emergency
domain. We will conclude with a concrete example of the health emergency training
system (JUST VR System) developed in frame of the EU IST JUST. As a project JUST
addresses the domain of training of civil non-professional health emergency operators
(citizens volunteers).
2 Problem: Requirements, Constraints, Open Issues
2.1 Health Emergency Training: Requirements
In health emergency care, a clear distinction is made between a) the advanced
training aimed at highly qualified professionals and b) the training of para-medical
personnel (including citizen volunteers). The scope of the second type of training
includes the following three key aspects:
- knowledge,
- skills,
- facing reality.
While knowledge and skills can be efficiently acquired through the methods
currently at hand (e.g. courses, books, multimedia applications, use of manikin and real
devices, etc.), the third aspect of the training is still being achieved through the risky,
leaving no place for mistakes, on-job training.
In course of the on-job training a trainee is to face an emergency situation reality in
usually difficult to predict context (e.g. day, night, busy street, metro station,
appartment, empty park, crying people, etc.). The main stress factors are uncertainty of a
health emergency situation (limited cues, confusion), pressure of passing time and
consequences of mistakes (risk, responsibility). In such conditions the trainee should be
able to asses the situation and make a sequence of decisions in order to perform the most
suitable medical rescue procedure. In effect of an on-job experience a trainee is
supposed to lower psychological barriers, handle stress, use knowledge, make decision,
take responsibility and as a result build self-confidence. Unfortunately the main inherent
problems of the on-job training are no room for mistakes, not easily available training
context (random, not controlled) making it in effect very risky, long and expensive.
2.2 VR Situation Training: Promising Option
Taking into account the very recent rapid advancements of the real-time computer
graphics and virtual character simulation technologies it seems that VR becomes a
promising option to address the current problem related to health emergency on-job
training. In short the main objective of a required VR situation training system would be
immersion of the trainee inside an interactive scenario. The system would need to
provide the following three functional elements:
(1) Immerse a trainee in virtual environment:
Taking into account the current state of the art of the VR immersion hardware, and
the main requirement of the VR situation training assuming various combinations of
medical cases and contexts, this part requires special attention and will be treated further
in more detail.
(2) Simulate a health emergency situation:
Provision of an interactive, multi-path, believable health emergency scenario is a
very challenging task. The system must seamlessly combine multiple hardware devices
and various heterogeneous simulation and interaction technologies that while being there
must at the certain moment become invisible to the trainee in order to generate the
required sense of presence. Special attention must be paid to: 3D graphical rendering,
3D surround sound rendering, realistic virtual human modeling and animation,
consistent and natural behaviors of the virtual humans and the elements of the virtual
scene, intuitive and possibly invisible multi-modal interaction and finally tools allowing
for authoring, execution and control of the interactive scenarios.
(3) Let the trainee act and handle stress factors:
When using the system actions of a trainee should be focused on the situation
assessment and decision making. The stress factors would be a direct result of the
uncertainty of a health emergency situation, surrounding virtual environment conditions,
pressure of passing time and a fear of making decisions combined with a need to take
responsibility of the decisions made.
2.3 VR Immersive Technologies: Constraints
From the technological point of view, the VR challenge did not change much since
Sutherland’s first experiments [7]: trying to achieve sense of presence as an interface
metaphor to a virtual world. Despite of the significant technological improvements over
the last 40 years, the intrinsic limitations of the immersive hardware and the man-
machine communication complexity are still there. As Gobbetti et al.[8] conclude: it is
common to have misconceptions on what VR can and cannot do, and to have negative
reactions when noticing that VR “is not that real”.
In the case of health emergency situation training, we face the same major classes of
problems. The interaction devices available do not allow for a large scene exploration
and a direct manipulation simultaneously. The two are actually mutually exclusive
because of the haptic feedback kinaesthetic interfaces cumbersomeness.
Finally the medical users expect a highly realistic simulation, particularly in case of
the human body appearance and behaviours. This of course conflicts with the real-time
requirements of such interactive applications and thus must lead to certain trade-offs.
2.4 Existing Approaches and Solutions: Some Examples
The existing systems targeting health emergency training vary substantially in scope
of immersion, interaction and scenarios offered to the trainees. Majority of them are
console based featuring strong multimedia character with 3D graphics and sound as a
main immersion methods. The Virtual Medical Trainer/Trauma Patient Simulator from
Research Triangle Institute is a console based application showing 3D model of a virtual
human with realistic visible injuries and internal trauma that exhibit medical signs and
symptoms with real-time, physiological behavior. LIMIT from HITL Washington
University uses Augmented Reality (AR) and focuses rather on medical data
representation providing only rudimentary models of virtual humans. The Medical
Readiness Trainer from University of Michigan focuses on combination of state of the
art instrumented manikin technology with some elements of VR.
Medical Emergency Training: MediSim and BioSimMER Projects
MediSim and BioSimMER from Sandia Laboratories are design for training first-
responders in military situations (battle field and bio-terrorism). In order to constrain VR
simulation requirements and objectives the range of MediSim cases has been limited to
the medical assessment and stabilization tasks. In BioSimMER, the trainees enter an
airport contaminated with an unknown agent to evacuate the victims.
Stanfield et al.[9] present the VR platform used for this application. The VR interface
is immersive; the medic/trainee wears a Head Mounted Display (HMD) and a set of four
position trackers. The navigation in the Virtual Environment (VE) is allowed by two
locomotion modes interfaced with the body movements (fly or walk modes). Through a
specific interaction paradigms employing inverse kinematics and interpretation of hand
motion, the trainee can act directly on the patient and perform such actions as head
bandaging or in mouth J-tube insertion. The BioSimMER system is also capable of
supporting multiple trainees. They could see each other and act in a collaborative way
using specific signs.
As for all VR application, the critical choices have been made in the design and the
development of the system. The hardware configuration actually uses SGI Octane
workstations running across a LAN multiple instances of the program. The simulation
software is based on SGI Performer (actually porting to PC OpenGL) and take full
advantage of Ethernet multicasting.
Then, decision was made to adopt fully immersive devices (HMD and sensors). This
is also motivated by the particular dressing of the emergency responders in contaminated
environment that limits the field of view in the same way as HMD. Natural navigation is
allowed for close interactions, but some locomotion paradigms are introduced to move
within the VE beyond the physical limitations imposed by the range of motion of the VR
tracking devices. The trainee is represented as tracked full-body avatar; he sees through
his eyes and controls his movements. To achieve this goal with a limited set of sensors
(on the head, the lower back, and each hand), the avatar driver computes pelvis and legs
positions with inverse kinematics algorithms and interpolates the torso and the head
positions. In order to fit the animation skeleton with the user’s body (i.e. resize height),
the system imposes a calibration phase in four steps. Then, the trainee can act in the VE
through his avatar; proximity-based grasping and “let go” gesture (twisting the wrist) are
the only requested paradigms. Indeed, smart objects are designed to automatically
perform the actions; pop into place (J-tube in the mouth), dock at the appropriate place
the appropriate time (injection), initiates an action of the avatar (unrolling the bandage),
or change its state. This simplifies the interactions but also introduce some artefacts. For
example, when performing an injection, the avatar remains bound in position to the auto-
injector even though the user moves his hand. Anyway, the only solution to this would
be a force-feedback interface, and this would introduce many other problems. A
particular attention is paid to the interactions with the virtual patients. Limited voice
interactions are possible thanks to a commercial speech-recognition software; given a
list of recognized words, the trainee may check the responsiveness of the patient that can
be programmed to answer to a specific key-word. The virtual humans are also
specifically designed for medical purpose (i.e. positive pupillary response test).
Over the 23 test subjects, a global satisfactory opinion was given over the system.
However, according to Stanfield et al., the comments collected from the questionnaires
“indicates need for improvements in input devices, naturalistic handling and use of
virtual objects”.
As there is no existing equivalent system, it is not possible to compare and to
determine whether or not this kind of training is efficient. Nevertheless, this shows the
complexity of such a system and presents interesting solutions to the VR intrinsic
difficulties.
2.5 Open Issues
Immersion
The “direct immersion” paradigm implemented usually through a head Mounted
Display (HMD) and various combinations of human body tracking technologies
introduces many limitations and artifacts: mapping between the virtual and the real
although direct is usually very imprecise leading to user confusion, very limited field of
view and cyber-sickness effects may lead even to injuries. Direct immersion, even if
combined with limited and cumbersome nowadays haptic devices, does not solve a
problem how to interact with the virtual objects. Moreover use of any haptic devices
automatically limits the range of the scenarios that the system may support which
conflicts with the main requirement of the situation training. In effects it seems that the
immersion must be based on certain metaphors and extensive use of audio-visual
sensory channel. Ideally, technologies should be hidden as much as possible; the
equipment should be light and the interface intuitive.
Haptic Feedback
As already mentioned above the haptic feedback devices use in case of generic
situation training is rather difficult if not prohibitive due to the functional constraints that
such devices impose. In case of health emergency training one could consider a use of
instrumented manikin as the most natural haptic feedback devices. Unfortunately in case
of HMD based immersion it would require tracking of the manikin posture leading to
virtual-real correspondence problems due to the tracking imprecision and after all still
such procedures like mouth-to-mouth breathing would be impossible due to the HMD
dimensions.
Interactive scenario
In majority of the VR applications and particularly in case of the skill training
systems a user faces an interactive scene. Such a scene contains multiple virtual objects
that undergo interaction and respond with certain behaviors (interaction with VR space).
In case of the situation training we need to bring this idea to the higher level: an
interactive scenario. Ideally and interactive scenario should tell a timeline story of
pedagogical nature leaving at the same time clear places for trainee’s interactions and
decisions that affect scenario direction (interaction with VR space and time). At present
interactive story telling is an intensive research topic with promising results but still
there is no clear and widely accepted guidelines how to capture the problem and deliver
a practical implementation.
Multi-modal interaction
Speech recognition/generation is an evolving and promising research topic.
However, its limitations force the user to adopt a limited vocabulary that is not sufficient
in many cases. Moreover, the users in an emergency situation may not have time to think
about the exact words or to repeat the command several times. Gestures are also often
used as a sign language in specific professional spheres. It would be appropriate to
investigate this possibility in addition to the vocal interaction.
3 JUST VR System
3.1 Approach and Concept
The main goal of the JUST VR situation training system is provision of a quasi-
random health emergency scenario that would expose a trainee to an immersive VR
simulation allowing check his decision-making capabilities under possibly close to real
stressful conditions.
As JUST project is targeting civilian applications we had to face a very challenging
list of requirements (advanced VR system, easy to use, easy to maintain, extendible) and
constraints ( limits of VR hardware, limited budget, affordable PC platform). Finally it
was important that the high-end VR hardware components could be replaced by low-end
ones and the system is still fully functional so that the particular installations of the same
system may range from high-VR-end (e.g. stereo vision, surround sound, natural
navigation) to low-multimedia-end (e.g. monitor, stereo speakers, mouse navigation).
Fig. 1 JUST VR System concept: key elements.
Being conscious of the limitations of the “direct immersion”, haptic feedback
technological limits and finally the above list of requirements and constraints we have
proposed a system concept captured schematically in (Fig. 1). In short, during the
interactive scenario the trainee faces a huge rear-projection screen displaying stereo
simulation PC
simulation
control PC
simulation supervisor
trainee
navigation inside
virtual environment
communication
virtual victim
virtual
assistant
rear stereo
projection
images of the simulation. He is immersed in surround sound. In course of the simulation
he is able to navigate freely around the virtual environment in order to locate an
emergency site. The intuitive navigation paradigm is based on a single magnetic tracker
attached to the trainee’s head. On the emergency site the trainee is interacting with his
Virtual Assistant (VA) using natural voice commands and hearing respective replies and
prompts from VA. The role of the trainee is to asses the situation and give commands to
the VA who is executing them showing proper manual skills. Simulation supervisor
stays behind the scenes and has the following tasks: direction of the scenario path,
“speech recognition” of the voice commands given by the trainee and triggering
respective actions to be executed by VA (the supervisor can be regarded as the “ears of
the VA”), putting pressure on the trainee by triggering of prompts spoken by VA or
triggering of “disturbing” virtual events.
Audio-Visual Immersion
The visual immersion is realized through the rear projection and active stereo. The
trainee wears lightweight (compared to HMD) shutter glasses and faces the screen of the
~3.0x2.8m size. Due to the rear projection he is able to approach the screen very closely
without any shadow being cast on the screen. Compared to HMD this solution is less
cumbersome, we avoid problems of very limited field of view, cyber sickness (related to
the lack of peripheral vision) and claustrophobic effects in case of some more sensitive
trainees. The trainee faces the virtual wall being a natural extension of the reality into a
different, virtual dimension. Finally compared to HMD this solution offer high
scalability from stereo capable high-end CRT project through cheap LCD projector
(loosing stereo) to simple monitor.
Concerning audio immersion the trainee is surrounded by 5.1 home cinema speaker
system that can be as well scaled down to simple stereo one.
Navigation in Virtual Environment
Navigation paradigm is based on a single magnetic tracker attached to the trainee’s
head (Fig. 2). Ascensions PC Bird is used for this purpose. In order to “walk around”
the virtual environment the trainee needs to step into the navigation ring which in effect
triggers camera motion in the desired direction. The trainee can still move inside the
central area of the ring without causing any camera motion. In order to “look around”
the trainee needs to look at the margins of the projection screen. This analogously
triggers horizontal and vertical camera rotation.
The paradigm is lightweight and easy intuitive to understand. Moreover following
the requirement of the modularity and scalability the magnetic tracker can be replaced
by a wireless hand held mouse or a normal mouse in case of scale down.
Multi-modal Interaction: Trainee, Virtual Assistant, Simulation Supervisor
While targeting situation training we had to address an inherent lack of generic
haptic feedback solution that would support whole spectrum of scenarios and simulation
contexts. In order to do so we have clearly separated decision making from decision
execution by introduction of the Virtual Assistant (VA) concept. In course of the
simulation the trainee (decision maker) navigates, assesses the health emergency
situation and issues natural voice commands to the VA. He may use any type of wording
or language (facilitates localization issues). VA (decision executor) waits for commands
and executes them showing correct medical skills. It refuses to execute erroneous
decisions prompting the trainee for retrial. In case the trainee is slow in reactions or
unable to act the VA prompts him to hurry up or auto-executes proper actions
respectively. A simulation supervisor (ears of VA) directs and advances interactive
scenario execution by triggering actions available at the given scenario step and ordered
by the trainee.
Fig. 2 JUST VR System navigation paradigm: “navigation ring” metaphor
allowing for “walking around” the virtual environment; “sliding margin”
metaphor allowing for “looking around” the virtual environment.
Interactive Scenario: Authoring and Execution
As it has been already mentioned above definition and handling of an interactive
scenario (as opposed to the interactive scene) is still an open issue. Nevertheless for the
purpose of the JUST VR System we elaborated our own semantics allowing for
representation and semantics of the interactive scenario as a tree-like graph of states.
During execution the scenario advances through the subsequent scenario steps. At each
scenario step there is a set of actions available (defined by medical procedures) and the
trainee must command his VA to execute one of them e.g “Do chest compressions”,
“Check responsiveness”, “Make mouth to mouth breathing”, etc.
Fig. 3 JUST VR System scenario execution: the main GUI used by the simulation
supervisor to direct interactive scenario execution
Only correct actions are available on each step. In this way the trainee is protected
from making bad decisions which has very important educational impact. In case of such
attempt to make a bad decision the supervisor may trigger one of the optional actions
that make the VA say e.g. “No, I do not think I should do that”, “Are you sure ?”, etc.
The optional actions which are always available are used by the simulation supervisor to
prompt the trainee to hurry up and to put in general some pressure on the trainee e.g
“OK, and now ?!”, “Hurry up, tell me what to do!”. In case the trainee is not able to
scenario tree expands
step by step in course
of the scenario execution
current
scenario
step
scenario actions available
at the current scenario step
optional actions
(always available)
make a correct decision the supervisor advances the scenario further automatically
which is another stress generating factor for the trainee who starts to understand that he
cannot keep up with the scenario pace.
3.2 Implementation Details
Hardware Platform
One of the challenges we had to face in case of JUST project was implementation of
the advanced immersive VR system on affordable PC platform with the use of off-the-
shelf VR devices. It was important as well that the system is modular enough so that if
necessary it is possible to migrate smoothly from the high VR end to low multimedia
end of the spectrum. As a result the system is implemented on two networked PCs
running under Windows 2000. One of the PCs equipped with the high-end graphics
board (Quadro7 XGL 900) and EAX compatible 5.1 surround sound board (Sound
Blaster Audigy) is responsible for the hosting and execution of the simulation. Another,
standard PC is responsible for simulation control and display of the respective control
GUIs operated by simulation supervisor.
Software Platform
The overall JUST VR System software architecture is developed based on the
VHD++ real-time development framework [10] being a proprietary middleware solution
of both MIRAlab and VRlab laboratories. VHD++ is a highly flexible and extendible
real-time framework supporting component based development of interactive
audio-visual simulation applications in the domain of VR/AR with particular focus on
virtual character simulation technologies. It uses C++ as the main implementation
language with possibility of scripting in Python. The most important features and
functionalities of the VHD++ framework are: a) support for real-time audio-visual
applications, b) extendible spectrum of heterogeneous simulation technologies provided
in from of plug-able vhdServices, c) extendibility and scalability, e) runtime flexibility:
XML based system and content configuration, f) complexity curbing: multiple design
patterns improve clarity while abstraction levels simplify implementation constructs, h)
large scale code reuse: fundamental components providing core system level
functionalities and readymade components (vhdServices) encapsulating heterogeneous
simulation level technologies.
The JUST system architecture is designed around a vhdRuntimeEngine being an
active software element power supplying set of plug-able vhdServices that encapsulate
the following main simulation technologies: 3D stereoscopic rendering, 3D sound, VR
navigation, virtual human animation, skin deformation, face animation, speech,
behaviors, interactive scenario authoring and execution, etc.
4 First Results
At the moment JUST VR System enters its validation phase focusing mainly on the
usability, sense of presence and educational aspects. Currently the system offers two
multi-path interactive scenarios built around Basic Life Support medical procedures and
involving cardiac arrest. The first scenario takes place late at night, on the empty office
floor. The second scenario is situated in the city park at night. Snapshots showing system
in use and the two respective scenarios are presented in (Fig. 4), (Fig. 5), (Fig. 6).
Already at the beginning it is important to note that given current state of the art of
motion capture technology (we used optical motion capture: Vicon) it is still very
difficult to generate high-quality human motion data required to visualize precise
medical procedures (especially in case of hands). Nevertheless it seems to be of
secondary importance in case of situation training where the focus is on situation
assessment and decision taking. It is as well very difficult to synchronize such high-
quality motion data when played at the same time on the Virtual Victim and the Virtual
Assistant. Finally, concerning the usability tests, the assessment may be difficult due to
the fact that there is no similar systems that could be used for comparison.
After the very first test it seems that the overall system is easy to use and maintain.
Navigation and interaction paradigms seem to be intuitive and trainees get used to them
quickly. As the system supports scalability it is possible to use it equally with high-end
VR immersion hardware as well as in the desktop configuration by just changing
configuration files (helps development). The technology is hidden from the trainee and
the use of trainee-VA natural speech communication, as expected, is natural and robust.
Fig. 4 A trainee interacting with the system: hiding technology, natural voice
interaction with the virtual assistant, natural navigation in virtual
environment.
Fig. 5 Crating a health emergency situation training context: late evening, empty
office, the trainee and his virtual assistant try to help the victim found on
the floor.
Fig. 6 Creating a health emergency situation training context: city at night, a
victim found in the public park, the trainee and his virtual assistant try to
asses the situation and handle the emergency situation.
5 Conclusions
In this paper we have presented the main requirement, constraints and challenges
involved in the practical realization of an immersive VR situation training system
targeting in particular health emergency training domain as a promising alternative for
risky, long and costly on-job training.
We showed as well that despite of revolutionary progress in real time graphics and
virtual character simulation technologies still many open issues remain and they are
related mainly to the VR immersion, haptic feedback , interactive scenario (as opposed
to the interactive scene) and multi-modal interaction paradigms limitations.
Finally, using an example of the JUST VR health emergency training system we
showed that some of the problems can be avoided by careful system conceptualization,
selection of a proper hardware and its use in context of proper interaction paradigms.
Concerning the future development of the JUST VR system we plan to a) elaborate
more complex health emergency scenarios, b) investigate system usability in more detail
(sense of presence, educational aspects, stress factors), c) try to use the system in
applications related to mental health and rehabilitation requiring immersion and active
role of the user as a main decision maker e.g. public speech, phobias etc.
The latter is based on our strong belief that the JUST VR system can address much
broader scope of applications, reaching far beyond the initially aimed health emergency
training domain.
6 Acknowledgments
Presented JUST VR system research and development is supported by the Swiss
Federal Office for Education and Science in frame of EU IST JUST project.
7 References
[1] R.S. Kalawsky, The Science of Virtual Reality and Virtual Environments,
Addison-Wesley, 1993, ISBN 0-201-63171-7
[2] G.Burdea, P.Coiffet, Virtual Reality Technology, Wiley Interscience, John Wiley
& Sons, Inc. 1993, ISBN 2-866601-386-7
[3] M. Dinsmore, N. Langrana, G. Burdea, J. Ladeji, Virtual Reality Training
Simulation for Palpation of Subsurface Tumors, Proceedings of the Virtual
Reality Annual International Symposium, VRAIS’95, March 1-5, Albuquerque,
New Mexico, 54-60, 1995
[4] N. J. Avis, I. P. Logan, D. P. M. Wills, The use of Virtual Environments to Assist
the Teaching of Knee Arthroscopy Procedures, Royal Society conference on VR
in Society, Engineering and Science, Chapman-Hall, London, July 1995
[5] S. L. Delp, J. P. Loan, M. G. Hoy, F. E. Zajac, E. L. Topp, and J. M. Rosen, An
Interactive Graphics-Based Model of the Lower Extremity to Study Orthopaedics
Surgical Procedures, IEEE Transactions on Biomedical Engineering, vol 37, no 8,
August, 1990
[6] H. Delingette, S. Cotin, and N. Ayache, A Hybrid Elastic Model Allowing Real-
Time Cutting, Deformations, and Force-Feedback for Surgery Training and
Simulation, Proceedings of Computer Animation’99, Geneva, 70-81, 1999
[7] Sutherland I. E. The ultimate display. In Proceedings of IFIPS Congress (New
York City, NY, May 1965), vol. 2, pp. 506508
[8] Enrico Gobbetti and Riccardo Scateni. Virtual reality: Past, present, and future. In
G. Riva, B. K. Wiederhold, and E. Molinari, editors, Virtual Environments in
Clinical Psychology and Neuroscience: Methods and Techniques in Advanced
Patient-Therapist Interaction, pages 3-20. IOS, Amsterdam, The Netherlands,
November 1998
[9] Stanfield S, Shawver D, Sobel A, Prasad M, Tapia L. Design and Implementation
of a Virtual Reality System and Its Application to Training Medical First
Responders. In PRESENCE Teleoperators and virtual environments, vol 9(6),
pp.524-556, MIT Press, December 2000
[10] Michal Ponder, George Papagiannakis, Tom Molet, Nadia Magnenat-Thalmann,
Daniel Thalmann, VHD++ Framework: Extendible Game Engine with Reusable
Components for VR/AR R&D featuring Advanced Virtual Character Simulation
Technologies, submitted to IEEE VR 2003.
... Life support training has become more important in recent decades, creating mass training possibilities that have the potential to significantly reduce the number of deaths due to sudden cardiac arrest [1,2]. In emergency health care, it is crucial to know the differences between advanced training targeted at qualified professionals and training of the general population and paramedics [3]. Qualified professionals need to know how to use mechanical tools and drugs, in addition to performing lifesaving tasks, and general population cohorts need to know and understand how to perform basic manual skills (eg, maintaining an open airway and performing chest compressions) while waiting for professional help. ...
... The first example of a semi-immersive VR tool was JUST VR, a semi-immersive tool used for the training of nonprofessionals [3]. In this simulation, users face a screen that presents an interactive medical emergency. ...
... In this simulation, users face a screen that presents an interactive medical emergency. During the simulation, trainees can move; this is enabled by a magnetic sensor placed on their head; also, they can vocally interact with a virtual assistant (controlled by a technician) who is giving commands on how to manage the emergency situation [3]. Even though this proof of concept might be helpful for learning within the CPR algorithm, it does not provide haptic feedback or effectively train for manual skills. ...
Article
Full-text available
The use of augmented reality (AR) and virtual reality (VR) for life support training is increasing. These technologies provide an immersive experience that supports learning in a safe and controlled environment. This review focuses on the use of AR and VR for emergency care training for health care providers, medical students, and nonprofessionals. In particular, we analyzed (1) serious games, nonimmersive games, both single-player and multiplayer; (2) VR tools ranging from semi-immersive to immersive virtual and mixed reality; and (3) AR applications. All the toolkits have been investigated in terms of application goals (training, assessment, or both), simulated procedures, and skills. The main goal of this work is to summarize and organize the findings of studies coming from multiple research areas in order to make them accessible to all the professionals involved in medical simulation. The analysis of the state-of-the-art technologies reveals that tools and studies related to the multiplayer experience, haptic feedback, and evaluation of user’s manual skills in the foregoing health care-related environments are still limited and require further investigation. Also, there is an additional need to conduct studies aimed at assessing whether AR/VR-based systems are superior or, at the minimum, comparable to traditional training methods.
... To teach CPR, different learning strategies have been proposed such as classical teaching strategies with mannequins, interactive videos and 3D simulation scenarios [13][14][15][16]. Below, we review some of them. ...
... Other applications are the Mini-Virtual Reality Enhanced Mannequin (Mini-VREM) [25] which is a CPR feedback device with motion detection technology including Kinect, sensor and software specifically designed to analyze chest compression performance and provide realtime feedback in a simulation training setting; AED Challenge [26], an application that provides online automated external defibrillation and CPR skill practice and testing with realistic scenarios; First-Aid game [28], a video game addressed to middle and high school students based on a low-cost software simulation with photo-realistic scenarios and simple interactions to introduce the theory of basic life support and the correct performance of its procedures; and LIFESAVER [16] a simulator proposed by the UK Resuscitation Council that fuses interactivity with live-action film. In the serious games context, some games for CPR training are JUST VR System [14] an immersive virtual reality situation training system for non-professional health emergency operators; MicroSim Prehospital [13] designed for pre-hospital training on emergency medical services, and Staying alive [27], an online 3D simulator which provides a learning experience of saving a virtual patient from cardiac arrest in 4 min; Relive [24] a first person 3D adventure where the player faces different rescue situations and can test the quality of his CPR, directly; Viva!Game [29], a Web-based serious game designed to create awareness on cardiac arrest and cardiopulmonary resuscitation [24]; and HeartRun [31], a mobile simulation game to train resuscitation and targeted at giving school children an understanding of CPR and getting them to take action. ...
... Continue with chest compressions and rescue breathes in a ratio of 30:2 and stop to recheck the victim only if he starts to wake up: to move, opens eyes and to breathe normally. Otherwise, do not interrupt resuscitation (evaluated in ITEM 11,12,13,14). (vii) If the victim recovers put in the lateral security position. This protocol varies when the AED arrives. ...
Article
Cardiopulmonary resuscitation (CPR) is a first aid key survival technique used to stimulate breathing and keep blood flowing to the heart. Its effective administration can significantly increase the chances of survival for victims of cardiac arrest. LISSA is a serious game designed to complement CPR teaching and also to refresh CPR skills in an enjoyable way. The game presents an emergency situation in a 3D virtual environment and the player has to save the victim applying the CPR actions. In this paper, we describe LISSA and its evaluation in a population composed of 109 nursing undergraduate students enrolled in the Nursing degree of our university. To evaluate LISSA we performed a randomized controlled trial that compares the classical teaching methodology, composed of self-directed learning for theory plus laboratory sessions with a mannequin for practice, with the one that uses LISSA after self-directed learning for theory and before laboratory sessions with a mannequin. From our evaluation we observed that students using LISSA (Group 2 and 3) gave significantly better learning acquisition scores than those following traditional classes (Group 1). To evaluate the differences between students of these groups we performed a paired samples t-test between Group 1 and 2 (μ1=35, 67, μ2=47, 50 and p<0.05) and between students of Group 1 and 3 (μ1=35, 67, μ3=50, 58 and p<0.05). From these tests we observed that there are significant differences in both cases. We also evaluated student performance of main steps of CPR protocol. Students that use LISSA performed better than the ones that did not use it. Copyright © 2015 Elsevier Ireland Ltd. All rights reserved.
... On the other hand, there are handheld device applications such as CPR Game [23], a cardiac arrest simulator on iOS platform focused on advanced CPR training; iResus [24], an application for smart phone, designed to improve International Journal of Computer Games Technology 3 Table 1: Some of the proposed applications to introduce CPR or some of the steps of the protocol. Application type Video training Save-a-Life simulator [21], CPR-and-Choking [22] Handheld device CPR Game [23], iResus [24], iCPR [25,26], M-AID [27], CPR simulator [21], and LifeSaver [28] PC platform Mini-VREM [29], AED Challenge [30] Serious games JUST [31], MicroSim [32], Staying Alive [33], LISSA [34], Relive [35], Viva!Game [36], and HeartRun [37] the performance of an advanced life support provider in a simulated emergency situation; iCPR [25,26], an iPhone application designed for both lay persons and healthcare professionals able to detect the rate of chest compressions performance by using the built-in accelerometer; M-AID [27], a first-aid application for mobile phones that uses yes or no questions to judge an ongoing situation giving to the user detailed instructions of how to proceed; CPR simulator [21], a set of CPR exercises including adult, child, and infant CPR simulator that runs through the CPR sequence; and LifeSaver [28] a simulator proposed by the UK Resuscitation Council that fuses interactivity with live-action film. In addition, some applications for PC platforms are Mini-Virtual Reality Enhanced Mannequin (Mini-VREM) [29] which is a CPR feedback device with motion detection technology including Kinect, sensor and software specifically designed to analyse chest compression performance and provide realtime feedback in a simulation training setting; and AED Challenge [30], an application that provides online automated external defibrillation and CPR skill practice and testing with realistic scenarios. ...
... In addition, some applications for PC platforms are Mini-Virtual Reality Enhanced Mannequin (Mini-VREM) [29] which is a CPR feedback device with motion detection technology including Kinect, sensor and software specifically designed to analyse chest compression performance and provide realtime feedback in a simulation training setting; and AED Challenge [30], an application that provides online automated external defibrillation and CPR skill practice and testing with realistic scenarios. Finally, in the serious games context, some games for CPR training are JUST [31], an immersive virtual reality situation training system for nonprofessional health emergency operators; MicroSim Prehospital [32] designed for prehospital training on emergency medical services; and Staying Alive [33], an online 3D simulator which provides a learning experience of saving a virtual patient from cardiac arrest in four minutes. LISSA [34] is a serious game with actions turning around an emergency situation represented in a 3D virtual environment with the victim, the helper, and all the auxiliary tools that may require the emergency. ...
Article
Full-text available
Cardiopulmonary resuscitation (CPR) is a first-aid key survival technique used to stimulate breathing and keep blood flowing to the heart. Its effective administration can significantly increase the survival chances of cardiac arrest victims. We propose 30 : 2, a videogame designed to introduce the main steps of the CPR protocol. It is not intended for certification and training purpose. Driven by the 2010 European Resuscitation Council guidelines we have designed a game composed of eight mini games corresponding to the main steps of the protocol. The player acts as a helper and has to solve a different challenge. We present a detailed description of the game creation process presenting the requirements, the design decisions, and the implementation details. In addition, we present some first impressions of our testing users (25 children, five of each age from 8 to 12 years old and 12 males and 13 females). We evaluated clarity of instructions and three settings of the game: the aesthetics of scenarios, the playability, and the enjoyability of each mini game. All games were well punctuated, and there are no significantly differences between their sex. The proposed game can be a suitable tool to disseminate and promote CPR knowledge.
... In the field of cardiopulmonary resuscitation (CPR), Jerin et al proposed the automated external defibrillator challenge, which is a web-based serious game for teaching and training automated external defibrillation and first-aid maneuvers to lay people and emergency medical service professionals [49]. Other proposals are JUST [50], an immersive virtual reality situation training system for nonprofessional health emergency operators; MicroSim Prehospital [51] designed for prehospital training on emergency medical services; and Staying alive [52], an online 3D simulator that provides a learning experience of saving a virtual patient from cardiac arrest in 4 minutes; LISSA [53,54], which presents an emergency situation wherein CPR actions have to be applied to save the person with choking; 30:2 [55], a game designed to educate on CPR protocol to nonexperts; Relive [56], a first person 3D adventure where the player faces different rescue situations; Viva!Game [57], a web-based serious game designed to create awareness on cardiac arrest and CPR; and HeartRun [58], a mobile simulation game to train resuscitation and targeted at giving school children an understanding of this protocol. Recently, Benkhedda and Bendella [59] presented FASim, a 3D serious game that combines health care simulations with serious games and the functionalities of the multiagent systems in a single framework in order to learn the first-aid procedure for and the signs of a cardiac arrest. ...
Article
Full-text available
Background Choking is one of the causes of unintentional injury death. Gaining the knowledge of the first-aid procedure that has to be applied in case of choking can increase the chances of survival of persons with choking. Serious games can be a good channel for educating people about choking scenarios and the actions to be taken to save the persons with choking. Objective The objective of this study is to present and evaluate the effectiveness of a serious game designed to prevent choking and to promote the first-aid procedure that needs to be applied in case of choking. Methods In this study, we present a serious game as a set of minigames that reproduces the main steps of the protocol for the first-aid performed in choking. In the proposed game, the player acquires the role of a helper who has to save the person in a choking emergency by applying the main steps of the protocol. Time and score restrictions are imposed to pass each minigame. To test this game, we performed a pilot study with 48 high school students. Different tests were performed to assess the students’ preferences and their knowledge on choking before and after playing the proposed game. The obtained results were analyzed using Mann-Whitney U test when a grade variable was involved and by using Fisher exact test when 2 categorical variables were involved. ResultsThe findings of our study showed that the players enjoyed the game. No statistical differences were detected when considering the gender of the player, their preferences for video games, or their previous experience in choking emergencies. By comparing the knowledge of these students before and after playing the game, we found that all the indicators of the knowledge about how to act in case of a choking emergency were improved through this serious game. Conclusions The findings of our study show that the proposed game is a good strategy for promoting and teaching first-aid procedures in choking emergencies to nonexperts in this field.
... The strategies proposed to teach CPR range from classical teaching with mannequins, through interactive videos to 3D simulation scenarios and serious games. Examples of virtual scenarios can be found in Ponder et al. (2002), Laerdal (2012), Ilumens (2011) and Wattanasoontorn, Boada, Sbert, Olivet, and Juvinya (2014), and in the context of serious games, in Semeraro et al. (2014), VivaGame (2013), Schmitz, Klemke, Walhout, and Specht (2015) and Boada, Rodr ıguez-Ben ıtez, Garc ıa-Gonz alez, Thi o-Henestrosa, and Sbert (2016). Unfortunately, visually impaired players have many difficulties to access these applications, since most of them rely on impressive graphics and immersive visual experiences. ...
Article
Cardiopulmonary resuscitation (CPR) is a first aid survival technique used to recover victims of cardiac arrest. To learn CPR different handheld device applications have been proposed. Most of them are based on sophisticated graphics and interaction techniques that make them inaccessible to visually impaired people. In this paper, we propose CPRforBlind, a video game designed to introduce the main steps of the CPR protocol to visually impaired people. The game is composed of different mini‐games inspired by the CPR steps proposed by European Resuscitation Council. The player acts as a helper who has to save the victim by using tactile interaction to solve the challenges assigned to each mini‐game. To test the game, 16 participants from which 8 are blind have been considered. After and before playing the game participants answered different questionnaires related to CPR knowledge, playability and enjoyability. The obtained results are very promising detecting a great improvement on CPR knowledge. In addition, blind players enjoyed the game and found it easy to play.
... MR games [4] are able to elude one's cognitive system and produce the sense of being physically present in the real as well as the virtual gaming world. This "sense of being there" is also known as presence [6,27]. Presence is considered as a paramount factor when using MR environments, especially considering digital heritage sites, since it gives an indication of whereas the virtual scenario has the ability to drain the subject into it, as a "trip to the past" [3]. ...
Chapter
Full-text available
In recent years, the popularity of mixed reality (MR) environments has increased as they provide attractive and immersive experiences for educational, entertainment, and training purposes. The increasing advances of Augmented Reality (AR) and Virtual Reality (VR) hardware and software technologies also constitute an interesting area of research. In this work, we present the main pipeline followed for creating virtual character-based AR experiences, specifically in cultural heritage environments. Our main goal in this chapter is to compare different software methodologies for creating VR environments and present a complete novel methodology for authoring life-sized AR virtual characters and life-sized AR crowd simulation using only modern mobile devices. One important aspect of these environments that we focus on is creating realistic and interactive virtual characters via procedurally generated body and facial animations which are illuminated with real environment light. Virtual characters' transformations are handled efficiently using a single mathematical framework, the 3D Euclidean geometric algebra (GA), and the conformal geometric algebra (CGA) which is able to handle translations, rotations, and dilations. Using such a single algebraic framework, we avoid conversions between different mathematical representations; as a result, we achieve more efficient performance. We also compare the efficiency of different GA code generators—(a) the Gaigen library, (b) libvsr, and (c) Gaalop—so that a future user of GA can choose the most appropriate, currently available s/w library that will provide the most optimal and efficient results. Our main research involves the following questions: (a) Are novel, low-cost, modern HMDs suitable as VR platforms? (b) Which are the most appropriate s/w platforms needed to realize such VR digital heritage gamified experiences? (c) Can we achieve more efficient AR scene authoring? (d) Can we achieve more efficient AR animation interpolation and skinning using a single mathematical framework employing GA?
... For the past decade, the field of health education and health care has started to investigate how digital games can assist in teaching first-aid concepts and cardiopulmonary resuscitation (CPR). The range of games currently available provides vital functional elements to address the problems related to health emergency training[18]. Staying alive 1 , for example, is a video-based 3D game experience that requires players to apply their CPR knowledge to a virtual person suffering from cardiac arrest. Other examples are LIVESAVER 2 , a crisis simulator that fuses interactivity with live-action film to show how to respond to heart attacks or Relive[23], a first person 3D adventure to increase kids and young adults' awareness on the need of instant CPR. ...
Conference Paper
Full-text available
Out of the project EMuRgency a game-based learning environment evolved, which trains school children in providing reanimation and cardiopul-monary resuscitation (CPR). The application gets players to act as if they were in a real case of emergency. This paper reports on a formal usability study con-ducted with two different groups of learners, regular learners and learners with special educational needs (SEN). With the study we compared the two groups of learners with regard to game usability and effectiveness of the intervention. Our intention was to better understand the different needs and requirements to learning materials that game designer need to take into consideration in order to make the learning experience successful for both groups. A total of 89 children played the game simulation. Results showed differences in perception and ef-fectiveness of individual mechanisms for the two groups with regard to usabil-ity or switching between tasks and mobile device.
Chapter
‘User-experience’ reviews key literature surrounding arguably the central piece of the VR puzzle, the user. The chapter begins with a brief outline of physiological issues, specifically adverse effects pertaining to cybersickness, before addressing psychological components of user-experience including flow, diegesis and fun. Throughout this chapter, the role of sound is considered as we observe how every facet of user-experience has a significant connection to sound. This chapter closes with a discussion of the implications of VR for identity and our understanding of ourselves. Taking notions from digital games a step further, the central assertion raised here is that VR is dramatically changing our concept of the Self by presenting us with seemingly limitless opportunities for customising our existing identity and forming entirely new ones.
Conference Paper
Nowadays, users are able to interact with digital content using their mobile devices almost everywhere and anytime due to the increased power, portability, and ubiquitous connectivity of mobile devices. This paper presents the design and implementation of a novel, remotely controlled for the purposes of edutainment and instructor-student interaction, three-dimensional full body avatar gamification framework. The main innovation introduced focuses on multi-presence gamified educational scenarios in multiple desktop computers and mobile devices. Thus the remotely-controlled avatar can act as a guide, assistant or information presenter for novel, cross-platform Ambient Intelligence (AmI) edutainment scenarios. In detail, the avatar's role depends on the requirements of the AmI client applications as these are propagated remotely (using remote procedure calls). Examples of remote function invocations include real-time 3D biped skinned animations, text-to-speech, producing facial expressions and presenting multimedia content.
Thesis
Full-text available
Today, pupils at the age of 15 have spent their entire life surrounded by and interacting with diverse forms of computers. It is a routine part of their day-to-day life and by now computer-literacy is common at very early age. Over the past five years, technology for teens has become predominantly mobile and ubiquitous within every aspect of their lives. To them, being online is an implicitness. In Germany, 88% of youth aged between 12-19 years own a smartphone and about 20% use the Internet via tablets. Meanwhile, more and more young learners bring their devices into the classroom and pupils increasingly demand for innovative and motivating learning scenarios that strongly respond to their habits of using media. With this development, a shift of paradigm is slowly under way with regard to the use of mobile technology in education. By now, a large body of literature exists, that reports concepts, use-cases and practical studies for effectively using technology in education. Within this field, a steadily growing body of research has developed that especially examines the use of digital games as instructional strategy. The core concern of this thesis is the design of mobile games for learning. The conditions and requirements that are vital in order to make mobile games suitable and effective for learning environments are investigated. The base for exploration is the pattern approach as an established form of templates that provide solutions for recurrent problems. Building on this acknowledged form of exchanging and re-using knowledge, patterns for game design are used to classify the many gameplay rules and mechanisms in existence. This research draws upon pattern descriptions to analyze learning game concepts and to abstract possible relationships between gameplay patterns and learning outcomes. The linkages that surface are the starting bases for a series of game design concepts and their implementations are subsequently evaluated with regard to learning outcomes. The findings and resulting knowledge from this research is made accessible by way of implications and recommendations for future design decisions.
Conference Paper
Full-text available
We describe the basic components of a surgery simulator prototype developed at INRIA. After a short presentation of the geometric modeling of anatomical structures from medical images, we insist on the physical modeling components which must allow realistic interaction with surgical instruments. We present three physical models which are well suited for surgery simulation. Those models are based on linear elasticity theory and finite element modeling. The first model pre-computes the deformations and forces applied on a finite element model, therefore allowing the deformation of large structures in real-time. Unfortunately, it does not allow any topology change of the mesh therefore forbids the simulation of cutting during surgery. The second physical model is based on a dynamic law of motion and allows to simulate cutting and tearing. We called this model “tensor-mass” since it is analogous to spring-mass models for linear elasticity. This model allows volumetric deformations and cuttings, but has to be applied to a limited number of nodes to run in real-time. Finally, we propose a method for combining those two approaches into a hybrid model which may allow real time deformations and cuttings of large enough anatomical structures. This model has been implemented in a simulation system and real-time experiments are described and illustrated
Article
Full-text available
This paper presents the design and implementation of a distributed virtual reality (VR) platform that was developed to support the training of multiple users who must perform complex tasks in which situation assessment and critical thinking are the primary components of success. The system is fully immersive and multimodal, and users are represented as tracked, full-body figures. The system supports the manipulation of virtual objects, allowing users to act upon the environment in a natural manner. The underlying intelligent simulation component creates an interac- tive, responsive world in which the consequences of such actions are presented within a realistic, time-critical scenario. The focus of this work has been on the training of medical emergency-response personnel. BioSimMER, an application of the system to training first responders to an act of bio-terrorism, has been imple- mented and is presented throughout the paper as a concrete example of how the underlying platform architecture supports complex training tasks. Finally, a prelimi- nary field study was performed at the Texas Engineering Extension Service Fire Protection Training Division. The study focused on individual, rather than team, in- teraction with the system and was designed to gauge user acceptance of VR as a training tool. The results of this study are presented.
Article
Full-text available
This report provides a short survey of the field of virtual reality, highlighting application domains, technological requirements, and currently available solutions. The report is organized as follows: section 1 presents the background and motivation of virtual environment research and identifies typical application domain, section 2 discusses the characteristics a virtual reality system must have in order to exploit the perceptual and spatial skills of users, section 3 surveys current input/output devices for virtual reality, section 4 surveys current software approaches to support the creation of virtual reality systems, and section 5 summarizes the report.
Article
We live in a physical world whose properties we have come to know well through long familiarity. We sense an involvement with this physical world which gives us the ability to predict its properties well. For example, we can predict where objects will fall, how well-known shapes look from other angles, and how much force is required to push objects against friction. We lack corresponding familiarity with the forces on charged particles, forces in non-uniform fields, the effects of nonprojective geometric transformations, and high-inertia, low friction motion. A display connected to a digital computer gives us a chance to gain familiarity with concepts not realizable in the physical world. It is a looking glass into a mathematical wonderland. Computer displays today cover a variety of capabilities. Some have only the fundamental ability to plot dots. Displays being sold now generally have built in line-drawing capability. An ability to draw simple curves would be useful. Some available displays are able to plot very short line segments in arbitrary directions, to form characters or more complex curves. Each of these abilities has a history and a known utility.
Article
This book goals hits basic idea: giving a review on Virtual Reality and on the theory at its basis. Starting from history, going trought science as medicine, arriving at technical stuff this is in syntesis this book. For everyone interested in "How it work?". Only one problem : sometimes is too difficoult to understand at no graduate persons dues to hi level techincal explanations in particoular in display chapter
  • G Burdea
  • P Coiffet
G.Burdea, P.Coiffet, Virtual Reality Technology, Wiley Interscience, John Wiley & Sons, Inc. 1993, ISBN 2-866601-386-7