Content uploaded by Robert Stone
Author content
All content in this area was uploaded by Robert Stone on Sep 03, 2014
Content may be subject to copyright.
Clinical review
Science, medicine, and the future
Virtual reality in surgery
Rory McCloy, Robert Stone
New technologies, in particular virtual reality and
robotics, will have a major impact on health care in the
next decade. Clinically validated, powerful medical
simulators are now available and in use across the
world. General surgery leads in the use of simulators,
and neurosurgery leads with augmented reality and
image guided surgery. Robotics are used in orthopaed-
ics and cardiology. Other virtual reality applications are
being used in mental health, anaesthetics, and
emergency medicine. Rapid developments in the
internet and “e-learning”domains have accelerated the
dissemination of simulation techniques, interactive 3D
images, and structured courseware. This review
describes the application of virtual reality and robotics
to surgical training and planning and the execution of
procedures in theatre and discusses the near term
future of this new technology.
Methods
A successful medical simulator or surgical system based
on virtual reality requires the participation of a team of
specialists including experts in ergonomics and applied
psychology, software engineering and digital 3D design,
electromechanical engineering, robotics, and microtech-
nology. Consequently, no single database adequately
covers all the issues involved. This review is based on our
experience supplemented with data from searches of
Medline and the Ergonomics Information Analysis
Centre (University of Birmingham) and of the internet
with various web search engines.
Historical perspective
Current virtual reality surgical systems owe their
existence to pioneering developments in the early and
middle 1980s. Organisations developing robots to
replace humans from hostile and hazardous environ-
ments
—
undersea, in nuclear installations, in space, and
on the battlefield
—
turned to an emerging technology
that seemed to offer the ideal solution. The developers
claimed that, with a special helmet equipped with head
tracking devices and 3D displays, it was possible to create
an illusion for the wearer that he or she was present in
such an environment (“telepresence”) and, with a fibre
optic glove,possible for them to use their natural skills to
control the robots to perform tasks safely and efficiently.
In the 1990s research teams, notably at the
University of North Carolina
1
and in the US Depart-
ment of Defense,
2
developed the concept of surgeons of
the future equipped with virtual reality headsets and
rehearsing real or robotic procedures using advanced
computer generated images. At the same time, the
growing market for virtual reality technologies encour-
aged pioneering (if somewhat optimistic) US compa-
nies, which were singularly responsible for fuelling the
obsession with “making surgical simulation real”.
Recent conferences and exhibitions (such as the US
hosted Medicine Meets Virtual Reality 2000
3
) suggest that
a plateau may now have been reached, with some of the
front running concepts undergoing consolidation
through clinical validation. Established products are
becoming available at prices that are affordable to most
surgical teaching institutions. In addition, many projects
are now receiving academic grant support or national
and EU funding (as with the Framework V initiative
4
).
What is virtual reality?
Virtual reality is best described as a collection of tech-
nologies that allow people to interact efficiently with
3D computerised databases in real time using their
natural senses and skills. This definition avoids any ref-
erence to a need for head mounted displays and
instrumented clothing such as gloves or suits, as was
the bias in the late 1980s and early 1990s. Although
Anticipated developments
Introduction of robot assisted and telerobotic
surgery into selected disciplines
Use of computerised simulations to train and
assess psychomotor skills needed to perform
invasive procedures
Use of virtual reality technologies to interact with
medical images for surgical planning and training
Use of virtual reality simulators to rationalise
surgical training and assessment of fitness to
practise
Use of virtual reality simulators to guide
micro-robots undertaking minimally invasive
interventions
Wolfson Centre for
Minimally Invasive
Therapy,
Manchester Royal
Infirmary,
Manchester
M13 9WL
Rory McCloy
senior lecturer in
surgery
Clinical Research
DivisionsI&II,
Surgery
(University),
University of
Manchester,
Manchester
Robert Stone
visiting professor of
virtual reality in
surgery
Correspondence to:
R McCloy
r.mccloy@
man.ac.uk
BMJ 2001;323:912–5
912 BMJ VOLUME 323 20 OCTOBER 2001 bmj.com
this so called immersive technology is still evident
today, only 10% of virtual reality applications warrant
its use. The key strength of virtual reality, be it in design
or training, is that it supports and enhances real time
interaction on the part of the user.
The application of this technology to surgical train-
ing is evident. Surgical training is expensive, and the
pressures from shortened training programmes and
reduced working hours for trainees demand that an
increasing proportion of the surgical expertise of
trainees has to be gained outside the operating theatre.
The crucial factor that will determine the uptake of vir-
tual reality technology by surgeons will be the demon-
stration that virtual reality is capable of delivering
reliable and valid training and assessment systems.
Recent evidence suggests that this is the case. Not only
has the virtual reality community produced experi-
mentally validated systems for the training and assess-
ment of surgical skills, it has done so using established
techniques that are now becoming recognised as inter-
national standards, such as the International Organis-
ation for Standardization’s ISO 13407 “Human
centred design for interactive systems” (see box).
5
Current applications
There is a substantial difference for surgical trainees
between training with artificial or inanimate tissues (such
as raw chicken) and supervised procedures on patients
in the operating theatre, with all the attendant pressures
such as time restraints and clinical governance. A wide
range of virtual reality training systems have been devel-
oped, but not all are widely available yet. A commercially
available simulator for venepuncture has force feedback
to simulate the feel of the cannula entering the skin and
vein.
6
This is suitable for training nurses, medical
students, phlebotomists, and paramedics. More complex
simulators for therapeutic gastroscopy, endoscopic
retrograde cholangiopancreatography, and colono-
scopic procedures are available for trainees in gastro-
enterology.
7
A radiological simulator provides training
in cardiac catheterisation and angiography, with real
time modelling of physiological parameters and blood
flow.
8
These virtual reality simulators offer repeatable,
logged, computerised training, often without the need
for supervision.
9
In addition, some interactive virtual reality simula-
tors that have been developed for procedures such as
lumbar puncture and brain ventricular tap are freely
available for use over the world wide web.
10
The MIST system
The MIST (minimally invasive surgical trainer) system,
a product for training and assessment of surgical
laparoscopic psychomotor skills, was originally devel-
oped by us in Britain and is now commercially
available from Mentice Medical Simulation AB,
Gothenburg, Sweden.
8
In the first step in its
development we made an ergonomic evaluation of the
psychomotor skills involved in performing laparo-
scopic surgery in theatre. This led to the identification
of a set of simplified minimally invasive part tasks that
represented the “toolkit” of skills needed for holding
tissue in an accurate and steady manner, adopting dif-
ferent styles of handling or separating tissue and
vessels, left hand and right hand instrument control,
and appropriate use of electrocautery. We found that
each simple part task could be implemented
reasonably easily within a proprietary virtual reality
software package and each could be associated with
equations of human perceptual and motor perform-
ance developed from applied psychology studies.
11 12
The MIST system’s training interface, based on
modified laparoscopic instruments, is translated into
quite simple real time 3D computer graphics that accu-
rately track and represent the movements of the instru-
ments within a virtual operating volume. In this volume,
geometric shapes that approximate to those faced
during actual operations on organs are generated for
display on the computer screen and subsequently
manipulated by a surgical trainee (fig 1 ).
13
Each task can
be programmed to deliver varying degrees of difficulty
to the surgical trainee, and his or her performance can
be recorded and saved for later replay by the supervisor
or for statistical analysis. The data can be analysed in
several ways, focusing on such aspects as accuracy and
errors, time to complete part tasks, right or left hand
performance, and even the trainee’s economy of
movement when handling the virtual instruments.
13
The
advantage of the part task approach to simulation, com-
pared with full anatomical simulation of organs and
operations, is that they train generic skills for minimal
access surgery common to many surgical disciplines
—
such as general surgery, gynaecology, thoracic and
cardiac surgery, urology, and orthopaedics.
11
Guidelines for ISO 13407 “Human centred
design for interactive systems”
• Active involvement of users
• Clear understanding of use and task requirements
• Appropriate allocation of function
• Iteration of design solutions
• Multidisciplinary design
Fig 1 Using the MIST system (Mentice Medical Simulation AB,
Gothenburg, Sweden) for training and assessment of psychomotor
skills for minimally invasive surgery
Clinical review
913BMJ VOLUME 323 20 OCTOBER 2001 bmj.com
Objective assessment of surgical skills
Many factors can influence surgical outcome, and psy-
chomotor ability is only a small part of the process of
surgery. However, objective assessment of operative
skill is fast becoming necessary.
14
The opportunities
presented by tools such as MIST go beyond those of a
computerised trainer. Randomised controlled studies
have shown that MIST can distinguish between experi-
enced surgeons and non-surgeons or inexperienced
surgeons (A G Gallagher et al, ninth annual medicine
meets virtual reality conference, Newport Beach, CA,
2001).
15 16
Other factors affecting surgical perform-
ance, such as alcohol intake and sleep, can also be
evaluated (A Chaudhry et al, medicine meets virtual
reality 7, San Francisco, CA, 1999).
17
The objective assessment of the psychomotor
aspects of surgical performance is now a practical and
affordable reality.
13
In Europe, Sweden is investing
heavily in medical virtual reality technologies: a new
part task virtual reality simulator for training
laparoscopic surgeons will be marketed in Sweden
later this year
18
as well as a full anatomical organ simu-
lator for laparoscopic surgery.
8
Training on a virtual
reality simulator has to be shown to translate into per-
formance in the operating theatre. Hence, the demon-
stration that the objective assessment of skills to
manipulate virtual 3D structures from a 2D monitor
display is a significant predictor of laparoscopic surgi-
cal performance represents an important advance (A
G Gallagher et al, ninth annual medicine meets virtual
reality conference, Newport Beach, CA, 2001). Another
possible use of virtual reality simulators might be to
select medical students or young graduates on their
aptitude for surgical skills.It may also be possible to use
this type of simulator to check on the psychomotor
skills of experienced surgeons to ensure their
competence to continue to practise. These systems are
attracting the attention of the Royal Colleges of
Surgery and are already an integral part of training
courses at the European Surgical Institute, Hamburg,
Germany, where the MIST system is already a manda-
tory component of basic and advanced courses cover-
ing a wide range of surgical techniques (T Buerger and
M Erdtmann, ninth annual medicine meets virtual
reality conference, Newport Beach, CA, 2001).
Surgical planning
Virtual reality technologies allow an operation to be
practised, and the outcome viewed, before the patient
undergoes surgery
—
such as in breast reconstruction
and corrective maxillofacial surgery). Thus, the surgical
approaches can be optimised and rehearsed, with
obvious advantages for patients and healthcare provid-
ers. Virtual reality for surgical planning and training
purposes is being investigated in an EU (Framework V)
funded project called IERAPSI (integrated environ-
ment for rehearsal and planning of surgical interven-
tions) (R J Stone, project IERAPSI: a human-centred
definition of surgical procedures. Deliverable D2 (Part
1) for EU contract No IST-1999-12175; May, 2000).
The investigators used the same human factors
techniques as those used during the early work on
MIST (detailed surgical task analyses, elicitation of sur-
geons’ knowledge during video replay and focused
interviews) to analyse actual surgical procedures
involving mastoidectomy, cochlear implantation, and
acoustic neuroma resection. The results of these analy-
ses highlighted the need for a virtual reality system to
plan operations and train surgeons. The IERAPSI
training system will be based on new technologies,
including a stereoscopic virtual reality microscope and
a special desktop stylus device called PHANTOM
(SensAble Technologies, Cambridge MA, USA). The
PHANTOM is being programmed to deliver a wide
range of force and touch (haptic) effects to trainees’
hands (fig 2), including the sensation of drilling
through different densities of bone.
Challenges of introducing virtual reality
into routine surg ical practice
Little is known about the computer literacy of the
medical profession, but computer anxiety and aliena-
tion are real problems
19
and healthcare professionals
over the age of 30 are regarded as the “lost generation”
with regard to information technology.
20
The NHS is
now making a substantial effort to introduce health
informatics to hospital doctors and has been piloting
Fig 2 The PHANTOM haptic feedback device (SensAble Technologies,
Cambridge MA, USA). The user holds a small stylus and explores, in
this case, a 3D virtual molecular structure. Each time the 3D cursor
makes contact with part of the structure, small motors work in
tandem to restrict the movement of the stylus, thereby creating a
sense of touch (reproduced with permission).
Fig 3 The da Vinci Surgical System (Intuitive Surgical, California,
USA) for performing minimally invasive surgery. The surgeon sits at
a control console with 3D visualisation of the surgical field and the
robotic surgical instruments (reproduced with permission)
Clinical review
914 BMJ VOLUME 323 20 OCTOBER 2001 bmj.com
the introduction of the European Computer Driving
Licence, which is set to become the baseline qualifica-
tion for future jobs involving information technology.
21
The wide acceptance of high tech surgery, such as
using videos in laparoscopic procedures, has overcome
many surgical prejudices, but virtual reality and robots
are still viewed as gimmicks or potentially dangerous
by most surgeons. More multidisciplinary teams will be
required to develop the use of these new technologies
in surgery. As with so many advances in health care,
however, these new technologies are likely to increase
costs, against which must be weighed the potential for
improved surgical competence and reduced medical
error, with reduced morbidity and mortality. Evidence
for these benefits is likely to take at least 5-10 years to
accumulate.
The future
Apart from the development of current applications,
virtual reality also probably has a role in such exciting
developments as microsurgery and nanosurgery. Just as
virtual reality was first developed to control macro-scale
robots for use in hazardous environments, future micro-
robots and even nanobots, such as the DNA “screw-
driver,”
22
designed for use within the human body will
need supervision by skilled operators equipped with
advanced virtual reality equipment. Already a German
company has produced a “micro-submarine” powered
by an induction motor; at 4 mm long and 650 ìmin
diameter, it is small enough to pass down a hypodermic
needle and has the potential for various diagnostic or
therapeutic applications.
23
Miniature cameras that can
be swallowed and transmit images of the gastrointestinal
tract to a viewing station are being used to survey the gut
in a variety of diseases.
24 25
The fact that European and US developments in
micro-lasers and micro-manipulators, such as the da
Vinci Surgical System (fig 3), are now being used for
master-slave robotic procedures such as minimally
invasive coronary artery bypass grafting and laparo-
scopic surgery
26
is proof of the application of virtual
reality technology and suggests that a revolution in
medical instruments and in the training of those who
use them is closer than many people realise. In keeping
with the early attempts to introduce technologies such
as virtual reality and robotics into other markets, a sea
change in medical opinion will be required and a mas-
sive learning curve will have to be overcome if the
advances already achieved, never mind those to come,
are to be translated into realities in health care.
Funding: None.
Competing interests: None declared.
1 Brooks FP. Grasping reality through illusion
—
interactive graphics serving
science. In: CHI ’88, Conference proceedings on human factors in computing
systems. New York: Association for Computing Machinery, 1988:1-11.
2 Satava RM. Medicine 2001: the king is dead. In: Morgan R, Satava RM,
Sieburg HB, Matthews R, Christensen JP, eds. Interactive technology and the
new paradigm for healthcare. Amsterdam: IOS Press, 1995:334-9.
3 Westwood JD, Hoffman HM, Mogel GT, Robb RA, Stredney D, eds. Medi-
cine meets virtual reality 2000. Envisioning healing: interactive technology and
the patient-practitioner dialogue. Amsterdam: IOS Press, 2000.
4 European Commission. Fifth framework programme. www.cordis.lu/
fp5/home.html (accessed 7 Sep 2001).
5 EMMUS (European MultiMedia Usability Services). Introduction to ISO
13407. www.ucc.ie/hfrg/emmus/methods/iso.html (updated 29 Sep
1999).
6 Reachin. Reachin API. www.reachin.se/products/reachinapi/ (accessed 7
Sep 2001).
7 Simbionix. www.simbionix.com (accessed 7 Sep 2001).
8 Mentice Medical Simulation. www.mentice.com (accessed 7 Sep 2001).
9 Stone RJ. The opportunities for virtual reality and simulation in the
training and assessment of technical surgical skills. In: Surgical competence:
challenges of assessment in training and practice. London: Royal College of
Surgeons of England, 1999:109-25.
10 John NW, Riding M, Phillips NI, Mackay S, Steineke L, Fontaine B, et al.
Web-based surgical education tools. In: Westwood JD, Hoffman HM,
Mogel GT, Stredney D, Robb RA, et al. Medicine meets virtual reality 2001.
Amsterdam: IOS Press, 2001:212-7.
11 Stone RJ, McCloy RF. Virtual environment training systems for
laparoscopic surgery; activities at the UK’s Wolfson Centre for Minimally
Invasive Therapy. J Med Virtual Reality 1996;1(2):42-51.
12 Boff KR, Lincoln JE, eds. Engineering data compendium: human perception
and performance. Vols 1-4. Dayton, OH: Armstrong Aerospace Medical
Research Laboratory, Wright-Patterson Air Force Base, 1988.
13 Wilson MS, Middlebrook A, Sutton C, Stone R, McCloy RF. MIST VR: a
virtual reality trainer for surgery assesses performance. Ann R Coll Surg
Engl 1997;79:403-4.
14 Darzi A, Smith S, Taffinder N. Assessing operative skills. BMJ
1999;318:887-8.
15 Chaudhry A, Sutton C, Wood J, Stone R, McCloy R. Learning rate for
laparoscopic surgical skills on MIST VR, a virtual reality simulator: qual-
ity of human-computer interface. Ann R Coll Surg Engl 1999;81:281-6.
16 Taffinder N, Sutton C, Fishwick RJ, McManus IC, Darzi A. Validation of
virtual reality to teach and assess psychomotor skills in laparoscopic sur-
gery: results from randomised controlled studies using the MIST VR
laparoscopic simulator. In: Westwood JD, Hoffman HM, Stredney D,
Weghost SJ, eds. Medicine meets virtual reality. Amsterdam: IOS Press,
1998:124-30.
17 Taffinder NJ, McManus IC, Gul Y, Russell RCG, Darzi A. Effect of sleep
deprivation on surgeons’ dexterity on laparoscopic simulator. Lancet
1998;352:1191.
18 Surgical Science. Surgical Science
—
cutting edge in surgical simulation.
www.surgical-science.com (accessed 7 Sep 2001).
19 Ray NM, Minch RP. Computer anxiety and alienation: towards a
definition and parsimonious measure. Hum Factors 1990;32:477-91.
20 Kidd MR, McPhee W. The “lost generation”: IT education for healthcare
professionals. Med J Aust 1999;171:510-1.
21 NHS Information Authority. Ways of working with information.
http://www.nhsia.nhs.uk/wowwi/pages/default.asp (accessed 7 Sep 2001).
22 Mao C, Sun W, Shen Z, Seeman NC. A nanomechanical device based on
the B-Z transition of DNA. Nature 1999;397:144-6.
23 MicroTec. The world’s smallest submarine. www.microtec-d.com/
microsubmarine.htm (accessed 7 Sep 2001).
24 Bradbury J. Journey to the center of the body. Lancet 2000;356:2074.
25 Appleyard M, Glukhovsky A, Swain P. Wireless-capsule diagnostic endos-
copy for recurrent small-bowel bleeding. N Engl J Med 2001;344:232-3.
26 Intuitive Surgical. Da Vinci Surgical System. www.intusurg.com/html/
davinci.html (accessed 7 Sep 2001).
Additional educational resources
Books
• Stanney KM, ed. Virtual environments handbook. Mahwah, NJ: Lawrence
Erlbaum Associates (in press). (see http://vehand.engr.ucf.edu/
revised2.htm)
• Karkowski W, ed. International encyclopaedia of ergonomics and human factors.
London: Taylor and Francis, 2001
Websites
• Surgery at the Manchester Royal Infirmary. www.biomedical.demon.
co.uk/vsurgery.htm (accessed 7 Sep 2001). (Links to laparoscopic surgery,
MIST, robotics, and other related surgical technology sites)
• Internet resources of computer aided surgery. http://homepage2.nifty.
com/cas/ (accessed 7 Sep 2001). (Includes surgical planning, surgical
navigation, image guided surgery, and surgical robotics)
• Emerson T, Prothero J, Weghorst S. Medicine and virtual reality: a guide
to the literature (MedVR). Washington DC: Human Interface Technology
Laboratory. www.hitl.washington.edu/projects/knowledge_base/medvr/
medvr.html (accessed 7 Sep 2001)
• NHS Information Authority. Ways of working with information.
www.nhsia.nhs.uk/wowwi/pages/default.asp (accessed 7 Sep 2001).
(Education, training, and development programme
—
includes details on
health information, information technology, basic information technology
skills, and the European Computer Driving Licence project)
• Mentice Medical Simulation. www.mentice.com (accessed 7 Sep 2001).
(Details of MIST system and surgical simulation)
• Surgical Science. Surgical Science
—
cutting edge in surgical simulation.
www.surgical-science.com (accessed 7 Sep 2001). (Details of the LapSim
laparoscopic simulator)
• Web-based surgical simulators and medical education tools
http://try.at/virtual.surgery (accessed 7 Sep 2001). (Details of lumbar
puncture, ventricular tap, etc)
Clinical review
915BMJ VOLUME 323 20 OCTOBER 2001 bmj.com
