Conference PaperPDF Available

Abstract and Figures

Haptics technologies are frequently used in virtual environments to allow participants to touch virtual objects. Medical applications are no exception and a wide variety of commercial and bespoke haptics hardware solutions have been employed to aid in the simulation of medical procedures. Intuitively the use of haptics will improve the training of the task. However, little evidence has been published to prove that this is indeed the case. In the paper we summarise the available evidence and use a case study from interventional radiology to discuss the question of how important is it to touch medical virtual environments?
Content may be subject to copyright.
The Need to Touch Medical Virtual Environments?
Fernando Bello Timothy R Coles* Derek A Gould Christopher J Hughes*
Nigel W. John* Franck P Vidal§ Simon Watt*
*Bangor University, UK
Imperial College London, UK
Royal Liverpool Hospital, UK
§University of California, San Diego
Haptics technologies are frequently used in virtual
environments to allow participants to touch virtual objects.
commercial and bespoke haptics hardware solutions have
Intuitively the use ofhaptics w ill improve the training of the
task. However, little evidence has been published to prove
available evidence and use a case study from interventional
INDEXTERMS: H.5.1[Information InterfacesAndPresentation]:
Multimedia Information Systems—Artificial, augmented, and
virtualrealities;J.3[ComputerApplications]:Lifeand Medical
Today, more than at any time in the history of medicine,
thereisunrelentingpressurefor changestoaccepted medical
practice, particularly as a consequence of legislation such as
theEuropeanworking timedirective,andtheCalmanreforms
intonovelminimal access approaches, in turnraising further
a part of such innovative practices. Safe, effective training of
the next generation healthcare professional can benefit by
force and/or tactile feedback within a simulator is a prime
haptics device. A detailed survey of the  current state‐of‐the‐
A complete evaluation of transfer of skills to patients,
comparing training of a control group( notusing simulation)
againsttwo study groups(oneusingsimulation with, andone
without haptics feedback) has yet to be published. This has
demonstrated that recall following visuohaptic training is
significantly more accurate than recall following visual or
conventional and robot‐assisted laparoscopic minimally
MISinterventions.Whilst amajorityofthe studiesresultedin
a positive assessment of the inclusion of force feedback,
are carried out using thin elongated instruments inserted
through ports that may create frictional forces in excess of 3
Newtons [6] capable of interfering with more subtle haptic
severalmedical specialtieswherereacting tohapticscues isa
interventional radiology, arthroscopy and internal
are true for any of these specialties has not yet been
investigated.Inthispaper wediscussfurtherifthere isa true
need for haptics devices within a medical simulator and use
The practice of interventionalradiology(IR)usesimaging
(fluoroscopic, computed tomography, and ultrasound) to
guide catheters (tubes) and wires through organ systems
usinga small portal of entry (suchas a needle)into thebody
[7]. As a result, IR techniques generally have less risk, less
post‐operative pain and shorter recovery time compared to
proceduresthat are minimally invasive.Reactingcorrectly to
practitioner’s ‘feel’ during a procedure is essential to avoid
cues are important and have been implemented in our
2.1 Palpation
Palpation is the use of a clinician’s sense of touch to probe
deeply beneath the patient’s skin, seeking evidence of any
pathology in the underlying anatomical structures. Palpation
andgeneralhapticresponse commonly requires multi finger,
multi contact tactile manipulations. Such an effect is difficult
to achieve within a medical virtual environment and when
 Itisnecessarytosimulatebothforcefeedbacktoconveythe
resistive force of the skin, organs and bones and, tactile
small abnormalities felt at the surface of the fingertips.
in the development of force feedback devices, a lack of
understanding of the large number of different tactile
available commercially. Possible technologies include use of
piezoelectric materials (as in Fig. 1), the vibrations from a
small audio speaker, pin arrays, and pneumatic solutions.
Practically, stimulating each of the fingertips’ receptors as
stimulated in a real palpation is infeasible, however, and an
approximationusingaforcefeedbackdevice combinedwitha
mannequinlike end effector appears to currently provide the
Figure 1. Palpation device combining force feedback from a
NovINT Falcon and tactile feedback at the fingertips from
piezoelectric materials.
2.2 Needle Insertion
Aneedle insertion is awidelyperformedprocedure which, in
wire into the femoral artery, liver, kidney, etc. The task
requires 6 DOF but force feedback can be realistically
theforces involved inrotatingthe needle shaftalongwhichit
is inserted. Many needle insertion simulations, both
commercial and academic opt to simulate only 3 degrees of
force feedback to reduce simulation cost for example
Mediseus Epidural from MedicVision (Kensington, Australia)
 UltrasoundimagesarecommonlyusedinIRtoguide
needles.Prior tothe needle insertion, an anaesthetic solution
feltduring the needleinsertion.Thenthe ultrasound probe is
showsa simulationofthisstage usingcommerciallyavailable
forcefeedbackdevices.Itshouldbevisible on the ultrasound
location within the patient. Once the needle is visible on the
should be identified on the ultrasoundimagesusedfor
needle tip, the resistance on the needle during the insertion
will provide invaluable information. For example, at the
interface between two kinds of tissue, the resistance will
increase until the surface of the deeper tissue is punctured,
thenit willdecrease.Also,the force required ontheneedleis
greater for the kidney than fat, and it is greater for the
diseased, cirrhotic liver than the normal kidney. Therefore,
such features are also required when teaching image guided
needlepuncture using a VRtraining simulator. The response
of real tissues during an actual needle insertion can be
Figure 2. Using PhanToM Omni Force feedback joysticks as an
ultrasound probe and virtual needle.
2.3 Guidewire and Catheter manipulation
greater tactile feedback, which is highly relevant to avoiding
complications and maintaining safe practice. For this reason,
realistic IR catheterisatonsimulations cannotusejoystick like
interfaces. Instead, frictional sensors provide essential
Replication of haptics in simulations is necessarily an
procedures performed in the real world. This in itself is
challengingwork, whichmustuse unobtrusive, novel sensors
procedure in a patient. At the same time, collection of
procedural force data might indicate the basis of haptics
during one or more actions. Such data can then be used as a
basis for refining algorithms, and identifying the actual
Figure 3. Simulator for the Seldinger Technique. Real guide wires
and catheters can be fed through a needle portal into the virtual
patient. Custom hardware has been designed to orientate the
needle (inset picture) and provide force feedback.
A major goal fo r a successful outco me is not to penetra te a
canresult in penetration (particularly incompleteocclusions,
where the force to continue along the lumen can be very
similar to that which allows wall perforation). Responding
need for multidisciplinary collaboration to build an effective
simulator is advocated. Other important points made are the
steps, unlimited deliberate practice with th is repeatability  of
procedures facilitating learning from mistakes, the provision
ofobjective feedback, andtheneed to integratethesimulator
always necessary to achieve ultra high fidelity in order to
 Basicresearchoncross‐modalperceptiondoes indicate the
likelyimportanceofhaptics in virtual medical environments.
Its potential contribution is most readily apparent when
visionand haptics providequalitativelydifferent information.
about spatial properties, whereas haptics provides
information about texture, and material properties such as
stiffness/compliance [12]. Recent research shows, however,
vision.In this case perceptionis not dominated by onesense
(i.e. vision). Instead, information from vision and haptics is
unconsciously and automatically integrated, with each signal
weighted according to how reliable it is in a given instance
morepreciselythanispossible from either sense alone. If
sensory integration occurs in real‐world bimodal tasks, it
process of sensory integration only operates appropriately,
however, if the brain can determine which visual and haptic
informationaboutthesameobjectand whichdo not[14][15].
integrationhas been observed whenusingtools,forexample,
which systematically alter the normal spatial relationships
betweenvisual and haptic signals [15].However,spatial(and
temporal) congruency between movements of the hand
and/or surgical instruments and the visual consequences of
to occur [14][15]. This suggests that it is critical to achieve
spatial and temporal co‐registration of visual and haptic
bimodaltaskssuch as surgical procedures. There is evidence
andhaptic information is presented simultaneously(asisthe
they are temporally separated [16]. It seems plausible, then,
thattraining inthesame(bimodal) sensory conditionsasthe
realtask could lead bothtobetterlearningandbetteroverall
The current published evidence clearly demonstrates that
VR simulation can improve intra‐operative performance. We
advocate that good use of haptics (e.g. force responses are
accurately modeled, the resolution of the haptics device is
reacting to haptics cues is a vital part of a successful
use the needle puncture simulator to set up a test study to
[1] K. Calman, “Hospital doctors: training for the future”, British
[2] F.D.Skidmore,“Juniorsurgeonsarebecomingdeskilledasresult
ofcalmanproposals”, Britishmedicaljournal, vol.327,1997, pp.
[3] T.Coles,DMeglanandN.W.John,“TheRoleofHapticsinMedical
[4] D.Morris,H.Tan,F.Barbagli,T.Chang,andK.Salisbury,"Haptic
Feedback Enhances Force Skill Learning", SecondJoint
VirtualEnvironmentandTeleoperatorSystems(WHC'07), 2007,
[5] O.A.J.vanderMeijdenandM.P.Schijven,“Thevalueofhaptic
feedback in conventional and robot‐assisted minimal invasive
surgery and virtual reality training: a current review”, Surg
[6] PicodG,JambonAC,VinatierD,DuboisP,“Whatcantheoperator
[7] P.F.Villard,F.P.Vidal,C.Hunt,F.Bello,N.W.John,S.Johnson,and
cholangiography training simulator with real‐time breathing
motion," InternationalJournalofComputerAssistedRadiology
[8] V.Luboz, C.Hughes,D. Gould,N.W.John,and F.Bello,"Real‐time
Seldinger technique simulation in complex vascular models,"
[9] S.E. Lakhan, A. Kaplan, C. Laird, and Y. Leiter Y, "The
interventionalism of medicine: interventional radiology, cardiology,
and neuroradiology". International Archives of Medicine 2 (27).
[10] F.P. Vidal, A.E. Healey, N.W. John and D.A. Gould, “Force
Guided Needle Puncture Training Simulator”, MICCAI2008
[11] A.G.GallagherandE.M.Ritter,“VirtualReality:Objective
Assessment, Education, and Training”, EmergingTechnologiesin
Surgery,M.Satava,A.GaspariandN.DiLorenzo(editors), ISBN
[12] S.J.LedermanandR.L.Klatzky,“Hapticperception:atutorial,”
[13] M.O.ErnstandM.S.Banks,“Humansintegratevisualandhaptic
[14] H.B.HelbigandM.O.Ernst.“Knowledgeaboutacommonsource
[15] C.Takahashi,J.Diedrichsen,andS.J.Watt,“Integrationofvision
[16] N.A.Giudice,R.L.Klatzky,andJ.M.Loomis,J.M.,“Evidencefor
This study aims to determine similarities and differences in affective experience when manipulating 3 physical user interfaces (PUIs): button, touch screen, and jog dial inputs. First, we conducted literature and expert reviews to identify key affective experiences of PUI manipulation. As a result, seven affective experiences (Softness, Elasticity, Heaviness, Smoothness, Sharpness, Clarity of Feedback, and Rattle) were identified, and they were categorized into two dimensions (Force and Feedback Vibration). Three experiments, including 51 types of buttons, 25 synthesized stimuli for touch screen, and 51 types of jog dials, were then conducted to determine the relationship between affective experiences, satisfaction, and physical parameters of each PUI. According to the results, it was common for all three PUIs that Clarity of Feedback was the most important factor for satisfaction. However, there were also differences between the effects of affective experiences and physical parameters. User satisfaction on manipulating jog dials was found to be less affected by affective experiences than buttons and touch screens. Among physical parameters, the strength of feedback was the most important factor for the affective experience of buttons and touch screens, whereas the period of feedback was the most important factor for jog dials. These similarities and differences could be explained by cognitive and manipulative factors (e.g., attention level, types of feedback, and hand posture of manipulation). The results of this study can help researchers and practitioners better understand and enhance the affective aspects of PUI manipulations.
Full-text available
Evidence for amodal representations after bimodal learning: Integration of haptic-visual layouts into a common spatial image. Spatial Cognition & Computation, 9(4), 287-304. Abstract: Participants learned circular layouts of six objects presented haptically or visually, then indicated the direction from a start target to an end target of the same or different modality (intramodal versus intermodal). When objects from the two modalities were learned separately, superior performance for intramodal trials indicated a cost of switching between modalities. When a bimodal layout intermixing modalities was learned, intra-and intermodal trials did not differ reliably. These findings indicate that a spatial image, independent of input modality, can be formed when inputs are spatially and temporally congruent, but not when modalities are temporally segregated in learning.
Conference Paper
Full-text available
Introduction and Motivations We present a method for modelling the force penetration of needles into anatomic structures that are encountered during visceral punctures. Our aim is to provide a validated haptic model that can be used for the insertion of needles within our developing medical simulations of visceral interventional needle puncture procedures. This preliminary study is focused on Chiba needles. These are commonly used for gaining access to the kidney to perform Nephrostomy. The force feedback delivered in current virtual environment (VE) medical simulator models is generally an approximation to a real procedure, as assessed by experts. Haptics based on real procedural forces will allow a more authentic simulation of the subtle cues perceived when carrying out an interventional procedure under real world conditions. In evaluating needle puncture procedures, in vitro studies are essential for detailed understanding of the physical components and effects of overall summated forces (soft tissue deformation force, clamping force, cutting force, etc.). Material and Methods We are collecting experimental data using a tensile tester used in vitro for pig and ox tissues obtained from a Butcher. However, due to the different physical properties of living tissues, in vitro data require verification by in vivo measurements. Until recently there were few devices available for measurement of instrument forces in vivo in humans, unobtrusively: flexible capacitance pads and other miniaturized sensors, however, present a novel opportunity to collect these data in vivo. Calibration in vivo has shown that the output of these devices is stable and reproducible. Results A Chiba needle was mounted and driven into the tissue at a fixed velocity, 500~mm per minute. This is an approximation of the speed of needle insertion during interventional radiological procedures. The needle orientation was orthogonal to the surface of the kidney. Ten punctures were repeated to obtain an average and the range of forces involved. Finally, the force is modeled analytically using a radial-basis function (RBF) network: \begin{equation*}\begin{split} F(x) & = \sum_{i = 1}^{N} w_i \varphi_i(x - x_i)\\ \varphi_i(r) & = -\exp^{\frac{r^2}{2\sigma_i^2}} \end{split}\end{equation*} Each RBF corresponds to a Gaussian function, for which three parameters have been estimated: the centre $x_i$, the weight $w_i$, and the full width at half maximum $\sigma_i$. Conclusion This model closely matches the experimental data and accurately reproduces the same trend as shown by the experimental data. The resulting haptics model has been embedded into the needle insertion component of our ultrasound guided needle puncture simulator.
Full-text available
Purpose We present here a simulator for interventional radiology focusing on percutaneous transhepatic cholangiography (PTC). This procedure consists of inserting a needle into the biliary tree using fluoroscopy for guidance. Methods The requirements of the simulator have been driven by a task analysis. The three main components have been identified: the respiration, the real-time X-ray display (fluoroscopy) and the haptic rendering (sense of touch). The framework for modelling the respiratory motion is based on kinematics laws and on the Chainmail algorithm. The fluoroscopic simulation is performed on the graphic card and makes use of the Beer-Lambert law to compute the X-ray attenuation. Finally, the haptic rendering is integrated to the virtual environment and takes into account the soft-tissue reaction force feedback and maintenance of the initial direction of the needle during the insertion. Results Five training scenarios have been created using patient-specific data. Each of these provides the user with variable breathing behaviour, fluoroscopic display tuneable to any device parameters and needle force feedback. Conclusions A detailed task analysis has been used to design and build the PTC simulator described in this paper. The simulator includes real-time respiratory motion with two independent parameters (rib kinematics and diaphragm action), on-line fluoroscopy implemented on the Graphics Processing Unit and haptic feedback to feel the soft-tissue behaviour of the organs during the needle insertion.
Conference Paper
Full-text available
This paper explores the use of haptic feedback to teach an abstract motor skill that requires recalling a sequence of forces. Participants are guided along a trajectory and are asked to learn a sequence of one- dimensional forces via three paradigms: haptic training, visual training, or combined visuohaptic training. The extent of learning is measured by accuracy of force recall. We find that recall following visuohaptic training is significantly more accurate than recall following visual or haptic training alone, although haptic training alone is inferior to visual training alone. This suggests that in conjunction with visual feedback, haptic training may be an effective tool for teaching sensorimotor skills that have a force- sensitive component to them, such as surgery. We also present a dynamic programming paradigm to align and compare spatiotemporal haptic trajectories.
Full-text available
This tutorial focuses on the sense of touch within the context of a fully active human observer. It is intended for graduate students and researchers outside the discipline who seek an introduction to the rapidly evolving field of human haptics. The tutorial begins with a review of peripheral sensory receptors in skin, muscles, tendons, and joints. We then describe an extensive body of research on "what" and "where" channels, the former dealing with haptic perception of objects, surfaces, and their properties, and the latter with perception of spatial layout on the skin and in external space relative to the perceiver. We conclude with a brief discussion of other significant issues in the field, including vision-touch interactions, affective touch, neural plasticity, and applications.
Full-text available
When integrating signals from vision and haptics the brain must solve a "correspondence problem" so that it only combines information referring to the same object. An invariant spatial rule could be used when grasping with the hand: here the two signals should only be integrated when the estimate of hand and object position coincide. Tools complicate this relationship, however, because visual information about the object, and the location of the hand, are separated spatially. We show that when a simple tool is used to estimate size, the brain integrates visual and haptic information in a near-optimal fashion, even with a large spatial offset between the signals. Moreover, we show that an offset between the tool-tip and the object results in similar reductions in cross-modal integration as when the felt and seen positions of an object are offset in normal grasping. This suggests that during tool use the haptic signal is treated as coming from the tool-tip, not the hand. The brain therefore appears to combine visual and haptic information, not based on the spatial proximity of sensory stimuli, but based on the proximity of the distal causes of stimuli, taking into account the dynamics and geometry of tools.
Computer-based simulation has been used for decades in aviation and other professional fields. However, the last 15 years have seen numerous attempts to introduce computer-based simulation into clinical medicine. Surgery, and specifically minimally invasive surgery (MIS), has led the way in the development and application of this technology in clinical practice. Recently, use of computer-based simulation for training has expanded into the multidisciplinary fields of catheter-based, image- guided intervention, enabling both surgeons and non-surgeons alike to train on new procedures. The widespread introduction and use of computer-based simulation is changing the way physicians are trained and positively affecting the treatments patients receive. We believe that this revolution represents a paradigm shift in the way procedural-based medicine will be learned and practiced.
This review paper discusses the role of haptics within virtual medical training applications, particularly, where it can be used to aid a practitioner to learn and practice a task. The review summarizes aspects to be considered in the deployment of haptics technologies in medical training. First, both force/torque and tactile feedback hardware solutions that are currently produced commercially and in academia are reviewed, followed by the available haptics-related software and then an in-depth analysis of medical training simulations that include haptic feedback. The review is summarized with scrutiny of emerging technologies and discusses future directions in the field.
Commercial interventional radiology vascular simulators emulate instrument navigation and device deployment, though none supports the Seldinger technique, which provides initial access to the vascular tree. This paper presents a novel virtual environment for teaching this core skill. Our simulator combines two haptic devices: vessel puncture with a virtual needle and catheter and guidewire manipulation. The simulation software displays the instrument interactions with the vessels. Instruments are modelled using a mass-spring approximation, while efficient collision detection and collision response allow real time interactions. Experienced interventional radiologists evaluated the haptic components of our simulator as realistic and accurate. The vessel puncture haptic device proposes a first prototype to simulate the Seldinger technique. Our simulator presents realistic instrument behaviour when compared to real instruments in a vascular phantom. This paper presents the first simulator to train the Seldinger technique. The preliminary results confirm its utility for interventional radiology training.