Augmented reality haptics system for dental surgical skills training

Abstract

We have developed a virtual reality (VR) and an augmented reality (AR) dental training simulator utilizing a haptic device. The simu-lators utilize volumetric force feedback computation and real time modification of the volumetric data. They include a virtual mir-ror to facilitate indirect vision during a simulated operation. The AR environment allows students to practice surgery in correct pos-tures by combining the 3D tooth and tool models with the real-world view and displaying the result through a video see-through head-mounted display (HMD). Preliminary results from an initial evaluation show that the system is a promising tool to supplement dental training and that there are advantages of the AR over the VR approach.

Full text

Copyright © 2010 by the Association for Computing Machinery, Inc.
Permission to make digital or hard copies of part or all of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for commercial advantage and that copies bear this notice and the full citation on the
first page. Copyrights for components of this work owned by others than ACM must be
honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on
servers, or to redistribute to lists, requires prior specific permission and/or a fee.
Request permissions from Permissions Dept, ACM Inc., fax +1 (212) 869-0481 or e-mail
permissions@acm.org.
VRST 2010, Hong Kong, November 22 – 24, 2010.
© 2010 ACM 978-1-4503-0441-2/10/0010 $10.00
Augmented Reality Haptics System for Dental Surgical Skills Training
Phattanapon Rhienmora
∗
Asian Institute of Technology
Kugamoorthy Gajananan
†
National Institute of Informatics
Peter Haddawy
‡
United Nations University
Matthew N. Dailey
§
Asian Institute of Technology
Siriwan Suebnukarn
¶
Thammasat University
Abstract
We have developed a virtual reality (VR) and an augmented reality
(AR) dental training simulator utilizing a haptic device. The simu-
lators utilize volumetric force feedback computation and real time
modification of the volumetric data. They include a virtual mir-
ror to facilitate indirect vision during a simulated operation. The
AR environment allows students to practice surgery in correct pos-
tures by combining the 3D tooth and tool models with the real-
world view and displaying the result through a video see-through
head-mounted display (HMD). Preliminary results from an initial
evaluation show that the system is a promising tool to supplement
dental training and that there are advantages of the AR over the VR
approach.
CR Categories: H.5.1 [Information Interfaces And Presenta-
tion]: Multimedia Information Systems—Artificial, augmented,
and virtual realities; H.5.2 [Information Interfaces And Presenta-
tion]: User Interfaces—Haptic I/O
Keywords: Dental training simulator, augmented reality, haptics
1 Introduction
Traditional methods for dental surgical training rely on practicing
procedural skills on plastic teeth or live patients under the supervi-
sion of dental experts. The limitations of this approach include a
lack of real-world cases and concerns for patient safety. Recently,
integration of virtual reality (VR) technology and haptic technology
has resulted in a number of dental simulators for clinical and surgi-
cal training [Yau et al. 2006; Kim and Park 2009]. The advantages
of these simulators are that the students are allowed to make errors
and are able to practice procedures as many times as they want at
no incremental cost.
A realistic dental simulator for surgical training would allow a stu-
dent to drill into a virtual tooth and feel the different stiffness of dif-
ferent anatomical structures. To achieve this, volumetric represen-
tations of tooth models can be applied as they provide information
about the internal structure of a tooth crucial for realistic graphical
display and haptic force feedback.
Many VR simulators do not implement a virtual dental mirror, de-
spite the fact that dental mirror is a vital instrument for many op-
∗
e-mail: phattanapon@gmail.com
†
e-mail: k-gajananan@nii.ac.jp
‡
e-mail: haddawy@iist.unu.edu
§
e-mail: mdailey@ait.ac.th
¶
e-mail: ssiriwan@tu.ac.th
Figure 1: Screenshot of the VR dental simulator.
erations, especially those requiring indirect vision. Another issue
with many VR simulators is that they are not co-located; users have
to look at the monitor instead of their hands during an operation.
This makes hand-eye coordination difficult and results in unrealis-
tic simulation. Thus, skills acquired from these simulator might not
transfer well to the operating room.
2 Dental Simulator with Volumetric Haptics
Our VR dental simulator (Figure 1) consists of a graphical display
and a haptic device for simulation of virtual dental tools. The sys-
tem allows dentists to practice using a probe to examine the surface
of a tooth, to feel its hardness, and to drill or cut it. We provide
an optional second haptic device to control the virtual dental mir-
ror. We use OpenGL’s stencil buffer to realize real-time reflection.
For a system with only one haptic device, we allow users to switch
between the dental handpiece and mirror.
The tooth model used in our simulator is acquired from a micro-CT
scanner at a voxel resolution of 128 × 128 × 256 [Menz 2006].
The tooth is stored in the form of a three dimensional grid of voxels
representing the density of the structure at each point with a value
between 0 and 255.
Our system performs visualization, haptic rendering, and real-time
modification of the volumetric data with the help of the the PolyVox
library [Williams 2010]. PolyVox provides a hybrid data struc-
ture consisting of a 3D volumetric grid, used for haptic rendering
and real time modification, and a triangular surface mesh, used for
graphic rendering. PolyVox extracts the surface mesh from the 3D
grid and updates it whenever a change to the 3D grid occurs. We
perform uniform volumetric sampling of a set of points from the bur
region of the handpiece; these sample points are used for collision
detection and haptic rendering.
A collision occurs when a volume sample point in the bur intersects
with the tooth volume. Once a collision is detected, we compute
force feedback based on the number of immersed sample points
and the tooth density in the colliding region. The direction of the
97

Figure 2: An AR scene displayed in the HMD screen
output force vector is based on a summation of force vectors cor-
responding to individual sample points. We also smooth the com-
puted force vectors using a weighted moving average technique to
reduce vibration effects due to abrupt changes in the direction of
the rendered force.
When the user activates cutting, on each iteration of the haptic loop,
every 3D model voxel in collision with a tool sample point is set to
a density of 0.
3 Augmented Reality Environment
We transformed our VR dental simulator into an AR environment
using a video see-through head-mounted display (HMD) with an
attached monocular camera. Figure 2 shows an example of an
image displayed on the HMD screen. The registration of the 3D
tooth in the actual environment is realized by ARToolKit [Kato and
Billinghurst 2010], an open source AR library.
Within this AR environment, the haptic device is co-located with
the 3D graphics, giving users a more natural way to practice dental
surgery, in which hand-eye coordination is crucial. Real-time head
tracking is made possible by continuously grabbing camera images,
detecting AR markers, and registering the 3D tooth accordingly.
By attaching another AR marker to a real dental mirror, as shown
in Figure 2 (inset), we can register the virtual mirror and render
reflections onto it. This technique allows a dentist to use a familiar
tool and also eliminates the need for a second haptic device.
4 Preliminary Evaluation and Discussion
In previous work, we asked dental students and a dental instructor
to evaluate the VR system in the context of a tooth preparation pro-
cedure. The users found the realism of the VR system’s graphical
and haptic rendering to be acceptable, but some evaluators found it
difficult to navigate and control the dental tool in the simulator. We
attribute this problem to the difficulty of hand-eye coordination in
non-co-located VR systems.
On completion of the AR prototype, we asked a dental instructor
from a the Faculty of Dentistry at Thammasat University, Thailand,
to give a preliminary evaluation of the new approach in the context
of crown preparation and a pulp access opening operations. The
expert agreed that the new environment is much closer to a real
clinical setting. She also suggested an ideal setting in which the
virtual tooth is overlaid on a traditional mannequin along with other
tangible real teeth.
There are few concerns regarding the use of a video see-trough
HMD. First, the HMD is relatively heavy for long-lasting simu-
lation sessions. However, all of the procedures we currently sim-
ulate can be completed in approximately two to five minutes, so
the weight is acceptable for this short period. Another issue is that
users’ depth perception is limited by the use of a monocular cam-
era. Accurate depth perception is important in dental surgery, and
stereo cameras would improve users’ sense of depth dramatically.
Unfortunately, for the time being, HMDs with stereo cameras are
prohibitively expensive. Finally, the display resolution is limited by
the camera’s specifications. However, compared to other solutions
for co-located visuo-haptic system such as a half-mirror, HMDs are
still our preferred technique, due to their mobility and performance.
5 Conclusion And Future Work
In this paper, we have given an overview of our VR and AR dental
training simulators. The simulators use volumetric force feedback
and allow real time modification of the volumetric data. To repre-
sent tools, we use a volumetric sampling approach that is compu-
tationally efficient yet provides for realistic, stable cutting simula-
tions. The virtual mirror implemented using basic computer graph-
ics techniques is quite valuable for dental simulations.
As expected by us and confirmed by an experienced dentist, there
are many advantages of the AR approach over the VR approach to
dental surgical simulation. The co-located visuo-haptic display in
the AR environment is closer to the actual clinical setting. As a
result, skills acquired using the simulator should transfer well to
the operating room.
We plan to combine the current setup with a mannequin with real
teeth as suggested by the expert for better realism. We will also look
for alternatives to ARToolKit’s fiducial markers, such as retroreflec-
tive markers or natural features. Finally, we are collecting logged
data from students and experts who perform operations with the
simulator towards constructing automatic performance assessment
tools and an intelligent tutor that can provide feedback during an
operation.
Acknowledgments
The authors would like to thank Ekarin Supataratarn and Poonam
Shrestha for discussion and help with software development.
References
KATO, H., AND BILLINGHURST, M., 2010. Artoolkit.
Open source software available at http://www.hitl.
washington.edu/artoolkit.
KIM, K., AND PARK, J. 2009. Virtual bone drilling for dental
implant surgery training. In VRST ’09: Proceedings of the 16th
ACM Symposium on Virtual Reality Software and Technology,
ACM, New York, NY, USA, 91–94.
MENZ, A. S. 2006. Semi-automatic transfer function generation
for non-domain specific direct volume rendering. Master’s the-
sis, Iowa State University.
WILLIAMS, D., 2010. Polyvox technology. Open source software
available at http://www.thermite3d.org.
YAU, H. T., TSOU, L. S., AND TSAI, M. J. 2006. Octree-based
virtual dental training system with a haptic device. Computer-
Aided Design & Applications 3, 415–424.
98