ArticlePDF Available

Abstract and Figures

We have developed a virtual reality (VR) and an augmented reality (AR) dental training simulator utilizing a haptic device. The simu-lators utilize volumetric force feedback computation and real time modification of the volumetric data. They include a virtual mir-ror to facilitate indirect vision during a simulated operation. The AR environment allows students to practice surgery in correct pos-tures by combining the 3D tooth and tool models with the real-world view and displaying the result through a video see-through head-mounted display (HMD). Preliminary results from an initial evaluation show that the system is a promising tool to supplement dental training and that there are advantages of the AR over the VR approach.
Content may be subject to copyright.
Copyright © 2010 by the Association for Computing Machinery, Inc.
Permission to make digital or hard copies of part or all of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for commercial advantage and that copies bear this notice and the full citation on the
first page. Copyrights for components of this work owned by others than ACM must be
honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on
servers, or to redistribute to lists, requires prior specific permission and/or a fee.
Request permissions from Permissions Dept, ACM Inc., fax +1 (212) 869-0481 or e-mail
VRST 2010, Hong Kong, November 2224, 2010.
© 2010 ACM 978-1-4503-0441-2/10/0010 $10.00
Augmented Reality Haptics System for Dental Surgical Skills Training
Phattanapon Rhienmora
Asian Institute of Technology
Kugamoorthy Gajananan
National Institute of Informatics
Peter Haddawy
United Nations University
Matthew N. Dailey
Asian Institute of Technology
Siriwan Suebnukarn
Thammasat University
We have developed a virtual reality (VR) and an augmented reality
(AR) dental training simulator utilizing a haptic device. The simu-
lators utilize volumetric force feedback computation and real time
modification of the volumetric data. They include a virtual mir-
ror to facilitate indirect vision during a simulated operation. The
AR environment allows students to practice surgery in correct pos-
tures by combining the 3D tooth and tool models with the real-
world view and displaying the result through a video see-through
head-mounted display (HMD). Preliminary results from an initial
evaluation show that the system is a promising tool to supplement
dental training and that there are advantages of the AR over the VR
CR Categories: H.5.1 [Information Interfaces And Presenta-
tion]: Multimedia Information Systems—Artificial, augmented,
and virtual realities; H.5.2 [Information Interfaces And Presenta-
tion]: User Interfaces—Haptic I/O
Keywords: Dental training simulator, augmented reality, haptics
1 Introduction
Traditional methods for dental surgical training rely on practicing
procedural skills on plastic teeth or live patients under the supervi-
sion of dental experts. The limitations of this approach include a
lack of real-world cases and concerns for patient safety. Recently,
integration of virtual reality (VR) technology and haptic technology
has resulted in a number of dental simulators for clinical and surgi-
cal training [Yau et al. 2006; Kim and Park 2009]. The advantages
of these simulators are that the students are allowed to make errors
and are able to practice procedures as many times as they want at
no incremental cost.
A realistic dental simulator for surgical training would allow a stu-
dent to drill into a virtual tooth and feel the different stiffness of dif-
ferent anatomical structures. To achieve this, volumetric represen-
tations of tooth models can be applied as they provide information
about the internal structure of a tooth crucial for realistic graphical
display and haptic force feedback.
Many VR simulators do not implement a virtual dental mirror, de-
spite the fact that dental mirror is a vital instrument for many op-
Figure 1: Screenshot of the VR dental simulator.
erations, especially those requiring indirect vision. Another issue
with many VR simulators is that they are not co-located; users have
to look at the monitor instead of their hands during an operation.
This makes hand-eye coordination difficult and results in unrealis-
tic simulation. Thus, skills acquired from these simulator might not
transfer well to the operating room.
2 Dental Simulator with Volumetric Haptics
Our VR dental simulator (Figure 1) consists of a graphical display
and a haptic device for simulation of virtual dental tools. The sys-
tem allows dentists to practice using a probe to examine the surface
of a tooth, to feel its hardness, and to drill or cut it. We provide
an optional second haptic device to control the virtual dental mir-
ror. We use OpenGLs stencil buffer to realize real-time reflection.
For a system with only one haptic device, we allow users to switch
between the dental handpiece and mirror.
The tooth model used in our simulator is acquired from a micro-CT
scanner at a voxel resolution of 128 × 128 × 256 [Menz 2006].
The tooth is stored in the form of a three dimensional grid of voxels
representing the density of the structure at each point with a value
between 0 and 255.
Our system performs visualization, haptic rendering, and real-time
modification of the volumetric data with the help of the the PolyVox
library [Williams 2010]. PolyVox provides a hybrid data struc-
ture consisting of a 3D volumetric grid, used for haptic rendering
and real time modification, and a triangular surface mesh, used for
graphic rendering. PolyVox extracts the surface mesh from the 3D
grid and updates it whenever a change to the 3D grid occurs. We
perform uniform volumetric sampling of a set of points from the bur
region of the handpiece; these sample points are used for collision
detection and haptic rendering.
A collision occurs when a volume sample point in the bur intersects
with the tooth volume. Once a collision is detected, we compute
force feedback based on the number of immersed sample points
and the tooth density in the colliding region. The direction of the
Figure 2: An AR scene displayed in the HMD screen
output force vector is based on a summation of force vectors cor-
responding to individual sample points. We also smooth the com-
puted force vectors using a weighted moving average technique to
reduce vibration effects due to abrupt changes in the direction of
the rendered force.
When the user activates cutting, on each iteration of the haptic loop,
every 3D model voxel in collision with a tool sample point is set to
a density of 0.
3 Augmented Reality Environment
We transformed our VR dental simulator into an AR environment
using a video see-through head-mounted display (HMD) with an
attached monocular camera. Figure 2 shows an example of an
image displayed on the HMD screen. The registration of the 3D
tooth in the actual environment is realized by ARToolKit [Kato and
Billinghurst 2010], an open source AR library.
Within this AR environment, the haptic device is co-located with
the 3D graphics, giving users a more natural way to practice dental
surgery, in which hand-eye coordination is crucial. Real-time head
tracking is made possible by continuously grabbing camera images,
detecting AR markers, and registering the 3D tooth accordingly.
By attaching another AR marker to a real dental mirror, as shown
in Figure 2 (inset), we can register the virtual mirror and render
reflections onto it. This technique allows a dentist to use a familiar
tool and also eliminates the need for a second haptic device.
4 Preliminary Evaluation and Discussion
In previous work, we asked dental students and a dental instructor
to evaluate the VR system in the context of a tooth preparation pro-
cedure. The users found the realism of the VR system’s graphical
and haptic rendering to be acceptable, but some evaluators found it
difficult to navigate and control the dental tool in the simulator. We
attribute this problem to the difficulty of hand-eye coordination in
non-co-located VR systems.
On completion of the AR prototype, we asked a dental instructor
from a the Faculty of Dentistry at Thammasat University, Thailand,
to give a preliminary evaluation of the new approach in the context
of crown preparation and a pulp access opening operations. The
expert agreed that the new environment is much closer to a real
clinical setting. She also suggested an ideal setting in which the
virtual tooth is overlaid on a traditional mannequin along with other
tangible real teeth.
There are few concerns regarding the use of a video see-trough
HMD. First, the HMD is relatively heavy for long-lasting simu-
lation sessions. However, all of the procedures we currently sim-
ulate can be completed in approximately two to five minutes, so
the weight is acceptable for this short period. Another issue is that
users’ depth perception is limited by the use of a monocular cam-
era. Accurate depth perception is important in dental surgery, and
stereo cameras would improve users’ sense of depth dramatically.
Unfortunately, for the time being, HMDs with stereo cameras are
prohibitively expensive. Finally, the display resolution is limited by
the camera’s specifications. However, compared to other solutions
for co-located visuo-haptic system such as a half-mirror, HMDs are
still our preferred technique, due to their mobility and performance.
5 Conclusion And Future Work
In this paper, we have given an overview of our VR and AR dental
training simulators. The simulators use volumetric force feedback
and allow real time modification of the volumetric data. To repre-
sent tools, we use a volumetric sampling approach that is compu-
tationally efficient yet provides for realistic, stable cutting simula-
tions. The virtual mirror implemented using basic computer graph-
ics techniques is quite valuable for dental simulations.
As expected by us and confirmed by an experienced dentist, there
are many advantages of the AR approach over the VR approach to
dental surgical simulation. The co-located visuo-haptic display in
the AR environment is closer to the actual clinical setting. As a
result, skills acquired using the simulator should transfer well to
the operating room.
We plan to combine the current setup with a mannequin with real
teeth as suggested by the expert for better realism. We will also look
for alternatives to ARToolKit’s fiducial markers, such as retroreflec-
tive markers or natural features. Finally, we are collecting logged
data from students and experts who perform operations with the
simulator towards constructing automatic performance assessment
tools and an intelligent tutor that can provide feedback during an
The authors would like to thank Ekarin Supataratarn and Poonam
Shrestha for discussion and help with software development.
KATO, H., AND BILLINGHURST, M., 2010. Artoolkit.
Open source software available at http://www.hitl.
KIM, K., AND PARK, J. 2009. Virtual bone drilling for dental
implant surgery training. In VRST ’09: Proceedings of the 16th
ACM Symposium on Virtual Reality Software and Technology,
ACM, New York, NY, USA, 91–94.
MENZ, A. S. 2006. Semi-automatic transfer function generation
for non-domain specific direct volume rendering. Master’s the-
sis, Iowa State University.
WILLIAMS, D., 2010. Polyvox technology. Open source software
available at
YAU, H. T., TSOU, L. S., AND TSAI, M. J. 2006. Octree-based
virtual dental training system with a haptic device. Computer-
Aided Design & Applications 3, 415–424.
... Numerous empirical and experimental studies have been conducted on providing realistic haptic feedback using VR dental simulators including [3, 4, 5, 12, 13, 14, 1]. Studies have evaluated surgical performance and skill acquisition during training and shown that major differences exist between experts and novices in force/torque magnitudes at the hand/tool interface [7, 11]. ...
... In this study, we employed the VR dental simulator developed by Rhienmora et al. [5]. The simulator operates on a standard PC connected to a PHANToM Omni haptic device as a dental handpiece. ...
... Training on the application of force while using instruments with verbal feedback such as " harder.. " or " 2 times higher.. " might be insufficient in guiding novices how to adjust their parameters [6]. Automatically moving with the haptic stylus in the student's hand along the recorded expert path will not solve the problem as it is a passive method and students still cannot learn the actual forces applied by the expert [5] . Similarly , tracking based haptic training approach shown no direct relationship between the trajectory tracking precision and the motor skill performance [2]. ...
Conference Paper
With the minute margins of error in endodontic surgery, training in manual dexterity and proper instrument handling are crucial components in the dental curriculum. Important parameters include tool path, tool angulation, and force applied. In this work, we focus on training of correct application of force. This is particularly challenging since the amounts of force used are on the order of tenths of Newtons, requiring a highly refined tactile sense and incorrect force can cause irreversible damage. Too great a force can cause overdrilling or in extreme cases perforation of the tooth. Too small a force can cause thermal irritation possibly resulting in tissue necrosis. Despite the importance of correct use of force, this is the dimension on which students receive the least tutorial feedback since force information is typically not available in traditional training settings. In this paper, we present an approach to using haptic feedback as a means to convey formative feedback on the correct application of force. Feedback is conveyed to the student graphically and the correct amount of force to apply is trained haptically. The simulator is rewound and the student is asked to redo the stage where the error occurred. Preliminary evaluation against a control group of students who received only feedback concerning outcome shows the feedback mechanism to be effective.
... They can be more readily available and cheaper, and there are no ethical and moral issues resulting from performing a surgery operation on human and animal corpses [38]. The advantage of virtual reality simulators is that the procedure can be started repeatedly even if mistakes are made [39]. Laparoscopy skills are achieved with practice and the feedback one get as result. ...
Conference Paper
Full-text available
From a strong practical point of view, we offer an overview of virtual and augmented reality solutions in medicine. We thus analyzed practical and industrial work included in peer-reviewed articles and conference proceedings.
Introduction In UK universities, caries removal teaching utilises plastic teeth. This format does not enable students to learn how to distinguish between tooth layers and caries via tactile feedback. The aim of this study was to the assess the applicability of a novel, 3D-printed carious tooth within caries removal teaching. Materials and methods Single-material 3D-printed teeth containing simulated tactile caries were developed and 14 final-year undergraduates were briefed to remove caries and minimise damage to healthy tissue within the tooth. Students completed evaluation questionnaires for their opinion of 3D-printed teeth in comparison to plastic teeth and perceived confidence to subsequently treat patients. Cavity preparation perimeters were measured, using photographs with a standard protocol. Heat map analysis illustrated variation in location and extent of cavity preparations produced by the cohort. Results Student feedback indicated the 3D-printed caries exercise was positively received, 71.4% agreed 3D-printed teeth would have better prepared students for patient treatment. 78.6% rated their pre-clinical stress/anxiety as ‘Very High’ or ‘High’ and 57.1% agreed that if pre-clinical teaching incorporated 3D-printed teeth, their stress/anxiety when treating their first caries patient would have been reduced. The average perimeter of cavity preparation indicated relative variation, with a maximum perimeter of 19.6mm and a minimum of 10.7mm, and a range of 8.9mm. Discussion Introducing 3D-printed teeth into preclinical teaching would allow students to gain confidence in clinically relevant experience in tactile aspects of caries treatment earlier in their training than currently possible. Conclusion This study demonstrates student acceptance of an alternative caries removal teaching method, with potential to increase aptitude in caries removal in a clinically relevant manner.
Full-text available
15 In December 2019, an outbreak of novel coronavirus pneumonia occurred, and subsequently attracted 16 worldwide attention when it bloomed into the COVID-19 pandemic. To limit the spread and 17 transmission of the novel coronavirus, governments, regulatory bodies, and health authorities across 18 the globe strongly enforced shut down of educational institutions including medical and dental schools. 19 The adverse effects of COVID-19 on dental education have been tremendous, including difficulties in 20 the delivery of practical courses such as restorative dentistry. As a solution to help dental schools adapt 21 to the pandemic, we have developed a compact and portable teaching-learning platform called 22 DenTeach. This platform is intended for remote teaching and learning pertaining to dental schools at 23 these unprecedented times. This device can facilitate fully remote and physical-distancing-aware 24 teaching and learning in dentistry. DenTeach platform consists of an instructor workstation (DT-25 Performer), a student workstation (DT-Student), advanced wireless networking technology, and cloud-26 based data storage and retrieval. The platform procedurally synchronizes the instructor and the student 27
Background Image registration lies in the core of augmented reality (AR), which aligns the virtual scene with the reality. In AR surgical navigation, the performance of image registration is vital to the surgical outcome. Methods This paper presents a practical marker-less image registration method for AR-guided oral and maxillofacial surgery where a virtual scene is generated and mixed with reality to guide surgical operation or provide surgical outcome visualization in the manner of video see-through overlay. An intraoral 3D scanner is employed to acquire the patient’s teeth shape model intraoperatively. The shape model is then registered with a custom-made stereo camera system using a novel 3D stereo matching algorithm and with the patient’s CT-derived 3D model using an iterative closest point scheme, respectively. By leveraging the intraoral 3D scanner, the CT space and the stereo camera space are associated so that surrounding anatomical models and virtual implants could be overlaid on the camera’s view to achieve AR surgical navigation. Results Jaw phantom experiments were performed to evaluate the target registration error of the overlay, which yielded an average error of less than 0.50 mm with the time cost less than 0.5 s. Volunteer trial was also conducted to show the clinical feasibility. Conclusions The proposed registration method does not rely on any external fiducial markers attached to the patient. It performs automatically so as to maintain a correct AR scene, overcoming the misalignment difficulty caused by patient’s movement. Therefore, it is noninvasive and practical in oral and maxillofacial surgery.
Virtual Reality (VR) Head-Mounted Displays (HMDs) are on the verge of becoming commodity hardware available to the average user and feasible to use as a tool for 3D work. Some HMDs include front-facing cameras, enabling Augmented Reality (AR) functionality. Apart from avoiding collisions with the environment, interaction with virtual objects may also be affected by seeing the real environment. However, whether these effects are positive or negative has not yet been studied extensively. For most tasks it is unknown whether AR has any advantage over VR. In this work we present the results of a user study in which we compared user performance measured in task completion time on a 9 degrees of freedom object selection and transformation task performed either in AR or VR, both with a 3D input device and a mouse. Our results show faster task completion time in AR over VR. When using a 3D input device, a purely VR environment increased task completion time by 22.5 percent on average compared to AR ( ${p}<0.024$ ). Surprisingly, a similar effect occurred when using a mouse: users were about 17.3 percent slower in VR than in AR ( ${p}<0.04$ ). Mouse and 3D input device produced similar task completion times in each condition (AR or VR) respectively. We further found no differences in reported comfort.
Conference Paper
Purpose. To develop and evaluate a low-cost, surgical navigation solution for periacetabular osteotomy (PAO) surgery. Methods. A commercially available low-cost miniature computer is used together with a camera board (Raspberry Pi 2 Model B, Camera Module PiNoir) to track planar markers (Aruco markers). The overall setup of the tracking unit is small enough to be attached directly to the patient’s pelvis. The patient’s pelvis is registered by estimating the pose of a planar marker which is attached to an anterior pelvic plane (APP) digitization device. Next, one marker is attached to the acetabular fragment and the initial orientation of the fragment is recorded. The estimated orientation of the fragment is transmitted to the host computer for visualization. Results. A plastic bone study (eight hip joints) was performed to validate the proposed system. The comparison with a previously developed optical tracking-based system showed no statistical significant difference between measurements obtained from the two systems. In all eight hip joints the mean absolute difference was below 2\(^\circ \) for both anteversion and inclination and a very strong correlation was observed. Conclusions. We show that with our proof-of-principle system, we are able to compute the acetabular orientation accurately.
Conference Paper
Ongoing progress in the area of virtual reality and computer simulation has been providing applications that show tremendous promise in overcoming most of the deficiencies associated with training using cadaver parts and plastic artifacts. In this work, we present a 3D Virtual Drilling Simulator of the edentulous space of the dental arch in oral rehabilitation, controlled by the Novint Falcon haptic device. We developed the application from a dental cast (maxilla and mandible), which was scanned for the generation of the 3D model. Additionally, we have used a CT scan for the simulation of different bone densities of the implant region and respective drilling resistances. Preliminary tests were also performed by a specialist in the field, who has validated the perceived tactile and visual feedback and recognized the relevance of the simulator as a support tool for training students in Implant Dentistry.
ResearchGate has not been able to resolve any references for this publication.