Content uploaded by Daniel Andersen
Author content
All content in this area was uploaded by Daniel Andersen on Apr 13, 2019
Content may be subject to copyright.
CE: R.R.; ANNSURG-D-17-02211; Total nos of Pages: 6;
ANNSURG-D-17-02211
Surgical Telementoring Without Encumbrance
A Comparative Study of See-through Augmented Reality-based Approaches
Edgar Rojas-Mun
˜oz, BS,
Maria Eugenia Cabrera, MSc,
Daniel Andersen, MS,yVoicu Popescu, PhD,y
Sherri Marley, BSN, RN,zBrian Mullis, MD,zBen Zarzaur, MD, MPH, FACS,zand Juan Wachs, PhD
Objective: This study investigates the benefits of a surgical telementoring
system based on an augmented reality head-mounted display (ARHMD) that
overlays surgical instructions directly onto the surgeon’s view of the operating
field, without workspace obstruction.
Summary Background Data: In conventional telestrator-based telementor-
ing, the surgeon views annotations of the surgical field by shifting focus to a
nearby monitor, which substantially increases cognitive load. As an alternative,
tablets have been used between the surgeon and the patient to display instruc-
tions; however, tablets impose additional obstructions of surgeon’s motions.
Methods: Twenty medical students performed anatomical marking (Task1)
and abdominal incision (Task2) on a patient simulator, in 1 of 2 telementoring
conditions: ARHMD and telestrator. The dependent variables were placement
error, number of focus shifts, and completion time. Furthermore, workspace
efficiency was quantified as the number and duration of potential surgeon-
tablet collisions avoided by the ARHMD.
Results: The ARHMD condition yielded smaller placement errors (Task1:
45%, P<0.001; Task2: 14%, P¼0.01), fewer focus shifts (Task1: 93%, P<
0.001; Task2: 88%, P¼0.0039), and longer completion times (Task1: 31%, P
<0.001; Task2: 24%, P¼0.013). Furthermore, the ARHMD avoided
potential tablet collisions (4.8 for 3.2 seconds in Task1; 3.8 for 1.3 seconds
in Task2).
Conclusion: The ARHMD system promises to improve accuracy and to
eliminate focus shifts in surgical telementoring. Because ARHMD partic-
ipants were able to refine their execution of instructions, task completion time
increased. Unlike a tablet system, the ARHMD does not require modifying
natural motions to avoid collisions.
Keywords: augmented reality, surgical telementoring, telemedicine,
teleproctoring
(Ann Surg 2018;xx:xxx– xxx)
Surgical telementoring is a method to deliver specialized expertise
to a mentee in scenarios where expertise is not readily available.
For example, telementoring can allow a remote expert surgeon to
convey specialized surgical expertise to a generalist surgeon in rural
hospitals, in disaster-affected regions and in the battlefield. Further-
more, telementoring can disseminate surgical procedure innovations
across the world.
Telestrators are the conventional approach for telementoring.
A remote expert receives and annotates video imagery from a
mentee’s operating field. These annotations are sent back to the
mentee where they are displayed on a nearby monitor. However, this
approach requires the mentee to shift focus from the surgical field to
the nearby monitor, and to memorize the annotations’ positions to
map them onto the patient’s anatomy. This introduces additional
cognitive load that can contribute to surgeon fatigue and error-prone
performance.
1,2
In previous work,
3
we leveraged augmented reality (AR)
technology to improve surgical telementoring. With our system,
dubbed the System for Telementoring with Augmented Reality
(STAR), the mentee views the operating field through a tablet-based
transparent display, which directly overlays mentor annotations onto
the mentee’s view of the surgical field (Fig. 1).
However, using a tablet at the mentee site possesses disad-
vantages. First, the tablet degrades the mentee’s depth perception
because of loss of stereopsis: binocular depth cues are not preserved
FIGURE 1. Top: STAR tablet-based setup. The tablet is held in
place between the patient and the user with a mechanical
bracket. Bottom: First-person view of an instruction received
with the STAR tablet-based setup. The augmentation consists of
2D lines and images. The view of the camera is displayed on the
device’s screen.
From the
School of Industrial Engineering, Purdue University, West Lafayette,
IN; yDepartment of Computer Science, Purdue University, West Lafayette, IN;
and zSchool of Medicine, Indiana University, Indianapolis, IN.
Sources of Funding: Supported by both the Office of the Assistant Secretary of
Defense for Health Affairs under Award No. W81XWH-14-1-0042 and the
National Science Foundation under Grant DGE-1333468.
Opinions, interpretations, conclusions and recommendations are those of the
author and are not necessarily endorsed by the funders.
The other authors report no conflicts of interest.
Reprints: Juan Wachs, PhD, Purdue University, 315 N. Grant Street, West Lafay-
ette, IN 47907. E-mail: jpwachs@purdue.edu.
Copyright ß2018 Wolters Kluwer Health, Inc. All rights reserved.
ISSN: 0003-4932/16/XXXX-0001
DOI: 10.1097/SLA.0000000000002764
Annals of Surgery Volume XX, Number XX, Month 2018 www.annalsofsurgery.com | 1
ORIGINAL ARTICLE
Copyright © 2018 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
CE: R.R.; ANNSURG-D-17-02211; Total nos of Pages: 6;
ANNSURG-D-17-02211
because of the 2D nature of the tablet screen.
4
This can impair hand–
eye coordination and increase hesitancy when performing precise
tasks: mentees must guess how far their movements should go before
reaching the destination, hindering task completion time. An exam-
ple of this behavior can be found in our previous work.
5
Second, the
tablet’s position in range of the mentee’s arms might impede the
mentee’s natural motion.
This article describes an enhancement of the STAR platform at
the mentee site that preserves binocular depth cues by replacing the
tablet with a see-through AR head-mounted display (ARHMD) worn
by the mentee. 3D graphical annotations are overlaid onto the
mentee’s view of the surgical field, remaining anchored as the
mentee moves. The ARHMD generates a different image for each
eye, making the annotations perceivable at the correct depth relative
to the patient’s body. In addition, proper depth cues are preserved
since see-through ARHMDs allow mentees to see the surgical field
directly. Moreover, the ARHMD does not obstruct the surgeon’s
hands, as it is entirely self-contained on the head. We have conducted
a user study that assesses the effectiveness of the new STAR platform
in surgery. This study indicates that ARHMDs have great potential in
surgical telementoring.
RELATED WORK
Telementoring has proved successful at developing surgeons’
surgical skills when experienced mentors are not available,
6,7
and to
avoid transportation-related delays of medical equipment and per-
sonnel.
8
Additional benefits are associated with telementoring in
austere environments.
9
A review evaluating the tradeoffs between
on-site mentoring and telementoring technologies revealed that each
approach has its own challenges (ie, travel and time costs vs
equipment cost and network issues).
10
Nonetheless, telementoring
is an effective way to provide access to expert mentors and advanced
surgical techniques that otherwise would be unavailable.
Telestrator-based technologies have demonstrated their use-
fulness in surgical telementoring.
11– 13
These approaches rely on
nearby monitors to show guidance sent by a remote expert. In recent
years, preferences between 2D and 3D telestration have been
researched. 2D telestration is more widely adopted because of its
simplicity.
14,15
However, 2D annotations introduce occlusion issues
and degrade binocular depth perception cues, hindering users’
orientation in a 3D environment. Investigating 3D telestration, Ali
et al
16
developed a video algorithm to translate instructions from
mentors from 2D to 3D in a da Vinci surgical robot’s console, and
found that 3D robotic telestration is feasible and does not negatively
influence performance in controlled tasks.
However, conventional telestrator-based approaches have a
substantial disadvantage: mentees must shift focus repeatedly
between the operating field and the monitor, which adds complexity
and increases cognitive load.
17
This disadvantage can be mitigated
by presenting information directly into users’ field of view (FOV).
AR has been used to accomplish this outside
18
and inside the surgical
domain.
19
Although navigation-related AR applications exist for
laparoscopic,
20
endoscopic,
21
and spinal surgery
22
procedures, AR
systems remain unavailable and therefore untested in most
surgical specialties.
State-of-the-art telementoring systems rely on tablet-based
devices to augment the users’ FOV.
3,23,24
However, this approach
introduces additional encumbrance by placing the tablet in the
surgeon’s workspace. Head-mounted displays (HMDs) can address
this drawback: while surgeons must wear a headset, workspace
encumbrance is avoided. Systems reported in urology
25,26
allow
mentors and mentees to exchange information, but either do not
consider complex annotations or only provide 2D imagery, instead of
3D. Our work leverages the capacity of an ARHMD platform to
display 3D annotations directly in the mentee’s FOV, providing
relevant instruction without focus shifts or workspace encumbrance.
SYSTEM DESCRIPTION
STAR is a surgical telementoring platform that displays
annotations sent by a remote expert surgeon directly into a mentee’s
FOV. It is composed of 2 subsystems: the Mentor System, used by the
mentors to deliver guidance to the mentees; and the Trainee System,
which non-specialist/mentee surgeons use to receive visual instruc-
tions from the mentors. The current system allows the placement and
anchoring of 3D imagery inside of the surgeon’s unencumbered
workspace. These annotations can be observed through the
ARHMD’s screen from any viewpoint (Fig. 2, top).
Consider the following example of how the STAR platform
can be utilized. Dr. Harrison is a surgeon situated in an operating
room who needs to perform a leg-fasciotomy on Mr. Smith, a patient
suffering from compartment syndrome. However, Dr. Harrison has
not received extensive training on this procedure and requires assis-
tance. In this case, Dr. Harrison, wearing the ARHMD Trainee
System, connects to the Mentor System, located at a Level 1 Trauma
Facility where Dr. Grover, an attending in Orthopedic Trauma, is
ready to provide support. The ARHMD broadcasts Dr. Harrison’s
FOV to Dr. Grover, who then uses the Mentor System to create
surgical annotations (incision lines, surgical instruments, among
others) in the live video sent by Dr. Harrison. These annotations
are transmitted back to the ARHMD, which projects and anchors
them in the right position over Mr. Smith’s leg. Aided with these
instructions, Dr. Harrison can successfully perform the leg-fasciot-
omy. Figure 2 (bottom) portrays an instruction as seen by the mentee
wearing the ARHMD.
FIGURE 2. Top: STAR ARHMD setup. No additional artifacts are
required except for the device worn by the user. Bottom: First-
person view of an instruction received with the STAR ARHMD
setup. The augmentation consists of 3D lines and 3D models
visible only through the device’s display.
Rojas-Mun
˜oz et al Annals of Surgery Volume XX, Number XX, Month 2018
2| www.annalsofsurgery.com ß2018 Wolters Kluwer Health, Inc. All rights reserved.
Copyright © 2018 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
CE: R.R.; ANNSURG-D-17-02211; Total nos of Pages: 6;
ANNSURG-D-17-02211
METHODS
A user study was conducted at Eskenazi Hospital (Indian-
apolis, Indiana), where 20 medical students performed telementored
tasks under 1 of 2 conditions: STAR ARHMD and conventional
telestrator. The participants had to complete 2 different tasks on a
patient simulator: an anatomical marker placement and a mock
abdominal incision. These tasks depict core surgical skills present
in most surgical procedures: landmark location and skin incision.
These tasks are atomic building blocks for more complex procedures,
and an initial evaluation of the system’s effectiveness in these tasks
would reveal insights for its application on more complex tasks.
Moreover, effectiveness in these tasks is mandatory to ensure subse-
quent effectiveness in more complex scenarios.
Participant performance was recorded and analyzed using the
following metrics: annotation placement error, task completion
time, and number of focus shifts. The ARHMD was also compared
against a tablet-based system
5
: by tracking the participants’ limb
motions and poses during the experiment, the number and duration
of potential collisions avoided by the ARHMD system had a tablet
device been there were calculated as a post-experiment metric.
To maintain consistency among participants, preprogrammed anno-
tations were used instead of live feedback from the Mentor
System. The following subsections elaborate on the details of the
experiment.
Participants
Twenty medical students (6 females, 14 males) ranging
from 23- to 29 years old were recruited. Participants were in their
second, third, or fourth year of medical school, and had no
previous experience with surgical telementoring systems. The
study was reviewed and approved by Indiana University Institu-
tional Review Board (#1409037680), and written participant
consent was acquired for each participant. Using medical students
in the role of mentees has high ecological validity,
27
as tele-
mentoring is likely to be deployed to support medical students,
new graduates, and other non-specialist surgeons. Moreover,
because of the simplicity of the tasks, it was required that the
participant’s level of expertise was not that of an expert surgeon or
a surgery resident: the objective was for the participants to rely in
the telementored guidance instead of being able to complete the
procedure alone. This configuration is an acceptable placeholder
for a medic or a nonspecialist attempting to do a procedure for
which they have little or no experience. Other studies have
leveraged the medical student population in medical telementor-
ing studies.
7,12,28,29
Apparatus and Setting
Figure 3 presents a diagram of the experiment’s setting. The
experiment setup included: the patient simulator on the operating
table, the participant acting as a mentee, an ARHMD (Microsoft
HoloLens), and a nearby monitor located 608to mentees’ right.
Surgical instruments to complete the tasks were located on a Mayo
stand to the side of the operating table.
In the telestrator condition, participants looked at the nearby
monitor to receive the instructions. The images shown on the
telestrator were a top-down view of the region of interest on the
patient simulator for each step of the procedure (Fig. 4, top).
Participants in the STAR condition wore the ARHMD, which con-
structs a virtual representation of the space it is in, and enables the
placement of virtual annotations in this representation, visible only to
the user wearing the device. Participants followed a 3D replica of the
instructions used in the telestrator condition, represented using 3D
models of surgical instruments, 3D lines, and spheres (Fig. 4,
bottom).
FIGURE 3. Experiment setting. For the telestrator condition,
the participant used the nearby screen to retrieve the instruc-
tions. In the ARHMD condition, the instructions were shown via
the device’s screen.
FIGURE 4. Top: Participant performing a task using the teles-
trator-based condition. The instructions for the task are
obtained by looking at a nearby screen. Bottom: Participant
performing a task using the STAR ARHMD condition. Inset is the
first-person view of the user wearing the ARHMD.
Annals of Surgery Volume XX, Number XX, Month 2018 Telementoring Without Encumbrance
ß2018 Wolters Kluwer Health, Inc. All rights reserved. www.annalsofsurgery.com | 3
Copyright © 2018 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
CE: R.R.; ANNSURG-D-17-02211; Total nos of Pages: 6;
ANNSURG-D-17-02211
Data Collection
Two Microsoft Kinect devices were used to record videos of
each participant’s performance. One Kinect was placed fronto-par-
allel to the patient simulator and the participant, recording infrared,
color, depth, and skeletal tracking data. The second Kinect was
placed to capture a top-down view of the procedure, recording color,
and audio. A calibration process was performed before the experi-
ment, allowing a mapping between the ARHMD reference frame and
the top-view Kinect. Time taken to complete each task was recorded
using a stopwatch.
After performing both tasks, participants answered a ques-
tionnaire on their experience with the telementoring systems. The
questionnaire (Fig. 5) contained 7 questions that evaluated each
system in terms of communication efficiency, available functionali-
ties, ease of use, and time required to complete the task. The
questionnaire included a section for comments and suggestions.
Procedure
The experiment consisted of 2 tasks: an anatomical marker
placement and a mock abdominal incision. The first task had 3 trials,
whereas the second task had only 1 trial. Participants were briefed
about both tasks before starting, and were encouraged to complete
the tasks as quickly and accurately as possible.
Task 1: Marker Placement
Participants had to mark different locations on the simulator’s
body with a dry erase marker. Each trial included 7 circular-shaped
annotations located around the patient’s neck and chest. Each trial
showed different locations to avoid recall.
Task 2: Abdominal Incision
Participants had to follow 16 instructions to cut through 2
simulated layers of skin and to spread the linea alba. Incisions on the
skin layers had to be done without puncturing a balloon that
simulated the stomach. Before the execution of each step, partic-
ipants had to mark the locations/incision lines depicted in
the instruction.
Experimental Design
A between-subject design was selected, randomly assigning
participants to each of the 2 telementoring conditions. Condition was
treated as an independent variable, whereas the performance metrics
were treated as dependent variables. In addition, potential tablet
collisions were calculated for the STAR condition. Finally, the
questionnaire responses were used as a subjective metric to assess
system usability.
Placement Error
The distance between the ground truth (represented by land-
marks over the images) and the locations marked by the participants
was measured. This distance was measured in centimeters: the depth
position in the workspace of each annotation was retrieved with the
Kinect and analyzed using a computer algorithm. For each annota-
tion, the placement error is the average Euclidean distance between
its ground truth and the mentee’s marked location, in the 3D work-
space. Averages per participant per trial were obtained for each task.
Task Completion Time
The time taken to complete each task was measured in seconds
for each subject and task.
Focus Shifts
A focus shift was defined as a noticeable change in head
orientation away from the operating field. Focus shifts were deter-
mined as an absolute value per participant per task. Focus shifts
demanded by the task (eg, looking for the next tool) were not
included.
Workspace Efficiency
This analysis determined the number of times and for how
long the mentee’s arms would have collided with the tablet, had one
been present. Before the participants’ arrival, the 3D position in the
workspace of a tablet that was placed over the patient simulator was
acquired with a depth camera. This position was treated as constant
throughout the entire experiment. The skeletal tracking data of each
participant was used to assess whether the participants’ forearm
intersected the hypothetical tablet’s position. The total number of
collisions and their duration (both as an absolute value and as a ratio
of the total completion time) were calculated.
Responses to the Questionnaire
Participants filled a usability questionnaire regarding the used
telementoring system. Participants answered the questions on a 5-
level Likert scale from ‘‘Strongly Disagree’’ to ‘‘Strongly Agree’’
(1– 5, respectively). Overall scores were calculated among all
the participants.
Statistical Analysis
The statistical analysis was based on the comparison between
two populations: participants using STAR and participants using
telestrator. The normality assumption of the data was evaluated using
the Shapiro-Wilk test.
30
When data pointed to non-normality, the
nonparametric Wilcoxon rank-sum test
31
was used to compare the
two populations. In contrast, a ttest using the Satterthwaite condition
FIGURE 5. Post experiment questionnaire distributed to par-
ticipants. The questionnaire assessed the telementoring system
used by the participants with a five-level Likert scale and gave
participants the option of providing additional comments
regarding the system’s features and usability.
Rojas-Mun
˜oz et al Annals of Surgery Volume XX, Number XX, Month 2018
4| www.annalsofsurgery.com ß2018 Wolters Kluwer Health, Inc. All rights reserved.
Copyright © 2018 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
CE: R.R.; ANNSURG-D-17-02211; Total nos of Pages: 6;
ANNSURG-D-17-02211
for nonequal variance
32
was used when the normality assumption
was supported. The data were summarized as mean (m)standard
deviation (s
x
) for normally distributed data, and as
median interquartile range (IQR) for non-normal distributed data.
For the placement error metric, the best and worst results were
extracted from each condition, giving n ¼24 for the first task and n ¼
8 for the second. For completion time and focus shifts, the compar-
isons were made with n ¼30 for the first task, and n ¼10 for
the second.
RESULTS
The following subsections present the study results.
Placement Error
For Task 1, participants in the STAR condition (m, 11.37 mm;
s
x
, 0.72 mm) presented 45% less (P<0.001) average placement
error than those in the conventional telestrator condition (m,
20.73 mm; s
x
, 5.11 mm). For Task 2, participants using STAR (m,
8.606 mm; s
x
, 0.806 mm) presented 14% less (P¼0.01) average
placement error than those using telestrator (m, 9.95; s
x
, 1.07 mm).
Task Completion Time
Participants in the STAR condition (median, 46.210; IQR,
21.8 s) completed Task 1 31% slower (P<0.001) than those in the
telestrator condition (median, 31.85; IQR, 12.8 seconds). Participants
using the STAR platform (m, 256.8; s
x
, 56.2 seconds) performed
Task 2 24% slower (P¼0.013) than those using telestrator (m, 194.1;
s
x
, 31.6 seconds).
Focus Shifts
For Task 1, participants in the STAR condition (m, 0.83; s
x
,
0.97) performed 92% less (P<0.001) focus shifts in average than
those using telestrator (m, 11.10; s
x
, 3.33). For Task 2, participants
using STAR (median, 4.00; IQR, 7.00) performed 88% less (P¼
0.0039) focus shifts in average than those using telestrator (median,
34.50; IQR, 14.75).
Workspace Efficiency
Table 1 summarizes the results of the workspace efficiency
analysis. The ARHMD avoided 4.8 collisions for Task 1 on average,
and 3.8 collisions for Task 2. The duration of the potential collisions
on average was 3.2 and 1.3 seconds, respectively. For some partic-
ipants, Task 1 implied as many as 27 collisions, totaling 51% of the
task completion time.
Questionnaire Responses
Participants considered the STAR platform to be more favor-
able in terms of telementoring capabilities offered (4.71 vs 4.50);
ease of use (4.43 vs 4.38); ease to follow instructions (4.71 vs 4.38);
information exchange efficiency (4.43 vs 3.88); and reduction of time
taken (4.29 vs 4.00). Likewise, participants commented that STAR
had a less negative impact in frustration (4.57 vs 3.88) and in time
taken to complete the procedure (4.14 vs 3.88). Participants com-
mented that the STAR platform is useful and interesting, but it had a
limited FOV and its imagery may produce head discomfort and
ocular fatigue when the device is not adjusted correctly.
DISCUSSION
When compared against conventional telestrators, placement
error and focus shifts were significantly reduced with the ARHMD.
Participants did not have to shift focus to receive instructions, leading
to reduced cognitive load and error accumulation.
5
In addition, the
platform’s 3D visualization preserved depth perception to afford a
more natural mapping between the annotation’s source and destina-
tion.
However, completion time was higher when using the
ARHMD. Because participants were observing the absolute 3D
position in which the annotation was supposed to be located, they
tried to match the location as closely as possible, which led to
increased task completion times. This is consistent with our previous
work
5
and with speed-accuracy tradeoff literature:
33– 35
providing
more opportunities to refine placement is linked to increased task
completion times.
The questionnaire revealed more positive attitudes toward the
ARHMD condition than to the telestrator condition. Most ARHMD
users expressed that the system created a more immerse experience,
both with their comments and scores in the questionnaire. Some
questionnaire responses do not match the results of the statistical
analysis performed, specifically those regarding the time taken to
complete the tasks. Nevertheless, the questionnaire suggests that
participants preferred using STAR and perceived an improvement in
their performance.
Switching from a 2D to a 3D environment enhanced the STAR
platform beyond its previous tablet-based interface. The ARHMD
acted as a fully transparent display, favoring immersion and creating
a more natural experience as it preserved binocular depth cues,
crucial when performing dexterous tasks. The workspace efficiency
analysis reveals that collisions would have occurred had a tablet been
present. Therefore, the presence of a tablet would have interfered
with mentees’ free selection of body poses and motions during their
performance; participants would have to assume less comfortable
poses to avoid collisions when a tablet is in use. In addition, the
ARHMD is easier to deploy, and lets the user move freely and
observe annotations from different angles. However, this HMD
system should not be seen as a replacement of the previous STAR
tablet-based platform, but rather as a portable, first-response version
of the system. For example, even if the ARHMD system excels in
austere scenarios that require in-situ attention to stabilize the patient,
the live video feedback it provides would not be in a static position,
which is a given when using the STAR tablet device.
TABLE 1. Summary of Workspace Efficiency Analysis. Collision Durations of Over 50% of the Total Time Taken Demonstrate
the Encumbrance of Tablet-based Systems
Task 1 Task 2
Metric Trial 1 Trial 2 Trial 3
Number of collisions AVG 5.7 4.6 4.1 3.8
MAX 24 27 14 24
Collision duration AVG 4.4s
8.7%
2.5s
7.6%
2.7s
7.8%
1.3s
0.68%
MAX 28.8s
43%
13s
51%
17s
39%
15.7s
8.8%
Annals of Surgery Volume XX, Number XX, Month 2018 Telementoring Without Encumbrance
ß2018 Wolters Kluwer Health, Inc. All rights reserved. www.annalsofsurgery.com | 5
Copyright © 2018 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
CE: R.R.; ANNSURG-D-17-02211; Total nos of Pages: 6;
ANNSURG-D-17-02211
Current limitations include ARHMD hardware constraints
such as intermittent 3D imagery drift and reduced FOV through
which the annotations can be seen. In addition, the graphical
annotations shown to the ARHMD user are based on an approxi-
mated model of the human visual system; a more robust calibration
process is required to ensure proper depth perception for each user.
Future work will explore an individually-tailored system calibration,
and broadcast of the mentee’s first-person perspective to the mentor.
Moreover, although medical students and a simple task setup are the
reasonable placeholders for a medic requiring telementored guid-
ance, as our system matures its usefulness needs to be validated in
more complex scenarios and with populations consisting of surgery
residents and general surgeons.
CONCLUSION
This work evaluated the telementoring capabilities of the
STAR platform based on an Augmented Reality Head-Mounted
Display (ARHMD). The system was compared against a conven-
tional telestrator device (guidance in a nearby monitor). Participants
completed tasks regarding anatomical landmarking and a mock
abdominal incision procedure. Although participants using the STAR
platform completed the task slightly slower than those using teles-
trator, the placement error and number of focus shifts were signifi-
cantly reduced. A post-experiment questionnaire revealed that
participants using STAR had a more immersive experience and an
improved information exchange with the mentor. Moreover, an
analysis of the participants’ arms movements revealed that the
ARHMD allowed them to perform more natural movements and
use their workspace more efficiently, an improvement when com-
pared to our previously reported tablet-based system. This study
suggests that ARHMD devices can improve surgical performance
during telementoring by displaying augmented 3D annotations
directly into the users’ FOV.
REFERENCES
1. Lavie N, Hirst A, De Fockert JW, et al. Load theory of selective attention and
cognitive control. J Exp Psychol Gen. 2004;133:339 –354.
2. Kahol K, Smith M, Mayes S, et al. The effect of fatigue on cognitive and
psychomotor skills of surgical residents. International Conference on Foun-
dations of Augmented Cognition. Berlin, Heidelberg: Springer; 2007, 304-313.
3. Andersen D, Popescu V, Cabrera ME, et al. Virtual annotations of the surgical
field through an augmented reality transparent display. Vis Comput.
2016;32:1481– 1498.
4. Luursema J-M, Verwey WB, Kommers PA, et al. The role of stereopsis in
virtual anatomical learning. Interact Comput. 2008;20:455 –460.
5. Andersen D, Popescu V, Cabrera ME, et al. Medical telementoring using an
augmented reality transparent display. Surgery. 2016;159:1646–1653.
6. Treter S, Perrier N, Sosa JA, et al. Telementoring: a multi-institutional
experience with the introduction of a novel surgical approach for adrenalec-
tomy. Ann Surg Oncol. 2013;20:2754–2758.
7. Snyderman CH, Gardner PA, Lanisnik B, et al. Surgical telementoring: a new
model for surgical training. The Laryngoscope. 2016;126:1334 –1338.
8. Garcia P. Telemedicine for the battlefield: present and future technologies.
Surgical Robotics. Boston, MA: Springer; 2011, 33-68.
9. Kirkpatrick AW, McKee JL, McBeth PB, et al. The Damage Control Surgery
in Austere Environments Research Group (DCSAERG): a dynamic program
to facilitate real-time telementoring/telediagnosis to address exsanguination in
extreme and austere environments. J Trauma Acute Care Surg. 2017;83:
S156– S163.
10. Hung AJ, Chen J, Shah A, et al. Telementoring and telesurgery for minimally
invasive surgery. JUrol. 2018;199:355– 369.
11. Budrionis A, Augestad KM, Patel HR, et al. An evaluation framework for
defining the contributions of telestration in surgical telementoring. Interact J
Med Res. 2013;2. e14.
12. Rafiq A, Moore JA, Zhao X, et al. Digital video capture and synchronous
consultation in open surgery. Ann Surg. 2004;239:567–573.
13. Bogen EM, Augestad KM, Patel HR, et al. Telementoring in education of
laparoscopic surgeons: an emerging technology. World J Gastrointest Endosc.
2014;6:148– 155.
14. Hinata N, Miyake H, Kurahashi T, et al. Novel telementoring system for robot-
assisted radical prostatectomy: impact on the learning curve. Urology.
2014;83:1088– 1092.
15. Shin DH, Dalag L, Azhar RA, et al. A novel interface for the telementoring of
robotic surgery. BJU Int. 2015;116:302– 308.
16. Ali MR, Loggins JP, Fuller WD, et al. 3-D telestration: a teaching tool for
robotic surgery. J Laparoendosc Adv Surg Tech. 2008;18:107–112.
17. Bilgic E, Turkdogan S, Watanabe Y, et al. Effectiveness of telementoring in
surgery compared with on-site mentoring: a systematic review. Surg Innov.
2017;24:379– 385.
18. Azuma RT. A survey of augmented reality. Presence Teleoperators Virtual
Environ. 1997;6:355–385.
19. Shuhaiber JH. Augmented reality in surgery. Arch Surg. 2004;139:170–174.
20. Bernhardt S, Nicolau SA, Agnus V, et al. Automatic localization of endoscope
in intraoperative CT image: a simple approach to augmented reality guidance
in laparoscopic surgery. Med Image Anal. 2016;30:130– 143.
21. Citardi MJ, Agbetoba A, Bigcas JL, et al. Augmented reality for endoscopic
sinus surgery with surgical navigation: a cadaver study. Int Forum Allergy
Rhinol. 2016;6:523– 528.
22. Nguyen NQ, Ramjist JM, Jivraj J, et al. Preliminary development of aug-
mented reality systems for spinal surgery. Clinical and Translational Neuro-
photonics. San Francisco, CA: International Society for Optics and Photonics;
2017, 100500K.
23. Vera AM, Russo M, Mohsin A, et al. Augmented reality telementoring (ART)
platform: a randomized controlled trial to assess the efficacy of a new surgical
education technology. Surg Endosc. 2014;28:3467–3472.
24. Shenai MB, Dillavou M, Shum C, et al. Virtual interactive presence and
augmented reality (VIPAR) for remote surgical assistance. Oper Neurosurg.
2011;68(suppl 1):200– 207.
25. Dickey RM, Srikishen N, Lipshultz LI, et al. Augmented reality assisted
surgery: a urologic training tool. Asian J Androl. 2016;18:732– 734.
26. Marescaux J, Rubino F, Arenas M, et al. Augmented-reality– assisted laparo-
scopic adrenalectomy. Jama. 2004;292:2211 –2215.
27. Campbell DT, Stanley JC. Experimental and quasi-experimental designs for
research. Handb Res Teach NL Gage Ed. 1966;171– 246.
28. Panait L, Rafiq A, Tomulescu V, et al. Telementoring versus on-site mentoring
in virtual reality-based surgical training. Surg Endosc Interv Tech. 2006;20:
113– 118.
29. Sereno S, Mutter D, Dallemagne B, et al. Telementoring for minimally
invasive surgical training by wireless robot. Surg Innov. 2007;14:184–191.
30. Shapiro SS, Wilk MB. An analysis of variance test for normality (complete
samples). Biometrika. 1965;52:591–611.
31. Mann HB, Whitney DR. On a test of whether one of two random variables is
stochastically larger than the other. The annals of mathematical statistics.
1947;18:50– 60.
32. Satterthwaite FE. An approximate distribution of estimates of variance
components. Biom Bull. 1946;2:110– 114.
33. Wickelgren WA. Speed-accuracy tradeoff and information processing dynam-
ics. Acta Psychol (Amst). 1977;41:67 –85.
34. Chien JH, Tiwari MM, Suh IH, et al. Accuracy and speed trade-off in robot-
assisted surgery. Int J Med Robot. 2010;6:324– 329.
35. Thura D, Cos I, Trung J, et al. Context-dependent urgency influences speed –
accuracy trade-offs in decision-making and movement execution. J Neurosci.
2014;34:16442– 16454.
Rojas-Mun
˜oz et al Annals of Surgery Volume XX, Number XX, Month 2018
6| www.annalsofsurgery.com ß2018 Wolters Kluwer Health, Inc. All rights reserved.
Copyright © 2018 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
