ArticlePDF Available

Abstract and Figures

Objective: This study investigates the benefits of a surgical telementoring system based on an augmented reality head-mounted display (ARHMD) that overlays surgical instructions directly onto the surgeon's view of the operating field, without workspace obstruction. Summary background data: In conventional telestrator-based telementoring, the surgeon views annotations of the surgical field by shifting focus to a nearby monitor, which substantially increases cognitive load. As an alternative, tablets have been used between the surgeon and the patient to display instructions; however, tablets impose additional obstructions of surgeon's motions. Methods: Twenty medical students performed anatomical marking (Task1) and abdominal incision (Task2) on a patient simulator, in 1 of 2 telementoring conditions: ARHMD and telestrator. The dependent variables were placement error, number of focus shifts, and completion time. Furthermore, workspace efficiency was quantified as the number and duration of potential surgeon-tablet collisions avoided by the ARHMD. Results: The ARHMD condition yielded smaller placement errors (Task1: 45%, P < 0.001; Task2: 14%, P = 0.01), fewer focus shifts (Task1: 93%, P < 0.001; Task2: 88%, P = 0.0039), and longer completion times (Task1: 31%, P < 0.001; Task2: 24%, P = 0.013). Furthermore, the ARHMD avoided potential tablet collisions (4.8 for 3.2 seconds in Task1; 3.8 for 1.3 seconds in Task2). Conclusion: The ARHMD system promises to improve accuracy and to eliminate focus shifts in surgical telementoring. Because ARHMD participants were able to refine their execution of instructions, task completion time increased. Unlike a tablet system, the ARHMD does not require modifying natural motions to avoid collisions.
Content may be subject to copyright.
CE: R.R.; ANNSURG-D-17-02211; Total nos of Pages: 6;
Surgical Telementoring Without Encumbrance
A Comparative Study of See-through Augmented Reality-based Approaches
Edgar Rojas-Mun
˜oz, BS,
Maria Eugenia Cabrera, MSc,
Daniel Andersen, MS,yVoicu Popescu, PhD,y
Sherri Marley, BSN, RN,zBrian Mullis, MD,zBen Zarzaur, MD, MPH, FACS,zand Juan Wachs, PhD
Objective: This study investigates the benefits of a surgical telementoring
system based on an augmented reality head-mounted display (ARHMD) that
overlays surgical instructions directly onto the surgeon’s view of the operating
field, without workspace obstruction.
Summary Background Data: In conventional telestrator-based telementor-
ing, the surgeon views annotations of the surgical field by shifting focus to a
nearby monitor, which substantially increases cognitive load. As an alternative,
tablets have been used between the surgeon and the patient to display instruc-
tions; however, tablets impose additional obstructions of surgeon’s motions.
Methods: Twenty medical students performed anatomical marking (Task1)
and abdominal incision (Task2) on a patient simulator, in 1 of 2 telementoring
conditions: ARHMD and telestrator. The dependent variables were placement
error, number of focus shifts, and completion time. Furthermore, workspace
efficiency was quantified as the number and duration of potential surgeon-
tablet collisions avoided by the ARHMD.
Results: The ARHMD condition yielded smaller placement errors (Task1:
45%, P<0.001; Task2: 14%, P¼0.01), fewer focus shifts (Task1: 93%, P<
0.001; Task2: 88%, P¼0.0039), and longer completion times (Task1: 31%, P
<0.001; Task2: 24%, P¼0.013). Furthermore, the ARHMD avoided
potential tablet collisions (4.8 for 3.2 seconds in Task1; 3.8 for 1.3 seconds
in Task2).
Conclusion: The ARHMD system promises to improve accuracy and to
eliminate focus shifts in surgical telementoring. Because ARHMD partic-
ipants were able to refine their execution of instructions, task completion time
increased. Unlike a tablet system, the ARHMD does not require modifying
natural motions to avoid collisions.
Keywords: augmented reality, surgical telementoring, telemedicine,
(Ann Surg 2018;xx:xxx– xxx)
Surgical telementoring is a method to deliver specialized expertise
to a mentee in scenarios where expertise is not readily available.
For example, telementoring can allow a remote expert surgeon to
convey specialized surgical expertise to a generalist surgeon in rural
hospitals, in disaster-affected regions and in the battlefield. Further-
more, telementoring can disseminate surgical procedure innovations
across the world.
Telestrators are the conventional approach for telementoring.
A remote expert receives and annotates video imagery from a
mentee’s operating field. These annotations are sent back to the
mentee where they are displayed on a nearby monitor. However, this
approach requires the mentee to shift focus from the surgical field to
the nearby monitor, and to memorize the annotations’ positions to
map them onto the patient’s anatomy. This introduces additional
cognitive load that can contribute to surgeon fatigue and error-prone
In previous work,
we leveraged augmented reality (AR)
technology to improve surgical telementoring. With our system,
dubbed the System for Telementoring with Augmented Reality
(STAR), the mentee views the operating field through a tablet-based
transparent display, which directly overlays mentor annotations onto
the mentee’s view of the surgical field (Fig. 1).
However, using a tablet at the mentee site possesses disad-
vantages. First, the tablet degrades the mentee’s depth perception
because of loss of stereopsis: binocular depth cues are not preserved
FIGURE 1. Top: STAR tablet-based setup. The tablet is held in
place between the patient and the user with a mechanical
bracket. Bottom: First-person view of an instruction received
with the STAR tablet-based setup. The augmentation consists of
2D lines and images. The view of the camera is displayed on the
device’s screen.
From the
School of Industrial Engineering, Purdue University, West Lafayette,
IN; yDepartment of Computer Science, Purdue University, West Lafayette, IN;
and zSchool of Medicine, Indiana University, Indianapolis, IN.
Sources of Funding: Supported by both the Office of the Assistant Secretary of
Defense for Health Affairs under Award No. W81XWH-14-1-0042 and the
National Science Foundation under Grant DGE-1333468.
Opinions, interpretations, conclusions and recommendations are those of the
author and are not necessarily endorsed by the funders.
The other authors report no conflicts of interest.
Reprints: Juan Wachs, PhD, Purdue University, 315 N. Grant Street, West Lafay-
ette, IN 47907. E-mail:
Copyright ß2018 Wolters Kluwer Health, Inc. All rights reserved.
ISSN: 0003-4932/16/XXXX-0001
DOI: 10.1097/SLA.0000000000002764
Annals of Surgery Volume XX, Number XX, Month 2018 | 1
Copyright © 2018 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
CE: R.R.; ANNSURG-D-17-02211; Total nos of Pages: 6;
because of the 2D nature of the tablet screen.
This can impair hand
eye coordination and increase hesitancy when performing precise
tasks: mentees must guess how far their movements should go before
reaching the destination, hindering task completion time. An exam-
ple of this behavior can be found in our previous work.
Second, the
tablet’s position in range of the mentee’s arms might impede the
mentee’s natural motion.
This article describes an enhancement of the STAR platform at
the mentee site that preserves binocular depth cues by replacing the
tablet with a see-through AR head-mounted display (ARHMD) worn
by the mentee. 3D graphical annotations are overlaid onto the
mentee’s view of the surgical field, remaining anchored as the
mentee moves. The ARHMD generates a different image for each
eye, making the annotations perceivable at the correct depth relative
to the patient’s body. In addition, proper depth cues are preserved
since see-through ARHMDs allow mentees to see the surgical field
directly. Moreover, the ARHMD does not obstruct the surgeon’s
hands, as it is entirely self-contained on the head. We have conducted
a user study that assesses the effectiveness of the new STAR platform
in surgery. This study indicates that ARHMDs have great potential in
surgical telementoring.
Telementoring has proved successful at developing surgeons’
surgical skills when experienced mentors are not available,
and to
avoid transportation-related delays of medical equipment and per-
Additional benefits are associated with telementoring in
austere environments.
A review evaluating the tradeoffs between
on-site mentoring and telementoring technologies revealed that each
approach has its own challenges (ie, travel and time costs vs
equipment cost and network issues).
Nonetheless, telementoring
is an effective way to provide access to expert mentors and advanced
surgical techniques that otherwise would be unavailable.
Telestrator-based technologies have demonstrated their use-
fulness in surgical telementoring.
11– 13
These approaches rely on
nearby monitors to show guidance sent by a remote expert. In recent
years, preferences between 2D and 3D telestration have been
researched. 2D telestration is more widely adopted because of its
However, 2D annotations introduce occlusion issues
and degrade binocular depth perception cues, hindering users’
orientation in a 3D environment. Investigating 3D telestration, Ali
et al
developed a video algorithm to translate instructions from
mentors from 2D to 3D in a da Vinci surgical robot’s console, and
found that 3D robotic telestration is feasible and does not negatively
influence performance in controlled tasks.
However, conventional telestrator-based approaches have a
substantial disadvantage: mentees must shift focus repeatedly
between the operating field and the monitor, which adds complexity
and increases cognitive load.
This disadvantage can be mitigated
by presenting information directly into users’ field of view (FOV).
AR has been used to accomplish this outside
and inside the surgical
Although navigation-related AR applications exist for
and spinal surgery
procedures, AR
systems remain unavailable and therefore untested in most
surgical specialties.
State-of-the-art telementoring systems rely on tablet-based
devices to augment the users’ FOV.
However, this approach
introduces additional encumbrance by placing the tablet in the
surgeon’s workspace. Head-mounted displays (HMDs) can address
this drawback: while surgeons must wear a headset, workspace
encumbrance is avoided. Systems reported in urology
mentors and mentees to exchange information, but either do not
consider complex annotations or only provide 2D imagery, instead of
3D. Our work leverages the capacity of an ARHMD platform to
display 3D annotations directly in the mentee’s FOV, providing
relevant instruction without focus shifts or workspace encumbrance.
STAR is a surgical telementoring platform that displays
annotations sent by a remote expert surgeon directly into a mentee’s
FOV. It is composed of 2 subsystems: the Mentor System, used by the
mentors to deliver guidance to the mentees; and the Trainee System,
which non-specialist/mentee surgeons use to receive visual instruc-
tions from the mentors. The current system allows the placement and
anchoring of 3D imagery inside of the surgeon’s unencumbered
workspace. These annotations can be observed through the
ARHMD’s screen from any viewpoint (Fig. 2, top).
Consider the following example of how the STAR platform
can be utilized. Dr. Harrison is a surgeon situated in an operating
room who needs to perform a leg-fasciotomy on Mr. Smith, a patient
suffering from compartment syndrome. However, Dr. Harrison has
not received extensive training on this procedure and requires assis-
tance. In this case, Dr. Harrison, wearing the ARHMD Trainee
System, connects to the Mentor System, located at a Level 1 Trauma
Facility where Dr. Grover, an attending in Orthopedic Trauma, is
ready to provide support. The ARHMD broadcasts Dr. Harrison’s
FOV to Dr. Grover, who then uses the Mentor System to create
surgical annotations (incision lines, surgical instruments, among
others) in the live video sent by Dr. Harrison. These annotations
are transmitted back to the ARHMD, which projects and anchors
them in the right position over Mr. Smith’s leg. Aided with these
instructions, Dr. Harrison can successfully perform the leg-fasciot-
omy. Figure 2 (bottom) portrays an instruction as seen by the mentee
wearing the ARHMD.
FIGURE 2. Top: STAR ARHMD setup. No additional artifacts are
required except for the device worn by the user. Bottom: First-
person view of an instruction received with the STAR ARHMD
setup. The augmentation consists of 3D lines and 3D models
visible only through the device’s display.
˜oz et al Annals of Surgery Volume XX, Number XX, Month 2018
2| ß2018 Wolters Kluwer Health, Inc. All rights reserved.
Copyright © 2018 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
CE: R.R.; ANNSURG-D-17-02211; Total nos of Pages: 6;
A user study was conducted at Eskenazi Hospital (Indian-
apolis, Indiana), where 20 medical students performed telementored
tasks under 1 of 2 conditions: STAR ARHMD and conventional
telestrator. The participants had to complete 2 different tasks on a
patient simulator: an anatomical marker placement and a mock
abdominal incision. These tasks depict core surgical skills present
in most surgical procedures: landmark location and skin incision.
These tasks are atomic building blocks for more complex procedures,
and an initial evaluation of the system’s effectiveness in these tasks
would reveal insights for its application on more complex tasks.
Moreover, effectiveness in these tasks is mandatory to ensure subse-
quent effectiveness in more complex scenarios.
Participant performance was recorded and analyzed using the
following metrics: annotation placement error, task completion
time, and number of focus shifts. The ARHMD was also compared
against a tablet-based system
: by tracking the participants’ limb
motions and poses during the experiment, the number and duration
of potential collisions avoided by the ARHMD system had a tablet
device been there were calculated as a post-experiment metric.
To maintain consistency among participants, preprogrammed anno-
tations were used instead of live feedback from the Mentor
System. The following subsections elaborate on the details of the
Twenty medical students (6 females, 14 males) ranging
from 23- to 29 years old were recruited. Participants were in their
second, third, or fourth year of medical school, and had no
previous experience with surgical telementoring systems. The
study was reviewed and approved by Indiana University Institu-
tional Review Board (#1409037680), and written participant
consent was acquired for each participant. Using medical students
in the role of mentees has high ecological validity,
as tele-
mentoring is likely to be deployed to support medical students,
new graduates, and other non-specialist surgeons. Moreover,
because of the simplicity of the tasks, it was required that the
participant’s level of expertise was not that of an expert surgeon or
a surgery resident: the objective was for the participants to rely in
the telementored guidance instead of being able to complete the
procedure alone. This configuration is an acceptable placeholder
for a medic or a nonspecialist attempting to do a procedure for
which they have little or no experience. Other studies have
leveraged the medical student population in medical telementor-
ing studies.
Apparatus and Setting
Figure 3 presents a diagram of the experiment’s setting. The
experiment setup included: the patient simulator on the operating
table, the participant acting as a mentee, an ARHMD (Microsoft
HoloLens), and a nearby monitor located 608to mentees’ right.
Surgical instruments to complete the tasks were located on a Mayo
stand to the side of the operating table.
In the telestrator condition, participants looked at the nearby
monitor to receive the instructions. The images shown on the
telestrator were a top-down view of the region of interest on the
patient simulator for each step of the procedure (Fig. 4, top).
Participants in the STAR condition wore the ARHMD, which con-
structs a virtual representation of the space it is in, and enables the
placement of virtual annotations in this representation, visible only to
the user wearing the device. Participants followed a 3D replica of the
instructions used in the telestrator condition, represented using 3D
models of surgical instruments, 3D lines, and spheres (Fig. 4,
FIGURE 3. Experiment setting. For the telestrator condition,
the participant used the nearby screen to retrieve the instruc-
tions. In the ARHMD condition, the instructions were shown via
the device’s screen.
FIGURE 4. Top: Participant performing a task using the teles-
trator-based condition. The instructions for the task are
obtained by looking at a nearby screen. Bottom: Participant
performing a task using the STAR ARHMD condition. Inset is the
first-person view of the user wearing the ARHMD.
Annals of Surgery Volume XX, Number XX, Month 2018 Telementoring Without Encumbrance
ß2018 Wolters Kluwer Health, Inc. All rights reserved. | 3
Copyright © 2018 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
CE: R.R.; ANNSURG-D-17-02211; Total nos of Pages: 6;
Data Collection
Two Microsoft Kinect devices were used to record videos of
each participant’s performance. One Kinect was placed fronto-par-
allel to the patient simulator and the participant, recording infrared,
color, depth, and skeletal tracking data. The second Kinect was
placed to capture a top-down view of the procedure, recording color,
and audio. A calibration process was performed before the experi-
ment, allowing a mapping between the ARHMD reference frame and
the top-view Kinect. Time taken to complete each task was recorded
using a stopwatch.
After performing both tasks, participants answered a ques-
tionnaire on their experience with the telementoring systems. The
questionnaire (Fig. 5) contained 7 questions that evaluated each
system in terms of communication efficiency, available functionali-
ties, ease of use, and time required to complete the task. The
questionnaire included a section for comments and suggestions.
The experiment consisted of 2 tasks: an anatomical marker
placement and a mock abdominal incision. The first task had 3 trials,
whereas the second task had only 1 trial. Participants were briefed
about both tasks before starting, and were encouraged to complete
the tasks as quickly and accurately as possible.
Task 1: Marker Placement
Participants had to mark different locations on the simulator’s
body with a dry erase marker. Each trial included 7 circular-shaped
annotations located around the patient’s neck and chest. Each trial
showed different locations to avoid recall.
Task 2: Abdominal Incision
Participants had to follow 16 instructions to cut through 2
simulated layers of skin and to spread the linea alba. Incisions on the
skin layers had to be done without puncturing a balloon that
simulated the stomach. Before the execution of each step, partic-
ipants had to mark the locations/incision lines depicted in
the instruction.
Experimental Design
A between-subject design was selected, randomly assigning
participants to each of the 2 telementoring conditions. Condition was
treated as an independent variable, whereas the performance metrics
were treated as dependent variables. In addition, potential tablet
collisions were calculated for the STAR condition. Finally, the
questionnaire responses were used as a subjective metric to assess
system usability.
Placement Error
The distance between the ground truth (represented by land-
marks over the images) and the locations marked by the participants
was measured. This distance was measured in centimeters: the depth
position in the workspace of each annotation was retrieved with the
Kinect and analyzed using a computer algorithm. For each annota-
tion, the placement error is the average Euclidean distance between
its ground truth and the mentee’s marked location, in the 3D work-
space. Averages per participant per trial were obtained for each task.
Task Completion Time
The time taken to complete each task was measured in seconds
for each subject and task.
Focus Shifts
A focus shift was defined as a noticeable change in head
orientation away from the operating field. Focus shifts were deter-
mined as an absolute value per participant per task. Focus shifts
demanded by the task (eg, looking for the next tool) were not
Workspace Efficiency
This analysis determined the number of times and for how
long the mentee’s arms would have collided with the tablet, had one
been present. Before the participants’ arrival, the 3D position in the
workspace of a tablet that was placed over the patient simulator was
acquired with a depth camera. This position was treated as constant
throughout the entire experiment. The skeletal tracking data of each
participant was used to assess whether the participants’ forearm
intersected the hypothetical tablet’s position. The total number of
collisions and their duration (both as an absolute value and as a ratio
of the total completion time) were calculated.
Responses to the Questionnaire
Participants filled a usability questionnaire regarding the used
telementoring system. Participants answered the questions on a 5-
level Likert scale from ‘‘Strongly Disagree’’ to ‘‘Strongly Agree’
(1– 5, respectively). Overall scores were calculated among all
the participants.
Statistical Analysis
The statistical analysis was based on the comparison between
two populations: participants using STAR and participants using
telestrator. The normality assumption of the data was evaluated using
the Shapiro-Wilk test.
When data pointed to non-normality, the
nonparametric Wilcoxon rank-sum test
was used to compare the
two populations. In contrast, a ttest using the Satterthwaite condition
FIGURE 5. Post experiment questionnaire distributed to par-
ticipants. The questionnaire assessed the telementoring system
used by the participants with a five-level Likert scale and gave
participants the option of providing additional comments
regarding the system’s features and usability.
˜oz et al Annals of Surgery Volume XX, Number XX, Month 2018
4| ß2018 Wolters Kluwer Health, Inc. All rights reserved.
Copyright © 2018 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
CE: R.R.; ANNSURG-D-17-02211; Total nos of Pages: 6;
for nonequal variance
was used when the normality assumption
was supported. The data were summarized as mean (m)standard
deviation (s
) for normally distributed data, and as
median interquartile range (IQR) for non-normal distributed data.
For the placement error metric, the best and worst results were
extracted from each condition, giving n ¼24 for the first task and n ¼
8 for the second. For completion time and focus shifts, the compar-
isons were made with n ¼30 for the first task, and n ¼10 for
the second.
The following subsections present the study results.
Placement Error
For Task 1, participants in the STAR condition (m, 11.37 mm;
, 0.72 mm) presented 45% less (P<0.001) average placement
error than those in the conventional telestrator condition (m,
20.73 mm; s
, 5.11 mm). For Task 2, participants using STAR (m,
8.606 mm; s
, 0.806 mm) presented 14% less (P¼0.01) average
placement error than those using telestrator (m, 9.95; s
, 1.07 mm).
Task Completion Time
Participants in the STAR condition (median, 46.210; IQR,
21.8 s) completed Task 1 31% slower (P<0.001) than those in the
telestrator condition (median, 31.85; IQR, 12.8 seconds). Participants
using the STAR platform (m, 256.8; s
, 56.2 seconds) performed
Task 2 24% slower (P¼0.013) than those using telestrator (m, 194.1;
, 31.6 seconds).
Focus Shifts
For Task 1, participants in the STAR condition (m, 0.83; s
0.97) performed 92% less (P<0.001) focus shifts in average than
those using telestrator (m, 11.10; s
, 3.33). For Task 2, participants
using STAR (median, 4.00; IQR, 7.00) performed 88% less (P¼
0.0039) focus shifts in average than those using telestrator (median,
34.50; IQR, 14.75).
Workspace Efficiency
Table 1 summarizes the results of the workspace efficiency
analysis. The ARHMD avoided 4.8 collisions for Task 1 on average,
and 3.8 collisions for Task 2. The duration of the potential collisions
on average was 3.2 and 1.3 seconds, respectively. For some partic-
ipants, Task 1 implied as many as 27 collisions, totaling 51% of the
task completion time.
Questionnaire Responses
Participants considered the STAR platform to be more favor-
able in terms of telementoring capabilities offered (4.71 vs 4.50);
ease of use (4.43 vs 4.38); ease to follow instructions (4.71 vs 4.38);
information exchange efficiency (4.43 vs 3.88); and reduction of time
taken (4.29 vs 4.00). Likewise, participants commented that STAR
had a less negative impact in frustration (4.57 vs 3.88) and in time
taken to complete the procedure (4.14 vs 3.88). Participants com-
mented that the STAR platform is useful and interesting, but it had a
limited FOV and its imagery may produce head discomfort and
ocular fatigue when the device is not adjusted correctly.
When compared against conventional telestrators, placement
error and focus shifts were significantly reduced with the ARHMD.
Participants did not have to shift focus to receive instructions, leading
to reduced cognitive load and error accumulation.
In addition, the
platform’s 3D visualization preserved depth perception to afford a
more natural mapping between the annotation’s source and destina-
However, completion time was higher when using the
ARHMD. Because participants were observing the absolute 3D
position in which the annotation was supposed to be located, they
tried to match the location as closely as possible, which led to
increased task completion times. This is consistent with our previous
and with speed-accuracy tradeoff literature:
33– 35
more opportunities to refine placement is linked to increased task
completion times.
The questionnaire revealed more positive attitudes toward the
ARHMD condition than to the telestrator condition. Most ARHMD
users expressed that the system created a more immerse experience,
both with their comments and scores in the questionnaire. Some
questionnaire responses do not match the results of the statistical
analysis performed, specifically those regarding the time taken to
complete the tasks. Nevertheless, the questionnaire suggests that
participants preferred using STAR and perceived an improvement in
their performance.
Switching from a 2D to a 3D environment enhanced the STAR
platform beyond its previous tablet-based interface. The ARHMD
acted as a fully transparent display, favoring immersion and creating
a more natural experience as it preserved binocular depth cues,
crucial when performing dexterous tasks. The workspace efficiency
analysis reveals that collisions would have occurred had a tablet been
present. Therefore, the presence of a tablet would have interfered
with mentees’ free selection of body poses and motions during their
performance; participants would have to assume less comfortable
poses to avoid collisions when a tablet is in use. In addition, the
ARHMD is easier to deploy, and lets the user move freely and
observe annotations from different angles. However, this HMD
system should not be seen as a replacement of the previous STAR
tablet-based platform, but rather as a portable, first-response version
of the system. For example, even if the ARHMD system excels in
austere scenarios that require in-situ attention to stabilize the patient,
the live video feedback it provides would not be in a static position,
which is a given when using the STAR tablet device.
TABLE 1. Summary of Workspace Efficiency Analysis. Collision Durations of Over 50% of the Total Time Taken Demonstrate
the Encumbrance of Tablet-based Systems
Task 1 Task 2
Metric Trial 1 Trial 2 Trial 3
Number of collisions AVG 5.7 4.6 4.1 3.8
MAX 24 27 14 24
Collision duration AVG 4.4s
MAX 28.8s
Annals of Surgery Volume XX, Number XX, Month 2018 Telementoring Without Encumbrance
ß2018 Wolters Kluwer Health, Inc. All rights reserved. | 5
Copyright © 2018 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
CE: R.R.; ANNSURG-D-17-02211; Total nos of Pages: 6;
Current limitations include ARHMD hardware constraints
such as intermittent 3D imagery drift and reduced FOV through
which the annotations can be seen. In addition, the graphical
annotations shown to the ARHMD user are based on an approxi-
mated model of the human visual system; a more robust calibration
process is required to ensure proper depth perception for each user.
Future work will explore an individually-tailored system calibration,
and broadcast of the mentee’s first-person perspective to the mentor.
Moreover, although medical students and a simple task setup are the
reasonable placeholders for a medic requiring telementored guid-
ance, as our system matures its usefulness needs to be validated in
more complex scenarios and with populations consisting of surgery
residents and general surgeons.
This work evaluated the telementoring capabilities of the
STAR platform based on an Augmented Reality Head-Mounted
Display (ARHMD). The system was compared against a conven-
tional telestrator device (guidance in a nearby monitor). Participants
completed tasks regarding anatomical landmarking and a mock
abdominal incision procedure. Although participants using the STAR
platform completed the task slightly slower than those using teles-
trator, the placement error and number of focus shifts were signifi-
cantly reduced. A post-experiment questionnaire revealed that
participants using STAR had a more immersive experience and an
improved information exchange with the mentor. Moreover, an
analysis of the participants’ arms movements revealed that the
ARHMD allowed them to perform more natural movements and
use their workspace more efficiently, an improvement when com-
pared to our previously reported tablet-based system. This study
suggests that ARHMD devices can improve surgical performance
during telementoring by displaying augmented 3D annotations
directly into the users’ FOV.
1. Lavie N, Hirst A, De Fockert JW, et al. Load theory of selective attention and
cognitive control. J Exp Psychol Gen. 2004;133:339 –354.
2. Kahol K, Smith M, Mayes S, et al. The effect of fatigue on cognitive and
psychomotor skills of surgical residents. International Conference on Foun-
dations of Augmented Cognition. Berlin, Heidelberg: Springer; 2007, 304-313.
3. Andersen D, Popescu V, Cabrera ME, et al. Virtual annotations of the surgical
field through an augmented reality transparent display. Vis Comput.
2016;32:1481– 1498.
4. Luursema J-M, Verwey WB, Kommers PA, et al. The role of stereopsis in
virtual anatomical learning. Interact Comput. 2008;20:455 460.
5. Andersen D, Popescu V, Cabrera ME, et al. Medical telementoring using an
augmented reality transparent display. Surgery. 2016;159:1646–1653.
6. Treter S, Perrier N, Sosa JA, et al. Telementoring: a multi-institutional
experience with the introduction of a novel surgical approach for adrenalec-
tomy. Ann Surg Oncol. 2013;20:2754–2758.
7. Snyderman CH, Gardner PA, Lanisnik B, et al. Surgical telementoring: a new
model for surgical training. The Laryngoscope. 2016;126:1334 –1338.
8. Garcia P. Telemedicine for the battlefield: present and future technologies.
Surgical Robotics. Boston, MA: Springer; 2011, 33-68.
9. Kirkpatrick AW, McKee JL, McBeth PB, et al. The Damage Control Surgery
in Austere Environments Research Group (DCSAERG): a dynamic program
to facilitate real-time telementoring/telediagnosis to address exsanguination in
extreme and austere environments. J Trauma Acute Care Surg. 2017;83:
S156– S163.
10. Hung AJ, Chen J, Shah A, et al. Telementoring and telesurgery for minimally
invasive surgery. JUrol. 2018;199:355– 369.
11. Budrionis A, Augestad KM, Patel HR, et al. An evaluation framework for
defining the contributions of telestration in surgical telementoring. Interact J
Med Res. 2013;2. e14.
12. Rafiq A, Moore JA, Zhao X, et al. Digital video capture and synchronous
consultation in open surgery. Ann Surg. 2004;239:567–573.
13. Bogen EM, Augestad KM, Patel HR, et al. Telementoring in education of
laparoscopic surgeons: an emerging technology. World J Gastrointest Endosc.
2014;6:148– 155.
14. Hinata N, Miyake H, Kurahashi T, et al. Novel telementoring system for robot-
assisted radical prostatectomy: impact on the learning curve. Urology.
2014;83:1088– 1092.
15. Shin DH, Dalag L, Azhar RA, et al. A novel interface for the telementoring of
robotic surgery. BJU Int. 2015;116:302– 308.
16. Ali MR, Loggins JP, Fuller WD, et al. 3-D telestration: a teaching tool for
robotic surgery. J Laparoendosc Adv Surg Tech. 2008;18:107–112.
17. Bilgic E, Turkdogan S, Watanabe Y, et al. Effectiveness of telementoring in
surgery compared with on-site mentoring: a systematic review. Surg Innov.
2017;24:379– 385.
18. Azuma RT. A survey of augmented reality. Presence Teleoperators Virtual
Environ. 1997;6:355–385.
19. Shuhaiber JH. Augmented reality in surgery. Arch Surg. 2004;139:170–174.
20. Bernhardt S, Nicolau SA, Agnus V, et al. Automatic localization of endoscope
in intraoperative CT image: a simple approach to augmented reality guidance
in laparoscopic surgery. Med Image Anal. 2016;30:130– 143.
21. Citardi MJ, Agbetoba A, Bigcas JL, et al. Augmented reality for endoscopic
sinus surgery with surgical navigation: a cadaver study. Int Forum Allergy
Rhinol. 2016;6:523– 528.
22. Nguyen NQ, Ramjist JM, Jivraj J, et al. Preliminary development of aug-
mented reality systems for spinal surgery. Clinical and Translational Neuro-
photonics. San Francisco, CA: International Society for Optics and Photonics;
2017, 100500K.
23. Vera AM, Russo M, Mohsin A, et al. Augmented reality telementoring (ART)
platform: a randomized controlled trial to assess the efficacy of a new surgical
education technology. Surg Endosc. 2014;28:3467–3472.
24. Shenai MB, Dillavou M, Shum C, et al. Virtual interactive presence and
augmented reality (VIPAR) for remote surgical assistance. Oper Neurosurg.
2011;68(suppl 1):200– 207.
25. Dickey RM, Srikishen N, Lipshultz LI, et al. Augmented reality assisted
surgery: a urologic training tool. Asian J Androl. 2016;18:732– 734.
26. Marescaux J, Rubino F, Arenas M, et al. Augmented-reality– assisted laparo-
scopic adrenalectomy. Jama. 2004;292:2211 2215.
27. Campbell DT, Stanley JC. Experimental and quasi-experimental designs for
research. Handb Res Teach NL Gage Ed. 1966;171– 246.
28. Panait L, Rafiq A, Tomulescu V, et al. Telementoring versus on-site mentoring
in virtual reality-based surgical training. Surg Endosc Interv Tech. 2006;20:
113– 118.
29. Sereno S, Mutter D, Dallemagne B, et al. Telementoring for minimally
invasive surgical training by wireless robot. Surg Innov. 2007;14:184–191.
30. Shapiro SS, Wilk MB. An analysis of variance test for normality (complete
samples). Biometrika. 1965;52:591–611.
31. Mann HB, Whitney DR. On a test of whether one of two random variables is
stochastically larger than the other. The annals of mathematical statistics.
1947;18:50– 60.
32. Satterthwaite FE. An approximate distribution of estimates of variance
components. Biom Bull. 1946;2:110– 114.
33. Wickelgren WA. Speed-accuracy tradeoff and information processing dynam-
ics. Acta Psychol (Amst). 1977;41:67 –85.
34. Chien JH, Tiwari MM, Suh IH, et al. Accuracy and speed trade-off in robot-
assisted surgery. Int J Med Robot. 2010;6:324– 329.
35. Thura D, Cos I, Trung J, et al. Context-dependent urgency influences speed
accuracy trade-offs in decision-making and movement execution. J Neurosci.
2014;34:16442– 16454.
˜oz et al Annals of Surgery Volume XX, Number XX, Month 2018
6| ß2018 Wolters Kluwer Health, Inc. All rights reserved.
Copyright © 2018 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
... The 41% (16/39) of articles included in this section (Multimedia Appendix 2 [31,35,38,39,46,52,53,[55][56][57][59][60][61]63,64,68]) described AR in remote nonsurgical and surgical contexts. The latter included studies that focused on surgical efficiency, long-distance consultation, and differences between telesurgical systems. ...
... Andersen et al [59,63], Rojas-Muñoz et al [61], and Zhang et al [68] developed trials to compare the procedural efficiencies of different AR tool setups during telesurgery. Andersen et al [63] compared conventional telestration, in which the hybrid video with the expert's annotations is displayed on a separate monitor outside the surgical field, with the STAR platform, in which the hybrid video is shown on a tablet directly above the field and the surgeon's hands. ...
... Less idle time (P<.001) and higher performance scores were observed (P<.05 for both raters) in the STAR group [59]. Rojas-Muñoz et al [61] later compared STAR with HMD-STAR, in which the remote viewer's modifications were displayed on a headset instead of a tablet. The RCT used 2 groups of medical students performing a similar marking and incision task. ...
Full-text available
Background: Over the last decade, augmented reality (AR) has emerged in health care as a tool for visualizing data and enhancing simulation learning. AR, which has largely been explored for communication and collaboration in nonhealth contexts, could play a role in shaping future remote medical services and training. This review summarized existing studies implementing AR in real-time telemedicine and telementoring to create a foundation for health care providers and technology developers to understand future opportunities in remote care and education. Objective: This review described devices and platforms that use AR for real-time telemedicine and telementoring, the tasks for which AR was implemented, and the ways in which these implementations were evaluated to identify gaps in research that provide opportunities for further study. Methods: We searched PubMed, Scopus, Embase, and MEDLINE to identify English-language studies published between January 1, 2012, and October 18, 2022, implementing AR technology in a real-time interaction related to telemedicine or telementoring. The search terms were “augmented reality” OR “AR” AND “remote” OR “telemedicine” OR “telehealth” OR “telementoring.” Systematic reviews, meta-analyses, and discussion-based articles were excluded from analysis. Results: A total of 39 articles met the inclusion criteria and were categorized into themes of patient evaluation, medical intervention, and education. In total, 20 devices and platforms using AR were identified, with common features being the ability for remote users to annotate, display graphics, and display their hands or tools in the local user’s view. Common themes across the studies included consultation and procedural education, with surgery, emergency, and hospital medicine being the most represented specialties. Outcomes were most often measured using feedback surveys and interviews. The most common objective measures were time to task completion and performance. Long-term outcome and resource cost measurements were rare. Across the studies, user feedback was consistently positive for perceived efficacy, feasibility, and acceptability. Comparative trials demonstrated that AR-assisted conditions had noninferior reliability and performance and did not consistently extend procedure times compared with in-person controls. Conclusions: Studies implementing AR in telemedicine and telementoring demonstrated the technology’s ability to enhance access to information and facilitate guidance in multiple health care settings. However, AR’s role as an alternative to current telecommunication platforms or even in-person interactions remains to be validated, with many disciplines and provider-to-nonprovider uses still lacking robust investigation. Additional studies comparing existing methods may offer more insight into this intersection, but the early stage of technical development and the lack of standardized tools and adoption have hindered the conduct of larger longitudinal and randomized controlled trials. Overall, AR has the potential to complement and advance the capabilities of remote medical care and learning, creating unique opportunities for innovator, provider, and patient involvement.
... The feasibility of telementoring applicability in the performance of chest thoracotomy, skin grafting, and fasciotomy has been evaluated using ex vivo animal models [78,87,88]. Telementoring has been used to great effect in different stages of surgical planning in various orthopedic, craniofacial, spinal cord, vascular, and cardiothoracic surgeries [6,[89][90][91][92][93][94][95][96][97][98][99][100]. Teleconsultation is a primary segment of telehealth services, broadly consisting of remote consultation services using ICT. ...
... The System for Telementoring with AR (STAR) platform now combines optical see-through display, HoloLens AR HMD. Similarly, this system allows for telementoring guidance by overlaying 3D graphical annotations onto the mentee's view of the surgical field, which remains anchored in the same place even after the mentee moves their head position [95]. ...
Full-text available
Background The distinctive features of the digital reality platforms, namely augmented reality (AR), virtual reality (VR), and mixed reality (MR) have extended to medical education, training, simulation, and patient care. Furthermore, this digital reality technology seamlessly merges with information and communication technology creating an enriched telehealth ecosystem. This review provides a composite overview of the prospects of telehealth delivered using the MR platform in clinical settings. Objective This review identifies various clinical applications of high-fidelity digital display technology, namely AR, VR, and MR, delivered using telehealth capabilities. Next, the review focuses on the technical characteristics, hardware, and software technologies used in the composition of AR, VR, and MR in telehealth. Methods We conducted a scoping review using the methodological framework and reporting design using the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews) guidelines. Full-length articles in English were obtained from the Embase, PubMed, and Web of Science databases. The search protocol was based on the following keywords and Medical Subject Headings to obtain relevant results: “augmented reality,” “virtual reality,” “mixed-reality,” “telemedicine,” “telehealth,” and “digital health.” A predefined inclusion-exclusion criterion was developed in filtering the obtained results and the final selection of the articles, followed by data extraction and construction of the review. Results We identified 4407 articles, of which 320 were eligible for full-text screening. A total of 134 full-text articles were included in the review. Telerehabilitation, telementoring, teleconsultation, telemonitoring, telepsychiatry, telesurgery, and telediagnosis were the segments of the telehealth division that explored the use of AR, VR, and MR platforms. Telerehabilitation using VR was the most commonly recurring segment in the included studies. AR and MR has been mainly used for telementoring and teleconsultation. The most important technical features of digital reality technology to emerge with telehealth were virtual environment, exergaming, 3D avatars, telepresence, anchoring annotations, and first-person viewpoint. Different arrangements of technology—3D modeling and viewing tools, communication and streaming platforms, file transfer and sharing platforms, sensors, high-fidelity displays, and controllers—formed the basis of most systems. Conclusions This review constitutes a recent overview of the evolving digital AR and VR in various clinical applications using the telehealth setup. This combination of telehealth with AR, VR, and MR allows for remote facilitation of clinical expertise and further development of home-based treatment. This review explores the rapidly growing suite of technologies available to users within the digital health sector and examines the opportunities and challenges they present.
... This exciting advancement was demonstrated by Rojas-Munoz et al. that utilized an AR head mount display allowing instructors to remotely demonstrate technical skills without the incumbrance of physically obstructing trainees [56]. A prospective observational study evaluating ureteroscopy demonstrated the use of AR technology in the form of the HoloLens-enhanced operative times and overall satisfaction scores for trainees. ...
Full-text available
Purpose of Review Surgical simulation has become a cornerstone for the training of surgical residents, especially for urology residents. Urology as a specialty bolsters a diverse range of procedures requiring a variety of technical skills ranging from open and robotic surgery to endoscopic procedures. While hands-on supervised training on patients still remains the foundation of residency training and education, it may not be sufficient to achieve proficiency for graduation even if case minimums are achieved. It has been well-established that simulation-based education (SBE) can supplement residency training and achieve the required proficiency benchmarks. Recent Findings Low-fidelity modules, such as benchtop suture kits or laparoscopic boxes, can establish a strong basic skills foundation. Eventually, residents progress to high-fidelity models to refine application of technical skills and improve operative performance. Human cadavers and animal models remain the gold standard for procedural SBE. Recently, given the well-recognized financial and ethical costs associated with cadaveric and animal models, residency programs have shifted their investments toward virtual and more immersive simulations. Summary Urology as a field has pushed the boundaries of SBE and has reached a level where unexplored modalities, e.g., 3D printing, augmented reality, and polymer casting, are widely utilized for surgical training as well as preparation for challenging cases at both the residents, attending and team training level.
... The breadth of medical specialties demonstrates the versatility and potential uses for AR HMDs and highlights underlying procedural commonalities that motivate their use for open and interventional procedures. (9): [8,[94][95][96][97][98][99][100][101] Other (14): [58,68,[102][103][104][105][106][107][108][109][110][111][112][113] Cardiac Surgery/Interventional Cardiology (16) AR HMD (4): [7,[114][115][116] Other (12): [105,[117][118][119][120][121][122][123][124][125][126][127] Oral and Maxillofacial Surgery (13) AR HMD (4): [128][129][130][131] Other (9): [58,77,[132][133][134][135][136][137][138] General Surgery (12) AR HMD (4): [139][140][141][142] Other (8): [143][144][145][146][147][148][149][150] Endovascular Surgery (7) AR HMD (3): [151][152][153] Other (4): [154][155][156][157] Otolaryngology (7) AR HMD (3): [158][159][160] Other (4): [161][162][163][164] Dermatology/Plastic Surgery (3) AR HMD: -Other (3): [165][166][167] Emergency Medicine/Trauma (3) AR HMD (2): [168,169] Other (1): [170] Anesthesiology (2) AR HMD: -Other (2): [171,172] Obstetrics (1) AR HMD: -Other (1): [173] XR+ technologies were used in both pre-and intraoperative settings. Fifty-eight percent of all analyzed articles (85) and 79% of AR HMD articles (42) were used to perform a real or simulated surgical procedure. ...
Full-text available
Augmented reality (AR) head-mounted displays (HMDs) are an increasingly popular technology. For surgical applications, the use of AR HMDs to display medical images or models may reduce invasiveness and improve task performance by enhancing understanding of the underlying anatomy. This technology may be particularly beneficial in open surgeries and interventional procedures for which the use of endoscopes, microscopes, or other visualization tools is insufficient or infeasible. While the capabilities of AR HMDs are promising, their usability for surgery is not well-defined. This review identifies current trends in the literature, including device types, surgical specialties, and reporting of user demographics, and provides a description of usability assessments of AR HMDs for open surgeries and interventional procedures. Assessments applied to other extended reality technologies are included to identify additional usability assessments for consideration when assessing AR HMDs. The PubMed, Web of Science, and EMBASE databases were searched through September 2022 for relevant articles that described user studies. User assessments most often addressed task performance. However, objective measurements of cognitive, visual, and physical loads, known to affect task performance and the occurrence of adverse events, were limited. There was also incomplete reporting of user demographics. This review reveals knowledge and methodology gaps for usability of AR HMDs and demonstrates the potential impact of future usability research.
... A simplified, non-stereoscopic system was subsequently used to support remote pediatric neurosurgical procedures with success [6]. Similarly, Rojas-Muñoz et al. found that medical students were able to make incisions with greater accuracy using a telepresence AR system [29]. ...
The central goal of surgical training is progressive development of a trainee’s surgical skill and judgment, culminating in the graduation of a safe, independent surgeon. Progressive autonomy, leading to independence, is the foundation of surgical training. In this chapter, we describe several practical approaches to teaching technical skills and judgment, with a focus on modalities that can be effectively employed outside the operating room. Optimization of opportunities to teach technical skills and judgment outside of the operating room will be high yield for any training program. Integrating teaching of technical skills and judgment into existing structures by development of new teaching forums that may be in-person, virtual, or via independent study by trainees is the ultimate goal to achieve success. Following early needs assessment, this chapter outlines local training program objectives, funding, teaching capacity, conference structure, and educational facilities that must be developed in collaboration with partners from the host institution. This will ensure that any new or updated educational activities will be appropriately received. This includes the concept of “training the trainer,” to sustain the training long term using new teaching methods, many digital, that the trainees and trainers buy into and want to carry forward for future generations.Keywords“Zwisch Scale”AutonomyNeeds assessmentCollaborationDevelopment“Train the trainer”Feedback“Show and tell”
Full-text available
Rapid developments in technology as part of the Fourth Industrial Revolution have created a demand for educational technology (EdTech) and a gradual transition from traditional teaching and learning to EdTech-assisted learning in medical education. EdTech is a portmanteau (blended word) combining the concepts of education and technology, and it refers to various attempts to solve education-related problems through information and communication technology. The aim of this study was to explore the use of key EdTech applications in medical education programs. A scoping review was conducted by searching three databases (PubMed, CINAHL, and Educational Sources) for articles published from 2000 to June 2021. Twenty-one studies were found that presented relevant descriptions of the effectiveness of EdTech in medical education programs. Studies on the application and effectiveness of EdTech were categorized as follows: (1) artificial intelligence with learner-adaptive evaluation and feedback, (2) augmented/virtual reality for improving learning participation and academic achievement through immersive learning, and (3) social media/social networking services with learner-directed knowledge generation, sharing, and dissemination in medical communities. Although this review reports the effectiveness of EdTech in various medical education programs, the number of studies and the validity of the identified research designs are insufficient to confirm the educational effects of EdTech. Future studies should utilize suitable research designs and examine the instructional objectives achievable by EdTech-based applications to strengthen the evidence base supporting the application of EdTech by medical educators and institutions.
Augmented reality is a technology that opens new possibilities in surgery. We present our experience in a hepatobiliary-pancreatic surgery unit in terms of preoperative planning, intraoperative support and teaching. For surgical planning, we have used 3D CT and MRI reconstructions to evaluate complex cases, which has made the interpretation of the anatomy more precise and the planning of the technique simpler. At an intraoperative level, it provides for remote holographic connection between specialists, the substitution of physical elements for virtual elements, and the use of virtual consultation models and surgical guides. In teaching, new lessons include sharing live video of surgery with the support of virtual elements for a better student understanding. As the experience has been satisfactory, augmented reality could be applied in the future to improve the results of hepatobiliary-pancreatic surgery.
Background Augmented reality (AR) navigation has been developed in recent years and can overcome some limitations of existing technologies. This study aimed to investigate a novel method of fibula free flap (FFF) osteotomy based on AR technology through a cadaver study. Methods One mandible, seven fibulas, and seven lower limb specimens underwent computed tomography (CT) examination. We used the professional software Proplan CMF 3.0 to design a defective mandible model and created fourteen virtual reconstruction plans using the fibulas and lower limb specimens. The AR-based intraoperative guidance software prototype was developed using the Unity Real-Time Development Platform, and virtual plans were transferred into this software prototype. We used AR-based surgical navigation to guide the FFF osteotomy and used these fibular segments to reconstruct the defective mandible model. After reconstruction, all segments were scanned by CT. Osteotomy accuracy was evaluated by measuring the length and angular deviation between the virtual plan and the final result. The reconstruction precision was reflected by the volume overlap rate and average surface distance between the planned and obtained reconstruction. Results The length difference, angular deviation, volume overlap rate and average surface distance of the in vitro group were 1.03±0.68 mm, 5.04±2.61°, 95.35±1.81%, and 1.02±0.27 mm, respectively. Those of the in vivo group were 1.18±0.84 mm, 5.45±1.47°, 95.31±2.09%, and 1.22±0.12 mm. Conclusions Due to the ideal result of cadaver experiments, an AR-based FFF osteotomy guided system may become a novel approach to assist FFF osteotomy for the reconstruction of defective mandibles.
Full-text available
Background: The goal of this study was to design and implement a novel surgical telementoring system called STAR (System for Telementoring with Augmented Reality) that uses a virtual transparent display to convey precise locations in the operating field to a trainee surgeon. This system was compared to a conventional system based on a telestrator for surgical instruction. Methods: A telementoring system was developed and evaluated in a study which used a 1 x 2 between-subjects design with telementoring system, i.e. STAR or Conventional, as the independent variable. The participants in the study were 20 pre-medical or medical students who had no prior experience with telementoring. Each participant completed a task of port placement and a task of abdominal incision under telementoring using either the STAR or the Conventional system. The metrics used to test performance when using the system were placement error, number of focus shifts, and time to task completion. Results: When compared to the Conventional system, participants using STAR completed the two tasks with less placement error (45% and 68%) and with fewer focus shifts (86% and 44%), but more slowly (19% for each task). Conclusions: Using STAR resulted in decreased annotation placement error, fewer focus shifts, but greater times to task completion. STAR placed virtual annotations directly onto the trainee surgeon’s field of view of the operating field by conveying location with great accuracy; this technology helped to avoid shifts in focus, decreased depth perception, and enabled fine-tuning execution of the task to match telementored instruction, but led to greater times to task completion.
Full-text available
Augmented reality is widely used in aeronautics and is a developing concept within surgery. In this pilot study, we developed an application for use on Google Glass ® optical head-mounted display to train urology residents in how to place an inflatable penile prosthesis. We use the phrase Augmented Reality Assisted Surgery to describe this novel application of augmented reality in the setting of surgery. The application demonstrates the steps of the surgical procedure of inflatable penile prosthesis placement. It also contains software that allows for detection of interest points using a camera feed from the optical head-mounted display to enable faculty to interact with residents during placement of the penile prosthesis. Urology trainees and faculty who volunteered to take part in the study were given time to experience the technology in the operative or perioperative setting and asked to complete a feedback survey. From 30 total participants using a 10-point scale, educational usefulness was rated 8.6, ease of navigation was rated 7.6, likelihood to use was rated 7.4, and distraction in operating room was rated 4.9. When stratified between trainees and faculty, trainees found the technology more educationally useful, and less distracting. Overall, 81% of the participants want this technology in their residency program, and 93% see this technology in the operating room in the future. Further development of this technology is warranted before full release, and further studies are necessary to better characterize the effectiveness of Augmented Reality Assisted Surgery in urologic surgical training.
Purpose Tremendous interest and need lie in the intersection of telemedicine and minimally invasive surgery. Robotics provides an ideal environment for surgical telementoring and telesurgery, given its endoscopic optics and mechanized instrument movement. We review the current status, challenges and future promise of telemedicine in endoscopic and minimally invasive surgery, with particular focus on urologic applications. Materials and Methods Two paired investigators screened Pubmed®, Scopus® and Web of Science® databases for all full-text English language articles published between 1995 and 2016, using the keywords: telemedicine, minimally invasive surgical procedure, robotic surgical procedure, education, distance. We categorized and included studies on level of interaction between proctors and trainees. Research design, special equipment, telecommunicating network bandwidth, and research outcomes of each study were demonstrated and analyzed. Results Of 65 identified articles, 38 peer-reviewed manuscripts qualified for inclusion. Studies were categorized into four advancing levels: verbal guidance, guidance with telestration, guidance with tele-assist, and telesurgery. More advanced levels of surgical telementoring provides more effective and experiential teaching, but are associated with higher telecommunicating network bandwidth requirement and expense. Concerns from patient safety, legal, financial, economic, and ethical standpoints remain to be reconciled. Conclusion Telementoring and telesurgery in minimally invasive surgery are becoming more practical and cost-effective in facilitating the teaching of advanced surgical skills worldwide and the delivery of surgical care to underserved areas. Yet, many challenges remain. Maturity of these modalities depends on financial incentives, favorable legislation and collaboration with cybersecurity experts to ensure safety and cost-effectiveness.
Background: Mentorship is important but may not be feasible for distance learning. To bridge this gap, telementoring has emerged. The purpose of this systematic review was to evaluate the effectiveness of telementoring compared with on-site mentoring. Methods: A search was done up to March 2015. Studies were included if they used telementoring between surgeons during a clinical encounter and if they compared on-site mentoring and telementoring. Results: A total of 11 studies were included. All reported no difference in complication rates, and 9 (82%) reported similar operative times; 4 (36%) reported technical issues, which was 3% of the total number of cases in the 11 studies. No study reported on higher levels of evidence for effectiveness of telementoring as an educational intervention. Conclusion: Studies reported that telementoring is associated with similar complication rates and operative times compared with on-site mentoring. However, the level of evidence to support the effectiveness of telementoring as a training tool is limited. There is a need for studies that provide evidence for the equivalence of the effectiveness of telementoring as an educational intervention in comparison with on-site mentoring.
Hemorrhage is the most preventable cause of post-traumatic death. Many cases are potentially anatomically salvageable, yet remain lethal without logistics or trained personnel to deliver diagnosis or Resuscitative-surgery in austere-environments. Revolutions in technology for remote-mentoring of ultrasound and surgery may enhance capabilities to utilize the skill-sets of non-physicians. Thus, our Research-Collaborative explored remote-mentoring to empower non-physicians to address junctional and torso hemorrhage-control in Austere-environments. Major studies involved using remote-telementored ultrasound (RTMUS) to identify torso and junctional exsanguination, remotely mentoring resuscitative-surgery for torso hemorrhage-control, understanding and mitigating physiological stress during such tasks; and the technical practicalities of conducting Damage Control Surgery (DCS) in Austere Environments. Iterative projects involved: randomized guiding of Firefighters to identify torso (RCT) and junctional (pilot) hemorrhage using RTMUS; Randomized remote-mentoring of MedTechs conducting Resuscitative-surgery for torso exsanguination in an anatomically-realistic surgical trainer ("Cut-Suit") including physiological monitoring; and trained surgeons conducting a comparative randomized study for torso hemorrhage-control in normal (1g) versus weightlessness (0g).This work demonstrated that Firefighters could be remotely mentored to perform just-in-time torso RTMUS on a simulator. Both Firefighters and mentors were confident in their abilities, the ultrasounds being 97% accurate. An ultrasound-naïve Firefighter in Memphis could also be remotely-mentored from Hawaii to identify and subsequently tamponade an arterial junctional hemorrhage using RTMUS in a live tissue model. Thereafter, (both mentored and unmentored MedTechs) and train-surgeons completed Resuscitative-surgery for Hemorrhage-control on the Cut-Suit, demonstrating practicality for all involved. While Remote-mentoring did not decrease blood loss among MedTechs it increased procedural confidence, and decreased physiologic stress. Therefore, remote-mentoring may increase the feasibility of non-physicians conducting a psychologically daunting task. Finally, DCS in weightlessness was feasible without fundamental differences from 1g. Overall, the collective evidence suggests that Remote-mentoring supports diagnosis, non-invasive therapy, and ultimately Resuscitative-surgery to potentially rescue those exsanguinating in Austere-environments and should be more rigorously studied.
Objectives/hypothesis: To assess the efficacy of a surgical telementoring program for endoscopic skull base surgery. Study design: Prospective case series with surveys of surgeons. Methods: A surgical telementoring program was established for mentoring of a skull base team at the University of Maribor in Slovenia by an experienced skull base team at the University of Pittsburgh Medical Center in Pennsylvania. Two-way video and audio streaming provided real-time communication with the surgical team. Over a period of 3 years, 10 endoscopic endonasal surgeries of the skull base were mentored preoperatively and during the key part of the procedure. Following each procedure, an evaluation form was used to document the mentoring interventions and rate the experience. Results: Procedures included endoscopic endonasal approaches to the sella, anterior cranial fossa, posterior cranial fossa, and orbit. Diagnoses included benign and malignant neoplasms, cerebrospinal fluid leak, and inflammatory disease. In nine of 10 cases, adequate audio and video communications were maintained. The most frequent mentoring interventions were for identification of anatomy, extent of exposure, extent of resection, and surgical technique. The median perceived value by the junior surgical team was 9.5 (range 8-10). A model for surgical telementoring is proposed. Conclusion: Surgical telementoring provides the ability to help surgeons develop their surgical skills to a greater level of proficiency for complex surgeries when experienced mentors are not available locally. The technology is reliable and available at most institutions. Perceived benefits of surgical telementoring include improved surgical exposure, increased extent of tumor resection, and decreased duration of surgery. Level of evidence: N/A. Laryngoscope, 2016.
Background: Augmented reality (AR) fuses computer-generated images of preoperative imaging data with real-time views of the surgical field. Scopis Hybrid Navigation (Scopis GmbH, Berlin, Germany) is a surgical navigation system with AR capabilities for endoscopic sinus surgery (ESS). Methods: Predissection planning was performed with Scopis Hybrid Navigation software followed by ESS dissection on 2 human specimens using conventional ESS instruments. Results: Predissection planning included creating models of relevant frontal recess structures and the frontal sinus outflow pathway on orthogonal computed tomography (CT) images. Positions of the optic nerve and internal carotid artery were marked on the CT images. Models and annotations were displayed as an overlay on the endoscopic images during the dissection, which was performed with electromagnetic surgical navigation. The accuracy of the AR images relative to underlying anatomy was better than 1.5 mm. The software's trajectory targeting tool was used to guide instrument placement along the frontal sinus outflow pathway. AR imaging of the optic nerve and internal carotid artery served to mark the positions of these structures during the dissection. Conclusion: Surgical navigation with AR was easily deployed in this cadaveric model of ESS. This technology builds upon the positive impact of surgical navigation during ESS, particularly during frontal recess surgery. Instrument tracking with this technology facilitates identifying and cannulation of the frontal sinus outflow pathway without dissection of the frontal recess anatomy. AR can also highlight "anti-targets" (ie, structures to be avoided), such as the optic nerve and internal carotid artery, and thus reduce surgical complications and morbidity.
Providing medical care in the field is very challenging because of the limited availability of medical resources. The current practice in military operations is to stabilize patients far forward and evacuate them to better equipped medical facilities, such as combat field hospitals. This strategy has proven very successful in recent conflicts. However, it is possible to save more lives and ameliorate the consequences of long term injuries by providing an accurate diagnosis earlier, specialized surgical care faster and more sophisticated intensive care during transport. This chapter focuses on technologies that allow augmenting the diagnostics and treatment capabilities of medical teams in the field.