Conference PaperPDF Available

ARMAGNI: Augmented Reality Enhanced Surgical Magnifying Glasses

Authors:
The 18th Scandinavian Conference on Health informatics, Tromsø, Norway, August 22-24, 2022. Organized by UiT The
Arctic University of Norway. Conference Proceedings published by Linköping University Electronic Press at
https://doi.org/10.3384/ecp187. © The Author(s). This work is licensed under the Creative Commons Attribution-
NonCommercial 4.0 International License. To view a copy of this license, visit http://creativecommons.org/licenses/by-
nc/4.0/
ARMAGNI: Augmented Reality Enhanced Surgical Magnifying Glasses
Situational Awareness during surgery with AR in the Loupe
Artur Liebrecht1, Roman Bibo1, Bastian Dewitz1, Sebastian Kalkhoff1, Sobhan Moazemi1, Markus Rollinger2,
Jean-Michel Asfour2, Klaus-Jürgen Janik3, Artur Lichtenberg1, Hug Aubin1*, Falko Schmid1*+
1Digital Health Lab Düsseldorf, Department of Cardiac Surgery, Medical Faculty and University Hospital Düsseldorf,
Heinrich-Heine-University Düsseldorf, Düsseldorf, Germany
2DIOPTIC GmbH, Weinheim, Germany
3JADENT GmbH, Aalen, Germany
*These authors contributed equally to this work as senior authors
+Corresponding author
Abstract
During cardiac surgery, in addition to their manual work, surgeons need to perceive a large amount of procedural
intraoperative data, including information which is not directly visible to them. Therefore, we developed an
augmented reality demonstrator that displays alphanumerical data into the loupe of surgical magnifying glasses.
Eight cardiac surgeons tested the demonstrator in a skill task that simulates the critical part of a typical surgical
procedure while being confronted to vital intraoperative parameters getting critical. The results showed a
decrease of missed critical phases and improved response times when using the demonstrator instead of a
customary monitor for tracking intraoperative parameters.
Keywords
Augmented Reality, Cardiac Surgery, Intraoperative Assistance
1 INTRODUCTION
Situational awareness, “the perception and understanding
of the surrounding environment” [13], is important in
humans’ performance fulfilling a complex task [6]. This
also applies to cardiac surgeries where safety and outcome
are dependent on information flow, concerning not only
preoperative data but also intraoperative procedural data
such as vital parameters. Although operation rooms are
equipped with multiple monitors displaying standard
parameters like vital signs, other information such as
heartlung machine procedural data and respiratory
parameters are not readily visible for the surgeon. When
needed, it requires intense communication efforts which
can lead to distraction and interruption of the procedure [4].
Augmented reality (AR) can be a solution to bridge the gap
between information needs, communication, and display
possibilities.
Head-mounted displays (HMD), including smart glasses
such as Google Glass, have been playing an increasing role
in the health care industry [12]. In a systematic review from
2019, Rahman et al. [12] determined 120 HMD
applications in surgery, of which most were used for image
guidance and AR. For instance, Liebert et al. [7] compared
traditional vital signs monitor to an HMD for monitoring
patient’s vitals, showing potential for increased situational
awareness and improved patient safety. The overall
feedback from users was positive. In a qualitative
descriptive study, Enlöf et al. [5] also showed that health
care professionals have a generally positive view of using
smart glasses in the medical field. In video assisted surgery
AR glasses have potential to reduce bad body posture
during procedure [8]. Arpaia et al. [1] presented a system
for retrieving and displaying patient’s vitals and evaluated
its effectiveness, transmission error rate, refresh rate and
latency with confirming technical feasibility of such a
system. The target group consisted of assistant surgeons
and anesthetists.
The related work mentioned in the previous lines indicate
benefits from using AR devices in surgery and high user
acceptance, but these applications did not consider cases
where users require surgery magnifying glasses. For
cardiac surgeons, wearing magnifying glasses and HMDs
simultaneously can be cumbersome due to interference. As
one solution, smart glasses can be mounted to surgical
loupes. Yoon et al. [14] successfully used such a setup for
image guidance during shunt placement. The AR images
were projected above the loupe and required quick eye
movement to see. Qian et al. [10], on the other hand,
combined a Magic Leap One with binocular magnifying
loupes. They evaluated a calibration method to align virtual
content to the real world in the magnified or minified field-
of-view of the user. But since we are projecting simple
alphanumerical procedural data that does not have to be
aligned to real objects, there is no need for such a
calibration method. Hence, we can use a simpler setup with
lightweight hardware.
The 18th Scandinavian Conference on Health informatics, Tromsø, Norway, August 22-24, 2022. 46
Figure 1. AR demonstrator consisting of an AR module
and Galilean loupes mounted on a custom-made glasses
frame mimicking surgical magnifying glasses.
Figure 2. Image of the field of view of the AR loupe.
Our approach for displaying procedural data in the field of
view of surgeons is different to the ones mentioned above.
Instead of extending existing HMDs which have limitations
due to weight, size, or short battery life [12], we develop
typical surgical magnifying glasses with an integrated AR
display into the loupe. For the proof of concept, we built an
AR demonstrator that can be adjusted to the user. The
prototype is shown in Figure 1 and the view of the loupe
with AR visualizations in Figure 2.
In a feasibility study, experienced cardiac surgeons tested
the usefulness of AR in the loupe with our demonstrator.
We propose that AR magnifying glasses can increase
situation awareness of surgeons. Beyond that, they will
have potential to reduce surgeons’ workload during cardiac
surgeries and encounter high user acceptance, as they
mimic the typical surgical magnifying glasses used during
cardiac surgery.
2 METHODS
For this study, we asked cardiac surgeons to test our
prototype referred as AR demonstrator, which we will
describe in detail below along with the corresponding
software, we will further outline the experimental setup.
2.1 AR Demonstrator
The prototype consists of two customary Galilean loupes
with 2.5x magnification and 300 mm working distance.
They were attached to a custom-built metallic mechanism,
which allows different users to adjust pupil distance, inter-
eye asymmetry or height of the loupes. The metallic
mechanism is mounted on a plastic glasses frame made
with a laser cutter.
The 2.5 cm x 2 cm x 1.5 cm sized AR module (Figure 3)
was built with a combination of a micro-OLED display, a
lens, and a beam splitter. For displaying AR visualizations,
we used a Sony ECX336AF-6 OLED micro display. The
0.23-inch diagonal sized RGB display has a 640 x 400
pixels resolution, 800 cd/m² maximal luminance and
10,000:1 contrast ratio. We installed the micro display on
the side of a 3D printed component. There, the displayed
images are projected through a lens magnifying them to
fully cover the view through the loupes. A beam splitter
centered between the display, the loupe, and the eye,
reflects the images into the eye of the user. The AR module
can be attached to one of the Galilean loupes.
Figure 3. AR module attachable to loupes.
The micro display is connected to a Raspberry Pi Zero
installed in the glasses frame. To be able to power the micro
display with the required 1.8 volts and send images to it,
the Raspberry Pi Zero was extended by a custom circuit
board. A 3.6 volts rechargeable battery is powering the
Raspberry Pi Zero through a 200 cm long cable. This
battery is bundled in a 3D printed case with integrated
electronics and a button, which allows us to turn the
Raspberry Pi on and off.
2.2 Base station
To keep the AR magnifying glasses lightweight and
conformable, we follow a client-server approach. One
endpoint is the mentioned Raspberry Pi Zero built in the
glasses frame. As the other endpoint, we are using a base
station on Raspberry Pi 4B basis and the official Raspberry
Pi 7-inch touchscreen (see Figure 4). The base station
serves as a mobile control center for fetching patient data
from simulated medical devices as well as generating
visualizations and streaming them to the AR display.
Furthermore, the base station is running a graphical user
interface frontend application for configuring the AR
visualizations using the touchscreen.
47 The 18th Scandinavian Conference on Health informatics, Tromsø, Norway, August 22-24, 2022.
AR visualizations are generated in the frontend application
and sent as PNG images to a REST server on the Raspberry
Pi Zero. Using a Raspberry Pi 4B with the 64-Bit version
of the Raspberry Pi OS as our base station, we can achieve
approximately 5 frames per second with a delay of 500
milliseconds.
Figure 4. Base station running the software.
Our frontend application developed with Vue.js enables
users to customize the AR visualizations. They can pick
different parameters from connected medical device
simulators and choose the desired position in the field of
view. We decided the visualized medical parameters to be
automatically arranged in a ring formation, so they do not
overlay the working surface of the user. Though, the user is
free to choose the size of the ring and by that the size of
parameters as well as the number of displayed parameters.
Another feature of the application allows users to finetune
the whole AR visualization by moving, resizing, or rotating
it with touch gestures while wearing the glasses and seeing
the changes near real-time. Each user can have multiple
settings profiles, which are saved in a PostgreSQL
database.
Communication with the database and medical devices is
performed by an Apollo GraphQL server, which exposes a
GraphQL API for the frontend application. Backend was
built in accordance with IEEE 11073 Service-oriented
Device Connectivity (SDC) for being capable of
communicating with real medical devices that also support
SDC. Using SDCLib, an open-source SDC implementation
by SurgiTAIX, we developed a SDC consumer micro
service, which serves as a bridge between medical devices
and our GraphQL server.
For the experiment, we developed a medical device
simulator, that generates random but reasonable values for
heart rate (labeled as HF in Figure 2), mean arterial pressure
(MAD), oxygen saturation (SPO2) and central venous
pressure (ZVD). The simulated values can continuously
increase and decrease every 3 seconds, and every 60 90
seconds one of the parameters becomes critical for 15
seconds. Parameters in a critical state are displayed with a
small warning sign and blink with a frequency
approximately 600 ms. By means of this simulation, we
evaluated the effect of the AR module on the user during a
skill exercise, the setup of which we will describe next.
2.3 Experimental setup
The experiment took place in the clinic for cardiac surgery
at the University Hospital Düsseldorf. We invited 8 cardiac
surgeons (2 female, 6 males, between 25 and 54 years old)
to perform a skill exercise while wearing our AR
magnifying glasses and reacting to critical parameters.
Directly afterwards, we asked them to take a survey, which
included the System Usability Scale (SUS), the NASA
Task Load Index (NASA-TLX) and further questions
regarding the usefulness and usability of the AR.
In the beginning of the experiment, the investigator taught
participants how they can adjust the AR magnifying glasses
via the metallic mechanism and finetuned the AR display
using the frontend application. During the experiment,
frontend and backend applications were running on a laptop
computer and used by investigator rather than the
participant.
We followed a within-subject design with randomized
order of conditions. In both cases, participants performed
an anastomosis on the Arroyo’s Anastomosis Simulator
[11] while wearing our prototype. In the test condition, the
AR module was turned on and the participants had to keep
track of the displayed parameters in the loupe. In the control
condition on the other hand, parameters were displayed on
a customary 27-inch display only. We positioned the
display on the right side of and 2 meters away from the
participants, so they had to turn their head to the right by
approximately 20-degree angle to see the values.
During performing the skill exercise, participants had to
react to critical values as soon as possible by telling the
investigator which parameter is critical and whether it is too
high or too low. We recorded their response time using a
timer implemented in the simulation software. Participants
had a time limit of 15 seconds to notice and react to a
critical phase and the investigator registered the answer as
“right” or “wrong” by pushing the corresponding button on
the keyboard. If the pre-set 15-second timeout passed
before the participant reacted, the software automatically
registered the answer as “missed” with a response time of
15 seconds.
After each trial, our simulation software generated a log file
containing participant’s response times to critical phases
and whether the response was right, wrong, or missed.
3 RESULTS
We performed t-tests to compare our metrics between test
condition (using AR) and control condition (using
monitor). All metrics were tested for homogeneity of
variance and normality before comparing them with a two-
tailed paired t-test for dependent means. If requirements for
homogeneity or normality were not met, Wilcoxon Singed-
Rank test was used.
3.1 Response performance
We used a Wilcoxon Signed-Rank test to compare response
performances between test and control conditions.
Response performance is the count of correct responses
divided by count of critical phases for each participant. The
calculated data for each participant can be seen in Figure 5.
It should be noted that no participant gave wrong responses.
The 18th Scandinavian Conference on Health informatics, Tromsø, Norway, August 22-24, 2022. 48
Figure 5. Rate for correct responses of each participant.
Participants showed an increase in response performance
when using the AR display (M=.8, SD=.2) compared to
customary display (M=.5, SD=.3), W=0, p < .05.
3.2 Response time
Data for response time is visualized in Figure 6. Since
missed critical phases mean that there was no reaction for
15 seconds, data is strictly speaking not normally
distributed. One participant even missed all critical phases
in control condition; hence median was 15 seconds in this
case.
Figure 6. Median response times in seconds.
The Levene's test for response times was not significant
(F(1, 7) = 1.313, p = .340) so variance homogeneity was
assumed. Despite data being limited between 0 and 15
seconds, Shapiro-Wilk test for the differences between the
pairs did not show a significant departure from the
normality (W(8) = .915, p = .433).
Participants responded faster to critical values in the AR
condition (M = 6.7, SD = 1.9) compared to monitor
condition (M = 10.1, SD = 3), (t(7) = 2.9, p = .025).
3.3 Total time for trials
Participants required different amount of time to complete
the anastomosis on the Arroyo’s simulator. We used a t-test
to compare the required time for test and control conditions.
The Levene's test for required times was not significant
(F(1, 7) = .079, p = .783) so variance homogeneity was
assumed. Shapiro-Wilk test for the differences between the
pairs did not show a significant departure from the
normality (W(8) = .984, p = .999).
The t-test did not show any significant difference in
required time for trials between AR (M = 12.4, SD = 5.2)
and monitor conditions (M = 11.4, SD = 5.5),
(t(7) = 1.4, p = .203).
3.4 NASA-TLX
The Levene’s test for global NASA-TLX, namely the sum
of individual NASA-TLX scores, was not significant
(F(1, 7) = .064, p = .804) and Shapiro-Wilk test did not
show significant departure from normality,
(W(8) = .92, p = .471). The t-test for global NASA-TLX
showed an improved workload for AR condition (M=52.9,
SD=22.1) compared to monitor condition (M = 65.4,
SD = 24.1), (t(7) = 2.8, p = .027).
Boxplots and significance in differences determined by
ttests for NASA-TLX subscales can be seen in Figure 7.
Variance homogeneity and normality criteria using
Levene’s test and Shapiro-Wilk test, respectively, were
met. Outliers are classified as being outside 1.5 times the
interquartile range.
Figure 7. Mean scores for mental demand (MD), physical
demand (PD), temporal demand, (TD), performance (P),
effort (E) and frustration (F). ns means that a paired t-test
result was not significant.
3.5 SUS Score
Mean SUS score was 66.875 (SD 12.02). Reponses for
individual statements are visualized in Figure 8.
3.6 Further questions
Responses to further questions about AR in the
demonstrator and about AR magnifying glasses in general
are summarized in Figure 9. Participants were asked to rate
whether they agree or disagree with the statements on a
scale of 0 100.
49 The 18th Scandinavian Conference on Health informatics, Tromsø, Norway, August 22-24, 2022.
Figure 8. Responses to SUS statements.
Figure 9. Please tell us whether you agree or disagree with
the following statements.
Q1. I preferred using the AR instead of looking at the
monitor to detect critical parameters.
Q2. The medical parameters in the AR were clearly
visible.
Q3. I found that the AR visualization obscured the
view over the working surface.
Q4. I can imagine myself using surgical magnifying
glasses with AR in daily OR work.
Q5. I think magnifying glasses with AR can improve
the outcome of surgeries.
4 DISCUSSION
4.1 Objective evaluation
The experiment’s results showed an improvement in
participant’s reactions to critical phases when using the AR
display compared to a common diagnostic monitor. In the
test condition, not only reaction times were 51 % faster, but
also the performance for detecting critical parameters was
increased by 60 %. On the other hand, given our sample
size of 8 participants being small, statistical conclusions
should be treated with care. Nevertheless, our initial
proposition that augmented reality can improve surgeons’
situational awareness could be confirmed, at least, in our
experimental setup.
Circumstances during real operation are different. Firstly,
OP light is much brighter than in our setup, which
negatively affects the visibility of the AR visualizations.
Technical suitability should be evaluated in further studies.
Secondly, the question arises whether an AR display in
magnifying glasses is necessary in the presence of auditory
alerts and assistant doctors or anesthetists. Our
experimental setup demonstrates that an AR display can
help surgeons keep track of selected parameters omitting
some communication efforts that can possibly interrupt the
operation procedure.
4.2 Subjective assessments
There was a significant difference in global NASA-TLX
between test and control condition, indicating that AR
magnifying glasses can also help to improve workload in
surgeons. Especially, frustration was reduced by 45 % and
physical demand by 36 % (borderline not significant
though) when using the AR display. Higher physical
demand in monitor condition could result from the fact that
participants had to turn their head to see the parameters.
Also, in the control condition, participants missed more
critical phases (response performance dropped by 38 %),
which explains higher frustration. It might be interesting to
further investigate these subscales under real
circumstances, where cardiac surgeons rarely look to
monitors but communicate with attendants about them.
4.3 Usability and opinions
Participant’s feedback to the AR demonstrator presented
were mixed and an average SUS score of 66.875 can be
considered as “marginally acceptable” according to
determined ranges by Bangor et al [2].
On the one hand, participants liked the AR functionality
and showed interest in using such a device in the future.
On the other hand, everyone had difficulties with the
prototype itself, mostly independent of the AR module.
Adjustment of the magnifying glasses could take over 10
minutes but even then, the alignment of the loupes was not
ideal. Sometimes participants had to re-adjust the loupes
during the skill task. Another common problem was that,
due to their weight, the glasses slightly slid down during
skill task so that the top two parameters disappeared. These
problems primarily occurred because of the mechanics for
adapting the glasses to different users. Normally, surgical
loupes are custom-made for each user, to fit pupil distance,
height, and preferred working distances. However, we
wanted one single device that could be tried out by multiple
people, which would not be possible with a tailormade
device.
In the future, we are building another prototype with a
different micro display that uses Bluetooth Low Energy for
data transmission and a lighter battery. This omits a
Raspberry Pi Zero in the glass frame reducing the weight
The 18th Scandinavian Conference on Health informatics, Tromsø, Norway, August 22-24, 2022. 50
of the device, which should resolve most issues of the
current AR demonstrator as discussed above.
5 CONCLUSION
This first feasibility study showed that ARMAGNI has the
potential to increase surgeons’ situation awareness during
operations requiring magnification glasses while
decreasing the workload for surgeons. Participants liked the
concept of the product but had difficulties with the current
prototype. In an upcoming improved prototype,
participants feedback will be taken into account to enhance
usability. In addition, a different micro display is intended
to improve overall ergonomics of the device.
6 REFERENCES
[1] Arpaia, P., Benedetto, E. D., Dodaro, C. A., Duraccio,
L. & Servillo, G. Metrology-Based Design of a
Wearable Augmented Reality System for Monitoring
Patient’s Vitals in Real Time. IEEE Sensors Journal
21, 1117611183 (2021).
[2] Bangor, A., Kortum, P. T. & Miller, J. T. An
Empirical Evaluation of the System Usability Scale.
International Journal of Human-Computer
Interaction 24, 574594 (2008).
[3] Cercenelli, L. et al. The Wearable VOSTARS System
for Augmented Reality-Guided Surgery: Preclinical
Phantom Evaluation for High-Precision Maxillofacial
Tasks. Journal of Clinical Medicine 9, 3562 (2020).
[4] ElBardissi, A. W., Wiegmann, D. A., Henrickson, S.,
Wadhera, R. & Sundt, T. M. Identifying methods to
improve heart surgery: an operative approach and
strategy for implementation on an organizational
level. European Journal of Cardio-Thoracic Surgery
34, 10271033 (2008).
[5] Enlöf, P., Romare, C., Jildenstål, P., Ringdal, M. &
Skär, L. Smart Glasses for Anesthesia Care: Initial
Focus Group Interviews with Specialized Health Care
Professionals. Journal of PeriAnesthesia Nursing 36,
4753 (2021).
[6] Hazlehurst, B., McMullen, C. K. & Gorman, P. N.
Distributed cognition in the heart room: How situation
awareness arises from coordinated communications
during cardiac surgery. Journal of Biomedical
Informatics 40, 539551 (2007).
[7] Liebert, C. A., Zayed, M. A., Aalami, O., Tran, J. &
Lau, J. N. Novel Use of Google Glass for Procedural
Wireless Vital Sign Monitoring. Surgical Innovation
23, 366373 (2016).
[8] Lim, A. K., Ryu, J., Yoon, H. M., Yang, H. C. & Kim,
S. Ergonomic effects of medical augmented reality
glasses in video-assisted surgery. Surgical Endoscopy
36, 988998 (2021).
[9] Maluf, M. A. et al. Cardiovascular Surgery Residency
Program: Training Coronary Anastomosis Using the
Arroyo Simulator and UNIFESP Models. Revista
Brasileira de Cirurgia Cardiovascular (2015).
[10] Qian, L., Song, T., Unberath, M. & Kazanzides, P.
AR-Loupe: Magnified Augmented Reality by
Combining an Optical See-Through Head-Mounted
Display and a Loupe. IEEE Transactions on
Visualization and Computer Graphics 11 (2020).
[11] Raet, J. M. D. et al. How to build your own coronary
anastomosis simulator from scratch†. Interactive
CardioVascular and Thoracic Surgery 16, 772776
(2013).
[12] Rahman, R. et al. Head-Mounted Display Use in
Surgery: A Systematic Review. Surgical Innovation
27, 88100 (2019).
[13] Woodward, J. & Ruiz, J. Analytic Review of Using
Augmented Reality for Situational Awareness. IEEE
Transactions on Visualization and Computer
Graphics 11 (2022).
[14] Yoon, J. W. et al. Technical feasibility and safety of
image-guided parieto-occipital ventricular catheter
placement with the assistance of a wearable head-up
display. The International Journal of Medical
Robotics and Computer Assisted Surgery 13, e1836
(2017).
7 ACKNOWLEDGEMENT
This Project is supported by the Federal Ministry for
Economic Affairs and Climate Action (BMWK) under
grant 16KN069341 on the basis of a decision by the
German Bundestag. Authors state no conflict of interest
51 The 18th Scandinavian Conference on Health informatics, Tromsø, Norway, August 22-24, 2022.
ResearchGate has not been able to resolve any citations for this publication.
Article
Full-text available
Background The aim of this study was to objectively compare medical augmented reality glasses (ARG) and conventional monitors in video-assisted surgery and to systematically analyze its ergonomic benefits. Methods Three surgeons (thoracic, laparoscopic, and thyroid surgeons) participated in the study. Six thoracoscopic metastasectomies, six subtotal laparoscopic gastrectomies, and six thyroidectomies were performed with and without ARG. The subjective experience was evaluated using a questionnaire-based NASA-Task Load Index (NASA-TLX). Postures during surgeries were recorded. The risk of musculoskeletal disorders associated with video-assisted surgery was assessed using rapid entire body assessment (REBA). Surface electromyography (EMG) was recorded. Muscle fatigue was objectively measured. Results NASA-TLX scores of three surgeons were lower when ARG was used compared to those with conventional monitor (66.4 versus 82.7). Less workload during surgery was reported with ARG. The laparoscopic surgeon exhibited a substantial decrease in mental and physical demand [− 21.1 and 12.5%)] and the thyroid surgeon did (− 40.0 and − 66.7%).Total REBA scores decreased with ARG (8 to 3.6). The risk of musculoskeletal disorders was improved in regions of the neck and shoulders. Root mean square (RMS) of the EMG signal decreased from 0.347 ± 0.150 to 0.286 ± 0.130 (p = 0.010) with usage of ARG; a decrease was observed in all surgeons. The greatest RMS decrease was observed in trapezius and sternocleidomastoid muscles. The decrease in brachioradialis muscle was small. Conclusion ARG assisted with correction of bad posture in surgeons during video-assisted surgery and reduced muscular fatigue of the upper body. This study highlights the superior ergonomic efficiency of ARG in video-assisted surgery.
Article
Full-text available
In this work, an augmented-reality (AR) system for monitoring patient’s vitals in real time during surgical procedures is proposed and metrologically characterized in terms of transmission error rates and latency. These specifications, in fact, are crucial to ensure real-time response which is a critical requirement for Health applications. The proposed system automatically acquires the vitals from the operating room instrumentation, and shows them in real time directly on a set of wearable AR glasses.In this way, the surgical team has a real-time visualization of a comprehensive set of patient’s information, without constantly looking at the instrumentation. The system was designed to ensure modularity, flexibility, ease of use and, most importantly, a reliable communication. The system was designed, implemented and validated experimentally through on-field tests carried out at the Academic Hospital of Federico II University, using instrumentation typically available in the operating room: namely, a respiratory ventilator and a patient monitor. Results of the experimental validation highlight the need for the metrological approach in conceiving advanced monitoring surgical procedures in operating room.
Article
Full-text available
Background: In the context of guided surgery, augmented reality (AR) represents a groundbreaking improvement. The Video and Optical See-Through Augmented Reality Surgical System (VOSTARS) is a new AR wearable head-mounted display (HMD), recently developed as an advanced navigation tool for maxillofacial and plastic surgery and other non-endoscopic surgeries. In this study, we report results of phantom tests with VOSTARS aimed to evaluate its feasibility and accuracy in performing maxillofacial surgical tasks. Methods: An early prototype of VOSTARS was used. Le Fort 1 osteotomy was selected as the experimental task to be performed under VOSTARS guidance. A dedicated set-up was prepared, including the design of a maxillofacial phantom, an ad hoc tracker anchored to the occlusal splint, and cutting templates for accuracy assessment. Both qualitative and quantitative assessments were carried out. Results: VOSTARS, used in combination with the designed maxilla tracker, showed excellent tracking robustness under operating room lighting. Accuracy tests showed that 100% of Le Fort 1 trajectories were traced with an accuracy of ±1.0 mm, and on average, 88% of the trajectory's length was within ±0.5 mm accuracy. Conclusions: Our preliminary results suggest that the VOSTARS system can be a feasible and accurate solution for guiding maxillofacial surgical tasks, paving the way to its validation in clinical trials and for a wide spectrum of maxillofacial applications.
Article
Full-text available
Purpose Smart glasses are a kind of wearable technology that gives users sustained, hands-free access to data and can transmit and receive information wirelessly. Earlier studies have suggested that smart glasses have the potential to improve patient safety in anesthesia care. Research regarding health care professionals' views of the potential use of smart glasses in anesthesia care is limited. The purpose of this study was to describe anesthesia health care professionals' views of smart glasses before clinical use. Design A qualitative descriptive study. Methods Data were collected from focus group interviews and analyzed using thematic content analysis. Findings Three categories of participants' views of smart glasses were created during the analysis: views of integrating smart glasses in clinical setting; views of customized functionality of smart glasses; and views of being a user of smart glasses. One theme, striving for situational control, was identified in the analysis. Conclusions Smart glasses were seen as a tool that can impact and improve access to patient-related information, and aid health care professionals in their struggle to gain situational control during anesthesia care. These are factors related to increased patient safety.
Article
Full-text available
Objective: Engage the UNIFESP Cardiovascular Surgery residents in coronary anastomosis, assess their skills and certify results, using the Arroyo Anastomosis Simulator and UNIFESP surgical models. Methods: First to 6th year residents attended a weekly program of technical training in coronary anastomosis, using 4 simulation models: 1. Arroyo simulator; 2. Dummy with a plastic heart; 3. Dummy with a bovine heart; and 4. Dummy with a beating pig heart. The assessment test was comprised of 10 items, using a scale from 1 to 5 points in each of them, creating a global score of 50 points maximum. Results: The technical performance of the candidate showed improvement in all items, especially manual skill and technical progress, critical sense of the work performed, confidence in the procedure and reduction of the time needed to perform the anastomosis after 12 weeks practice. In response to the multiplicity of factors that currently influence the cardiovascular surgeon training, there have been combined efforts to reform the practices of surgical medical training. Conclusion: 1 - The four models of simulators offer a considerable contribution to the field of cardiovascular surgery, improving the skill and dexterity of the surgeon in training. 2 - Residents have shown interest in training and cooperate in the development of innovative procedures for surgical medical training in the art.
Article
Situational awareness is the perception and understanding of the surrounding environment. Maintaining situational awareness is vital for performance and error prevention in safety critical domains. Prior work has examined applying augmented reality (AR) to the context of improving situational awareness, but has mainly focused on the applicability of using AR rather than on information design. Hence, there is a need to investigate how to design the presentation of information, especially in AR headsets, to increase users situational awareness. We conducted a Systematic Literature Review to research how information is currently presented in AR, especially in systems that are being utilized for situational awareness. Comparing current presentations of information to existing design recommendations aided in identifying future areas of design. In addition, this survey further discusses opportunities and challenges in applying AR to increasing users situational awareness.
Article
Head-mounted loupes can increase the user’s visual acuity to observe the details of an object. On the other hand, optical see-through head-mounted displays (OST-HMD) are able to provide virtual augmentations registered with real objects. In this article, we propose AR-Loupe, combining the advantages of loupes and OST-HMDs, to offer augmented reality in the user’s magnified field-of-vision. Specifically, AR-Loupe integrates a commercial OST-HMD, Magic Leap One, and binocular Galilean magnifying loupes, with customized 3D-printed attachments. We model the combination of user’s eye, screen of OST-HMD, and the optical loupe as a pinhole camera. The calibration of AR-Loupe involves interactive view segmentation and an adapted version of stereo single point active alignment method (Stereo-SPAAM). We conducted a two-phase multi-user study to evaluate AR-Loupe. The users were able to achieve sub-millimeter accuracy ( $0.82\;\mathrm{mm}$ ) on average, which is significantly ( $p < 0.001$ ) smaller compared to normal AR guidance ( $1.49\;\mathrm{mm}$ ). The mean calibration time was $268.46\;s$ . With the increased size of real objects through optical magnification and the registered augmentation, AR-Loupe can aid users in high-precision tasks with better visual acuity and higher accuracy.
Article
Purpose. We analyzed the literature to determine (1) the surgically relevant applications for which head-mounted display (HMD) use is reported; (2) the types of HMD most commonly reported; and (3) the surgical specialties in which HMD use is reported. Methods. The PubMed, Embase, Cochrane Library, and Web of Science databases were searched through August 27, 2017, for publications describing HMD use during surgically relevant applications. We identified 120 relevant English-language, non-opinion publications for inclusion. HMD types were categorized as “heads-up” (nontransparent HMD display and direct visualization of the real environment), “see-through” (visualization of the HMD display overlaid on the real environment), or “non–see-through” (visualization of only the nontransparent HMD display). Results. HMDs were used for image guidance and augmented reality (70 publications), data display (63 publications), communication (34 publications), and education/training (18 publications). See-through HMDs were described in 55 publications, heads-up HMDs in 41 publications, and non–see-through HMDs in 27 publications. Google Glass, a see-through HMD, was the most frequently used model, reported in 32 publications. The specialties with the highest frequency of published HMD use were urology (20 publications), neurosurgery (17 publications), and unspecified surgical specialty (20 publications). Conclusion. Image guidance and augmented reality were the most commonly reported applications for which HMDs were used. See-through HMDs were the most commonly reported type used in surgically relevant applications. Urology and neurosurgery were the specialties with greatest published HMD use.
Article
Background: Wearable technology is growing in popularity as a result of its ability to interface with normal human movement and function. Methods: Using proprietary hardware and software, neuronavigation images were captured and transferred wirelessly via a password-encrypted network to the head-up display. The operating surgeon wore a loupe-mounted wearable head-up display during image-guided parieto-occipital ventriculoperitoneal shunt placement in two patients. Results: The shunt placement was completed successfully without complications. The tip of the catheter ended well within the ventricles away from the ventricular wall. The wearable device allowed for continuous monitoring of neuronavigation images in the right upper corner of the surgeon's visual field without the need for the surgeon to turn his head to view the monitors. Conclusions: The adaptable nature of this proposed system permits the display of video data to the operating surgeon without diverting attention away from the operative task. This technology has the potential to enhance image-guided procedures.
Article
Purpose: This study investigates the feasibility and potential utility of head-mounted displays for real-time wireless vital sign monitoring during surgical procedures. Methods: In this randomized controlled pilot study, surgery residents (n = 14) performed simulated bedside procedures with traditional vital sign monitors and were randomized to addition of vital sign streaming to Google Glass. Time to recognition of preprogrammed vital sign deterioration and frequency of traditional monitor use was recorded. User feedback was collected by electronic survey. Results: The experimental group spent 90% less time looking away from the procedural field to view traditional monitors during bronchoscopy (P = .003), and recognized critical desaturation 8.8 seconds earlier; the experimental group spent 71% (P = .01) less time looking away from the procedural field during thoracostomy, and recognized hypotension 10.5 seconds earlier. Trends toward earlier recognition of deterioration did not reach statistical significance. The majority of participants agreed that Google Glass increases situational awareness (64%), is helpful in monitoring vitals (86%), is easy to use (93%), and has potential to improve patient safety (85%). Conclusion: In this early feasibility study, use of streaming to Google Glass significantly decreased time looking away from procedural fields and resulted in a nonsignificant trend toward earlier recognition of vital sign deterioration. Vital sign streaming with Google Glass or similar platforms is feasible and may enhance procedural situational awareness.