ARMAGNI: Augmented Reality Enhanced Surgical Magnifying Glasses

Full text

The 18th Scandinavian Conference on Health informatics, Tromsø, Norway, August 22-24, 2022. Organized by UiT The
Arctic University of Norway. Conference Proceedings published by Linköping University Electronic Press at
https://doi.org/10.3384/ecp187. © The Author(s). This work is licensed under the Creative Commons Attribution-
NonCommercial 4.0 International License. To view a copy of this license, visit http://creativecommons.org/licenses/by-
nc/4.0/
ARMAGNI: Augmented Reality Enhanced Surgical Magnifying Glasses
Situational Awareness during surgery with AR in the Loupe
Artur Liebrecht1, Roman Bibo1, Bastian Dewitz1, Sebastian Kalkhoff1, Sobhan Moazemi1, Markus Rollinger2,
Jean-Michel Asfour2, Klaus-Jürgen Janik3, Artur Lichtenberg1, Hug Aubin1*, Falko Schmid1*+
1Digital Health Lab Düsseldorf, Department of Cardiac Surgery, Medical Faculty and University Hospital Düsseldorf,
Heinrich-Heine-University Düsseldorf, Düsseldorf, Germany
2DIOPTIC GmbH, Weinheim, Germany
3JADENT GmbH, Aalen, Germany
*These authors contributed equally to this work as senior authors
+Corresponding author
Abstract
During cardiac surgery, in addition to their manual work, surgeons need to perceive a large amount of procedural
intraoperative data, including information which is not directly visible to them. Therefore, we developed an
augmented reality demonstrator that displays alphanumerical data into the loupe of surgical magnifying glasses.
Eight cardiac surgeons tested the demonstrator in a skill task that simulates the critical part of a typical surgical
procedure while being confronted to vital intraoperative parameters getting critical. The results showed a
decrease of missed critical phases and improved response times when using the demonstrator instead of a
customary monitor for tracking intraoperative parameters.
Keywords
Augmented Reality, Cardiac Surgery, Intraoperative Assistance
1 INTRODUCTION
Situational awareness, “the perception and understanding
of the surrounding environment” [13], is important in
humans’ performance fulfilling a complex task [6]. This
also applies to cardiac surgeries where safety and outcome
are dependent on information flow, concerning not only
preoperative data but also intraoperative procedural data
such as vital parameters. Although operation rooms are
equipped with multiple monitors displaying standard
parameters like vital signs, other information such as
heartlung machine procedural data and respiratory
parameters are not readily visible for the surgeon. When
needed, it requires intense communication efforts which
can lead to distraction and interruption of the procedure [4].
Augmented reality (AR) can be a solution to bridge the gap
between information needs, communication, and display
possibilities.
Head-mounted displays (HMD), including smart glasses
such as Google Glass, have been playing an increasing role
in the health care industry [12]. In a systematic review from
2019, Rahman et al. [12] determined 120 HMD
applications in surgery, of which most were used for image
guidance and AR. For instance, Liebert et al. [7] compared
traditional vital signs monitor to an HMD for monitoring
patient’s vitals, showing potential for increased situational
awareness and improved patient safety. The overall
feedback from users was positive. In a qualitative
descriptive study, Enlöf et al. [5] also showed that health
care professionals have a generally positive view of using
smart glasses in the medical field. In video assisted surgery
AR glasses have potential to reduce bad body posture
during procedure [8]. Arpaia et al. [1] presented a system
for retrieving and displaying patient’s vitals and evaluated
its effectiveness, transmission error rate, refresh rate and
latency with confirming technical feasibility of such a
system. The target group consisted of assistant surgeons
and anesthetists.
The related work mentioned in the previous lines indicate
benefits from using AR devices in surgery and high user
acceptance, but these applications did not consider cases
where users require surgery magnifying glasses. For
cardiac surgeons, wearing magnifying glasses and HMDs
simultaneously can be cumbersome due to interference. As
one solution, smart glasses can be mounted to surgical
loupes. Yoon et al. [14] successfully used such a setup for
image guidance during shunt placement. The AR images
were projected above the loupe and required quick eye
movement to see. Qian et al. [10], on the other hand,
combined a Magic Leap One with binocular magnifying
loupes. They evaluated a calibration method to align virtual
content to the real world in the magnified or minified field-
of-view of the user. But since we are projecting simple
alphanumerical procedural data that does not have to be
aligned to real objects, there is no need for such a
calibration method. Hence, we can use a simpler setup with
lightweight hardware.
The 18th Scandinavian Conference on Health informatics, Tromsø, Norway, August 22-24, 2022. 46

Figure 1. AR demonstrator consisting of an AR module
and Galilean loupes mounted on a custom-made glasses
frame mimicking surgical magnifying glasses.
Figure 2. Image of the field of view of the AR loupe.
Our approach for displaying procedural data in the field of
view of surgeons is different to the ones mentioned above.
Instead of extending existing HMDs which have limitations
due to weight, size, or short battery life [12], we develop
typical surgical magnifying glasses with an integrated AR
display into the loupe. For the proof of concept, we built an
AR demonstrator that can be adjusted to the user. The
prototype is shown in Figure 1 and the view of the loupe
with AR visualizations in Figure 2.
In a feasibility study, experienced cardiac surgeons tested
the usefulness of AR in the loupe with our demonstrator.
We propose that AR magnifying glasses can increase
situation awareness of surgeons. Beyond that, they will
have potential to reduce surgeons’ workload during cardiac
surgeries and encounter high user acceptance, as they
mimic the typical surgical magnifying glasses used during
cardiac surgery.
2 METHODS
For this study, we asked cardiac surgeons to test our
prototype referred as AR demonstrator, which we will
describe in detail below along with the corresponding
software, we will further outline the experimental setup.
2.1 AR Demonstrator
The prototype consists of two customary Galilean loupes
with 2.5x magnification and 300 mm working distance.
They were attached to a custom-built metallic mechanism,
which allows different users to adjust pupil distance, inter-
eye asymmetry or height of the loupes. The metallic
mechanism is mounted on a plastic glasses frame made
with a laser cutter.
The 2.5 cm x 2 cm x 1.5 cm sized AR module (Figure 3)
was built with a combination of a micro-OLED display, a
lens, and a beam splitter. For displaying AR visualizations,
we used a Sony ECX336AF-6 OLED micro display. The
0.23-inch diagonal sized RGB display has a 640 x 400
pixels resolution, 800 cd/m² maximal luminance and
10,000:1 contrast ratio. We installed the micro display on
the side of a 3D printed component. There, the displayed
images are projected through a lens magnifying them to
fully cover the view through the loupes. A beam splitter
centered between the display, the loupe, and the eye,
reflects the images into the eye of the user. The AR module
can be attached to one of the Galilean loupes.
Figure 3. AR module attachable to loupes.
The micro display is connected to a Raspberry Pi Zero
installed in the glasses frame. To be able to power the micro
display with the required 1.8 volts and send images to it,
the Raspberry Pi Zero was extended by a custom circuit
board. A 3.6 volts rechargeable battery is powering the
Raspberry Pi Zero through a 200 cm long cable. This
battery is bundled in a 3D printed case with integrated
electronics and a button, which allows us to turn the
Raspberry Pi on and off.
2.2 Base station
To keep the AR magnifying glasses lightweight and
conformable, we follow a client-server approach. One
endpoint is the mentioned Raspberry Pi Zero built in the
glasses frame. As the other endpoint, we are using a base
station on Raspberry Pi 4B basis and the official Raspberry
Pi 7-inch touchscreen (see Figure 4). The base station
serves as a mobile control center for fetching patient data
from simulated medical devices as well as generating
visualizations and streaming them to the AR display.
Furthermore, the base station is running a graphical user
interface frontend application for configuring the AR
visualizations using the touchscreen.
47 The 18th Scandinavian Conference on Health informatics, Tromsø, Norway, August 22-24, 2022.

AR visualizations are generated in the frontend application
and sent as PNG images to a REST server on the Raspberry
Pi Zero. Using a Raspberry Pi 4B with the 64-Bit version
of the Raspberry Pi OS as our base station, we can achieve
approximately 5 frames per second with a delay of 500
milliseconds.
Figure 4. Base station running the software.
Our frontend application developed with Vue.js enables
users to customize the AR visualizations. They can pick
different parameters from connected medical device
simulators and choose the desired position in the field of
view. We decided the visualized medical parameters to be
automatically arranged in a ring formation, so they do not
overlay the working surface of the user. Though, the user is
free to choose the size of the ring and by that the size of
parameters as well as the number of displayed parameters.
Another feature of the application allows users to finetune
the whole AR visualization by moving, resizing, or rotating
it with touch gestures while wearing the glasses and seeing
the changes near real-time. Each user can have multiple
settings profiles, which are saved in a PostgreSQL
database.
Communication with the database and medical devices is
performed by an Apollo GraphQL server, which exposes a
GraphQL API for the frontend application. Backend was
built in accordance with IEEE 11073 Service-oriented
Device Connectivity (SDC) for being capable of
communicating with real medical devices that also support
SDC. Using SDCLib, an open-source SDC implementation
by SurgiTAIX, we developed a SDC consumer micro
service, which serves as a bridge between medical devices
and our GraphQL server.
For the experiment, we developed a medical device
simulator, that generates random but reasonable values for
heart rate (labeled as HF in Figure 2), mean arterial pressure
(MAD), oxygen saturation (SPO2) and central venous
pressure (ZVD). The simulated values can continuously
increase and decrease every 3 seconds, and every 60 – 90
seconds one of the parameters becomes critical for 15
seconds. Parameters in a critical state are displayed with a
small warning sign and blink with a frequency
approximately 600 ms. By means of this simulation, we
evaluated the effect of the AR module on the user during a
skill exercise, the setup of which we will describe next.
2.3 Experimental setup
The experiment took place in the clinic for cardiac surgery
at the University Hospital Düsseldorf. We invited 8 cardiac
surgeons (2 female, 6 males, between 25 and 54 years old)
to perform a skill exercise while wearing our AR
magnifying glasses and reacting to critical parameters.
Directly afterwards, we asked them to take a survey, which
included the System Usability Scale (SUS), the NASA
Task Load Index (NASA-TLX) and further questions
regarding the usefulness and usability of the AR.
In the beginning of the experiment, the investigator taught
participants how they can adjust the AR magnifying glasses
via the metallic mechanism and finetuned the AR display
using the frontend application. During the experiment,
frontend and backend applications were running on a laptop
computer and used by investigator rather than the
participant.
We followed a within-subject design with randomized
order of conditions. In both cases, participants performed
an anastomosis on the Arroyo’s Anastomosis Simulator
[11] while wearing our prototype. In the test condition, the
AR module was turned on and the participants had to keep
track of the displayed parameters in the loupe. In the control
condition on the other hand, parameters were displayed on
a customary 27-inch display only. We positioned the
display on the right side of and 2 meters away from the
participants, so they had to turn their head to the right by
approximately 20-degree angle to see the values.
During performing the skill exercise, participants had to
react to critical values as soon as possible by telling the
investigator which parameter is critical and whether it is too
high or too low. We recorded their response time using a
timer implemented in the simulation software. Participants
had a time limit of 15 seconds to notice and react to a
critical phase and the investigator registered the answer as
“right” or “wrong” by pushing the corresponding button on
the keyboard. If the pre-set 15-second timeout passed
before the participant reacted, the software automatically
registered the answer as “missed” with a response time of
15 seconds.
After each trial, our simulation software generated a log file
containing participant’s response times to critical phases
and whether the response was right, wrong, or missed.
3 RESULTS
We performed t-tests to compare our metrics between test
condition (using AR) and control condition (using
monitor). All metrics were tested for homogeneity of
variance and normality before comparing them with a two-
tailed paired t-test for dependent means. If requirements for
homogeneity or normality were not met, Wilcoxon Singed-
Rank test was used.
3.1 Response performance
We used a Wilcoxon Signed-Rank test to compare response
performances between test and control conditions.
Response performance is the count of correct responses
divided by count of critical phases for each participant. The
calculated data for each participant can be seen in Figure 5.
It should be noted that no participant gave wrong responses.
The 18th Scandinavian Conference on Health informatics, Tromsø, Norway, August 22-24, 2022. 48

Figure 5. Rate for correct responses of each participant.
Participants showed an increase in response performance
when using the AR display (M=.8, SD=.2) compared to
customary display (M=.5, SD=.3), W=0, p < .05.
3.2 Response time
Data for response time is visualized in Figure 6. Since
missed critical phases mean that there was no reaction for
15 seconds, data is strictly speaking not normally
distributed. One participant even missed all critical phases
in control condition; hence median was 15 seconds in this
case.
Figure 6. Median response times in seconds.
The Levene's test for response times was not significant
(F(1, 7) = 1.313, p = .340) so variance homogeneity was
assumed. Despite data being limited between 0 and 15
seconds, Shapiro-Wilk test for the differences between the
pairs did not show a significant departure from the
normality (W(8) = .915, p = .433).
Participants responded faster to critical values in the AR
condition (M = 6.7, SD = 1.9) compared to monitor
condition (M = 10.1, SD = 3), (t(7) = 2.9, p = .025).
3.3 Total time for trials
Participants required different amount of time to complete
the anastomosis on the Arroyo’s simulator. We used a t-test
to compare the required time for test and control conditions.
The Levene's test for required times was not significant
(F(1, 7) = .079, p = .783) so variance homogeneity was
assumed. Shapiro-Wilk test for the differences between the
pairs did not show a significant departure from the
normality (W(8) = .984, p = .999).
The t-test did not show any significant difference in
required time for trials between AR (M = 12.4, SD = 5.2)
and monitor conditions (M = 11.4, SD = 5.5),
(t(7) = 1.4, p = .203).
3.4 NASA-TLX
The Levene’s test for global NASA-TLX, namely the sum
of individual NASA-TLX scores, was not significant
(F(1, 7) = .064, p = .804) and Shapiro-Wilk test did not
show significant departure from normality,
(W(8) = .92, p = .471). The t-test for global NASA-TLX
showed an improved workload for AR condition (M=52.9,
SD=22.1) compared to monitor condition (M = 65.4,
SD = 24.1), (t(7) = 2.8, p = .027).
Boxplots and significance in differences determined by
ttests for NASA-TLX subscales can be seen in Figure 7.
Variance homogeneity and normality criteria using
Levene’s test and Shapiro-Wilk test, respectively, were
met. Outliers are classified as being outside 1.5 times the
interquartile range.
Figure 7. Mean scores for mental demand (MD), physical
demand (PD), temporal demand, (TD), performance (P),
effort (E) and frustration (F). ns means that a paired t-test
result was not significant.
3.5 SUS Score
Mean SUS score was 66.875 (SD 12.02). Reponses for
individual statements are visualized in Figure 8.
3.6 Further questions
Responses to further questions about AR in the
demonstrator and about AR magnifying glasses in general
are summarized in Figure 9. Participants were asked to rate
whether they agree or disagree with the statements on a
scale of 0 – 100.
49 The 18th Scandinavian Conference on Health informatics, Tromsø, Norway, August 22-24, 2022.

Figure 8. Responses to SUS statements.
Figure 9. Please tell us whether you agree or disagree with
the following statements.
Q1. I preferred using the AR instead of looking at the
monitor to detect critical parameters.
Q2. The medical parameters in the AR were clearly
visible.
Q3. I found that the AR visualization obscured the
view over the working surface.
Q4. I can imagine myself using surgical magnifying
glasses with AR in daily OR work.
Q5. I think magnifying glasses with AR can improve
the outcome of surgeries.
4 DISCUSSION
4.1 Objective evaluation
The experiment’s results showed an improvement in
participant’s reactions to critical phases when using the AR
display compared to a common diagnostic monitor. In the
test condition, not only reaction times were 51 % faster, but
also the performance for detecting critical parameters was
increased by 60 %. On the other hand, given our sample
size of 8 participants being small, statistical conclusions
should be treated with care. Nevertheless, our initial
proposition that augmented reality can improve surgeons’
situational awareness could be confirmed, at least, in our
experimental setup.
Circumstances during real operation are different. Firstly,
OP light is much brighter than in our setup, which
negatively affects the visibility of the AR visualizations.
Technical suitability should be evaluated in further studies.
Secondly, the question arises whether an AR display in
magnifying glasses is necessary in the presence of auditory
alerts and assistant doctors or anesthetists. Our
experimental setup demonstrates that an AR display can
help surgeons keep track of selected parameters omitting
some communication efforts that can possibly interrupt the
operation procedure.
4.2 Subjective assessments
There was a significant difference in global NASA-TLX
between test and control condition, indicating that AR
magnifying glasses can also help to improve workload in
surgeons. Especially, frustration was reduced by 45 % and
physical demand by 36 % (borderline not significant
though) when using the AR display. Higher physical
demand in monitor condition could result from the fact that
participants had to turn their head to see the parameters.
Also, in the control condition, participants missed more
critical phases (response performance dropped by 38 %),
which explains higher frustration. It might be interesting to
further investigate these subscales under real
circumstances, where cardiac surgeons rarely look to
monitors but communicate with attendants about them.
4.3 Usability and opinions
Participant’s feedback to the AR demonstrator presented
were mixed and an average SUS score of 66.875 can be
considered as “marginally acceptable” according to
determined ranges by Bangor et al [2].
On the one hand, participants liked the AR functionality
and showed interest in using such a device in the future.
On the other hand, everyone had difficulties with the
prototype itself, mostly independent of the AR module.
Adjustment of the magnifying glasses could take over 10
minutes but even then, the alignment of the loupes was not
ideal. Sometimes participants had to re-adjust the loupes
during the skill task. Another common problem was that,
due to their weight, the glasses slightly slid down during
skill task so that the top two parameters disappeared. These
problems primarily occurred because of the mechanics for
adapting the glasses to different users. Normally, surgical
loupes are custom-made for each user, to fit pupil distance,
height, and preferred working distances. However, we
wanted one single device that could be tried out by multiple
people, which would not be possible with a tailormade
device.
In the future, we are building another prototype with a
different micro display that uses Bluetooth Low Energy for
data transmission and a lighter battery. This omits a
Raspberry Pi Zero in the glass frame reducing the weight
The 18th Scandinavian Conference on Health informatics, Tromsø, Norway, August 22-24, 2022. 50

of the device, which should resolve most issues of the
current AR demonstrator as discussed above.
5 CONCLUSION
This first feasibility study showed that ARMAGNI has the
potential to increase surgeons’ situation awareness during
operations requiring magnification glasses while
decreasing the workload for surgeons. Participants liked the
concept of the product but had difficulties with the current
prototype. In an upcoming improved prototype,
participants feedback will be taken into account to enhance
usability. In addition, a different micro display is intended
to improve overall ergonomics of the device.
6 REFERENCES
[1] Arpaia, P., Benedetto, E. D., Dodaro, C. A., Duraccio,
L. & Servillo, G. Metrology-Based Design of a
Wearable Augmented Reality System for Monitoring
Patient’s Vitals in Real Time. IEEE Sensors Journal
21, 11176–11183 (2021).
[2] Bangor, A., Kortum, P. T. & Miller, J. T. An
Empirical Evaluation of the System Usability Scale.
International Journal of Human-Computer
Interaction 24, 574–594 (2008).
[3] Cercenelli, L. et al. The Wearable VOSTARS System
for Augmented Reality-Guided Surgery: Preclinical
Phantom Evaluation for High-Precision Maxillofacial
Tasks. Journal of Clinical Medicine 9, 3562 (2020).
[4] ElBardissi, A. W., Wiegmann, D. A., Henrickson, S.,
Wadhera, R. & Sundt, T. M. Identifying methods to
improve heart surgery: an operative approach and
strategy for implementation on an organizational
level. European Journal of Cardio-Thoracic Surgery
34, 1027–1033 (2008).
[5] Enlöf, P., Romare, C., Jildenstål, P., Ringdal, M. &
Skär, L. Smart Glasses for Anesthesia Care: Initial
Focus Group Interviews with Specialized Health Care
Professionals. Journal of PeriAnesthesia Nursing 36,
47–53 (2021).
[6] Hazlehurst, B., McMullen, C. K. & Gorman, P. N.
Distributed cognition in the heart room: How situation
awareness arises from coordinated communications
during cardiac surgery. Journal of Biomedical
Informatics 40, 539–551 (2007).
[7] Liebert, C. A., Zayed, M. A., Aalami, O., Tran, J. &
Lau, J. N. Novel Use of Google Glass for Procedural
Wireless Vital Sign Monitoring. Surgical Innovation
23, 366–373 (2016).
[8] Lim, A. K., Ryu, J., Yoon, H. M., Yang, H. C. & Kim,
S. Ergonomic effects of medical augmented reality
glasses in video-assisted surgery. Surgical Endoscopy
36, 988–998 (2021).
[9] Maluf, M. A. et al. Cardiovascular Surgery Residency
Program: Training Coronary Anastomosis Using the
Arroyo Simulator and UNIFESP Models. Revista
Brasileira de Cirurgia Cardiovascular (2015).
[10] Qian, L., Song, T., Unberath, M. & Kazanzides, P.
AR-Loupe: Magnified Augmented Reality by
Combining an Optical See-Through Head-Mounted
Display and a Loupe. IEEE Transactions on
Visualization and Computer Graphics 1–1 (2020).
[11] Raet, J. M. D. et al. How to build your own coronary
anastomosis simulator from scratch†. Interactive
CardioVascular and Thoracic Surgery 16, 772–776
(2013).
[12] Rahman, R. et al. Head-Mounted Display Use in
Surgery: A Systematic Review. Surgical Innovation
27, 88–100 (2019).
[13] Woodward, J. & Ruiz, J. Analytic Review of Using
Augmented Reality for Situational Awareness. IEEE
Transactions on Visualization and Computer
Graphics 1–1 (2022).
[14] Yoon, J. W. et al. Technical feasibility and safety of
image-guided parieto-occipital ventricular catheter
placement with the assistance of a wearable head-up
display. The International Journal of Medical
Robotics and Computer Assisted Surgery 13, e1836
(2017).
7 ACKNOWLEDGEMENT
This Project is supported by the Federal Ministry for
Economic Affairs and Climate Action (BMWK) under
grant 16KN069341 on the basis of a decision by the
German Bundestag. Authors state no conflict of interest
51 The 18th Scandinavian Conference on Health informatics, Tromsø, Norway, August 22-24, 2022.