Conference PaperPDF Available

An Augmented Reality Framework for Soft Tissue Surgery

  • Odin Vision
  • european patent office, Munich, Germany

Abstract and Figures

Augmented reality for soft tissue laparoscopic surgery is a growing topic of interest in the medical community and has potential application in intra-operative planning and image guidance. Delivery of such systems to the operating room remains complex with theoretical challenges related to tissue deformation and the practical limitations of imaging equipment. Current research in this area generally only solves part of the registration pipeline or relies on fiducials, manual model alignment or assumes that tissue is static. This paper proposes a novel augmented reality framework for intra-operative planning: the approach co-registers pre-operative CT with stereo laparoscopic images using cone beam CT and fluoroscopy as bridging modalities. It does not require fiducials or manual alignment and compensates for tissue deformation from insufflation and respiration while allowing the laparoscope to be navigated. The paper's theoretical and practical contributions are validated using simulated, phantom, ex vivo, in vivo and non medical data.
Content may be subject to copyright.
An Augmented Reality Framework for Soft
Tissue Surgery
Peter Mountney1, Johannes Fallert2, Stephane Nicolau3, Luc Soler3,4and
Philip W. Mewes5
1Imaging and Computer Vision, Siemens Corporate Technology, Princeton, NJ, USA
2Imaging Technologies Research, Karl Storz, Tuttlingen, Germany
3Institut de Recherche contre les Cancers de l’Appareil Digestif (IRCAD)
4Institut Hospitalo-Universitaire de Strasbourg (IHU Strasbourg)
5Angiography & Interventional X-Ray Systems, Siemens Healthcare, Germany
Abstract. Augmented reality for soft tissue laparoscopic surgery is a
growing topic of interest in the medical community and has potential
application in intra-operative planning and image guidance. Delivery of
such systems to the operating room remains complex with theoretical
challenges related to tissue deformation and the practical limitations of
imaging equipment. Current research in this area generally only solves
part of the registration pipeline or relies on fiducials, manual model
alignment or assumes that tissue is static. This paper proposes a novel
augmented reality framework for intra-operative planning: the approach
co-registers pre-operative CT with stereo laparoscopic images using cone
beam CT and fluoroscopy as bridging modalities. It does not require fidu-
cials or manual alignment and compensates for tissue deformation from
insufflation and respiration while allowing the laparoscope to be navi-
gated. The paper’s theoretical and practical contributions are validated
using simulated, phantom, ex vivo,in vivo and non medical data.
1 Introduction
Interest in augmented reality (AR) for soft tissue surgery, such as liver resection
and partial nephrectomy, has grown steadily within the medical community. The
role of AR in this context is procedure- and workflow-dependent. It can be used
at the beginning of the surgical procedure for intra-operative planning to rapidly
identify target anatomy and critical sub surface vessels, or it can facilitate image
guidance to display tumor resection margins and improve dissection accuracy [1].
A number of theoretical and practical challenges remain for the translation
of such systems into the operating room. The core challenge is registration of
the pre-operative image (CT/MRI) with the intra-operative laparoscopic im-
age. This in itself is challenging due to the lack of cross modality landmarks
and the laparoscopic camera’s small viewing field. Furthermore, surgical proce-
dures require insufflation of the abdomen causing an initial organ shift and tissue
deformation, which must be reconciled. The registration problem is further com-
plicated during the procedure itself due to continuous tissue deformation caused
by respiration and tool-tissue interaction.
Due to the complex registration pipeline required to deliver AR to the op-
erating room, current research tends to focus on individual components of the
process and do not provide complete solutions. For example, notable work ex-
ists in deformable tissue modeling [2, 3], dense reconstruction [4, 3], non-rigid
registration of CT to cone beam CT (CBCT) [5], tissue tracking [6], surface
registration [7] and laparoscopic camera pose estimation [8, 9].
A handful of end-to-end systems have been proposed for the operating room
that rely on additional fiducials, manual registration, or the baseline assumption
that tissue is static. Challenges persist in each scenario. Fiducials act as cross
modality landmarks and have been attached externally on the patient’s skin [10]
and to the organ itself [11]. Their use however, can be disruptive to the clinical
workflow. Manual registration, on the other hand, requires experts to visually
align a 3D model to the laparoscopic image [12]. Accuracy is user dependent
even when alignment is constrained with a single cross modality landmark [13].
Finally, as per the static environment assumption, a comprehensive system has
been proposed for skull surgery [8], but deformation compromises its accuracy.
This paper proposes an AR framework for intra-operative planning in liver
surgery1. The novel system registers pre-operative CT and stereo laparoscopic
images to a common coordinate system using CBCT and fluoroscopy as bridg-
ing modalities. It does not require fiducials or manual model alignment. Tissue
deformation caused by insufflation, organ shift and respiration are accounted for
along with laparoscopic camera motion. The framework is evaluated on simu-
lated, phantom, ex vivo,in vivo and non medical data.
2 Method
A key component of the AR system is the introduction of CBCT into the operat-
ing room. CBCT machines capture 3D CT-like images and 2D fluoroscopy —in
the same coordinate system —while the patient is on the operating table. CBCT
and fluoroscopy are used as bridging modalities to co-register pre-operative CT
and laparoscopic images. The framework consists of three registration phases:
1) a registration of CT to CBCT (Fig. 1), which takes into account tissue defor-
mation resulting from insufflation 2) a registration of the laparoscope to CT via
CBCT coordinate system (Fig. 2), accounting for tissue deformation caused by
respiration and 3) a temporal registration of laparoscopic images (Fig. 3), which
deals with camera motion and tissue deformation caused by respiration.
2.1 Non Rigid Registration of CT to CBCT
Pre-operative CT and organ segmentation are performed in the days or weeks
prior to the operation. With the patient in the supine position, two CT images are
captured using a contrast injection at the arterial and venous phases. The images
are registered together and segmented2into 3D anatomical models including the
liver, tumor, vessels and abdomen wall as shown in Fig. 1.
1Not currently commercially available.
Fig. 1: Registration of pre-operative CT to intra-operative CBCT.
During the procedure, the patient is positioned for easy of access (e.g. reverse
Trendelenburg) and the abdomen is insufflated with CO2causing organ shift
and deformation. The tools and laparoscope are removed or positioned safely
and a CBCT is acquired during an inhale breath hold. Fig. 1, shows the signifi-
cant difference between the CT and CBCT images. The CT is registered to the
CBCT using a non-rigid biomechanically driven registration technique [5]. This
registration approach consists of three steps: 1) rigid alignment of the spine, 2)
biomechanical insufflation modeling, and 3) diffeomorphic non-rigid registration.
The final deformation field can be applied to the pre-operative planning data
and models, thus bringing this information into the CBCT coordinate system.
2.2 Registration of the Laparoscope to CBCT Coordinate System
With the CT to CBCT registration complete, the next task is registering the
laparoscope to the CBCT coordinate system. This is challenging due to the lack
of cross modality landmarks and the camera’s small field of view. A two step
registration is proposed- an initial position estimation and local refinement.
The initial position of the laparoscope in the CBCT coordinate system is
estimated using fluoroscopic images. A mechanical device holds the laparoscope
in position and two mono fluoroscopic images are acquired, each 90apart.
A semi-automated method is used to select two points along the shaft which
are triangulated to estimate the laparoscope’s position and pose with 5 degrees
of freedom. The rotation around the laparoscope’s optical imaging axis is not
estimated due to its symmetrical appearance in the fluoroscopic images. Further-
more, the physical position of the camera center along the shaft is not known,
and this introduces additional errors.
A local registration refinement is performed directly between the laparoscopic
images and the 3D surface model of the organ in the CBCT coordinate system.
At this point in the surgical workflow the patient is not at breath hold. Their
breathing is periodic and controlled by a ventilator. This respiration causes the
abdominal tissue to deform periodically. The first challenge, therefore, lies in the
registration of the laparoscopic images to a 3D model representing the tissue at
an inhale breath hold. Registering to any other point in the respiration cycle
would introduce error into the system.
The temporal motion of the tissue in the laparoscopic images is used to es-
timate the current point in the respiration cycle. Features are detected on the
tissue surface and matched in the left and right stereo laparoscope images to es-
timate their 3D position relative to the camera. The 3D features are transformed
Fig. 2: Registration of laparoscope to CBCT coordinate system.
into CBCT space using the initial laparoscope alignment and features which are
not position near the liver are removed. The features are tracked from frame to
frame and their 3D position is computed. Principal Component Analysis (PCA)
is applied to extract a 1D respiration signal from the 3D motion of the features
[9]. The first component corresponds to respiration, this data is smoothed using
a moving average filter to obtain a 1D respiration signal for each feature.
The maximum inhalation position is estimated by fitting a respiration model
z(t) = z0bcos2n(Πt
τφ) (1)
where z0is the position of the liver at the exhale, bis the amplitude, τis the res-
piration frequency, φis the phase and ndescribes the gradient of the model and
is empirically set to 4. The parameters of Eq. 1 are estimated using Levenberg-
Marquardt minimization algorithm. Before the model is fit, outliers are removed
by applying RANSAC to the orientation of the PCA transformation and thresh-
olding the periodicity of the respiration signal which corresponds to τand φ.
The remaining inliers are averaged and the model parameters are estimated to
identify the point in the respiration cycle corresponding to maximum inhale.
Given the initial estimate of the laparoscope’s position and the point in
the respiration cycle, the final step remains to perform the direct registration
between stereo images and the 3D model. A 3D-3D registration aligns a stereo
reconstruction [4] to a point set extracted from the 3D model surface. This point
set is extracted using the initial estimate of the laparoscope’s position from the
previous step, the camera’s intrinsic parameters, and z-buffering.
The accurate registration of the 3D model point set and the stereo recon-
struction is challenging. At a macro level the point sets represent the same shape,
however at a local level they are structurally different because of the way the
point sets are generated. The 3D model is continuous, smooth and isotropic. The
stereo reconstruction is discretized, contains steps due to pixel level disparity es-
timates, is anisotropic and may not be a complete surface representation. As a
result, even after correct alignment it is impossible to get an exact match for
each point. This can cause point-to-point algorithms such as Iterative Closest
Point (ICP) to converge a sub-optimal solution as shown in [7].
A probabilistic approach is used [14] that models noise in both the target and
source point sets. It makes use of the underlying surface structure while remain-
ing computationally efficient by combines point-to-point and point-to-plane ICP
in a single framework. The goal is to align two point sets A={ai}i=1,...,n and
Fig. 3: Temporal registration of laparoscope and tissue.
B={bi}i=1,...,n0. The proposed approach replaces the traditional ICP minimiza-
tion step TargminTPi{Tbimik2}which finds the optimal transformation
Tbetween point biand mi(the closest corresponding point in A) with
T= argmin
where dT
i=biTaiand CA
iand CB
iare the covariance matrices used to
model noise in the system. By setting high covariance along the local plane and
a low covariance along the surface normal, the registration algorithm is guided
to use the surface information in both the 3D model point set and the stereo
reconstruction point set. The stereo point set is a subset of the 3D model point
set. A maximum correspondence distance is empirically set to account for the
fact that some points do not have matches.
2.3 Temporal Alignment
Section 2.2 outlined an approach for registering the laparoscope to the CBCT
system where the laparoscope is static and the tissue is temporally static, i.e.
at maximum inhale. However, during abdominal surgery, tissue and organs are
continuously deforming and the surgeon is free to move the laparoscopic camera.
The position of the laparoscopic camera and tissue deformation are jointly
estimated using a modified Simultaneous Localization and Mapping (SLAM)
technique [9]. This approach models the position and orientation of the camera
in conjunction with a dynamic 3D tissue model which is driven by a respiration
model. Within an Extended Kalman Filter (EKF) framework the state vector ˆxis
comprised of the camera position rW, its orientation RRW , translational velocity
vWand angular velocity wRand the respiration model parameters estimated in
section 2.2 {z0, b, τ, φ}. In addition, for each feature, the state contains ˆyi=
y, eig ) where ¯yis the average 3D position of the feature and eig is the PCA
transformation. As shown in Fig. 3, the system iterates between prediction and
update steps to estimate the camera’s position and tissue deformation. Further
details can be found in [9].
The SLAM algorithm initalization follows the registration in section 2.2. As a
result, the 3D SLAM features are co-registered to the CBCT coordinate system.
In the subsequent image frames, computing the transformation, using singular
value decomposition, between the feature positions at time tand time 0 yeilds
the estimated 3D model position.
Fig. 4: Laparoscope to CBCT registration: Fiducials shown in green (ground
truth), blue (before registration), yellow (after registration). a) non medical, b)
sim, c) ex vivo, d) phantom. In vivo SRE (mm) e) before and f) after registration.
3 Experiments and Results
A range of experiments were performed to validate the proposed framework
on simulated, phantom, ex vivo,in vivo and non medical data. The phases of
the pipeline are evaluated separately here, both for clarity and because not all
data contain temporal deformation. CT to CBCT obtains accuracy of <1mm
on liver, due to space constraints the reader is directed to [5] for evaluation. A
description of the datasets follows. Simulated: a mesh generated form a CT and
textured with laparoscopic images. Phantom: a visually realistic silicon liver
phantom with surface fiducials for ground truth. Ex vivo: porcine with fiducials
for ground truth. In vivo: two porcine without fiducials. Non Medical: meshes
from Stanford dataset3textured with laparoscopic images.
Registration of laparoscopic camera to CBCT. 50 datasets with ground
truth were available- simulated (20), phantom (10), ex vivo (10) and non medical
(10). Random noise (Up to ±20mm) was added to the initial position of the
laparoscope in the CBCT system to quantitatively evaluate the registration. 10
noisy datasets were created for each ground truth dataset, making a total of 500
datasets. 11 in vivo datasets were evaluated without ground truth fiducials. The
results are shown in Table 1 and illustrated in Fig. 4.
Table 1: Quantitative validation: Registration of laparoscope to CBCT.
Dataset Before After Before After
Sim 5.3mm 0.8mm 10.4mm, 289.9 px 1.69mm, 56.8 px
Phantom 5.7mm 1.1mm 10.2mm, 90.5 px 4.1mm, 29.9 px
Ex vivo 4.7mm 1.3mm 10.28mm, 136.5 px 3.4mm, 48.7 px
In vivo 5.4mm 0.9mm N/A N/A
Non Medical 5.5mm 0.9mm 10.2mm, 321.2 px 0.3mm, 10.6 px
Fig. 5: Augmented reality overlay of a virtual tumor for intra-operative planning.
The metrics Surface Registration Error (SRE) =1
Target Registration Error (TRE) =RM SError(F iducial s1F iducials2) are
used for evaluation. The registration refinement process reduces the TRE for
all datasets converging to results of between 0.3-4.1mm. The phantom data has
the largest error which is attributed to its homogenous shape. Additional errors
may be introduced by manual fiducial annotation. The 2D TRE is dependent
on the proximity of the fiducials to the camera and image size. The in vivo
and ex vivo image size is 1280x720 and all others are 1920x1080. The 2D TRE is
visualized in Fig. 4. Fig. 4a) shows a successful registration where the added noise
is 10around the optical axis and 10mm along the optical axis. The registration
reduces the SRE for all datasets. Fig. 4 e-f) show the SRE for in vivo data
before and after registration with Fig. 4 f) demonstrates a converged registration.
Stereo reconstruction takes 5.1s and registration takes 7.2s however, the proposed
surgical workflow does not requrie these step to be real-time.
Temporal registration was quantitatively evaluated on 20 simulated and
five in vivo datasets. Simulated data was generated by applying a realistic biome-
chanical deformation to the organ model and moving the camera. Evaluation
with respect to TRE and camera position are shown in Table 2. For in vivo data
ground truth was obtained by annotating the position of the scope in fluoro
images at the start and end of each sequence. The annotation contains absolute
positional errors in the CBCT coordinate system but it can be considered accu-
rate relative to the camera coordinate system. The results are shown in Table 2.
Qualitative validation is provided for in vivo data in Fig. 5 where a segmented
virtual tumor is augmented. This illustrates the accurate estimation of the cam-
era’s position and the point in the respiration cycle. The respiration models are
visualized in Fig 6. Temporal registration runs at 15fps.
Fig. 6: Respiration model. in vivo
(top), sim (bot). 1D respiration sig-
nal (b), smoothed data (r), model(g)
TRE Camera
Dataset Error 3D Position 3D
Sim 3.6mm 1.9 mm
In vivo n/a 4.1 mm
Table 2: Quantitative evaluation
of temporal registration of laparo-
scopic images.
4 Conclusion
In this paper, an augmented reality framework for intra-operative planning is
proposed which co-registering pre-operative CT to laparoscope images. It does
not require fiducials, manual model alignment and accounts for camera motion
and tissue deformation. The framework has been validated on simulated, phan-
tom, ex vivo (porcine), in vivo (porcine) and non medical data. Future work will
focus on improving computational efficiency and more complex tissue modelling.
1. Hughes-Hallett, A., Mayer, E.K., Marcus, H.J., Cundy, T.P., Pratt, P.J., Darzi,
A.W., Vale, J.A.: Augmented reality partial nephrectomy: Examining the current
status and future perspectives. Urology (2013) 266–273
2. Allard, J., Cotin, S., Faure, F., Bensoussan, P.J., Poyer, F., Duriez, C., Delingette,
H., Grisoni, L.: Sofa-open framework for medical simulation. In: MMVR. (2007)
3. Collins, T., Bartoli, A.: Towards live monocular 3d laparoscopy using shading and
specularity information. In Abolmaesumi, P., Joskowicz, L., Navab, N., Jannin,
P., eds.: IPCAI. Volume 7330 of LNCS. Springer (2012) 11–21
4. Stoyanov, D., Scarzanella, M., Pratt, P., Yang, G.Z.: Real-time stereo reconstruc-
tion in robotically assisted minimally invasive surgery. In Jiang, T., Navab, N.,
Pluim, J., Viergever, M., eds.: MICCAI. Volume 6361 of LNCS. Springer (2010)
5. Oktay, O., Zhang, L., Mansi, T., Mountney, P., Mewes, P., Nicolau, S., Soler, L.,
Chefdhotel, C.: Biomechanically driven registration of pre- to intra-operative 3d
images for laparoscopic surgery. In Mori, K., Sakuma, I., Sato, Y., Barillot, C.,
Navab, N., eds.: MICCAI. Volume 8150 of LNCS. Springer (2013) 1–9
6. Puerto Souza, G.A., Adibi, M., Cadeddu, J.A., Mariottini, G.L.: Adaptive multi-
affine (AMA) feature-matching algorithm and its application to minimally-invasive
surgery images. In: IROS. (2011) 2371–2376
7. Maier-Hein, L., Franz, A., dos Santos, T., Schmidt, M., Fangerau, M., Meinzer,
H., Fitzpatrick, J.: Convergent iterative closest-point algorithm to accomodate
anisotropic and inhomogenous localization error. PAMI 34(8) (2012) 1520–1532
8. Mirota, D., Uneri, A., Schafer, S., Nithiananthan, S., Reh, D., Ishii, M., Gallia, G.,
Taylor, R., Hager, G., Siewerdsen, J.: Evaluation of a system for high-accuracy 3D
image-based registration of endoscopic video to c-arm cone-beam CT for image-
guided skull base surgery. Transactions on Medical Imaging 32 (2013) 1215–1226
9. Mountney, P., Yang, G.Z.: Motion compensated slam for image guided surgery.
In Jiang, T., Navab, N., Pluim, J., Viergever, M., eds.: MICCAI. Volume 6362 of
LNCS. Springer (2010) 496–504
10. Nicolau, S.A., Pennec, X., Soler, L., Buy, X., Gangi, A., Ayache, N., Marescaux,
J.: An augmented reality system for liver thermal ablation: Design and evaluation
on clinical cases. Medical Image Analysis 13(3) (2009) 494–506
11. Teber, D., Guven, S., Simpfendrfer, T., Baumhauer, M., Gven, E.O., Yencilek,
F., Gzen, A.S., Rassweiler, J.: Augmented reality: a new tool to improve surgical
accuracy during laparoscopic partial nephrectomy? preliminary in vitro and in vivo
results. European Urology 56(2) (2009) 332–338
12. Su, L.M., Vagvolgyi, B.P., Agarwal, R., Reiley, C.E., Taylor, R.H., Hager, G.D.:
Augmented reality during robot-assisted laparoscopic partial nephrectomy: toward
real-time 3D-CT to stereoscopic video registration. Urology 73(4) (2009) 896–900
13. Pratt, P., Mayer, E., Vale, J., Cohen, D., Edwards, E., Darzi, A., Yang, G.Z.:
An effective visualisation and registration system for image-guided robotic partial
nephrectomy. Journal of Robotic Surgery 6(1) (2012) 23–31
14. Segal, A., Haehnel, D., Thrun, S.: Generalized-ICP. In: RSS. (2009) 4

Supplementary resources (2)

... Intraoperative approach requires imaging device like Cone Beam Computer Tomography (CBCT) along with external tracking devices with markers. The need of external tracking device with marker itself is a limitation [17]. So, lots of researches were ongoing in preoperative approach. ...
... deformable registration). Rigid registration is a simple transform and doesn't cover dynamic deformation [17,29]. So, to support the dynamic deformation of organs, non-rigid registration approach is the best one and is challenging. ...
... Furthermore, box grid filter is applied to reduce noise and gain down-sampled point cloud. The ICP registration algorithm is applied to get coarse registration based on the down-sampled point cloud and pre-operative model point cloud, which is obtained by stitched surface [17,36]. ...
Full-text available
Augmented reality (AR) based bowel or liver surgery still has not been implemented successfully due to limitations of accurate and proper image registration of uterus and gallbladder during surgery. This research aims to improve target registration error, which helps to navigate through hidden uterus and gallbladder during surgery. Therefore, it will reduce risk of cutting uterus or common bile duct during surgery, which can be fatal and cause devastating effects on the patient. The proposed system integrates the enhanced Coherent Point Drift (CPD) Algorithm with hybrid optimization scheme that incorporates Nelder-Mead simplex and genetic algorithm, to optimize the obtained weight parameter, which in turns improves the target image registration error and processing time of image registration. The system has minimized the target registration error by 0.31 mm in average. It provides a substantial accuracy in terms of target registration error, where the root mean square error is enhanced from 1.28 ± 0.68 mm to 0.97 ± 0.41 mm and improves processing time from 16 ~ 18 ms/frame to 11 ~ 12 ms/frame. The proposed system is focused on improving the accuracy of deformable image registration accuracy of soft tissues and hidden organs, which then helps in proper navigation and localization of the uterus hidden behind bowel and gallbladder hidden behind liver.
... Data transmission may include different communication protocols designed for IoT communication (e.g., [6]). Multifunctional operator platforms for control and monitoring have a variety of application domains, including interactions with industrial devices [2], intelligent instruments [7] and healthcare, including surgery [3,4]. ...
... Following a description of the state-of-the-art in the domain, we first present design principles for MOS implementation based on the solid Tcl/Tk scripting programming stack. We then discuss new possibilities for MOS provided by Augmented Reality (AR) and Virtual Reality (VR), and develop an extension of the MOS design based on AR/VR (MOSAR) and one demo implementation based on a modern gaming platform, Unity which includes extended AR/VR capabilities and other useful characteristics such as physical simulation of real objects and their interactions [1][2][3][4][5]. ...
Full-text available
Design principles of a novel Multifunctional Operation Station (MOS) using Augmented Reality (AR) technology (MOSAR) are proposed in this paper. AR-based design allows more ergonomic remote instrument control in real time in contrast to classical instrument-centered interfaces. Another advantage is its hierarchical software structure including multiple programming interpreters. The MOSAR approach is illustrated with a remote surgical operating station that controls intelligent surgical instruments. The implementation of the Operation Station (MOS) is based on the multiplatform open-source library Tcl/Tk, and an AR extension has been developed on a Unity platform, using Vuforia SDK.
... The extent of inaccuracy varies due to different setups, algorithms and organ of interest, although, for laparoscopic liver navigation, overall inaccuracies, reported in the literature, ranging from 8.7 to 42 mm [45] [47] [26] [24]. Approached utilizing intraoperative CBCT/fluoroscopy and stereo laparoscopic images for registration have been researched and tested although without a TRE measurment during in vivo investigation [27]. The aim of this study is to assess a novel navigation solution for liver laparoscopy which requires a simple, yet accurate, registration procedure, with a less user-dependent update possibility which can work even in the occurrence of large soft tissue deformations. ...
... Technological achievements has led to modernization of the ORs with new intraoperative imaging technologies [25]. The use of intraoperative CBCT, associated to fluoroscopy has been proposed in the literature [31] [27]. Mountney et al. combined non-rigid biomechanically driven registration between pre-operative CT and intra-operative CBCT together with stereo laparoscopic reconstruction [4], and triangulated fluoroscopic images to detect the tip of the laparoscope camera (similarly to [6]) to create an augmented reality system. ...
Full-text available
In laparoscopic liver resection, surgeons conventionally rely on anatomical landmarks detected through a laparoscope, preoperative volumetric images and laparoscopic ultrasound to compensate for the challenges of minimally invasive access. Image guidance using optical tracking and registration procedures is a promising tool, although often undermined by its inaccuracy. This study evaluates a novel surgical navigation solution that can compensate for liver deformations using an accurate and effective registration method. The proposed solution relies on a robotic C-arm to perform registration to preoperative CT/MRI image data and allows for intraoperative updates during resection using fluoroscopic images. Navigation is offered both as a 3D liver model with real-time instrument visualization, as well as an augmented reality overlay on the laparoscope camera view. Testing was conducted through a pre-clinical trial which included four porcine models. Accuracy of the navigation system was measured through two evaluation methods: liver surface fiducials reprojection and a comparison between planned and navigated resection margins. Target Registration Error with the fiducials evaluation shows that the accuracy in the vicinity of the lesion was 3.78±1.89 mm. Resection margin evaluations resulted in an overall median accuracy of 4.44 mm with a maximum error of 9.75 mm over the four subjects. The presented solution is accurate enough to be potentially clinically beneficial for surgical guidance in laparoscopic liver surgery.
... To handle respiratory motion the researchers improve SLAM by including asymmetric respiration model [130]. In [149], the authors improve their work further by using CBCT scan as a connective element to register a preoperative CT and a laparoscopic camera in the global coordinate system. The preoperative data is registered on the CBCT using biomechanically driven approach [165]. ...
Full-text available
The purpose of the work is to find a way to estimate the boundary conditions of the liver. They play an essential role in forming the predictive capacity of the biomechanical model, but are presented mainly by ligaments, vessels, and surrounding organs, the properties of which are "patient specific" and cannot be measured reliably. We propose to present the boundary conditions as nonlinear springs and estimate their parameters. Firstly, we create a generalized initial approximation using the constitutive law available in the literature and a statistical atlas, obtained from a set of models with segmented ligaments. Then, we correct the approximation based on the nonlinear Kalman filtering approach, which assimilates data obtained from a modality during surgical intervention. To assess the approach, we performed experiments for both synthetic and real data. The results show a certain improvement in simulation accuracy for the cases with estimated boundaries.
... There are many applications of both VR and AR in medicine including pre-operative planning [10,11], surgical simulations [12,13], intra-operative guidance [11,[14][15][16], surgical navigation [17][18][19], and trainee education [13,20]. Although AR and VR have been gaining momentum over the past few years, experiences using VR/AR technologies in medicine remain fairly sparse and the role of AR in medicine is yet to be defined. ...
Full-text available
Augmented reality (AR) and virtual reality (VR) are burgeoning technologies that have the potential to greatly enhance patient care. Visualizing patient-specific three-dimensional (3D) imaging data in these enhanced virtual environments may improve surgeons’ understanding of anatomy and surgical pathology, thereby allowing for improved surgical planning, superior intra-operative guidance, and ultimately improved patient care. It is important that radiologists are familiar with these technologies, especially since the number of institutions utilizing VR and AR is increasing. This article gives an overview of AR and VR and describes the workflow required to create anatomical 3D models for use in AR using the Microsoft HoloLens device. Case examples in urologic oncology (prostate cancer and renal cancer) are provided which depict how AR has been used to guide surgery at our institution.
... Trends in medical technologies are novel techniques and methods for study of biological tissue and determination of the presence of an anomaly in the tissue structure [4]. Recent efforts to improve medical technology are related to obtaining information and data in real time and their visualization, as their use makes diagnosis and treatment faster and more reliable [5,6]. An attempt in this direction is based on augmented reality surgical systems and devices that allow doctors to visualize data from diagnostic and surgical procedures, and improves work efficiency [7,8,9]. ...
Conference Paper
Trends in medical technologies are novel techniques and methods for study of biological tissue and determination of the presence of an anomaly in the tissue structure. An attempt in this direction is based on augmented reality surgical systems and devices that allow doctors to visualize data from diagnostic and surgical procedures, and improve work efficiency. The primary aim of this work is to study biological tissue using Augmented Reality surgical systems and to model the behavior of an instrument-organ in an environment close to its real one. To solve these problems the following is done: i) an original tactile model of an instrument for robot-assisted surgery is designed and produced ii) Unified Modeling Language (UML) design of augmented reality surgical system, and iii) an extension program MOSAR ICTPro 2.0 is design as an augmented reality upgrade of the Multifunctional Operation Station (MOS) for laparoscopic manipulations. This work is a continuation of previous research in the field of surgical robotics.
... Active research is going on to minimize the error rate. [51] Computer technology has also changed scenarios, in other areas of medicine. The most notable computer-assisted technology is what is called 3D printing or additive manufacturing. ...
After the revolution of minimal access surgery, a new surgery area is rising: Augmented Surgery. It aims at augmenting surgeon vision, surgeon gesture and surgeon decision, introducing the Augmented Surgery concept. Augmented surgical vision is based on 3D/4D patient-specific modelling. The first step consists in preoperative patient-specific 3D modelling of organs and pathologies from a patient’s medical image (CT or MRI). Preoperatively, the resulting numerical clone can be used to plan and simulate the surgical procedure thanks to user-friendly mobile software. Intraoperative assistance will then consist in Augmented Reality that provides a kind of virtual transparency of the patient. Main limits of this technique are linked to organ movement and deformation between the preoperative image and the intraoperative position and shape. To overcome this limit, the introduction of 3D-medical imaging systems in the Operating Room is then mandatory. The intraoperative medical image is registered with the preoperative image in order to correct organ deformations. By adding the laparoscopic image analysis, it is then possible to compute in real-time the precise location and shape of organs and pathologies. These technologies of Augmented Reality, preoperatively and intraoperatively, can thus be compared to a GPS for surgeon. It is clearly an inevitable step in the progress of Minimally Invasive Surgery before its combination with a robotic system and Artificial Intelligence to develop the next generation of automated surgical robots.
Minimally invasive surgery represents one of the main evolutions of surgical techniques. However, minimally invasive surgery adds difficulty that can be reduced by computer technology. Indeed, from a patient’s medical image (US, CT, or MRI), virtual reality (VR) and augmented reality (AR) can increase the surgeon’s preoperative and intraoperative vision by providing virtual transparency of the patient. VR consists of the 3D visualization of the anatomical or pathological structures visible in the medical image through direct volume rendering or 3D surface rendering of organs and pathologies extracted and modeled from medical images. This 3D modeling can also be used to plan and simulate preoperatively the surgical procedure without imparting risk for the patient. As we will illustrate, VR represents a first major progress of surgery. AR is an extension of VR that consists of fusing the VR view with the real view of the patient in the same position and shape: thus, the patient becomes virtually transparent. To be efficient, the VR view must be perfectly registered onto the real view provided by the surgeon eye (direct AR) or by minimally access camera (indirect AR). Registration can be rigid or nonrigid and can be manual or automatic, the main goal being today to have accurate nonrigid and automatic registration. This registration can and will be increasingly used in link with robotic systems to automatize part of complex or repetitive surgical gestures. In this chapter, we will illustrate how computer-aided surgery will be an inevitable step in the progress of minimally invasive procedures through several applications and results of such innovations.
PurposeThe surface-based registration approach to laparoscopic augmented reality (AR) has clear advantages. Nonrigid point-set registration paves the way for surface-based registration. Among current non-rigid point set registration methods, the coherent point drift (CPD) algorithm is rarely used because of two challenges: (1) volumetric deformation is difficult to predict, and (2) registration from intraoperative visible tissue surface to whole anatomical preoperative model is a “part-to-whole” registration that CPD cannot be applied directly to. We preliminarily applied CPD on surgical navigation for laparoscopic partial nephrectomy (LPN). However, it introduces normalization errors and lacks navigation robustness. This paper presents important advances for more effectively applying CPD to LPN surgical navigation while attempting to quantitatively evaluate the accuracy of CPD-based surgical navigation.Methods First, an optimized volumetric deformation (Op-VD) algorithm is proposed to achieve accurate prediction of volume deformation. Then, a projection-based partial selection method is presented to conveniently and robustly apply the CPD to LPN surgical navigation. Finally, kidneys with different deformations in vitro, phantom and in vivo experiments are performed to evaluate the accuracy and effectiveness of our approach.ResultsThe average root-mean-square error of volume deformation was refined to 0.84 mm. The mean target registration error (TRE) of the surface and inside markers in the in vitro experiments decreased to 1.51 mm and 1.29 mm, respectively. The robustness and precision of CPD-based navigation were validated in phantom and in vivo experiments, and the mean navigation TRE of the phantom experiments was found to be \(1.69 \pm 0.31\) mm.Conclusion Accurate volumetric deformation and robust navigation results can be achieved in AR navigation of LPN by using surface-based registration with CPD. Evaluation results demonstrate the effectiveness of our proposed methods while showing the clinical application potential of CPD. This work has important guiding significance for the application of the CPD in laparoscopic AR.
Conference Paper
Full-text available
Minimally invasive laparoscopic surgery is widely used for the treatment of cancer and other diseases. During the procedure, gas insufflation is used to create space for laparoscopic tools and operation. Insufflation causes the organs and abdominal wall to deform significantly. Due to this large deformation, the benefit of surgical plans, which are typically based on pre-operative images, is limited for real time navigation. In some recent work, intra-operative images, such as cone-beam CT or interventional CT, are introduced to provide updated volumetric information after insufflation. Other works in this area have focused on simulation of gas insufflation and exploited only the pre-operative images to estimate deformation. This paper proposes a novel registration method for pre- and intra-operative 3D image fusion for laparoscopic surgery. In this approach, the deformation of pre-operative images is driven by a biomechanical model of the insufflation process. The proposed method was validated by five synthetic data sets generated from clinical images and three pairs of in vivo CT scans acquired from two pigs, before and after insufflation. The results show the proposed method achieved high accuracy for both the synthetic and real insufflation data.
Full-text available
Since its introduction in the early 1990s, the Iterative Closest Point (ICP) algorithm has become one of the most well-known methods for geometric alignment of 3D models. Given two roughly aligned shapes represented by two point sets, the algorithm iteratively establishes point correspondences given the current alignment of the data and computes a rigid transformation accordingly. From a statistical point of view, however, it implicitly assumes that the points are observed with isotropic Gaussian noise. In this paper, we show that this assumption may lead to errors and generalize the ICP such that it can account for anisotropic and inhomogenous localization errors. We 1) provide a formal description of the algorithm, 2) extend it to registration of partially overlapping surfaces, 3) prove its convergence, 4) derive the required covariance matrices for a set of selected applications, and 5) present means for optimizing the runtime. An evaluation on publicly available surface meshes as well as on a set of meshes extracted from medical imaging data shows a dramatic increase in accuracy compared to the original ICP, especially in the case of partial surface registration. As point-based surface registration is a central component in various applications, the potential impact of the proposed method is high.
Conference Paper
Full-text available
The effectiveness and clinical benefits of image guided surgery are well established for procedures where there is manageable tissue motion. In minimally invasive cardiac, gastrointestinal, or abdominal surgery, large scale tissue deformation prohibits accurate registration and fusion of pre- and intraoperative data. Vision based techniques such as structure from motion and simultaneous localization and mapping are capable of recovering 3D structure and laparoscope motion. Current research in the area generally assumes the environment is static, which is difficult to satisfy in most surgical procedures. In this paper, a novel framework for simultaneous online estimation of laparoscopic camera motion and tissue deformation in a dynamic environment is proposed. The method only relies on images captured by the laparoscope to sequentially and incrementally generate a dynamic 3D map of tissue motion that can be co-registered with pre-operative data. The theoretical contribution of this paper is validated with both simulated and ex vivo data. The practical application of the technique is further demonstrated on in vivo procedures.
Conference Paper
We present steps toward the first real-time system for computing and visualising 3D surfaces viewed in live monocular laparoscopy video. Our method is based on estimating 3D shape using shading and specularity information, and seeks to push current Shape from Shading (SfS) boundaries towards practical, reliable reconstruction. We present an accurate method to model any laparoscope's light source, and a highly-parallelised SfS algorithm that outperforms the fastest current method. We give details of its GPU implementation that facilitates realtime performance of an average frame-rate of 23fps. Our system also incorporates live 3D visualisation with virtual stereoscopic synthesis. We have evaluated using real laparoscopic data with ground-truth, and we present the successful in-vivo reconstruction of the human uterus. We however draw the conclusion that the shading cue alone is insufficient to reliably handle arbitrary laparoscopic images.
A minimal access approach to partial nephrectomy has historically been under-utilized, but is now becoming more popular with the growth of robot-assisted laparoscopy. One of the criticisms of minimal access partial nephrectomy is the loss of haptic feedback. Augmented reality operating environments are forecast to play a major enabling role in the future of minimal access partial nephrectomy by integrating enhanced visual information to supplement this loss of haptic sensation. In this article, we systematically examine the current status of augmented reality in partial nephrectomy by identifying existing research challenges and exploring future agendas for this technology to achieve wider clinical translation.
Robotic partial nephrectomy is presently the fastest-growing robotic surgical procedure, and in comparison to traditional techniques it offers reduced tissue trauma and likelihood of post-operative infection, while hastening recovery time and improving cosmesis. It is also an ideal candidate for image guidance technology since soft tissue deformation, while still present, is localised and less problematic compared to other surgical procedures. This work describes the implementation and ongoing development of an effective image guidance system that aims to address some of the remaining challenges in this area. Specific innovations include the introduction of an intuitive, partially automated registration interface, and the use of a hardware platform that makes sophisticated augmented reality overlays practical in real time. Results and examples of image augmentation are presented from both retrospective and live cases. Quantitative analysis of registration error verifies that the proposed registration technique is appropriate for the chosen image guidance targets.
The safety of endoscopic skull base surgery can be enhanced by accurate navigation in preoperative CT or, more recently, intraoperative cone-beam CT (CBCT). The ability to register real-time endoscopic video with CBCT offers an additional advantage by rendering information directly within the visual scene to account for intraoperative anatomical change. However, tracker localization error (12 mm) limits the accuracy with which video and tomographic images can be registered. This paper reports the first implementation of image-based video-CBCT registration, conducts a detailed quantitation of the dependence of registration accuracy on system parameters, and demonstrates improvement in registration accuracy achieved by the image-based approach. Performance was evaluated as a function of parameters intrinsic to the image-based approach, including system geometry, CBCT image quality, and computational runtime. Overall system performance was evaluated in a cadaver study simulating transsphenoidal skull base tumor excision. Results demonstrated significant improvement (p < 0.001) in registration accuracy with a mean reprojection distance error of 1.28 mm for the image-based approach versus 1.82 mm for the conventional tracker-based method. Image-based registration was highly robust against CBCT image quality factors of noise and resolution, permitting integration with low-dose intraoperative CBCT.
Conference Paper
In this paper we combine the Iterative Closest Point ('ICP') and 'point-to-plane ICP' algorithms into a single probabilistic framework. We then use this framework to model locally planar surface structure from both scans instead of just the "model" scan as is typically done with the point-to-plane method. This can be thought of as 'plane-to-plane'. The new approach is tested with both simulated and real-world data and is shown to outperform both standard ICP and point-to-plane. Furthermore, the new approach is shown to be more robust to incorrect correspondences, and thus makes it easier to tune the maximum match distance parameter present in most variants of ICP. In addition to the demonstrated performance improvement, the proposed model allows for more expressive probabilistic models to be incorporated into the ICP framework. While maintaining the speed and simplicity of ICP, the Generalized-ICP could also allow for the addition of outlier terms, measurement noise, and other probabilistic techniques to increase robustness. I. INTRODUCTION
Conference Paper
We present our novel Adaptive Multi-Affine (AMA) feature-matching algorithm that finds correspondences between two views of the same non-planar object. The proposed method only uses monocular images to robustly match clusters of 2-D features according to their relative position on the object surface; finally, AMA adaptively finds the best number of clusters that maximizes the number of matching features. We use AMA to recover a feature tracker from failure (e.g., loss of points due to occlusions or deformations), by robustly matching the features in the images before and after such events. This is paramount in Augmented-Reality (AR) systems for Minimally- Invasive Surgery (MIS) to cope for frequent occlusions and organ deformations that can cause the tracked image-points to drastically reduce (or even disappear) in the current video. We validated our approach on a large set of MIS videos of partial- nephrectomy surgery; AMA achieves an increased number of matches, as well as a reduced feature-matching error when compared to state-of-the-art method. I. INTRODUCTION