Conference Paper

Non Rigid Registration of 3D Images to Laparoscopic Video for Image Guided Surgery

Authors:
  • Odin Vision
To read the full-text of this research, you can request a copy directly from the authors.

Abstract

Image guidance and the visualization of sub surface structures during laparoscopic procedures have the potential to change the current capabilities of surgery. Increased target localization accuracy and the identification of critical structures can reduce resection margins , procedure time and tissue trauma while simplifying procedures and enabling new functional capabilities. Image guidance requires the registration of 3D images to the laparoscopic video. Tissue deformation and lack of cross modality landmarks make this challenging. Registration can be performed by aligning the 3D image to a surface reconstructed from stereo laparoscopic images. Current research is focused on creating more generic stereo reconstruction techniques and rigid registration methods. This paper proposes a novel stereo reconstruction approach which exploits prior knowledge of patient specific organ models and outlier robust non rigid registration. The approach is validated on phantom data and the practical application of the reconstruction is demonstrated on in vivo data.

No full-text available

Request Full-text Paper PDF

To read the full-text of this research,
you can request a copy directly from the authors.

... In recent years, nonrigid registration methods have been developed for the laparoscopic environment. Allan et al. 21 developed a nonrigid registration method to stereo-reconstructed laparoscopic surfaces using coherent point drift. 22 Suwelack et al. 23 employed a model that mixed elastic mechanical response with electrostatic attractive forces to match the shapes of preoperative and intraoperative models of the liver. ...
... Another variant reported by Reichard et al. 25 projected spring force boundary conditions from a stereo-reconstructed depth map onto a biomechanical model. However, these approaches either fail to use mechanics-based models 18,21,22 or treat deformation correction through direct application of digitized intraoperative surfaces as boundary conditions. 19,20,[23][24][25] While the former methods do not accurately model deformation beyond the immediate neighborhood of intraoperative data, the latter methods may not adequately align regions with poor data localization and can have unfavorable responses to untreated sources of intraoperative surface noise. ...
Article
Laparoscopic liver surgery is challenging to perform due to a compromised ability of the surgeon to localize subsurface anatomy in the constrained environment. While image guidance has the potential to address this barrier, intraoperative factors, such as insufflation and variable degrees of organ mobilization from supporting ligaments, may generate substantial deformation. The severity of laparoscopic deformation in humans has not been characterized, and current laparoscopic correction methods do not account for the mechanics of how intraoperative deformation is applied to the liver. We first measure the degree of laparoscopic deformation at two insufflation pressures over the course of laparoscopic-to-open conversion in 25 patients. With this clinical data alongside a mock laparoscopic phantom setup, we report a biomechanical correction approach that leverages anatomically load-bearing support surfaces from ligament attachments to iteratively reconstruct and account for intraoperative deformations. Laparoscopic deformations were significantly larger than deformations associated with open surgery, and our correction approach yielded subsurface target error of [Formula: see text] and surface error of [Formula: see text] using only sparse surface data with realistic surgical extent. Laparoscopic surface data extents were examined and found to impact registration accuracy. Finally, we demonstrate viability of the correction method with clinical data.
... Un autre type de recalage désigné parfois lui aussi comme mono-image peut aussi être considéré. Il s'agit de méthodes utilisant des données provenant d'une caméra stéréo pour se recaler [176,4]. De l'information sur la profondeur de la scène est ainsi accessible et utilisable pour des contraintes supplémentaires à appliquer sur le modèle à recaler. ...
Thesis
Cette thèse concerne les techniques de recalage déformable de données pré-opératoires dans la scène peropératoire en tant qu’étape indispensable à la réalisation de réalité augmentée pour la chirurgie abdominale. De telles techniques sont ainsi discutées, de même que les méthodologies d’évaluation associées à ces dernières.Deux contextes sont considérés : le recalage pour la chirurgie coelioscopique assistée par ordinateur et le recalage postural de patient sur la table d’opération. Pour ces deux contextes, les besoins auxquels doivent répondre les algorithmes de recalage considérés sont discutés, ainsi que les principales limitations des solutions existantes.Des algorithmes réalisés au cours de cette thèse, permettant de répondre à ces besoins sont ainsi proposés et discutés. Une attention toute particulière est alors accordée à leur évaluation. Différents jeux de données permettant une évaluation quantitative de la précision des algorithmes de recalage, créés eux aussi durant cette thèse, et rendu publics, sont ainsi présentés. De telles données sont extrêmement importantes car elles répondent à un manque de données standardisées permettant d’évaluer l’erreur de recalage de façon quantitative, et ainsi de comparer les différents algorithmes. La modélisation de l’éclairage de la scène coelioscopique, permettant d’extraire des contraintes fortes sur les données à recaler et la surface de l’organe observé, et ainsi d’être utilisée pour contraindre ces problématiques de recalage, est aussi discutée. Ce manuscrit est séparé en sept parties. La première traite du contexte de la thèse. La chirurgie mini-invasive est présentée ainsi que différents problèmes de vision par ordinateur généraux qui, une fois appliqués au contexte médical permettent de définir la chirurgie assistée par ordinateur. La seconde partie traite des prérequis à la lecture de la thèse. Le prétraitement des données pré-opératoires et per-opératoires, avant utilisation par les algorithmes de recalage présentés,est ainsi discuté. La troisième partie correspond au recalage de données hépatiques en coelioscopie, et de l’évaluation associée à cette méthode. La quatrième partie correspond quant à elle à la problématique du recalage postural. La cinquième partie propose une modélisation de l’éclairage en coelioscopie pouvant être utilisée pour obtenir des contraintes fortes entre la surface observée et les images coelioscopiques. La sixièmepartie propose une utilisation des modèles de lumière discutés dans la partie précédente afin de raffiner et densifier des reconstructions de la scène coelioscopique. Enfin, la septième et dernière partie correspond à nos conclusions vis-à-vis des problématiques abordées au cours de la thèse, et aux travaux futurs.
... A global convex optimization based method for real time dense reconstruction is presented in. 10 The proposed method overcomes the staircasing effect, but computing speed can be degraded for high resolution images as a result of the exhaustive search step of the approach. Allan et al. 15 regard the disparity estimation as a labeling problem on a Markov Random Field (MRF) and optimize it by a graph-cut algorithm which results in staircasing effect. ...
Conference Paper
Full-text available
Time-of-Flight (ToF) sensors have become a considerable alternative to conventional surface acquisition techniques such as laser range scanning and stereo vision. Application of ToF cameras for the purpose of intra-operative registration requires matching of the noisy surfaces generated from ToF range data onto pre-interventionally acquired high-resolution surfaces. The contribution of this paper is two-fold: Firstly, we present a novel method for fine rigid registration of noisy ToF data with high-resolution surface meshes taking into account both, the noise characteristics of ToF cameras and the resolution of the target mesh. Secondly, we introduce an evaluation framework for assessing the performance of ToF registration methods based on physically realistic ToF range data generated from a virtual scence. According to experiments within the presented evaluation framework, the proposed method outperforms the standard ICP algorithm with respect to correspondence search and transformation computation, leading to a decrease in the target registration error (TRE) of more than 70%.
Conference Paper
Full-text available
We propose an image-based motion tracking algorithm that can be used with stereo endoscopie and microscope systems. The tracking problem is considered to be a time-varying optimization of a parametric function describing the disparity map. This algorithm could be used as part of a virtual stabilization system that can be employed to compensate residual motion of the heart during robot-assisted off-pump coronary artery bypass surgery (CABG). To test the appropriateness of our methods for this application, we processed an image sequence of a beating pig heart obtained by the stereo endoscope used in the da Vinci robotic surgery system. The tracking algorithm was able to detect the beating of the heart itself as well as the respiration of the lungs.
Article
Full-text available
The trend toward minimally invasive surgical interventions has created new challenges for visualization during surgical procedures. However, at the same time, the introduction of high-definition digital endoscopy offers the opportunity to apply methods from computer vision to provide visualization enhancements such as anatomic reconstruction, surface registration, motion tracking, and augmented reality. This review provides a perspective on this rapidly evolving field. It first introduces the clinical and technical background necessary for developing vision-based algorithms for interventional applications. It then discusses several examples of clinical interventions where computer vision can be applied, including bronchoscopy, rhinoscopy, transnasal skull-base neurosurgery, upper airway interventions, laparoscopy, robotic-assisted surgery, and Natural Orifice Transluminal Endoscopic Surgery (NOTES). It concludes that the currently reported work is only the beginning. As the demand for minimally invasive procedures rises, computer vision in surgery will continue to advance through close interdisciplinary work between interventionists and engineers.
Article
Full-text available
Point set registration is a key component in many computer vision tasks. The goal of point set registration is to assign correspondences between two sets of points and to recover the transformation that maps one point set to the other. Multiple factors, including an unknown nonrigid spatial transformation, large dimensionality of point set, noise, and outliers, make the point set registration a challenging problem. We introduce a probabilistic method, called the Coherent Point Drift (CPD) algorithm, for both rigid and nonrigid point set registration. We consider the alignment of two point sets as a probability density estimation problem. We fit the Gaussian mixture model (GMM) centroids (representing the first point set) to the data (the second point set) by maximizing the likelihood. We force the GMM centroids to move coherently as a group to preserve the topological structure of the point sets. In the rigid case, we impose the coherence constraint by reparameterization of GMM centroid locations with rigid parameters and derive a closed form solution of the maximization step of the EM algorithm in arbitrary dimensions. In the nonrigid case, we impose the coherence constraint by regularizing the displacement field and using the variational calculus to derive the optimal transformation. We also introduce a fast algorithm that reduces the method computation complexity to linear. We test the CPD algorithm for both rigid and nonrigid transformations in the presence of noise, outliers, and missing points, where CPD shows accurate results and outperforms current state-of-the-art methods.
Conference Paper
Full-text available
The recovery of 3D tissue structure and morphology during robotic assisted surgery is an important step towards accurate deployment of surgical guidance and control techniques in minimally invasive therapies. In this article, we present a novel stereo reconstruction algorithm that propagates disparity information around a set of candidate feature matches. This has the advantage of avoiding problems with specular highlights, occlusions from instruments and view dependent illumination bias. Furthermore, the algorithm can be used with any feature matching strategy allowing the propagation of depth in very disparate views. Validation is provided for a phantom model with known geometry and this data is available online in order to establish a structured validation scheme in the field. The practical value of the proposed method is further demonstrated by reconstructions on various in vivo images of robotic assisted procedures, which are also available to the community. Additional material can be found at http://ubimon.doc.ic.ac.uk/dvs/m857.html .
Conference Paper
Full-text available
The use of physically-based models combined with image constraints for intraoperative guidance is important for surgical procedures that involve large-scale tissue deformation. A biomechanical model of tissue deformation is described in which surface positional constraints and internally generated forces are derived from endoscopic images and preoperative 4D CT data, respectively. Considering cardiac motion, a novel technique is presented which minimises the average registration error over one or more complete cycles. Features tracked in the stereo video stream provide surface constraints, and an inverse finite element simulation is presented which allows internal forces to be recovered from known preoperative displacements. The accuracy of surface texture, segmented mesh and volumetrically rendered overlays is evaluated with detailed phantom experiments. Results indicate that by combining preoperative and intraoperative images in this manner, accurate intraoperative tissue deformation modelling can be achieved.
Conference Paper
Full-text available
The design of physiological motion compensation systems for robotic-assisted cardiac Minimally Invasive Surgery (MIS) is a challenging research topic. In this domain, vision-based techniques have proven to be a practical way to retrieve the motion of the beating heart. However due to the complexity of the heart motion and its surface characteristics, efficient tracking is still a complicated task. In this paper, we propose an algorithm for tracking the 3D motion of the beating heart, based on a Thin-Plate Splines (TPS) parametric model. The novelty of our approach lies in that no explicit matching between the stereo camera images is required and consequently no intermediate steps such as rectification are needed. Experiments conducted on ex-vivo and in-vivo tissue show the effectiveness of the proposed algorithm for tracking surfaces undergoing complex deformations.
Conference Paper
Full-text available
One of the problems tightly linked to endoscopic surgery is the fact that, because of the narrow field of view, it is sometimes quite difficult to locate the objects that can be seen through the endoscope. This is especially true in cardiac surgery, where it is difficult to not confuse two coronary arteries on a beating heart. We propose a methodology to achieve coronary localisation by augmented reality on a robotized stereoscopic endoscope. The method we propose involves five steps: making a time-variant 3D model of the beating heart using coronarography and CT-scan or MRI, calibrating the stereoscopic endoscope, reconstructing the 3D operating field, registering the operating field surface with the 3D heart model, and adding information on the endoscopic images by augmented reality. The da Vinci<sup>TM</sup> surgical system was used for our first experiments with the Cardiac Surgery team at Hopital Europeen Georges Pompidou, Paris, France
Article
Cambridge Core - Computer Graphics, Image Processing and Robotics - Computer Vision - by Simon J. D. Prince
Conference Paper
Shape-from-Shading (SfS) is one of the fundamental techniques to recover depth from a single view. Such a method has shown encouraging but limited results in laparoscopic surgery due to the complex reflectance properties of the organ tissues. On the other hand, Template-Based Deformable-Shape-from-Motion (DSfM) has been recently used to recover a coarse 3D shape in laparoscopy. We propose to combine both geometric and photometric cues to robustly reconstruct 3D human organs. Our method is dubbed Deformable-Shape-from-Motion-and-Shading (DSfMS). It tackles the limits of classical SfS and DSfM methods: First the photometric template is reconstructed using rigid SfM (Shape-from-Motion) while the surgeon is exploring – but not deforming – the peritoneal environment. Second a rough 3D deformed shape is computed using a recent method for elastic surface from a single laparoscopic image. Third a fine 3D deformed shape is recovered using shading and specularities. The proposed approach has been validated on both synthetic data and in-vivo laparoscopic videos of a uterus. Experimental results illustrate its effectiveness compared to SfS and DSfM.
Conference Paper
Range imaging modalities, such as time-of-flight cameras (ToF), are becoming very popular for the acquisition of intra-operative data, which can be used for registering the patient's anatomy with pre-operative data, such as 3D images generated by computed tomographies (CT) or magnetic resonance imaging (MRI). However, due to the distortions that appear because of the different acquisition principles of the input surfaces, the noise, and the deformations that may occur in the intra-operative environment, we face different surface properties for points lying on the same anatomical locations and unreliable feature points detection, which are crucial for most surface matching algorithms. In order to overcome these issues, we present a method for automatically finding correspondences between surfaces that searches for minimally deformed configurations. For this purpose, an error metric that expresses the reliability of a correspondence set based on its spatial configuration is employed. The registration error is minimized by a combinatorial analysis through search-trees. Our method was evaluated with real and simulated ToF and CT data, and showed to be reliable for the registration of partial multi-modal surfaces with noise and distortions.
Article
Time-of-Flight (ToF) sensors have become a considerable alternative to conventional surface acquisition techniques such as laser range scanning and stereo vision. Application of ToF cameras for the purpose of intra-operative registration requires matching of the noisy surfaces generated from ToF range data onto pre-interventionally acquired high-resolution surfaces. The contribution of this paper is twofold: Firstly, we present a novel method for fine rigid registration of noisy ToF data with high-resolution surface meshes taking into account both, the noise characteristics of ToF cameras and the resolution of the target mesh. Secondly, we introduce an evaluation framework for assessing the performance of ToF registration methods based on physically realistic ToF range data generated from a virtual scence. According to experiments within the presented evaluation framework, the proposed method outperforms the standard ICP algorithm with respect to correspondence search and transformation computation, leading to a decrease in the target registration error (TRE) of more than 70%.
Article
The current protocol for image guidance in open abdominal liver tumor removal surgeries involves a rigid registration between the patient's operating room space and the pre-operative diagnostic image-space. Systematic studies have shown that the liver can deform up to 2 cm during surgeries in a non-rigid fashion thereby compromising the accuracy of these surgical navigation systems. Compensating for intra-operative deformations using mathematical models has shown promising results. In this work, we follow up the initial rigid registration with a computational approach that is geared towards minimizing the residual closest point distances between the un-deformed pre-operative surface and the rigidly registered intra-operative surface. We also use a surface Laplacian equation based filter that generates a realistic deformation field. Preliminary validation of the proposed computational framework was performed using phantom experiments and clinical trials. The proposed framework improved the rigid registration errors for the phantom experiments on average by 43%, and 74% using partial and full surface data, respectively. With respect to clinical data, it improved the closest point residual error associated with rigid registration by 54% on average for the clinical cases. These results are highly encouraging and suggest that computational models can be used to increase the accuracy of image-guided open abdominal liver tumor removal surgeries.
Article
To investigate a markerless tracking system for real-time stereo-endoscopic visualization of preoperative computed tomographic imaging as an augmented display during robot-assisted laparoscopic partial nephrectomy. Stereoscopic video segments of a patient undergoing robot-assisted laparoscopic partial nephrectomy for tumor and another for a partial staghorn renal calculus were processed to evaluate the performance of a three-dimensional (3D)-to-3D registration algorithm. After both cases, we registered a segment of the video recording to the corresponding preoperative 3D-computed tomography image. After calibrating the camera and overlay, 3D-to-3D registration was created between the model and the surgical recording using a modified iterative closest point technique. Image-based tracking technology tracked selected fixed points on the kidney surface to augment the image-to-model registration. Our investigation has demonstrated that we can identify and track the kidney surface in real time when applied to intraoperative video recordings and overlay the 3D models of the kidney, tumor (or stone), and collecting system semitransparently. Using a basic computer research platform, we achieved an update rate of 10 Hz and an overlay latency of 4 frames. The accuracy of the 3D registration was 1 mm. Augmented reality overlay of reconstructed 3D-computed tomography images onto real-time stereo video footage is possible using iterative closest point and image-based surface tracking technology that does not use external navigation tracking systems or preplaced surface markers. Additional studies are needed to assess the precision and to achieve fully automated registration and display for intraoperative use.
Real-time stereo reconstruction in robotically assisted minimally invasive surgery
  • D Stoyanov
D. Stoyanov et al, "Real-time stereo reconstruction in robotically assisted minimally invasive surgery," MICCAI, vol. 13, no. Pt 1, pp. 275-282, 2010.
Vision-based navigation in image-guided interventions
  • D J Mirota
  • DJ Mirota