Content uploaded by Heinz Handels
Author content
All content in this area was uploaded by Heinz Handels
Content may be subject to copyright.
A preview of the PDF is not available
Automatic Landmark Detection and Non-linear Landmark-and Surface-based Registration of Lung CT Images
Abstract
Registration of the lungs in thoracic CT images is required in many fields of application in medical imaging, for example for motion estimation or analysis of pathology progression. In this paper, we present a feature-based registration approach for lung CT images based on lung surfaces and automatically detected inner-lung landmark pairs. In a first step, an affine pre-registration of surface models generated from lung segmentation masks is performed. Following, an au-tomatic algorithm is used for the landmark identification and landmark transfer between fixed and moving image. The result of this landmark detection and the result of a non-linear diffusion-based surface registra-tion are used to generate the final deformation field by thin-plate-splines interpolation. The approach is evaluated based on 20 CT scans provided for the EM-PIRE10 study for pulmonary image registration. In this study, the ap-proach reached a final placement of 21 out of 34 participating algorithms. The evaluation shows a very good alignment of lung boundaries in con-trast to a disappointing matching of inner lung structures, although land-mark pairs were detected correctly with the automatic algorithm.
... Deformable registration with an initial keypoint correspondence search has been considered in previous work [21]- [24]. However, the matching of each keypoint has been performed independently, which thus generally resulted in a considerable number of outliers. ...
... Keypoints are widely used in image recognition [39] and multi-view scene reconstruction [40]. In order to cope with large motion, searching for sparse keypoint matches has been proposed in previous work [21]- [23]. However, these approaches have in common that an unconstrained optimum is found for each keypoint independently, potentially leading to outliers. ...
... 1) Sparse Keypoint Extraction: As suggested by previous approaches for interest point localization in lung CT scans [21], [23], a Förstner operator [41] is applied to find a sparse set of distinctive keypoints K ⊂ R 3 in the fixed image. The spatial gradients of the fixed scan ∇F are smoothed with a Gaussian kernel G σ , which yields a distinctiveness volume ...
We present a novel algorithm for the registration of pulmonary CT scans. Our method is designed for large respiratory motion by integrating sparse keypoint correspondences into a dense continuous optimization framework. The detection of keypoint correspondences enables robustness against large deformations by jointly optimizing over a large number of potential discrete displacements, whereas the dense continuous registration achieves subvoxel alignment with smooth transformations. Both steps are driven by the same normalized gradient fields data term. We employ curvature regularization and a volume change control mechanism to prevent foldings of the deformation grid and restrict the determinant of the Jacobian to physiologically meaningful values. Keypoint correspondences are integrated into the dense registration by a quadratic penalty with adaptively determined weight. Using a parallel matrix-free derivative calculation scheme, a runtime of about 5 min was realized on a standard PC. The proposed algorithm ranks first in the EMPIRE10 challenge on pulmonary image registration. Moreover, it achieves an average landmark distance of 0.82 mm on the DIR-Lab COPD database, thereby improving upon the state of the art in accuracy by 15%. Our algorithm is the first to reach the inter-observer variability in landmark annotation on this dataset.
... Deformable registration with an initial keypoint correspon- dence search has been considered in previous work [21]- [24]. However, the matching of each keypoint has been performed independently, which thus generally resulted in a considerable number of outliers. ...
... Keypoints are widely used in image recognition [39] and multi-view scene reconstruction [40]. In order to cope with large motion, searching for sparse keypoint matches has been proposed in previous work [21]- [23]. However, these approaches have in common that an unconstrained optimum is found for each keypoint independently, potentially leading to outliers. ...
... 1) Sparse Keypoint Extraction: As suggested by previ- ous approaches for interest point localization in lung CT scans [21], [23], a Förstner operator [41] is applied to find a sparse set of distinctive keypoints K ⊂ R 3 in the fixed image. The spatial gradients of the fixed scan ∇F are smoothed with a Gaussian kernel G σ , which yields a distinctiveness volume ...
... Large deformation lung registration: Both iconic and geometric approaches have often been found to yield relative large residual errors for large motion lung registration (forced inhale-to-exhale): e.g. 4.68 mm for the discrete optimization algorithm in [7] applied to the DIR-lab COPD data [5] and 3.61 mm (on the inhale-exhale pairs of the EMPIRE10 challenge) for [6], which used both keypoint-and intensity-based information. Learning the alignment of such difficult data appears to be so far impossible with intensity-driven CNN approaches that already struggle with more shallow breathing in 4D-CT [14]. ...
... learning shows another small but significant improvement to 4.3±3.6 mm. These alignment errors cannot be directly compared to the large variety of image-and feature-based registration algorithms that reached 3.6 mm[6], 4.7 mm[7] or 1.1 mm ...
Deformable registration continues to be one of the key challenges in medical image analysis. While iconic registration methods have started to benefit from the recent advances in medical deep learning, the same does not yet apply for the registration of point sets, e.g. registration based on surfaces, keypoints or landmarks. This is mainly due to the restriction of the convolution operator in modern CNNs to densely gridded input. However, with the newly developed methods from the field of geometric deep learning suitable tools are now emerging, which enable powerful analysis of medical data on irregular domains. In this work, we present a new method that enables the learning of regularized feature descriptors with dynamic graph CNNs. By incorporating the learned geometric features as prior probabilities into the well-established coherent point drift (CPD) algorithm, formulated as differentiable network layer, we establish an end-to-end framework for robust registration of two point sets. Our approach is evaluated on the challenging task of aligning keypoints extracted from lung CT scans in inhale and exhale states with large deformations and without any additional intensity information. Our results indicate that the inherent geometric structure of the extracted keypoints is sufficient to establish descriptive point features, which yield a significantly improved performance and robustness of our registration framework.
... The resulting transformed keypoints can then be compared with the keypoints of the fixed image to create a loss. This keypoint supervision has been utilized in optimization-based registration methods to improve performance, as demonstrated in a number of studies (Ehrhardt et al., 2010;Polzin et al., 2013;Rühaak et al., 2017;Heinrich et al., 2015;Fischer and Modersitzki, 2003a). Hering et al. (2021) were the first to incorporate keypoint supervision into a DNN framework by comparing the MSE between the transformed and target keypoints, which resulted in a substantial improvement in the target registration error of the keypoint. ...
Over the past decade, deep learning technologies have greatly advanced the field of medical image registration. The initial developments, such as ResNet-based and U-Net-based networks, laid the groundwork for deep learning-driven image registration. Subsequent progress has been made in various aspects of deep learning-based registration, including similarity measures, deformation regularizations, and uncertainty estimation. These advancements have not only enriched the field of deformable image registration but have also facilitated its application in a wide range of tasks, including atlas construction, multi-atlas segmentation, motion estimation, and 2D-3D registration. In this paper, we present a comprehensive overview of the most recent advancements in deep learning-based image registration. We begin with a concise introduction to the core concepts of deep learning-based image registration. Then, we delve into innovative network architectures, loss functions specific to registration, and methods for estimating registration uncertainty. Additionally, this paper explores appropriate evaluation metrics for assessing the performance of deep learning models in registration tasks. Finally, we highlight the practical applications of these novel techniques in medical imaging and discuss the future prospects of deep learning-based image registration.
... The results also showed that the additional guidance by automatic landmark correspondences improved the performance of DIR irrespective of the variance in the number, spatial matching errors, and spatial distribution of the automatic landmarks in both simulated as well as clinical deformations test sets. These findings are in line with the existing literature on the use of automatic landmarks for the improvement of DIR in chest CT, 11,12,35 head and neck CT, 36 retinal images, 13 and brain MRI images. 14,37 A study on DIR of thoracic CT scans 38 reported that automatic landmarks-based optimization of the regularization parameter reduced the TRE of expert landmarks on average by 0.07 mm. ...
Purpose:
Deformable image registration (DIR) can benefit from additional guidance using corresponding landmarks in the images. However, the benefits thereof are largely understudied, especially due to the lack of automatic landmark detection methods for three-dimensional (3D) medical images.
Approach:
We present a deep convolutional neural network (DCNN), called DCNN-Match, that learns to predict landmark correspondences in 3D images in a self-supervised manner. We trained DCNN-Match on pairs of computed tomography (CT) scans containing simulated deformations. We explored five variants of DCNN-Match that use different loss functions and assessed their effect on the spatial density of predicted landmarks and the associated matching errors. We also tested DCNN-Match variants in combination with the open-source registration software Elastix to assess the impact of predicted landmarks in providing additional guidance to DIR.
Results:
We tested our approach on lower abdominal CT scans from cervical cancer patients: 121 pairs containing simulated deformations and 11 pairs demonstrating clinical deformations. The results showed significant improvement in DIR performance when landmark correspondences predicted by DCNN-Match were used in the case of simulated ( p = 0 e 0 ) as well as clinical deformations ( p = 0.030 ). We also observed that the spatial density of the automatic landmarks with respect to the underlying deformation affect the extent of improvement in DIR. Finally, DCNN-Match was found to generalize to magnetic resonance imaging scans without requiring retraining, indicating easy applicability to other datasets.
Conclusions:
DCNN-match learns to predict landmark correspondences in 3D medical images in a self-supervised manner, which can improve DIR performance.
... For conventional image registration, previous work (e.g. Ehrhardt et al. (2010); Polzin et al. (2013); Rühaak et al. (2017)) has shown that the integration of sparse keypoints during the optimization of the deformation field yields better registration results. In contrast to conventional registration approaches, keypoints can be integrated into the loss function and are therefore, similar to the segmentation masks for the mask penalty, only needed for training but not during inference. ...
Deep-learning-based registration methods emerged as a fast alternative to conventional registration methods. However, these methods often still cannot achieve the same performance as conventional registration methods, because they are either limited to small deformation or they fail to handle a superposition of large and small deformations without producing implausible deformation fields with foldings inside. In this paper, we identify important strategies of conventional registration methods for lung registration and successfully developed the deep-learning counterpart. We employ a Gaussian-pyramid-based multilevel framework that can solve the image registration optimization in a coarse-to-fine fashion. Furthermore, we prevent foldings of the deformation field and restrict the determinant of the Jacobian to physiologically meaningful values by combining a volume change penalty with a curvature regularizer in the loss function. Keypoint correspondences are integrated to focus on the alignment of smaller structures. We perform an extensive evaluation to assess the accuracy, the robustness, the plausibility of the estimated deformation fields, and the transferability of our registration approach. We show that it archives state-of-the-art results on the COPDGene dataset compared to the challenge winning conventional registration method with much shorter execution time.
Deep-learning-based registration methods emerged as a fast alternative to conventional registration methods. However, these methods often still cannot achieve the same performance as conventional registration methods because they are either limited to small deformation or they fail to handle a superposition of large and small deformations without producing implausible deformation fields with foldings inside.
In this paper, we identify important strategies of conventional registration methods for lung registration and successfully developed the deep-learning counterpart. We employ a Gaussian-pyramid-based multilevel framework that can solve the image registration optimization in a coarse-to-fine fashion. Furthermore, we prevent foldings of the deformation field and restrict the determinant of the Jacobian to physiologically meaningful values by combining a volume change penalty with a curvature regularizer in the loss function. Keypoint correspondences are integrated to focus on the alignment of smaller structures.
We perform an extensive evaluation to assess the accuracy, the robustness, the plausibility of the estimated deformation fields, and the transferability of our registration approach. We show that it achieves state-of-the-art results on the COPDGene dataset compared to conventional registration method with much shorter execution time. In our experiments on the DIRLab exhale to inhale lung registration, we demonstrate substantial improvements (TRE below 1.2 mm) over other deep learning methods. Our algorithm is publicly available at https://grand-challenge.org/algorithms/deep-learning-based-ct-lung-registration/.
Accurate registration of lung computed tomography (CT) image is a significant task in thorax image analysis. Recently deep learning-based medical image registration methods develop fast and achieve promising performance on accuracy and speed. However, most of them learned the deformation field through intensity similarity but ignored the importance of aligning anatomical landmarks (e.g., the branch points of airway and vessels). Accurate alignment of anatomical landmarks is essential for obtaining anatomically correct registration. In this work, we propose landmark constrained learning with a convolutional neural network (CNN) for lung CT registration. Experimental results of 40 lung 3D CT images show that our method achieves 0.93 in terms of Dice index and 3.54 mm of landmark Euclidean distance on lung CT registration task, which outperforms state-of-the-art methods in registration accuracy.
Deformable registration continues to be one of the key challenges in medical image analysis. While iconic registration methods have started to benefit from the recent advances in medical deep learning, the same does not yet apply for the registration of point sets, e.g. registration based on surfaces, keypoints or landmarks. This is mainly due to the restriction of the convolution operator in modern CNNs to densely gridded input. However, with the newly developed methods from the field of geometric deep learning suitable tools are now emerging, which enable powerful analysis of medical data on irregular domains. In this work, we present a new method that enables the learning of regularized feature descriptors with dynamic graph CNNs. By incorporating the learned geometric features as prior probabilities into the well-established coherent point drift (CPD) algorithm, formulated as differentiable network layer, we establish an end-to-end framework for robust registration of two point sets. Our approach is evaluated on the challenging task of aligning keypoints extracted from lung CT scans in inhale and exhale states with large deformations and without any additional intensity information. Our results indicate that the inherent geometric structure of the extracted keypoints is sufficient to establish descriptive point features, which yield a significantly improved performance and robustness of our registration framework.
KeywordsGeometric deep learningDeformable point set registration
Es wird ein Verfahren zur automatischen Detektion und Übertragung von
Landmarken in 4D Lungen-CTs präsentiert. Die Landmarken können zB zur Evaluation von
Registrierungsverfahren eingesetzt werden. Um charakteristische Punkte der Lunge als
Landmarkenkandidaten zu ermitteln, wird ein krümmungsbasierter Differentialoperator
genutzt. Weitere Anforderungen an die Detektion wie eine gleichmäßige Verteilung der
Landmarken in der Lunge werden berücksichtigt. Zur Landmarkenübertragung wird ein ...
The Visualization Toolkit (VTK) is an open-source, object-oriented software system providing a toolkit of functionality for 3D data visualization. As visualization inherently involves the capabilities from computer graphics, image processing, volume rendering, computational geometry, human/computer interaction, and other fields, VTK supports these capabilities in an integrated, robust manner. VTK is packaged as a part of a sophisticated development environment that includes advanced interface, build, and test tool. It is this combination of openness, extensive features, reliability, and comprehensive development environment that makes VTK one of today's premier visualization tools for academic, research, and commercial applications. This chapter provides insight into the key features of the toolkit. The focus is on architecture and system concepts. The chapter also provides the motivation, history, and goals for VTK. Further, it describes important architectural features and how VTK is used in large-data visualization, one of the key challenges facing visualization systems of the future. The chapter concludes with a brief survey of important applications developed using VTK.
The decomposition of deformations by principal warps is demonstrated. The method is extended to deal with curving edges between landmarks. This formulation is related to other applications of splines current in computer vision. How they might aid in the extraction of features for analysis, comparison, and diagnosis of biological and medical images is indicated.
Point-based registration of images strongly depends on the extraction of suitable landmarks. Recently, different 3D operators have been proposed in the literature for the detection of anatomical point landmarks in 3D images. In this paper, we investigate nine 3D differential operators for the detection of point landmarks in 3D MR and CT images. These operators are based on either first, second, or first and second order partial derivatives of an image. In our investigation, we use measures which reflect different aspects of the detection performance of the operators. In the first part of the investigation, we analyze the number of corresponding detections under elastic deformations and noise, in the second part, we use statistical measures to determine the detection performance for landmarks within regions of interest (ROIs), and in the third part, we investigate the separability of the detections. It turns out that operators based on only first order partial derivatives of an image (i) yield a larger number of corresponding points than the other operators, (ii) that their performance on the basis of the statistical measures is better, and (iii) that the separability of the detections is better so that a suitably chosen threshold can significantly decrease the number of false detections.
This paper describes methods for the atlas-based segmentation of bone structures of the hip, the automatic detection of anatomical
point landmarks and the computation of orthopedic parameters. An anatomical atlas was designed to replace interactive, time-consuming
pre-processing steps needed for the virtual planning of hip operations. Furthermore, a non-linear gray value registration
of CT data is used to recognize different bone structures of the hip. A surface based registration algorithm enables the robust
and precise detection of anatomical point landmarks. Furthermore the determination of quantitative parameters, like angles,
distances or sizes of contact areas, is important for the planning of hip operations. Based on segmented bone structures and
detected landmarks algorithms for the automatic computation of orthopedic parameters were implemented. A first evaluation
of the presented methods will be given at the end of the paper.
A method for an automatic reliable and precise detection of anatomical landmarks based on triangulated surface models of bone structures is presented. By means of a surface-based registration procedure, manually specified landmarks on the atlas surface are transferred automatically to the patient's surface. In a pre-processing step, initial landmark positions on the patient's surface are automatically determined to restrict the registration to a surface area. During the registration process, apart from the Euclidean distances of the surface points, the surface-normals and a new local curvature measure for triangulated surfaces are used. Finally, an evaluation of the proposed method is presented.
This paper describes a general purpose, representation independent
method for the accurate and computationally efficient registration of
3-D shapes including free-form curves and surfaces. The method handles
the full six-degrees of freedom and is based on the iterative closest
point (ICP) algorithm, which requires only a procedure to find the
closest point on a geometric entity to a given point. The ICP algorithm
always converges monotonically to the nearest local minimum of a
mean-square distance metric, and experience shows that the rate of
convergence is rapid during the first few iterations. Therefore, given
an adequate set of initial rotations and translations for a particular
class of objects with a certain level of 'shape complexity', one can
globally minimize the mean-square distance metric over all six degrees
of freedom by testing each initial registration. For examples, a given
'model' shape and a sensed 'data' shape that represents a major portion
of the model shape can be registered in minutes by testing one initial
translation and a relatively small set of rotations to allow for the
given level of model complexity. One important application of this
method is to register sensed data from unfixtured rigid objects with an
ideal geometric model prior to shape inspection. The described method is
also useful for deciding fundamental issues such as the congruence
(shape equivalence) of different geometric representations as well as
for estimating the motion between point sets where the correspondences
are not known. Experimental results show the capabilities of the
registration algorithm on point sets, curves, and surfaces.