No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Slice-to-volume medical image registration: A survey
Abstract
During the last decades, the research community of medical imaging has witnessed continuous advances in image registration methods, which pushed the limits of the state-of-the-art and enabled the development of novel medical procedures. A particular type of image registration problem, known as slice-to-volume registration, played a fundamental role in areas like image guided surgeries and volumetric image reconstruction. However, to date, and despite the extensive literature available on this topic, no survey has been written to discuss this challenging problem. This paper introduces the first comprehensive survey of the literature about slice-to-volume registration, presenting a categorical study of the algorithms according to an ad-hoc taxonomy and analyzing advantages and disadvantages of every category. We draw some general conclusions from this analysis and present our perspectives on the future of the field.
... The problem of S2V registration is best known from applications in medical imaging [7,13]. These applications include correlative multi-instrument analysis (e.g., crossreferencing 3D computer tomographic (CT) scans with 2D tissue classifications from histology), localization of 2D imaging instruments in image-guided surgeries, and volumetric reconstruction from 2D imaging modalities. ...
... Often an initial approximation of a more complex registration task, this formulation allows e.g. to acquire a strong initial guess for subsequent non-rigid S2V registration. This class of registration problems has been traditionally approached as a dissimilarity minimization problem [52], the comprehensive survey on which is provided in [13]. ...
... Rigid S2V registration is defined [13] as the problem of finding the affine transformation A that aligns a 2D slice I with a 3D volume V . To perform this alignment, we allocate the initial pose of I to the Z = 0 plane in 3D space. ...
We address domain-agnostic slice-to-volume (S2V) registration, the alignment of 2D sliced/tomographic images into 3D volumes without prior knowledge of structure, shape, or orientation. While S2V registration is well-studied in medical imaging, which often relies on auxiliary information (e.g. landmarks, segmentation masks, predefined orientations, canonical/atlas volumes), applications such as micro-structure characterization in materials science lack such domain-specific aids. This leaves the task inherently ill-posed due to noise, unstructured regions, repetitive patterns, rotational and translational symmetries. To address this challenge, we present “Needles & Haystacks,” Project page: https://xaf-cv.github.io/nh-rs2v/ a novel multi-domain algorithm development dataset with 158, 436 unique registration problems and ground-truth solutions, based on diverse and openly licensed real-world volumetric data. Additionally, we provide an online platform with 8461 test problems for reproducible evaluation of competing methods. We also propose strong baseline solutions with public implementations and highlight opportunities for further algorithmic advancements.
... However, these methods struggle with individual variability and often require a wellinitialized position in large 3D space, complicating their utility in real-world scenarios. Also, the learning-based approaches only focus on aligning 2D slices of arbitrary orientation into a 3D fixed (or common) CT volume space [16]. For example, Hou et al. [17], [18] propose a learning-based rigid slice-to-volume registration method for fetal brain images, which automatically learns the mapping function that aligns arbitrary slices into a canonical coordinate. ...
... Slice-to-volume registration is the process of aligning the slices (corresponding to arbitrary planes) into a unique coordinate of a fixed 3D volume [16]. Similar to our study is the rigid slice-to-volume registration, which is mainly divided into optimization-based and learning-based approaches. ...
Automatic view positioning is crucial for cardiac computed tomography (CT) examinations, including disease diagnosis and surgical planning. However, it is highly challenging due to individual variability and large 3D search space. Existing work needs labor-intensive and time-consuming manual annotations to train view-specific models, which are limited to predicting only a fixed set of planes. However, in real clinical scenarios, the challenge of positioning semantic 2D slices with any orientation into varying coordinate space in arbitrary 3D volume remains unsolved. We thus introduce a novel framework, AVP-AP, the first to use Atlas Prompting for self-supervised Automatic View Positioning in the 3D CT volume. Specifically, this paper first proposes an atlas prompting method, which generates a 3D canonical atlas and trains a network to map slices into their corresponding positions in the atlas space via a self-supervised manner. Then, guided by atlas prompts corresponding to the given query images in a reference CT, we identify the coarse positions of slices in the target CT volume using rigid transformation between the 3D atlas and target CT volume, effectively reducing the search space. Finally, we refine the coarse positions by maximizing the similarity between the predicted slices and the query images in the feature space of a given foundation model. Our framework is flexible and efficient compared to other methods, outperforming other methods by 19.8% average structural similarity (SSIM) in arbitrary view positioning and achieving 9% SSIM in two-chamber view compared to four radiologists. Meanwhile, experiments on a public dataset validate our framework's generalizability.
... Our focus is mainly on the 3D to 3D deformable registration problem where the input and the output images are 3D volumes. However, the same framework can be trivially used for other registration problems as well, as 2D to 2D, or even for slice-to-volume registration [26]. One of the key issues we face in all these problems is that the ground truth deformations are not known. ...
... Pyramidal Approach Similar to [26], [29], we adopt a pyramidal approach (detailed in Algorithm 1) to refine the search space at every level and, at the same time, capture a big range of deformations. The pyramidal approach consists in registering the images incrementally, at different resolutions (constructing a Gaussian pyramid of downsampled images) and associating to every level a different grid spacing. ...
Deformable registration has been one of the pillars of biomedical image computing. Conventional approaches refer to the definition of a similarity criterion that, once endowed with a deformation model and a smoothness constraint, determines the optimal transformation to align two given images. The definition of this metric function is among the most critical aspects of the registration process. We argue that incorporating semantic information (in the form of anatomical segmentation maps) into the registration process will further improve the accuracy of the results. In this paper, we propose a novel weakly supervised approach to learn domain specific aggregations of conventional metrics using anatomical segmentations. This combination is learned using latent structured support vector machines (LSSVM). The learned matching criterion is integrated within a metric free optimization framework based on graphical models, resulting in a multi-metric algorithm endowed with a spatially varying similarity metric function conditioned on the anatomical structures. We provide extensive evaluation on three different datasets of CT and MRI images, showing that learned multi-metric registration outperforms single-metric approaches based on conventional similarity measures.
... It plays an important role in clinical examination of prenatal brain disorders due to its superior image resolution and diverse tissue contrasts, which are considered more informative than ultrasonography [3], [4]. The advancements in fetal MRI were attributed to the development of fast imaging techniques [5], improved motion correction and advanced reconstruction algorithms [6], [7]. These technological improvements enabled the generation of detailed, high-resolution three-dimensional volumes of the fetal brain, providing benefits for both diagnostic and quantitative assessments [8]. ...
... (c) Proposed MA2-Fuse. This fusion module took image and atlas features from corresponding branches as inputs, and outputs fused feature for further segmentation.A group of convolutions with kernel sizes of[7,5,3,1] were used to extract multi-scale features. Late concatenation was always used for deep fusion. ...
Accurate tissue segmentation in fetal brain MRI remains challenging due to the dynamically changing anatomical anatomy and contrast during fetal development. To enhance segmentation accuracy throughout gestation, we introduced AtlasSeg, a dual-U-shape convolution network incorporating gestational age (GA) specific information as guidance. By providing a publicly available fetal brain atlas with segmentation label at the corresponding GA, AtlasSeg effectively extracted the contextual features of age-specific patterns in atlas branch and generated tissue segmentation in segmentation branch. Multi-scale attentive atlas feature fusions were constructed in all stages during encoding and decoding, giving rise to a dual-U-shape network to assist feature flow and information interactions between two branches. AtlasSeg outperformed six well-known segmentation networks in both our internal fetal brain MRI dataset and the external FeTA dataset. Ablation experiments demonstrate the efficiency of atlas guidance and the attention mechanism. The proposed AtlasSeg demonstrated superior segmentation performance against other convolution networks with higher segmentation accuracy, and may facilitate fetal brain MRI analysis in large-scale fetal brain studies.
... This model covers six degrees of freedom and only considers translation t and rotation r with respect to the x-, y-and z-axes. Consequently, this model is very simple, yet able to estimate in-plane and through-plane motion-induced changes [88]. A rigid body spatial transformation matrix can be expressed as following: ...
... However, the increase to twelve degrees of freedom also means an increase in calculation time and a reduction in the confidence limits of parameter estimates. Other non-rigid models such as in-plane or out-of-plane deformable models are available [88], but these are not very applicable in neurological imaging. As the shape of the head and brain does not change significantly with motion, a rigid body model is suitable [91]. ...
Magnetic resonance imaging (MRI) at the magnetic field strength of 7 Tesla (7T) enhances the quality of images available for research and clinical use. The improvements are however accompanied by novel challenges that are specific to ultra high-field MRI, which includes field strengths of 7T and above. Transmit B1+ field inhomogeneity is also higher, causing uneven signal intensity and linking to an uneven SAR distribution, which is also higher than at lower field strengths. The potential for higher spatial resolution imaging can also result in more pronounced motion artefacts. To address these issues in routine clinical use, motion correction strategies are required. This thesis describe the implementation of real-time, image-based Multislice Prospective Acquisition Correction (MS-PACE) technique for 7T MRI. Firstly, developmental work was done to establish a 7T-specific MS-PACE implementation. Pulse sequence and image reconstruction pipeline work was implemented using the Siemens Integrated Development Environment for Applications (IDEA) and Image Calculation Environment (ICE) framework. The technique was then validated in a task-based functional MRI study with healthy subjects. It was also integrated with parallel transmit imaging using slice-by-slice B1+ shimming. Validation experiments were performed in vivo using the Siemens MAGNETOM Terra 7T MRI scanner (Siemens Healthineers, Erlangen, Germany) at the Imaging Centre of Excellence (ICE).
... In addition, other MRI-based methods have enabled reconstruction of sparse histological images. 18,19,[25][26][27] Although entailing less-accurate slice-to-volume registrations, 28 this option avoids dense sampling of the whole specimen. Unfortunately, in vivo MRI is available only for special cases or in planned followup initiatives such as the Alzheimer's Disease Neuroimaging Initiative (ADNI). ...
... Direct slice-to-volume registration of a histological section to MNI space is extremely ill posed and ambiguous, particularly for nonlinear registration. 28 To circumvent this challenge, our pipeline takes advantage of prior knowledge of the brain slices from which tissue blocks (and histology sections) were derived. This correspondence is used to register each histological section to its approximate location and rotation in the brain slice. ...
INTRODUCTION
Three‐dimensional (3D) histology analyses are essential to overcome sampling variability and understand pathological differences beyond the dissection axis. We present Path2MR, the first pipeline allowing 3D reconstruction of sparse human histology without a magnetic resonance imaging (MRI) reference. We implemented Path2MR with post‐mortem hippocampal sections to explore pathology gradients in Alzheimer's disease.
METHODS
Blockface photographs of brain hemisphere slices are used for 3D reconstruction, from which an MRI‐like image is generated using machine learning. Histology sections are aligned to the reconstructed hemisphere and subsequently to an atlas in standard space.
RESULTS
Path2MR successfully registered histological sections to their anatomic position along the hippocampal longitudinal axis. Combined with histopathology quantification, we found an expected peak of tau pathology at the anterior end of the hippocampus, whereas amyloid‐beta (Aβ) displayed a quadratic anterior‐posterior distribution.
CONCLUSION
Path2MR, which enables 3D histology using any brain bank data set, revealed significant differences along the hippocampus between tau and Aβ.
Highlights
Path2MR enables three‐dimensional (3D) brain reconstruction from blockface dissection photographs.
This pipeline does not require dense specimen sampling or a subject‐specific magnetic resonance (MR) image.
Anatomically consistent mapping of hippocampal sections was obtained with Path2MR.
Our analyses revealed an anterior‐posterior gradient of hippocampal tau pathology.
In contrast, the peak of amyloid‐beta (Aβ) deposition was closer to the hippocampal body.
... Additionally, other MRI-based methods have enabled reconstruction of sparse histological images 18,19,[25][26][27] . Although entailing less accurate slice-to-volume registrations 28 , this option avoids dense sampling of the whole specimen. Unfortunately, in vivo MRI is only available for special cases or in planned follow-up initiatives such as the Alzheimer's Disease Neuroimaging Initiative (ADNI) 29 . ...
... Direct slice-to-volume registration of a histological section to MNI space is extremely illposed and ambiguous, particularly for nonlinear registration 28 . To circumvent this challenge, our pipeline takes advantage of prior knowledge of the brain slices from which tissue blocks (and histology sections) were derived. ...
INTRODUCTION: Three-dimensional (3D) histology analyses are essential to overcome sampling variability and understand pathological differences beyond the dissection axis. We present Path2MR, the first pipeline allowing 3D reconstruction of sparse human histology without an MRI reference. We implemented Path2MR with post-mortem hippocampal sections to explore pathology gradients in Alzheimer's Disease.
METHODS: Blockface photographs of brain hemisphere slices are used for 3D reconstruction, from which an MRI-like image is generated using machine learning. Histology sections are aligned to the reconstructed hemisphere and subsequently to an atlas in standard space.
RESULTS: Path2MR successfully registered histological sections to their anatomical position along the hippocampal longitudinal axis. Combined with histopathology quantification, we found an expected peak of tau pathology at the anterior end of the hippocampus, while amyloid-beta displayed a quadratic anterior-posterior distribution.
CONCLUSION: Path2MR, which enables 3D histology using any brain bank dataset, revealed significant differences along the hippocampus between tau and amyloid-beta.
... Fusion of preoperative 3D CT/MRI images with intraoperative 2D US images based on image registration technology can provide real-time and all-round clear guidance for needle-based interventions [16]. 2D US -3D CT/MRI registration belongs to the cross-modal and cross-dimensional image registration problem [17], and the images to be aligned are very different in terms of not only appearance but also fields of view. In commercial navigation systems, such as GE V-Nav [18], Siemens eSieFusion [19] and Samsung S-fusion [20], a rigid transformation that optimally aligns an US slice with a preoperative CT/MRI volume is firstly determined based on some manually identified anatomic fiducials. ...
... In the field of image registration, the methods combing graph model with discrete optimization have the nature of global search [17]. In order to obtain the rigid transformation 2DUS 3DUS T that can optimally align the initial 3D US with the intraoperative 2D US, inspired by Zikic et al. [48] and Porchetto et al. [49], a three-level pyramid registration framework based on the Markov Random Field (MRF) is adopted and multiple iterations are performed at each level. ...
Puncture robots pave a new way for stable, accurate and safe percutaneous liver tumor puncture operation. However, affected by respiratory motion, intraoperative accurate location of the tumor and its surrounding anatomical structures remains a difficult problem in existing robot-assisted puncture operations. In this paper, a dual-arm robotic needle insertion system with guidance of intraoperative 2D ultrasound (US) and preoperative 3D computed tomography (CT) fusion is proposed, addressing the shortcomings of existing puncture robots. To deal with the challenge of cross-modal and cross-dimensional registration between 2D US and 3D CT, a decoupled two-stage registration approach combining initial vessel structure-based 3D US – 3D CT registration with intraoperative intensity-based 2D US -3D US registration is proposed. To achieve fast and robust ultrasound probe calibration, a method based on an improved N-wire phantom is proposed. Twenty puncture experiments are performed in different breath-holding positions on a respiratory motion simulation platform, and experimental results show that the mean puncture error is 2.48 mm, which can meet the requirements in a wide of clinical scenarios
Note to Practitioners
—In clinical percutaneous liver tumor puncture operation, due to the lack of real-time and clear image guidance, it is difficult to locate the tumor and its surrounding vital anatomical structures. In addition, the stability and accuracy of manual operation are poor. The development of a puncture robot is an effective solution for these problems. However, existing CT and magnetic resonance imaging (MRI) guided robots do not consider the tumor localization errors caused by inconsistent breath-holding positions between preoperative scan period and intraoperative puncture period, and US guided robots are limited by the poor image quality and the narrow field of vision. In this paper, a dual-arm robotic needle insertion system with guidance of intraoperative 2D US and preoperative 3D CT fusion is proposed. This system can take advantage of the real-time ultrasound and clear CT images at the same time, and can provide real-time, clear and all-round guidance for percutaneous liver tumor puncture operation, which has obvious advantages over the existing puncture robots. Phantom experiments have been completed and animal experiments will be carried out in the future.
... However, this approach has limited out-of-plane resolution and suffers from artifacts during staining and sectioning (Pichat et al., 2018). As a result, any correlation with MRI can involve complex slice-to-volume registration (Ferrante & Paragios, 2017). Visualizing the three-dimensional arrangement of hippocampal cells is important for understanding the progression of mTLE (Kowalski et al., 2010;Häussler et al., 2012;Marx et al., 2013), but MRI is suboptimal for this task due to its relatively low spatial resolution. ...
... Slice-to-volume registration between histological sections and volumetric images, which is the process of finding dense spatial correspondences between the two modalities, is a challenging task due to the large search space, limited twodimensional data, appearance differences due to multimodality, and topology changes (e.g. cracks) (Ferrante & Paragios, 2017). Here, histology-to-3D X-ray histology matching may prove simpler than histology-to-MRI matching, thanks to the comparable resolution and more similar appearance between label-free 3D X-ray histology and common stains, cf. Figure 6. ...
The most common form of epilepsy among adults is mesial temporal lobe epilepsy (mTLE), with seizures often originating in the hippocampus due to abnormal electrical activity. The gold standard for the histopathological analysis of mTLE is histology, which is a two-dimensional technique. To fill this gap, we propose complementary three-dimensional (3D) X-ray histology. Herein, we used synchrotron radiation-based phase-contrast microtomography with 1.6 μm-wide voxels for the post mortem visualization of tissue microstructure in an intrahippocampal-kainate mouse model for mTLE. We demonstrated that the 3D X-ray histology of unstained, unsectioned, paraffin-embedded brain hemispheres can identify hippocampal sclerosis through the loss of pyramidal neurons in the first and third regions of the Cornu ammonis as well as granule cell dispersion within the dentate gyrus. Morphology and density changes during epileptogenesis were quantified by segmentations from a deep convolutional neural network. Compared to control mice, the total dentate gyrus volume doubled and the granular layer volume quadrupled 21 days after injecting kainate. Subsequent sectioning of the same mouse brains allowed for benchmarking 3D X-ray histology against well-established histochemical and immunofluorescence stainings. Thus, 3D X-ray histology is a complementary neuroimaging tool to unlock the third dimension for the cellular-resolution histopathological analysis of mTLE.
... The n number of BATS could have d dimensional solutions. The fitness of food is [22] checkmark Perez-Cham et al. [45] Rajasekhar et al. [47] Wang et al. [70] Salavati and Abdollahpouri [51] Roshanzamir et al. [49] Singh et al. [54] Sundgaard et al. [62] Ali et al. [4] Li et al. [38] Dou et al. [21] Ferrante and Paragios [24] Mahmood and Durr [40] Mookiah et al. [41] Stolte and Fang [61] Razavi Zadegan et al. [48] Deng et al. [19] Huang et al. [29] Yin et al. [77] Tabakhi and Moradi [63] Teraiya and Shah [65] Fang et al. [23] Das et al. [15] Thakkar and Lohiya [66] determined in the BAT population as: ...
... The Local solution has been searched by using equation 24. ...
Cluster labeling is a problem of finding optimal clusters by maximizing the similarity within clusters. Generally, classical optimization methods are used for clustering, which depends upon initial estimates. Meta-heuristic optimization techniques came into existence to avoid this shortcoming as they are not problem-specific and do not dependent on the initial estimates. Optimal threshold selection is a widely known problem for image clustering that optimizes the entropy of image clusters. But for region-wise clustering, the sheer amount of literature is available that uses evolutionary techniques. Sub-optimal cluster selection and slow convergence are the problems that arise in meta-heuristics approaches. In this paper, we optimize the region-wise cluster labels for an efficient objective function. In particular, we consider the cluster label optimization problems in tumour-based magnetic resonance (MR) images. In order to solve the problem, a meta-heuristic approaches is used. The stable random walk is incorporated with the BAT algorithm to avoid the potential endangerment of local minima trapping. Further, the objective improvement is carried out by hybridizing the intra-cluster and inter-cluster-based objectives. To assess the efficiency of the proposed approach, we evaluated the proposed method against the recently proposed techniques. The evolutionary technique is well performed in various aspects of cluster label evaluation and paves new ideas for future research lines.
... Through image registration, the registration of two or more images with partially identical scenes can be obtained so that they can be fused into one image, solving the problems of the narrow field of view and incomplete information of a single image. Image fusion by image registration technology may successfully resolve the issue of severe negative distortion of the image captured by the lens and also efficiently resolve the issue of large distortion of the panoramic image obtained by the traditional way of camera ring shooting (Ferrante et al. 2017). ...
This research aims to create an image registration system specific to art design, employing an upgraded version of the Speeded-Up Robust Features algorithm, known as Gradient Speeded-Up Robust Features. Optimizing computational efficiency during the processing of images, particularly for real-time analysis in scenarios in art design, is its key purpose. In its proposed algorithm, traditional rectangular templates have been replaced with circular ones, and a significant drop in computational intensity and improvements in feature detection and matching capabilities have been seen. Experimental results reveal a 25% drop in computation time and a 15% boost in correct matching when contrasted with the traditional Speeded-Up Robust Features algorithm. In addition, its average processing time for processing an image has been reduced by 1.2 s, and therefore, it is particularly ideal for use in scenarios such as artwork installations, multimedia, and augmented reality environments. This work puts into prominence the growing role of computational approaches in art design and raises demand for continued improvements in image processing technology. The theory proposed in this work forms a basis for combining technology with registrations in images in art design and promotes innovation in works of digital and interactive artwork. Overall, these findings present avenues for improvements in even sophisticated processing in picture processing systems utilized in scenarios in art design.
... All these applications are based on plausibly correcting the spatial distortion between the corresponding anatomical tissues from different images. Over the nearly two decades, lots of non-rigid registration approaches have been proposed to estimate the dense deformation fields [5], [6]. ...
Objective: Non-rigid image registration with high accuracy and efficiency is still a challenging task for medical image analysis. In this work, we present the spatially region-weighted correlation ratio (SRWCR) as a novel similarity measure to improve the registration performance. Methods: SRWCR is rigorously deduced from a three-dimension joint probability density function combining the intensity channels with an extra spatial information channel. SRWCR estimates the optimal functional dependence between the intensities for each spatial bin, in which the spatial distribution modeled by a cubic B-spline function is used to differentiate the contribution of voxels. We also analytically derive the gradient of SRWCR with respect to the transformation parameters and optimize it using a quasi-Newton approach. Furthermore, we propose a GPU-based parallel mechanism to accelerate the computation of SRWCR and its derivatives. Results: The experiments on synthetic images, public 4-D thoracic computed tomography (CT) dataset, retinal optical coherence tomography (OCT) data, and clinical CT and positron emission tomography (PET) images confirm that SRWCR significantly outperforms some state-of-the-art techniques such as spatially encoded mutual information and Robust PaTch-based cOrrelation Ration. Conclusion: This study demonstrates the advantages of SRWCR in tackling the practical difficulties due to distinct intensity changes, serious speckle noise, or different imaging modalities. Significance: The proposed registration framework might be more reliable to correct the non-rigid deformations and more potential for clinical applications.
... In contrast, the latter is concerned with aligning multiple misaligned 2D slices into a unique co-ordinate system of a reference volume. A recent review of slice-to-volume registration techniques is given in [5]. ...
Limited capture range, and the requirement to provide high quality initialization for optimization-based 2D/3D image registration methods, can significantly degrade the performance of 3D image reconstruction and motion compensation pipelines. Challenging clinical imaging scenarios, which contain significant subject motion such as fetal in-utero imaging, complicate the 3D image and volume reconstruction process. In this paper we present a learning based image registration method capable of predicting 3D rigid transformations of arbitrarily oriented 2D image slices, with respect to a learned canonical atlas co-ordinate system. Only image slice intensity information is used to perform registration and canonical alignment, no spatial transform initialization is required. To find image transformations we utilize a Convolutional Neural Network (CNN) architecture to learn the regression function capable of mapping 2D image slices to a 3D canonical atlas space. We extensively evaluate the effectiveness of our approach quantitatively on simulated Magnetic Resonance Imaging (MRI), fetal brain imagery with synthetic motion and further demonstrate qualitative results on real fetal MRI data where our method is integrated into a full reconstruction and motion compensation pipeline. Our learning based registration achieves an average spatial prediction error of 7 mm on simulated data and produces qualitatively improved reconstructions for heavily moving fetuses with gestational ages of approximately 20 weeks. Our model provides a general and computationally efficient solution to the 2D/3D registration initialization problem and is suitable for real-time scenarios.
... ; https://doi.org/10.1101/2024.11.29.626074 doi: bioRxiv preprint proposed for registering T1 and T2 MRI images [35]. Algorithms for image stack registration have also been actively explored [36]. ...
Electron microscopy is essential for the quantitative study of synaptic ultrastructure. At present, the correlation of functional and structural properties of the same synapse is extremely challenging. We introduce a novel integrated workflow designed to simplify sample navigation across spatial scales, allowing the identification of individual synapses from optical microscopy mouse brain image stacks that can be targeted for analysis using electron tomography imaging. We developed a software which has a function to register multimodal images using a novel segmentation-based image registration algorithm as well as a function to visualize all the registration results. Using our newly designed software we streamline mapping of high-resolution optical imaging onto reference maps using blood vessels as endogenous fiducial marks. Further we demonstrate significant improvements on the ultramicrotomy stage of volume Correlative Light and Electron Microscopy (vCLEM) workflows, providing real time guidance to targeted trimming to match previously acquired Regions Of Interest (ROIs), and reliable estimates of cutting depth relative to ROI, based on fluorescence imaging of TEM ready ultrathin sections. Using this workflow, we successfully targeted TEM tomography to the proximal axonal region containing the Axon Initial Segment identified using fluorescent light microscopy.
... To provide a complete guidance view for cardiac interventions, it is necessary to explore frame-to-volume registration that fuse intraoperative 2D ultrasound images and preoperative 3D ultrasound volumes in real time, which shortens the learning curve of ultrasound-guided cardiac interventions. The ultrasound frame-to-volume registration aims to seek a transformation that optimally aligns the resampled slice from the given volume by the transformation with the 2D input image [5,6], as shown in Fig. 1. Existing registration methods are divided into mathematical methods and deep learning-based methods. ...
... Multimodal image matching is a fundamental problem that involves identifying and pairing similar features or patterns across images from different modalities, with significant appearance changes [11]. It has a wide range of applications in medical imaging, including image retrieval and classification [11,14], slice-tovolume alignment [7] and image registration [6,18,17,10,12]. When used during image-guided surgery, it can provide surgeons with complementary imaging information from various modalities, facilitating the identification of key anatomical and surgical structures for improved surgical outcomes. ...
We propose in this paper a texture-invariant 2D keypoints descriptor specifically designed for matching preoperative Magnetic Resonance (MR) images with intraoperative Ultrasound (US) images. We introduce a matching-by-synthesis strategy, where intraoperative US images are synthesized from MR images accounting for multiple MR modalities and intraoperative US variability. We build our training set by enforcing keypoints localization over all images then train a patient-specific descriptor network that learns texture-invariant discriminant features in a supervised contrastive manner, leading to robust keypoints descriptors. Our experiments on real cases with ground truth show the effectiveness of the proposed approach, outperforming the state-of-the-art methods and achieving 80.35% matching precision on average.
... Fetal brain MRI is a powerful tool in prenatal diagnosis and developmental neuroscience for its high spatial resolution and strong tissue contrast, 1,2 compared to ultrasound. 3 However, the presence of irregular fetal movement and maternal abdominal motion poses challenges for direct 3D volume acquisition. 4,5 To address this issue, fast 2D multi-slice imaging methods, such as single-shot fast spin-echo (SSFSE) or balanced steady-state free precession (bSSFP) are employed to acquire stacks of slices in multiple orthogonal views, which are then combined to reconstruct a 3D volume of the fetal brain. While fast 2D multi-slice acquisition effectively freezes intra-slice motion, 6 inter-slice motion that occurs during the slice acquisition interval is still inevitable and remains challenging for accurate 3D volume reconstruction. ...
Slice‐to‐volume registration and super‐resolution reconstruction are commonly used to generate 3D volumes of the fetal brain from 2D stacks of slices acquired in multiple orientations. A critical initial step in this pipeline is to select one stack with the minimum motion among all input stacks as a reference for registration. An accurate and unbiased motion assessment (MA) is thus crucial for successful selection. Here, we presented an MA method that determines the minimum motion stack based on 3D low‐rank approximation using CANDECOMP/PARAFAC (CP) decomposition. Compared to the current 2D singular value decomposition (SVD) based method that requires flattening stacks into matrices to obtain ranks, in which the spatial information is lost, the CP‐based method can factorize 3D stack into low‐rank and sparse components in a computationally efficient manner. The difference between the original stack and its low‐rank approximation was proposed as the motion indicator. Experiments on linearly and randomly simulated motion illustrated that CP demonstrated higher sensitivity in detecting small motion with a lower baseline bias, and achieved a higher assessment accuracy of 95.45% in identifying the minimum motion stack, compared to the SVD‐based method with 58.18%. CP also showed superior motion assessment capabilities in real‐data evaluations. Additionally, combining CP with the existing SRR‐SVR pipeline significantly improved 3D volume reconstruction. The results indicated that our proposed CP showed superior performance compared to SVD‐based methods with higher sensitivity to motion, assessment accuracy, and lower baseline bias, and can be used as a prior step to improve fetal brain reconstruction.
... The ultrasound frame-to-volume registration aims to seek a transformation that optimally aligns the resampled slice from the given volume by the transformation with the 2D input image [6,5], as shown in Fig. 1. Existing registration methods are divided into mathematical methods and deep learning-based methods. ...
A comprehensive guidance view for cardiac interventional surgery can be provided by the real-time fusion of the intraoperative 2D images and preoperative 3D volume based on the ultrasound frame-to-volume registration. However, cardiac ultrasound images are characterized by a low signal-to-noise ratio and small differences between adjacent frames, coupled with significant dimension variations between 2D frames and 3D volumes to be registered, resulting in real-time and accurate cardiac ultrasound frame-to-volume registration being a very challenging task. This paper introduces a lightweight end-to-end Cardiac Ultrasound frame-to-volume Registration network, termed CU-Reg. Specifically, the proposed model leverages epicardium prompt-guided anatomical clues to reinforce the interaction of 2D sparse and 3D dense features, followed by a voxel-wise local-global aggregation of enhanced features, thereby boosting the cross-dimensional matching effectiveness of low-quality ultrasound modalities. We further embed an inter-frame discriminative regularization term within the hybrid supervised learning to increase the distinction between adjacent slices in the same ultrasound volume to ensure registration stability. Experimental results on the reprocessed CAMUS dataset demonstrate that our CU-Reg surpasses existing methods in terms of registration accuracy and efficiency, meeting the guidance requirements of clinical cardiac interventional surgery.
... However, areabased methods face challenges with geometrical transformations and local deformations. Research works presented in [17,[19][20][21] hint at the integration of deep learning into areabased matching for improved efficacy, a topic to be reviewed in the learning-based matching section. Graph matching (GM) involves associating feature points to nodes, forming a graph for investigating the image data structure. ...
Efficient image stitching plays a vital role in the Non-Destructive Evaluation (NDE) of infrastructures. An essential challenge in the NDE of infrastructures is precisely visualizing defects within large structures. The existing literature predominantly relies on high-resolution close-distance images to detect surface or subsurface defects. While the automatic detection of all defect types represents a significant advancement, understanding the location and continuity of defects is imperative. It is worth noting that some defects may be too small to capture from a considerable distance. Consequently, multiple image sequences are captured and processed using image stitching techniques. Additionally, visible and infrared data fusion strategies prove essential for acquiring comprehensive information to detect defects across vast structures. Hence, there is a need for an effective image stitching method appropriate for infrared and visible images of structures and industrial assets, facilitating enhanced visualization and automated inspection for structural maintenance. This paper proposes an advanced image stitching method appropriate for dual-sensor inspections. The proposed image stitching technique employs self-supervised feature detection to enhance the quality and quantity of feature detection. Subsequently, a graph neural network is employed for robust feature matching. Ultimately, the proposed method results in image stitching that effectively eliminates perspective distortion in both infrared and visible images, a prerequisite for subsequent multi-modal fusion strategies. Our results substantially enhance the visualization capabilities for infrastructure inspection. Comparative analysis with popular state-of-the-art methods confirms the effectiveness of the proposed approach.
... Slicing is a commonly used analytical algorithm for observing the cross-sectional structure of a sample. This algorithm has already been applied in a wide range of fields, such as pathological section studies (Ferrante and Paragios, 2017), 3D printing (Qiu et al., 2011), and industrial component inspection (Xu et al., 2022). The application of slicing in volume quantification has increased with the development and application of point cloud acquisition technology. ...
Three-dimensional (3D) point clouds are widely used for geomorphic change detection. However, the lack of efficient change-detection algorithms for complex terrain limits the use of 3D point clouds in area-wide morphological change studies. In this study, a complex terrain development process was simulated on a natural slope in the hilly and gully Loess Plateau in China using 28 runoff scouring experiments conducted in two plots. Highly precise point clouds were obtained using terrestrial laser scanning (TLS) before and after each experiment. A slice contraction change detection (SCCD) algorithm was developed based on slicing, Laplacian-based contraction, and differential principles for detecting geomorphic and volumetric changes on complex terrain, and the level of detection (LoD) was derived with respect to that of the multiscale model to model the cloud comparison (M3C2) algorithm. The accuracy of SCCD was compared with that of the 3D-M3C2 algorithm (i.e., a 3D volumetric change calculation algorithm based on M3C2) and the digital elevation model (DEM) of difference (DoD) algorithm based on the measured sediment yield from the plots. The comparison was performed also under different point cloud densities and morphologies. Results showed the following: (1) The precisions of SCCD and 3D-M3C2 were comparable and considerably higher than that of DoD. The mean relative errors of SCCD, 3D-M3C2, and DoD for the two plots were 13.32% and 10.37%, 10.07% and 10.84%, and 35.30% and 27.23%, respectively. The relative error fluctuations of the three algorithms for the individual experiments followed the sequence of DoD (standard deviation (std.): 10.18) > 3D-M3C2 (std.: 8.29) > SCCD (std.: 5.79). (2) The sensitivity to point cloud density changes followed the sequence of 3D-M3C2 > SCCD > DoD as the point cloud density varied between 10,000 and 1000 points m − 2. The mean relative errors of 3D-M3C2, SCCD, and DoD for the two plots were 10.07-18.59% and 10.84-13.62%, 13.32-16.83% and 10.37-15.50%, and 35.30-38.33% and 26.52-27.26%, respectively. (3) The accuracy of 3D-M3C2 decreased significantly (p < 0.05), whereas those of SCCD and DoD either changed insignificantly (p > 0.05) or increased significantly for substantial morphologic changes. As the experiments progressed, the relative errors of 3D-M3C2, SCCD, and DoD for the two plots were 10.87-93.77% and 30.76-167.89%, 20.04-9.95% and 5.54-7.96%, and 42.49-11.94% and 3.89-4.96%, respectively. Overall, the SCCD algorithm provides a reliable means of conducting geomorphic change detection in complex terrain and thus facilitates future studies on detecting and characterizing land surface processes.
... Dong et al. used an improved convex hull algorithm to extract tree crown contours and calculate volumes [22]. The slicing algorithm degrades the issue on calculating volumes from complicated irregularly-shaped 3D geometry into relative simple multiple 2D shapes [23], which significantly reduces both the space and time complexity [24,25]. ...
... While multi-modal slice-to-volume registration is of high interest to the medical community [12,24], the registration of image slices to statistical shape representations has not been explored extensively. Ghanavati et al. first proposed registering US slices to an image atlas generated by deformable registration of CT images [15,14]. ...
Thyroid disorders are most commonly diagnosed using high-resolution Ultrasound (US). Longitudinal nodule tracking is a pivotal diagnostic protocol for monitoring changes in pathological thyroid morphology. This task, however, imposes a substantial cognitive load on clinicians due to the inherent challenge of maintaining a mental 3D reconstruction of the organ. We thus present a framework for automated US image slice localization within a 3D shape representation to ease how such sonographic diagnoses are carried out. Our proposed method learns a common latent embedding space between US image patches and the 3D surface of an individual's thyroid shape, or a statistical aggregation in the form of a statistical shape model (SSM), via contrastive metric learning. Using cross-modality registration and Procrustes analysis, we leverage features from our model to register US slices to a 3D mesh representation of the thyroid shape. We demonstrate that our multi-modal registration framework can localize images on the 3D surface topology of a patient-specific organ and the mean shape of an SSM. Experimental results indicate slice positions can be predicted within an average of 1.2 mm of the ground-truth slice location on the patient-specific 3D anatomy and 4.6 mm on the SSM, exemplifying its usefulness for slice localization during sonographic acquisitions. Code is publically available: \href{https://github.com/vuenc/slice-to-shape}{https://github.com/vuenc/slice-to-shape}
... Some research has focused on developing methods specifically for 2D misalignment correction in order to facilitate the following 3D reconstruction task. These include slice-to-slice registration (Goshtasby and Turner, 1996;McLeish et al., 2002;Villard et al., 2016), sliceto-volume-registration (Chandler et al., 2008;Su et al., 2014;Ferrante and Paragios, 2017), probabilistic segmentation maps generated with decision forests , combined image slice segmentation and alignment correction (Villard et al., 2018b), and statistical shape model based misalignment correction . Considerable research efforts have also focused on directly addressing these challenges as part of the 3D surface reconstruction task (Villard et al., 2018a;Mauger et al., 2019;Banerjee et al., 2021a). ...
Cine magnetic resonance imaging (MRI) is the current gold standard for the assessment of cardiac anatomy and function. However, it typically only acquires a set of two-dimensional (2D) slices of the underlying three-dimensional (3D) anatomy of the heart, thus limiting the understanding and analysis of both healthy and pathological cardiac morphology and physiology. In this paper, we propose a novel fully automatic surface reconstruction pipeline capable of reconstructing multi-class 3D cardiac anatomy meshes from raw cine MRI acquisitions. Its key component is a multi-class point cloud completion network (PCCN) capable of correcting both the sparsity and misalignment issues of the 3D reconstruction task in a unified model. We first evaluate the PCCN on a large synthetic dataset of biventricular anatomies and observe Chamfer distances between reconstructed and gold standard anatomies below or similar to the underlying image resolution for multiple levels of slice misalignment. Furthermore, we find a reduction in reconstruction error compared to a benchmark 3D U-Net by 32% and 24% in terms of Hausdorff distance and mean surface distance, respectively. We then apply the PCCN as part of our automated reconstruction pipeline to 1000 subjects from the UK Biobank study in a cross-domain transfer setting and demonstrate its ability to reconstruct accurate and topologically plausible biventricular heart meshes with clinical metrics comparable to the previous literature. Finally, we investigate the robustness of our proposed approach and observe its capacity to successfully handle multiple common outlier conditions.
Brain slicing is a commonly used technique in brain science research. In order to study the spatial distribution of labeled information, such as specific types of neurons and neuronal circuits, it is necessary to register the brain slice images to the 3D standard brain space defined by the reference atlas. However, the registration of 2D brain slice images to a 3D reference brain atlas still faces challenges in terms of accuracy, computational throughput, and applicability. In this paper, we propose the SlicesMapi, an interactive 3D registration method for brain slice sequence. This method corrects linear and non-linear deformations in both 3D and 2D spaces by employing dual constraints from neighboring slices and corresponding reference atlas slices and guarantees precision by registering images with full resolution, which avoids the information loss of image down-sampling implemented in the deep learning based registration methods. This method was applied to deal the challenges of unknown slice angle registration and non-linear deformations between the 3D Allen Reference Atlas and slices with cytoarchitectonic or autofluorescence channels. Experimental results demonstrate Dice scores of 0.9 in major brain regions, highlighting significant advantages over existing methods. Compared with existing methods, our proposed method is expected to provide a more accurate, robust, and efficient spatial localization scheme for brain slices. Therefore, the proposed method is capable of achieving enhanced accuracy in slice image spatial positioning.
In this paper, we introduce a Universal Motion Correction (UniMo) framework, leveraging deep neural networks to tackle the challenges of motion correction across diverse imaging modalities. Our approach employs advanced neural network architectures with equivariant filters, overcoming the limitations of current models that require iterative inference or retraining for new image modalities. UniMo enables one-time training on a single modality while maintaining high stability and adaptability for inference across multiple unseen image modalities. We developed a joint learning framework that integrates multimodal knowledge from both shape and images that faithfully improve motion correction accuracy despite image appearance variations. UniMo features a geometric deformation augmenter that enhances the robustness of global motion correction by addressing any local deformations whether they are caused by object deformations or geometric distortions, and also generates augmented data to improve the training process. Our experimental results, conducted on various datasets with four different image modalities, demonstrate that UniMo surpasses existing motion correction methods in terms of accuracy. By offering a comprehensive solution to motion correction, UniMo marks a significant advancement in medical imaging, especially in challenging applications with wide ranges of motion, such as fetal imaging. The code for this work is available online, https://github.com/IntelligentImaging/UNIMO/.
Histopathological findings depend on the distinction between normal and subtle changes in mucosal tissues. These distinctions are made based on the nature of the cell lining, nuclear hyperchromasia and cyto-architecture of glands. Diagnosis is mostly based on the experience of the pathologists, which is a subjective procedure. This research focuses on automating the process through 3D reconstruction of gastric histopathology images enhancing quantitative diagnosis. It emphasizes the advantages of 3D reconstruction from light microscopic serial section gastric images over whole slide imaging for clinical decision-making. Comparative analysis shows that processing and analyzing WSI scans are more expensive than light microscopy. While 2D imaging with light microscopy faces challenges due to limited focal depth, especially for thin sections, the objective of this research is to highlight the importance of transforming 2D gastric images into 3D by layering sequential sections. It will offer insights into glandular texture and volume, aiding in distinguishing between malignant glands and benign mimickers. The paper presents a summary and comparative examination of the registration techniques necessary for reconstructing 3D histopathology images. Additionally, it discusses future directions in the field of 3D histopathology reconstruction and acknowledges its potential benefits for research scholars. This will hold potential interest for pathologists when grading gastric carcinoma.
Surgical robotics application in the field of minimally invasive surgery has developed rapidly and has been attracting increasingly more research attention in recent years. A common consensus has been reached that surgical procedures are to become less traumatic and with the implementation of more intelligence and higher autonomy, which is a serious challenge faced by the environmental sensing capabilities of robotic systems. One of the main sources of environmental information for robots are images, which are the basis of robot vision. In this review article, we have divided clinical image into direct and indirect based on the object of information acquisition, and into continuous, intermittent continuous and discontinuous according to the target-tracking frequency. The characteristics and applications of the existing surgical robots in each category are introduced based on these two dimensions. Our purpose of conducting this review was to analyze, summarize, and discuss the current evidence on the general rules on the application of the image technologies for medical purpose. Our analysis insights and provides guidance conducive to the development of more advanced surgical robotics systems in the future.
Local feature matching has been a critical task in computer vision applications. In the early days of computer vision, local feature matching relied heavily on detector-based methods, where keypoint detectors were used to extract and describe the salient features of an image. However, with the advent of deep learning, detector-free methods that do not rely on keypoint detection have become increasingly popular. These methods directly learn feature descriptors from the image data, leading to improved performance and faster computation times. In this review, we explore the evolution of local feature matching from detector-based to detector-free methods, discussing the advantages and disadvantages of each approach and highlighting the recent advancements in the field. We also discuss the challenges and opportunities for future research in this area.
Magnetic resonance imaging (MRI) is the standard tool to image the human brain in vivo . In this domain, digital brain atlases are essential for subject-specific segmentation of anatomical regions of interest (ROIs) and spatial comparison of neuroanatomy from different subjects in a common coordinate frame. High-resolution, digital atlases derived from histology (e.g., Allen atlas [3], BigBrain [4], Julich [5]), are currently the state of the art and provide exquisite 3D cytoarchitectural maps, but lack probabilistic labels throughout the whole brain. Here we present NextBrain , a next-generation probabilistic atlas of human brain anatomy built from serial 3D histology and corresponding highly granular delineations of five whole brain hemispheres. We developed AI techniques to align and reconstruct ∼10,000 histological sections into coherent 3D volumes, as well as to semi-automatically trace the boundaries of 333 distinct anatomical ROIs on all these sections. Comprehensive delineation on multiple cases enabled us to build an atlas with probabilistic labels throughout the whole brain. Further, we created a companion Bayesian tool for automated segmentation of the 333 ROIs in any in vivo or ex vivo brain MRI scan using the NextBrain atlas. We showcase two applications of the atlas: automated segmentation of ultra-high-resolution ex vivo MRI and volumetric analysis of brain ageing based on ∼4,000 publicly available in vivo MRI scans. We publicly release the raw and aligned data (including an online visualisation tool), probabilistic atlas, and segmentation tool. By enabling researchers worldwide to analyse brain MRI scans at a superior level of granularity without manual effort or highly specific neuroanatomical knowledge, NextBrain will accelerate our quest to understand the human brain in health and disease.
Complicated deformation problems are frequently encountered in medical image registration tasks. Although various advanced registration models have been proposed, accurate and efficient deformable registration remains challenging, especially for handling the large volumetric deformations. To this end, we propose a novel recursive deformable pyramid (RDP) network for unsupervised non-rigid registration. Our network is a pure convolutional pyramid, which fully utilizes the advantages of the pyramid structure itself, but does not rely on any high-weight attentions or transformers. In particular, our network leverages a step-by-step recursion strategy with the integration of high-level semantics to predict the deformation field from coarse to fine, while ensuring the rationality of the deformation field. Meanwhile, due to the recursive pyramid strategy, our network can effectively attain deformable registration without separate affine pre-alignment. We compare the RDP network with several existing registration methods on three public brain magnetic resonance imaging (MRI) datasets, including LPBA, Mindboggle and IXI. Experimental results demonstrate our network consistently outcompetes state of the art with respect to the metrics of Dice score, average symmetric surface distance, Hausdorff distance, and Jacobian. Even for the data without the affine pre-alignment, our network maintains satisfactory performance on compensating for the large deformation.
The code is publicly available at
https://github.com/ZAX130/RDP.
Thyroid disorders are most commonly diagnosed using high-resolution Ultrasound (US). Longitudinal nodule tracking is a pivotal diagnostic protocol for monitoring changes in pathological thyroid morphology. This task, however, imposes a substantial cognitive load on clinicians due to the inherent challenge of maintaining a mental 3D reconstruction of the organ. We thus present a framework for automated US image slice localization within a 3D shape representation to ease how such sonographic diagnoses are carried out. Our proposed method learns a common latent embedding space between US image patches and the 3D surface of an individual’s thyroid shape, or a statistical aggregation in the form of a statistical shape model (SSM), via contrastive metric learning. Using cross-modality registration and Procrustes analysis, we leverage features from our model to register US slices to a 3D mesh representation of the thyroid shape. We demonstrate that our multi-modal registration framework can localize images on the 3D surface topology of a patient-specific organ and the mean shape of an SSM. Experimental results indicate slice positions can be predicted within an average of 1.2 mm of the ground-truth slice location on the patient-specific 3D anatomy and 4.6 mm on the SSM, exemplifying its usefulness for slice localization during sonographic acquisitions. Code is publically available: https://github.com/vuenc/slice-to-shape.
Multimodal 2D-3D co-registration is a challenging problem with numerous clinical applications, including improved diagnosis, radiation therapy, or interventional radiology. In this paper, we present StructuRegNet, a deep-learning framework that addresses this problem with three novel contributions. First, we combine a 2D-3D deformable registration network with an adversarial modality translation module, allowing each block to benefit from the signal of the other. Second, we solve the initialization challenge for 2D-3D registration by leveraging tissue structure through cascaded rigid areas guidance and distance field regularization. Third, StructuRegNet handles out-of-plane deformation without requiring any 3D reconstruction thanks to a recursive plane selection. We evaluate the quantitative performance of StructuRegNet for head and neck cancer between 3D CT scans and 2D histopathological slides, enabling pixel-wise mapping of low-quality radiologic imaging to gold-standard tumor extent and bringing biological insights toward homogenized clinical guidelines. Additionally, our method can be used in radiation therapy by mapping 3D planning CT into the 2D MR frame of the treatment day for accurate positioning and dose delivery. Our framework demonstrates superior results to traditional methods for both applications. It is versatile to different locations or magnitudes of deformation and can serve as a backbone for any relevant clinical context.
Pancreatic diseases are difficult to treat with high doses of radiation, as they often present both periodic and aperiodic deformations. Nevertheless, we expect that these difficulties can be overcome, and treatment results may be improved with the practical use of a device that can capture 2-D slices of organs during irradiation. However, since only a few 2-D slices can be taken, the 3-D motion needs to be estimated from partially observed information. In this study, we propose a physics-based framework for estimating the 3-D motion of organs, regardless of periodicity, from motion information obtained by 2-D slices in one or more directions and a regression model that estimates the accuracy of the proposed framework to select the optimal slice. Using information obtained by slice-to-slice registration and setting the surrounding organs as boundaries, the framework drives the physical models for estimating 3-D motion. The R2 score of the proposed regression model was greater than 0.9, and the RMSE was 0.357 mm. The mean errors were
mm using an axial slice and
mm using concurrent axial, sagittal, and coronal slices. Our results suggest that the proposed framework is comparable to volume-to-volume registration and is feasible.
In this paper we propose a deep learning approach for segmenting sub-cortical structures of the human brain in Magnetic Resonance (MR) image data. We draw inspiration from a state-of-the-art Fully-Convolutional Neural Network (F-CNN) architecture for semantic segmentation of objects in natural images, and adapt it to our task. Unlike previous CNN-based methods that operate on image patches, our model is applied on a full blown 2D image, without any alignment or registration steps at testing time. We further improve segmentation results by interpreting the CNN output as potentials of a Markov Random Field (MRF), whose topology corresponds to a volumetric grid. Alpha-expansion is used to perform approximate inference imposing spatial volumetric homogeneity to the CNN priors. We compare the performance of the proposed pipeline with a similar system using Random Forest-based priors, as well as state-of-art segmentation algorithms, and show promising results on two different brain MRI datasets.
Deep Neural Networks (DNNs) have recently shown outstanding performance on image classification tasks [14]. In this paper we go one step further and address the problem of object detection using DNNs, that is not only classifying but also precisely localizing objects of various classes. We present a simple and yet powerful formulation of object detection as a regression problem to object bounding box masks. We define a multi-scale inference procedure which is able to produce high-resolution object detections at a low cost by a few network applications. State-of-the-art performance of the approach is shown on Pascal VOC.
Deformable image registration plays a fundamental role in many clinical applications. In this paper we investigate the use of graphical models in the context of a particular type of image registration problem, known as slice-to-volume registration. We introduce a scalable, modular and flexible formulation that can accommodate low-rank and high order terms, that simultaneously selects the plane and estimates the in-plane deformation through a single shot optimization approach. The proposed framework is instantiated into different variants seeking either a compromise between computational efficiency (soft plane selection constraints and approximate definition of the data similarity terms through pair-wise components) or exact definition of the data terms and the constraints on the plane selection. Simulated and real-data in the context of ultrasound and magnetic resonance registration (where both framework instantiations as well as different optimization strategies are considered) demonstrate the potentials of our method.
We propose a new deformable slice-to-volume registration method to register a 2D Transvaginal Ultrasound (TVUS) to a 3D Magnetic Resonance (MR) volume. Our main goal is to find a cross-section of the MR volume such that the endometrial implants and their depth of infiltration can be mapped from TVUS to MR. The proposed TVUS-MR registration method uses contour to surface correspondences through a novel variational one-step deformable Iterative Closest Point (ICP) method. Specifically, we find a smooth deformation field while establishing point correspondences automatically. We demonstrate the accuracy of the proposed method by quantitative and qualitative tests on both semi-synthetic and clinical data. To generate semi-synthetic data sets, 3D surfaces are deformed with 4-40% degrees of deformation and then various intersection curves are obtained at 0-20° cutting angles. Results show an average mean square error of 5.7934±0.4615mm, average Hausdorff distance of 2.493±0.14mm, and average Dice similarity coefficient of 0.9750±0.0030.
Copyright © 2015 Elsevier Ltd. All rights reserved.
Deep learning allows computational models that are composed of multiple processing layers to learn representations of data with multiple levels of abstraction. These methods have dramatically improved the state-of-the-art in speech recognition, visual object recognition, object detection and many other domains such as drug discovery and genomics. Deep learning discovers intricate structure in large data sets by using the backpropagation algorithm to indicate how a machine should change its internal parameters that are used to compute the representation in each layer from the representation in the previous layer. Deep convolutional nets have brought about breakthroughs in processing images, video, speech and audio, whereas recurrent nets have shone light on sequential data such as text and speech.
Convolutional neural networks (CNNs) have recently been very successful in a
variety of computer vision tasks, especially on those linked to recognition.
Optical flow estimation has not been among the tasks where CNNs were
successful. In this paper we construct appropriate CNNs which are capable of
solving the optical flow estimation problem as a supervised learning task. We
propose and compare two architectures: a generic architecture and another one
including a layer that correlates feature vectors at different image locations.
Since existing ground truth data sets are not sufficiently large to train a
CNN, we generate a synthetic Flying Chairs dataset. We show that networks
trained on this unrealistic data still generalize very well to existing
datasets such as Sintel and KITTI, achieving competitive accuracy at frame
rates of 5 to 10 fps.
Purpose/Aims of the study: Bone's hierarchical structure can be visualized using a variety of methods. Many techniques, such as light and electron microscopy generate two-dimensional (2D) images, while micro-computed tomography (µCT) allows a direct representation of the three-dimensional (3D) structure. In addition, different methods provide complementary structural information, such as the arrangement of organic or inorganic compounds. The overall aim of the present study is to answer bone research questions by linking information of different 2D and 3D imaging techniques. A great challenge in combining different methods arises from the fact that they usually reflect different characteristics of the real structure.
We investigated bone during healing by means of µCT and a couple of 2D methods. Backscattered electron images were used to qualitatively evaluate the tissue's calcium content and served as a position map for other experimental data. Nanoindentation and X-ray scattering experiments were performed to visualize mechanical and structural properties.
We present an approach for the registration of 2D data in a 3D µCT reference frame, where scanning electron microscopies serve as a methodic link. Backscattered electron images are perfectly suited for registration into µCT reference frames, since both show structures based on the same physical principles. We introduce specific registration tools that have been developed to perform the registration process in a semi-automatic way.
By applying this routine, we were able to exactly locate structural information (e.g. mineral particle properties) in the 3D bone volume. In bone healing studies this will help to better understand basic formation, remodeling and mineralization processes.
Capturing an enclosing volume of moving subjects and organs using fast individual image slice acquisition has shown promise in dealing with motion artefacts. Motion between slice acquisitions results in spatial inconsistencies that can be resolved by slice-to-volume reconstruction (SVR) methods to provide high quality 3D image data. Existing algorithms are, however, typically very slow, specialised to specific applications and rely on approximations, which impedes their potential clinical use. In this paper, we present a fast multi-GPU accelerated framework for slice-to-volume reconstruction. It is based on optimised 2D/3D registration, super-resolution with automatic outlier rejection and an additional (optional) intensity bias correction. We introduce a novel and fully automatic procedure for selecting the image stack with least motion to serve as an initial registration target. We evaluate the proposed method using artificial motion corrupted phantom data as well as clinical data, including tracked freehand ultrasound of the liver and fetal Magnetic Resonance Imaging. We achieve speed-up factors greater than 30 compared to a single CPU system and greater than 10 compared to currently available state-of-the-art multi-core CPU methods. We ensure high reconstruction accuracy by exact computation of the pointspread function for every input data point, which has not previously been possible due to computational limitations. Our framework and its implementation is scalable for available computational infrastructures and tests show a speed-up factor of 1.70 for each additional GPU. This paves the way for the online application of image based reconstruction methods during clinical examinations. The source code for the proposed approach is publicly available.
The aim of this study was the registration of digitized thin 2D sections of mouse vertebrae and tibiae used for histomorphometry of trabecular bone structure into 3D micro computed tomography (μCT) datasets of the samples from which the sections were prepared. Intensity-based and segmentation-based registrations (SegRegs) of 2D sections and 3D μCT datasets were applied. As the 2D sections were deformed during their preparation, affine registration for the vertebrae was used instead of rigid registration. Tibiae sections were additionally cut on the distal end, which subsequently undergone more deformation so that elastic registration was necessary. The Jaccard distance was used as registration quality measure. The quality of intensity-based registrations and SegRegs was practically equal, although precision errors of the elastic registration of segmentation masks in tibiae were lower, while those in vertebrae were lower for the intensity-based registration. Results of SegReg significantly depended on the segmentation of the μCT datasets. Accuracy errors were reduced from approximately 64% to 42% when applying affine instead of rigid transformations for the vertebrae and from about 43% to 24% when using B-spline instead of rigid transformations for the tibiae. Accuracy errors can also be caused by the difference in spatial resolution between the thin sections (pixel size: 7.25 μm) and the μCT data (voxel size: 15 μm). In the vertebrae, average deformations amounted to a 6.7% shortening along the direction of sectioning and a 4% extension along the perpendicular direction corresponding to 0.13-0.17 mm. Maximum offsets in the mouse tibiae were 0.16 mm on average.
In muscle dystrophy studies, registration of histological image with MRI image volume enables cross validation of MRI biomarkers using pathological result. However, correlation of 2D histology slice with 3D MRI volume is technically challenging due to the potentially non-orthogonal slice plane and incomplete or distorted histological slice. This paper presents an efficient method to directly perform the 2D-3D registration. The method is unique in that it uses smart phone as a navigation tool for initial alignment followed by an overlap invariant mutual information-based refinement. Experimental results using animal muscle samples images from a 3T MRI and HE stained histological images show that the proposed method is capable of aligning the histological slice with an oblique slice in MR volume.
A continuous two‐dimensional region is partitioned into a fine rectangular array of sites or “pixels”, each pixel having a particular “colour” belonging to a prescribed finite set. The true colouring of the region is unknown but, associated with each pixel, there is a possibly multivariate record which conveys imperfect information about its colour according to a known statistical model. The aim is to reconstruct the true scene, with the additional knowledge that pixels close together tend to have the same or similar colours. In this paper, it is assumed that the local characteristics of the true scene can be represented by a non‐degenerate Markov random field. Such information can be combined with the records by Bayes' theorem and the true scene can be estimated according to standard criteria. However, the computational burden is enormous and the reconstruction may reflect undesirable large‐scale properties of the random field. Thus, a simple, iterative method of reconstruction is proposed, which does not depend on these large‐scale characteristics. The method is illustrated by computer simulations in which the original scene is not directly related to the assumed random field. Some complications, including parameter estimation, are discussed. Potential applications are mentioned briefly.
We study the question of feature sets for robust visual object recognition, adopting linear SVM based human detection as a test case. After reviewing existing edge and gradient based descriptors, we show experimentally that grids of Histograms of Oriented Gradient (HOG) descriptors significantly outperform existing feature sets for human detection. We study the influence of each stage of the computation on performance, concluding that fine-scale gradients, fine orientation binning, relatively coarse spatial binning, and high-quality local contrast normalization in overlapping descriptor blocks are all important for good results. The new approach gives near-perfect separation on the original MIT pedestrian database, so we introduce a more challenging dataset containing over 1800 annotated human images with a large range of pose variations and backgrounds.
Consistency of image edge filtering is of prime importance for 3D interpretation of image sequences using feature tracking algorithms. To cater for image regions containing texture and isolated features, a combined corner and edge detector based on the local auto-correlation function is utilised, and it is shown to perform with good consistency on natural imagery.
Convolutional networks are powerful visual models that yield hierarchies of
features. We show that convolutional networks by themselves, trained
end-to-end, pixels-to-pixels, exceed the state-of-the-art in semantic
segmentation. Our key insight is to build "fully convolutional" networks that
take input of arbitrary size and produce correspondingly-sized output with
efficient inference and learning. We define and detail the space of fully
convolutional networks, explain their application to spatially dense prediction
tasks, and draw connections to prior models. We adapt contemporary
classification networks (AlexNet, the VGG net, and GoogLeNet) into fully
convolutional networks and transfer their learned representations by
fine-tuning to the segmentation task. We then define a novel architecture that
combines semantic information from a deep, coarse layer with appearance
information from a shallow, fine layer to produce accurate and detailed
segmentations. Our fully convolutional network achieves state-of-the-art
segmentation of PASCAL VOC (20% relative improvement to 62.2% mean IU on 2012),
NYUDv2, and SIFT Flow, while inference takes one third of a second for a
typical image.
This paper aims to solve a fundamental problem in intensity-based 2D/3D registration, which concerns the limited capture range and need for very good initialization of state-of-the-art image registration methods. We propose a regression approach that learns to predict rotations and translations of arbitrary 2D image slices from 3D volumes, with respect to a learned canonical atlas co-ordinate system. To this end, we utilize Convolutional Neural Networks (CNNs) to learn the highly complex regression function that maps 2D image slices into their correct position and orientation in 3D space. Our approach is attractive in challenging imaging scenarios, where significant subject motion complicates reconstruction performance of 3D volumes from 2D slice data. We extensively evaluate the effectiveness of our approach quantitatively on simulated MRI brain data with extreme random motion. We further demonstrate qualitative results on fetal MRI where our method is integrated into a full reconstruction and motion compensation pipeline. With our CNN regression approach we obtain an average prediction error of 7 mm on simulated data, and convincing reconstruction quality of images of very young fetuses where previous methods fail. We further discuss applications to Computed Tomography (CT) and X-Ray projections. Our approach is a general solution to the 2D/3D initialization problem. It is computationally efficient, with prediction times per slice of a few milliseconds, making it suitable for real-time scenarios.
Multimodal registration is a challenging problem due the high variability of tissue appearance under different imaging modalities. The crucial component here is the choice of the right similarity measure. We make a step towards a general learning-based solution that can be adapted to specific situations and present a metric based on a convolutional neural network. Our network can be trained from scratch even from a few aligned image pairs. The metric is validated on intersubject deformable registration on a dataset different from the one used for training, demonstrating good generalization. In this task, we outperform mutual information by a significant margin.
We present a method to predict image deformations based on patch-wise image appearance. Specifically, we design a patch-based deep encoder-decoder network which learns the pixel/voxel-wise mapping between image appearance and registration parameters. Our approach can predict general deformation parameterizations, however, we focus on the large deformation diffeomorphic metric mapping (LDDMM) registration model. By predicting the LDDMM momentum-parameterization we retain the desirable theoretical properties of LDDMM, while reducing computation time by orders of magnitude: combined with patch pruning, we achieve a / speed-up compared to GPU-based optimization for 2D/3D image registration. Our approach has better prediction accuracy than predicting deformation or velocity fields and results in diffeomorphic transformations. Additionally, we create a Bayesian probabilistic version of our network, which allows evaluation of deformation field uncertainty through Monte Carlo sampling using dropout at test time. We show that deformation uncertainty highlights areas of ambiguous deformations. We test our method on the OASIS brain image dataset in 2D and 3D.
3D cardiac MR imaging enables accurate analysis of cardiac morphology and physiology. However, due to the requirements for long acquisition and breath-hold, the clinical routine is still dominated by multi-slice 2D imaging, which hamper the visualization of anatomy and quantitative measurements as relatively thick slices are acquired. As a solution, we propose a novel image super-resolution (SR) approach that is based on a residual convolutional neural network (CNN) model. It reconstructs high resolution 3D volumes from 2D image stacks for more accurate image analysis. The proposed model allows the use of multiple input data acquired from different viewing planes for improved performance. Experimental results on 1233 cardiac short and long-axis MR image stacks show that the CNN model outperforms state-of-the-art SR methods in terms of image quality while being computationally efficient. Also, we show that image segmentation and motion tracking benefits more from SR-CNN when it is used as an initial upscaling method than conventional interpolation methods for the subsequent analysis.
Registration of histopathology volumes to Magnetic Resonance Images(MRI) is a crucial step for finding correlations in Prostate Cancer (PCa) and assessing tumor agressivity. This paper proposes a two-stage framework aimed at registering both modalities. Firstly, Speeded-Up Robust Features (SURF) algorithm and a context-based search is used to automatically determine slice correspondences between MRI and histology volumes. This step initializes a multimodal nonrigid registration strategy, which allows to propagate histology slices to MRI. Evaluation was performed on 5 prospective studies using a slice index score and landmark distances. With respect to a manual ground truth, the first stage of the framework exhibited an average error of 1,54 slice index and 3,51 mm in the prostate specimen. The reconstruction of a three-dimensional Whole-Mount Histology (WMH) shows promising results aimed to perform later PCa pattern detection and staging.
Simultaneous multi-slice (SMS) echo-planar imaging has had a huge impact on the acceleration and routine use of diffusion-weighted MRI (DWI) in neuroimaging studies in particular the human connectome project; but also holds the potential to facilitate DWI of moving subjects, as proposed by the new technique developed in this paper. We present a novel registration-based motion tracking technique that takes advantage of the multi-plane coverage of the anatomy by simultaneously acquired slices to enable robust reconstruction of neural microstructure from SMS DWI of moving subjects. Our technique constitutes three main components: (1) motion tracking and estimation using SMS registration, (2) detection and rejection of intra-slice motion, and (3) robust reconstruction. Quantitative results from 14 volunteer subject experiments and the analysis of motion-corrupted SMS DWI of 6 children indicate robust reconstruction in the presence of continuous motion and the potential to extend the use of SMS DWI in very challenging populations.
A popular technique to reduce respiratory motion for cardiovascular magnetic resonance is to perform a multi-slice acquisition in which a patient holds their breath multiple times during the scan. The feasibility of rigid slice-to-volume registration to correct for misalignments of slice stacks in such images due to differing breath-hold positions is explored. Experimental results indicate that slice-to-volume registration can compensate for the typical misalignments expected. Correction of slice misalignment results in anatomically more correct images, as well as improved left ventricular volume measurements. The interstudy reproducibility has also been improved reducing the number of samples needed for cardiac MR studies.
Rigid slice-to-volume registration is a challenging task, which finds application in medical imaging problems like image fusion for image guided surgeries and motion correction for volume reconstruction. It is usually formulated as an optimization problem and solved using standard continuous methods. In this paper, we discuss how this task be formulated as a discrete labeling problem on a graph. Inspired by previous works on discrete estimation of linear transformations using Markov Random Fields (MRFs), we model it using a pairwise MRF, where the nodes are associated to the rigid parameters, and the edges encode the relation between the variables. We compare the performance of the proposed method to a continuous formulation optimized using simplex, and we discuss how it can be used to further improve the accuracy of our approach. Promising results are obtained using a monomodal dataset composed of magnetic resonance images (MRI) of a beating heart.
We present a method to predict image deformations based on patch-wise image appearance. Specifically, we design a patch-based deep encoder-decoder network which learns the pixel/voxel-wise mapping between image appearance and registration parameters. Our approach can predict general deformation parameterizations, however, we focus on the large deformation diffeomorphic metric mapping (LDDMM) registration model. By predicting the LDDMM momentum-parameterization we retain the desirable theoretical properties of LDDMM, while reducing computation time by orders of magnitude: combined with patch pruning, we achieve a 1500x/66x speed up compared to GPU-based optimization for 2D/3D image registration. Our approach has better prediction accuracy than predicting deformation or velocity fields and results in diffeomorphic transformations. Additionally, we create a Bayesian probabilistic version of our network, which allows evaluation of deformation field uncertainty through Monte Carlo sampling using dropout at test time. We show that deformation uncertainty highlights areas of ambiguous deformations. We test our method on the OASIS brain image dataset in 2D and 3D.
Computational vision, visual computing and biomedical image analysis have made tremendous progress over the past two decades. This is mostly due the development of efficient learning and inference algorithms which allow better and richer modeling of image and visual understanding tasks. Hyper-Graph representations are among the most prominent tools to address such perception through the casting of perception as a graph optimization problem. In this paper, we briefly introduce the importance of such representations, discuss their strength and limitations, provide appropriate strategies for their inference and present their application to address a variety of problems in biomedical image analysis.
Convolutional networks are powerful visual models that yield hierarchies of features. We show that convolutional networks by themselves, trained end-to-end, pixels-to-pixels, improve on the previous best result in semantic segmentation. Our key insight is to build "fully convolutional" networks that take input of arbitrary size and produce correspondingly-sized output with efficient inference and learning. We define and detail the space of fully convolutional networks, explain their application to spatially dense prediction tasks, and draw connections to prior models. We adapt contemporary classification networks (AlexNet, the VGG net, and GoogLeNet) into fully convolutional networks and transfer their learned representations by fine-tuning to the segmentation task. We then define a skip architecture that combines semantic information from a deep, coarse layer with appearance information from a shallow, fine layer to produce accurate and detailed segmentations. Our fully convolutional network achieves improved segmentation of PASCAL VOC (30% relative improvement to 67.2% mean IU on 2012), NYUDv2, SIFT Flow, and PASCAL-Context, while inference takes one tenth of a second for a typical image.
Histological images provide reliable information on tissue characteristics which can be used to validate and improve our understanding for developing radiological imaging analysis methods. However, due to the large amount of deformation in histology stemming from resected tissues, estimating spatial correspondence with other imaging modalities is a challenging image registration problem. In this work we develop a three-stage framework for nonlinear registration between ex vivo MRI and histology of rectal cancer. For this multi-modality image registration task, two similarity metrics from patch-based feature transformations were used: the dense Scale Invariant Feature Transform (dense SIFT) and the Modality Independent Neighbourhood Descriptor (MIND). The potential of our method is demonstrated on a dataset of eight rectal histology images from two patients using annotated landmarks. The mean registration error was 1.80 mm after the rigid registration steps which improved to 1.08 mm after nonlinear motion correction using dense SIFT and to 1.52 mm using MIND.
This work proposes a novel approach for motion-robust diffusion-weighted (DW) brain MRI reconstruction through tracking temporal head motion using slice-to-volume registration. The slice-level motion is estimated through a filtering approach that allows tracking the head motion during the scan and correcting for out-of-plane inconsistency in the acquired images. Diffusion-sensitized image slices are registered to a base volume sequentially over time in the acquisition order where an outlier-robust Kalman filter, coupled with slice-to-volume registration, estimates head motion parameters. Diffusion gradient directions are corrected for the aligned DWI slices based on the computed rotation parameters and the diffusion tensors are directly estimated from the corrected data at each voxel using weighted linear least squares. The method was evaluated in DWI scans of adult volunteers who deliberately moved during scans as well as clinical DWI of 28 neonates and children with different types of motion. Experimental results showed marked improvements in DWI reconstruction using the proposed method compared to the state-of-the-art DWI analysis based on volume-to-volume registration. This approach can be readily used to retrieve information from motion-corrupted DW imaging data.
Image-based ultrasound to magnetic resonance image (US-MRI) registration can be an invaluable tool in image-guided neuronavigation systems. State-of-the-art commercial and research systems utilize image-based registration to assist in functions such as brain-shift correction, image fusion, and probe calibration.
Since traditional US-MRI registration techniques use reconstructed US volumes or a series of tracked US slices, the functionality of this approach can be compromised by the limitations of optical or magnetic tracking systems in the neurosurgical operating room. These drawbacks include ergonomic issues, line-of-sight/magnetic interference, and maintenance of the sterile field. For those seeking a US vendor-agnostic system, these issues are compounded with the challenge of instrumenting the probe without permanent modification and calibrating the probe face to the tracking tool.
To address these challenges, this paper explores the feasibility of a real-time US-MRI volume registration in a small virtual craniotomy site using a single slice. We employ the Linear Correlation of Linear Combination (LC2) similarity metric in its patch-based form on data from MNI’s Brain Images for Tumour Evaluation (BITE) dataset as a PyCUDA enabled Python module in Slicer. By retaining the original orientation information, we are able to improve on the poses using this approach. To further assist the challenge of US-MRI registration, we also present the BOXLC2 metric which demonstrates a speed improvement to LC2, while retaining a similar accuracy in this context.
Purpose:
As an inexpensive, noninvasive, and portable clinical imaging modality, ultrasound (US) has been widely employed in many interventional procedures for monitoring potential tissue deformation, surgical tool placement, and locating surgical targets. The application requires the spatial mapping between 2D US images and 3D coordinates of the patient. Although positions of the devices (i.e., ultrasound transducer) and the patient can be easily recorded by a motion tracking system, the spatial relationship between the US image and the tracker attached to the US transducer needs to be estimated through an US calibration procedure. Previously, various calibration techniques have been proposed, where a spatial transformation is computed to match the coordinates of corresponding features in a physical phantom and those seen in the US scans. However, most of these methods are difficult to use for novel users.
Methods:
We proposed an ultrasound calibration method by constructing a phantom from simple Lego bricks and applying an automated multi-slice 2D-3D registration scheme without volumetric reconstruction. The method was validated for its calibration accuracy and reproducibility.
Results:
Our method yields a calibration accuracy of [Formula: see text] mm and a calibration reproducibility of 1.29 mm.
Conclusion:
We have proposed a robust, inexpensive, and easy-to-use ultrasound calibration method.
High resolution MRI images of the beating heart permit observation of detailed anatomical features and enable quantification of small changes in metrics of cardiac function. To obtain approximately isotropic sampling with an adequate spatial and temporal resolution, these images need to be acquired in multiple breath-holds. They are, therefore, often affected by through-plane discontinuities due to inconsistent breath-hold positions. This paper presents a method to correct for these discontinuities by performing breath-hold-by-breath-hold registration of high resolution 3D data to radial long axis images. The corrected images appear free of discontinuities, and it was found that they could be delineated more reproducibly than uncorrected images. This reduces the sample size required to detect systematic changes in blood pool volume by 57% at end systole and 78% at end diastole.
In this paper, we present a Convolutional Neural Network (CNN) regression approach to address the two major limitations of existing intensity-based 2-D/3-D registration technology: 1) slow computation and 2) small capture range. Different from optimization-based methods, which iteratively optimize the transformation parameters over a scalar-valued metric function representing the quality of the registration, the proposed method exploits the information embedded in the appearances of the digitally reconstructed radiograph and Xray images, and employs CNN regressors to directly estimate the transformation parameters. An automatic feature extraction step is introduced to calculate 3-D pose-indexed features that are sensitive to the variables to be regressed while robust to other factors. The CNN regressors are then trained for local zones and applied in a hierarchical manner to break down the complex regression task into multiple simpler sub-tasks that can be learned separately. Weight sharing is furthermore employed in the CNN regression model to reduce the memory footprint. The proposed approach has been quantitatively evaluated on 3 potential clinical applications, demonstrating its significant advantage in providing highly accurate real-time 2-D/3-D registration with a significantly enlarged capture range when compared to intensitybased methods.
In research in cardiac development it is unavoidable to use sections of biological tissue, and it is difficult to interpret these sections because of the complexity of the developing heart. 3D computer reconstructions can provide an anatomical context of such sections and make their interpretation easier. However, for practical reasons, researchers often do not stain a complete series of sections and, therefore cannot make a 3D reconstruction. We are developing a program for tracing the anatomical context of individual tissue sections, by automatically fitting 2D sections into 3D reference reconstructions and thus enabling the retrieval of their right location and orientation. In this paper we show that a basic version of the program, using a primarily brute force pixel-based approach, already gives promising results. The performance of this basic program can substantially be improved if the program is extended with the use of relatively simple image features.
In this paper we propose a novel method based on discrete optimization of high order graphs, to perform deformable slice-to-volume registration of 2D images and 3D volumes. To this end, a 2D grid superimposed to the image is considered with control points deforming in 3D and their deformations corresponding to the label space. Geometrical consistency (unique plane selection) and deformation smoothness (in-plane deformations) as well as image similarity (visual matching) are encoded in different third order cliques. The proposed formulation is optimized through its mapping to a factor graph using conventional graph optimization methods. A dataset composed of 2D slices and 3D MRI volumes of the heart was used to evaluate its accuracy leading to very promising results.
We present a rigid registration framework for freehand 2D ultrasound sweeps to 3D CT of liver tumours. The method registers the 2D sweeps in a group-wise manner, without the need for prior 3D ultrasound compounding or probe tracking during acquisition. We first introduce a specific acquisition model to keep the dimension of this problem reasonable. Only seven parameters are indeed required to register the images. These are estimated using simulated annealing optimization of a robust modality-independent similarity measure. The framework contrasts the current methods that rely on tracking devices and phantom calibration, which are often difficult to use routinely in clinical practice. Our results on both synthetic and real data show that the method is well-suited for such ultrasound-CT registration of liver tumours.
please refer to the later journal version / preprint
This paper describes a novel respiratory motion compensation (MC) technique for hybrid PET/MR system. Existing PET/MR respiratory MC techniques mainly rely on respiratory gating to reconstruct 3D MR images for different respiratory phases, and use image registration techniques to estimate the deformation between different respiratory phases. A well known limitation of respiratory gating is its assumption that breathing motion is perfectly periodic, therefore MC techniques based on gated MR are sensitive to irregular breathing patterns. To address this limitation, the proposed technique uses dynamic 2D MRIs for PET/MR respiratory motion estimation. A static 3D MRI and a series of dynamic 2D MRIs are acquired before and during the PET data acquisition, respectively. The dynamic 2D MRIs are registered against the static 3D MRI using a novel deformable 2D+t to 3D image registration method to derive a 3D+t deformation field, which is then used for motion corrected PET reconstruction. The proposed method is validated on synthetic PET/MR data and real MR data, demonstrating its efficacy to compensate for irregular respiratory motion and to deblur the PET image.
This paper introduces a novel decomposed graphical model to deal with slice-to-volume registration in the context of medical images and image-guided surgeries.
We present a new non-rigid slice-to-volume registration method whose main contribution is the ability to decouple the plane selection and the in-plane deformation parts of the transformation-through two distinct graphs-toward reducing the complexity of the model while being able to obtain simultaneously the solution for both of them. To this end, the plane selection process is expressed as a local graph-labeling problem endowed with planarity satisfaction constraints, which is then directly linked with the deformable part through the data registration likelihoods. The resulting model is modular with respect to the image metric, can cope with arbitrary in-plane regularization terms and inherits excellent properties in terms of computational efficiency.
The proof of concept for the proposed formulation is done using cardiac MR sequences of a beating heart (an artificially generated 2D temporal sequence is extracted using real data with known ground truth) as well as multimodal brain images involving ultrasound and computed tomography images. We achieve state-of-the-art results while decreasing the computational time when we compare with another method based on similar techniques.
We confirm that graphical models and discrete optimization techniques are suitable to solve non-rigid slice-to-volume registration problems. Moreover, we show that decoupling the graphical model and labeling it using two lower-dimensional label spaces, we can achieve state-of-the-art results while substantially reducing the complexity of our method and moving the approach close to real clinical applications once considered in the context of modern parallel architectures.
In this paper we show how to learn directly from image data (i.e., without
resorting to manually-designed features) a general similarity function for
comparing image patches, which is a task of fundamental importance for many
computer vision problems. To encode such a function, we opt for a CNN-based
model that is trained to account for a wide variety of changes in image
appearance. To that end, we explore and study multiple neural network
architectures, which are specifically adapted to this task. We show that such
an approach can significantly outperform the state-of-the-art on several
problems and benchmark datasets.
