No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
ANHIR: Automatic Non-rigid Histological Image Registration Challenge
Abstract
Automatic Non-rigid Histological Image Registration (ANHIR) challenge was organized to compare the performance of image registration algorithms on several kinds of microscopy histology images in a fair and independent manner. We have assembled 8 datasets, containing 355 images with 18 different stains, resulting in 481 image pairs to be registered. Registration accuracy was evaluated using manually placed landmarks. In total, 256 teams registered for the challenge, 10 submitted the results, and 6 participated in the workshop. Here, we present the results of 7 well-performing methods from the challenge together with 6 well-known existing methods. The best methods used coarse but robust initial alignment, followed by non-rigid registration, used multiresolution, and were carefully tuned for the data at hand. They outperformed off-the-shelf methods, mostly by being more robust. The best methods could successfully register over 98 % of all landmarks and their mean landmark registration accuracy (TRE) was 0.44 % of the image diagonal. The challenge remains open to submissions and all images are available for download.
... Popular unsupervised DL registration methods with traditional similarity measures as loss functions are inefficient in handling such a registration task [20] because traditional similarity measures depend on the intensity patterns of images, which vary largely between multiple stained histological images. To deal with it, researchers introduce geometrically-discriminative structural information such as contours, lines, and keypoints to guide the training process of the network [19][20][21][22]. The most popular structural information is keypoints because they are easy to extract and match [23]. ...
... The proposed method was evaluated on the Automatic Nonrigid Histology Image Registration (ANHIR) [21] website and AutomatiC Registration Of Breast cAncer Tissue (ACROBAT) website [40]. It ranked 1st with a team name 'ddlucky123' as of August 6th, 2024. ...
... Daly et al. proposed a supervised hybrid classification-segmentation architecture [44]. Zhao et al. proposed a supervised method [21]. Besides, Gatenbee et al. [45] performed high-resolution non-rigid registration based on overlapping tissue masks and provided a free, fast, easy-to-use pipeline (i.e., VALIS) for registering histological images. ...
Histological image registration is a fundamental task in histological image analysis. It is challenging because of substantial appearance differences due to multiple staining. Keypoint correspondences, i.e., matched keypoint pairs, have been introduced to guide unsupervised deep learning (DL) based registration methods to handle such a registration task. This paper proposes an iterative keypoint correspondence-guided (IKCG) unsupervised network for non-rigid histological image registration. Fixed deep features and learnable deep features are introduced as keypoint descriptors to automatically establish keypoint correspondences, the distance between which is used as a loss function to train the registration network. Fixed deep features extracted from DL networks that are pre-trained on natural image datasets are more discriminative than handcrafted ones, benefiting from the deep and hierarchical nature of DL networks. The intermediate layer outputs of the registration networks trained on histological image datasets are extracted as learnable deep features, which reveal unique information for histological images. An iterative training strategy is adopted to train the registration network and optimize learnable deep features jointly. Benefiting from the excellent matching ability of learnable deep features optimized with the iterative training strategy, the proposed method can solve the local non-rigid large displacement problem, an inevitable problem usually caused by misoperation, such as tears in producing tissue slices. The proposed method is evaluated on the Automatic Non-rigid Histology Image Registration (ANHIR) website and AutomatiC Registration Of Breast cAncer Tissue (ACROBAT) website. It ranked 1st on both websites as of August 6th, 2024.
... Current CT-MR registration methods lack optimization for these challenges or integration of a priori knowledge of such deformations into their models. Furthermore, the annotation process for histological images demands more manpower and resources compared to CT/MR images [15,16]. Consequently, existing methods struggle in histological image registration. ...
... The ANHIR-dataset, introduced by Borovec et al. [15], encompasses eight subsets of histological images namely breast, chronic obstructive air way disease (COAD), gastric, kidney, lung lesion, lung lobes, mammary-gland, and mice-kidney respectively. It comprises a total of 481 cross-modal image pairs sourced from 18 different tissue staining operations. ...
... The quantitative indicator utilized in this article is the relative target registration error (rTRE) evaluation system [15], which is assessed following the evaluation system proposed in the Automatic Non-rigid Histological Image Registration Challenge (ANHIR). The calculation method is outlined as follows: ...
Multi-modal histological image registration tasks pose significant challenges due to tissue staining operations causing partial loss and folding of tissue. Convolutional neural network (CNN) and generative adversarial network (GAN) are pivotal in medical image registration. However, existing methods often struggle with severe interference and deformation, as seen in histological images of conditions like Cushing’s disease. We argue that the failure of current approaches lies in underutilizing the feature extraction capability of the discriminator in GAN. In this study, we propose a novel multi-modal registration approach GAN-DIRNet based on GAN for deformable histological image registration. To begin with, the discriminators of two GANs are embedded as a new dual parallel feature extraction module into the unsupervised registration networks, characterized by implicitly extracting feature descriptors of specific modalities. Additionally, modal feature description layers and registration layers collaborate in unsupervised optimization, facilitating faster convergence and more precise results. Lastly, experiments and evaluations were conducted on the registration of the Mixed National Institute of Standards and Technology database (MNIST), eight publicly available datasets of histological sections and the Clustering-Registration-Classification-Segmentation (CRCS) dataset on the Cushing’s disease. Experimental results demonstrate that our proposed GAN-DIRNet method surpasses existing approaches like DIRNet in terms of both registration accuracy and time efficiency, while also exhibiting robustness across different image types.
... In addition, image registration is an important inverse problem in the domain of medical imaging which is summarized in Refs. [5][6][7], where radiation therapy, computational anatomy, intervention and treatment planning, computer-aided diagnosis, fusion of different modalities, monitoring of diseases, or motion correction. ...
... We train CNN 1 by minimizing the loss function L 1 defined in (6). The purpose of this loss function is to ensure that the network produces accurate geometric information for each image. ...
... The values of the above errors are summarized in Table 1. In addition to the above two measures, to evaluate the quality visually, we use the curves of evaluation metrics (6) and (11), that illustrate their values during the training iterations for the detection and registration model, and the fused image before and after registration. We also assess if the map ϕ is diffeomorphic by computing the Jacobian determinant of each displacement field i.e. by checking the minimum of the Jacobian determinant det (∇ϕ(x). ...
Medical image registration is a crucial step in computer-assisted medical diagnosis, and has seen significant progress with the adoption of deep learning methods like convolutional neural networks (CNN). Creating a deep learning network for image registration is complex because humans can’t easily prepare or supervise the training data unless it’s very basic. This article presents an innovative approach to unsupervised deep learning-based multilevel image registration approach. We propose to develop a CNN to detect the geometric features, such as edges and thin structures, from images using a loss function derived from the Blake-Zisserman energy. This method enables the detection of discontinuities at different scales without relying on labeled data. Subsequently, we use this geometric information extracted from the input images, to define a second loss function and to perform our multimodal image registration process. Furthermore, we introduce a novel deep neural network architecture for multilevel image registration, offering enhanced precision and efficiency compared to traditional methods. Numerical simulations are employed to demonstrate the accuracy and relevance of our approach. We perform some numerical simulations to show the accuracy and the relevance of our approach for multimodal registration and its multilevel implementation.
... This complicates an accurate registration between each neighboring section with increasing complexity as the distance between sections increases. Although several algorithms have been proposed to solve the task of registering serial sections [5][6][7][8][9][10][11][12][13][14][15][16][17] , they are largely based on FFPE tissue sectioned at close distance (e.g., 4-6 µm). FFPE sections are far less fragile and prone to artifacts compared to frozen tissue required for many spatial omics measurements, therefore requiring robust registration methods. ...
... Ensuring that molecular imaging experiments are performed on tissue sections as close as possible is crucial to reduce the effects of tissue heterogeneity, a problem not unique to MIIT. Another limitation is the occurrence of tissue damage 8,14 . Although GreedyFHist achieved high accuracy in registration, some images could not be registered accurately due to tissue damage (Table 3, 4, Figure 4), which can become problematic during groupwise registration. ...
... There were 3 annotators in total and each set of annotations were cross validated. These metrics have previously been used in the ANHIR challenge for evaluating the accuracy of different registration algorithms 8,9 . The TRE is defined as where and are a pair of matching landmarks from warped and fixed landmarks. ...
To truly understand the cancer biology of heterogenous tumors in the context of precision medicine, it is crucial to use analytical methodology capable of capturing the complexities of multiple omics levels, as well as the spatial heterogeneity of cancer tissue. Different molecular imaging techniques, such as mass spectrometry imaging (MSI) and spatial transcriptomics (ST) achieve this goal by spatially detecting metabolites and mRNA, respectively. To take full analytical advantage of such multi-omics data, the individual measurements need to be integrated into one dataset. We present MIIT (Multi-Omics Imaging Integration Toolset), a Python framework for integrating spatially resolved multi-omics data. MIIT's integration workflow consists of performing a grid projection of spatial omics data, registration of stained serial sections, and mapping of MSI-pixels to the spot resolution of Visium 10x ST data. For the registration of serial sections, we designed GreedyFHist, a registration algorithm based on the Greedy registration tool. We validated GreedyFHist on a dataset of 245 pairs of serial sections and reported an improved registration performance compared to a similar registration algorithm. As a proof of concept, we used MIIT to integrate ST and MSI data on cancer-free tissue from 7 prostate cancer patients and assessed the spot-wise correlation of a gene signature activity for citrate-spermine secretion derived from ST with citrate, spermine, and zinc levels obtained by MSI. We confirmed a significant correlation between gene signature activity and all three metabolites. To conclude, we developed a highly accurate, customizable, computational framework for integrating spatial omics technologies and for registration of serial tissue sections.
... 13 This class of optimization-based methods is widely used in medical imaging 14,15 and has also been applied to problems in pathology. 10,[16][17][18][19] These energy-minimizing methods make explicit model assumptions through the choice of distance measure and regularization scheme. When applying a method to a new dataset, model refinements can be made by adjusting the model's parameters. ...
... Below, we first describe the registration method and its application to restained and consecutive slide images. We then describe an evaluation framework based on landmark accuracies on two datasets, the "Automatic Nonlinear Histological Image Registration (ANHIR) 17,25 Challenge" and on a new dataset "Hybrid Restained and Consecutive Data (HyReCo)" 26 that contains both consecutive and restained slides and that we make publicly available. Finally, we analyze the accuracy of the image registration method with respect to image resolution and sectioning in both datasets. ...
... 33 Instead of the relative registration error from the challenge, we computed the registration error in absolute values based on the pixel sizes given in the challenge publication. 17 The public part of the ANHIR challenge dataset consists of 230 image pairs from 8 different tissue types (lung lesions, whole mice lung lobes, mammary glands, mice kidney, colon adenocarcinoma, gastric mucosa and adenocarcinoma, human breast, and human kidney) with 18 different stains. For a subset of the ANHIR dataset, the slices are reported to be cut with a distance of 3 μm, for the remaining sections, no thickness is given. ...
Significance:
Although the registration of restained sections allows nucleus-level alignment that enables a direct analysis of interacting biomarkers, consecutive sections only allow the transfer of region-level annotations. The latter can be achieved at low computational cost using coarser image resolutions.
Purpose:
In digital histopathology, virtual multistaining is important for diagnosis and biomarker research. Additionally, it provides accurate ground truth for various deep-learning tasks. Virtual multistaining can be obtained using different stains for consecutive sections or by restaining the same section. Both approaches require image registration to compensate for tissue deformations, but little attention has been devoted to comparing their accuracy.
Approach:
We compared affine and deformable variational image registration of consecutive and restained sections and analyzed the effect of the image resolution that influences accuracy and required computational resources. The registration was applied to the automatic nonrigid histological image registration (ANHIR) challenge data (230 consecutive slide pairs) and the hyperparameters were determined. Then without changing the parameters, the registration was applied to a newly published hybrid dataset of restained and consecutive sections (HyReCo, 86 slide pairs, 5404 landmarks).
Results:
We obtain a median landmark error after registration of 6.5 μm (HyReCo) and 24.1 μm (ANHIR) between consecutive sections. Between restained sections, the median registration error is 2.2 and 0.9 μm in the two subsets of the HyReCo dataset. We observe that deformable registration leads to lower landmark errors than affine registration in both cases (p<0.001), though the effect is smaller in restained sections.
Conclusion:
Deformable registration of consecutive and restained sections is a valuable tool for the joint analysis of different stains.
... The automatic registration of microscopy images is still an active research area [4,21]. Recently, several notable contributions were proposed to automatic registration of whole slide images (WSIs) acquired using different stains, varying from contributions that focus on the quality of deformable registration [26,24,8,25], through methods that address the robustness of initial alignment [16,12,17,23], ending with methods that propose ready-to-use software packages that can perform automatic registration without time-consuming parameter tuning or deep network retraining [7,22,14]. ...
... So far, the work on SHG and H&E was rather limited. The researchers focused primarily on the registration of H&E to IHC slides [4,21] or the H&E to magnetic resonance images (MRIs) [20,2,1]. There are several different methods dedicated to the initial alignment, including both intensity-based methods [12,3,18], as well as the feature-based contributions [17]. ...
... The automatic registration of microscopy images is still an active research area [4,21]. Recently, several notable contributions were proposed to automatic registration of whole slide images (WSIs) acquired using different stains, varying from contributions that focus on the quality of deformable registration [26,24,8,25], through methods that address the robustness of initial alignment [16,12,17,23], ending with methods that propose ready-to-use software packages that can perform automatic registration without time-consuming parameter tuning or deep network retraining [7,22,14]. ...
... So far, the work on SHG and H&E was rather limited. The researchers focused primarily on the registration of H&E to IHC slides [4,21] or the H&E to magnetic resonance images (MRIs) [20,2,1]. There are several different methods dedicated to the initial alignment, including both intensity-based methods [12,3,18], as well as the feature-based contributions [17]. ...
The automatic registration of noninvasive second-harmonic generation microscopy to hematoxylin and eosin slides is a highly desired, yet still unsolved problem. The task is challenging because the second-harmonic images contain only partial information, in contrast to the stained H&E slides that provide more information about the tissue morphology. Moreover, both imaging methods have different intensity distributions. Therefore, the task can be formulated as a multi-modal registration problem with missing data. In this work, we propose a method based on automatic keypoint matching followed by deformable registration based on instance optimization. The method does not require any training and is evaluated using the dataset provided in the Learn2Reg challenge by the COMULIS organization. The method achieved relatively good generalizability resulting in 88% of success rate in the initial alignment and average target registration error equal to 2.48 on the external validation set. We openly release the source code and incorporate it in the DeeperHistReg image registration framework.
... For instance, the dataset from de Haan et al. (2021) was not shared [6]. Additionally, the availability of high-quality paired data is limited; typically, datasets such as AHNIR [39] are compiled from adjacent slides, resulting in imperfectly matched samples. Specifically, the AHNIR kidney dataset contains only a limited set of slides, with five slides for each stain type: H&E, PAS, PASM, and MAS. ...
... The ANHIR [39] dataset includes five sets of high-resolution human kidney tissue slides, each containing four slides of consecutive tissues stained with different types (H&E, MAS, PAS, and PASM staining). Although these slides are structurally similar, they are not pixel-level paired and all are magnified at x40. ...
Chemical staining methods are dependable but require extensive time, expensive chemicals, and raise environmental concerns. These challenges highlight the need for alternative solutions like virtual staining, which accelerates the diagnostic process and enhances stain application flexibility. Generative AI technologies are pivotal in addressing these issues. However, the high-stakes nature of healthcare decisions, especially in computational pathology, complicates the adoption of these tools due to their opaque processes. Our work introduces the use of generative AI for virtual staining, aiming to enhance performance, trustworthiness, scalability, and adaptability in computational pathology. The methodology centers on a singular H&E encoder supporting multiple stain decoders. This design focuses on critical regions in the latent space of H&E, enabling precise synthetic stain generation. Our method, tested to generate 8 different stains from a single H&E slide, offers scalability by loading only necessary model components during production. We integrate label-free knowledge in training, using loss functions and regularization to minimize artifacts, thus improving paired/unpaired virtual staining accuracy. To build trust, we use real-time self-inspection with discriminators for each stain type, providing pathologists with confidence heat-maps. Automatic quality checks on new H&E slides ensure conformity to the trained distribution, ensuring accurate synthetic stains. Recognizing pathologists' challenges with new technologies, we have developed an open-source, cloud-based system, that allows easy virtual staining of H&E slides through a browser, addressing hardware/software issues and facilitating real-time user feedback. We also curated a novel dataset of 8 paired H&E/stains related to pediatric Crohn's disease, comprising 480 WSIs to further stimulate computational pathology research.
... We made use of freely available registration algorithms: Advanced Normalisation Tools (ANTs; Avants et al., 2008) and Ni yReg (Modat et al., 2014). These tools have demonstrated their efficiency for registering mono-and multimodal biomedical 2D and 3D datasets in various applications (Niedworok et al., 2016;Murakami et al., 2018;Nazib et al., 2018;Balakrishnan et al., 2019;Mano et al., 2020;Borovec et al., 2020;Iglesias et al., 2023). The ANTs algorithm is well-suited for precisely aligning small datasets, such as 2D histological slices, using nonlinear registration (Goubran et al., 2019;Krepl et al., 2021), especially its symmetric image normalization (SyN) transformation. ...
Brain atlases are essential for quantifying cellular composition in mouse brain regions. The Allen Institute's Common Coordinate Framework version 3 (CCFv3) is widely used, delineating over 600 anatomical regions, but it lacks coverage for the most rostral and caudal brain parts, including the main olfactory bulb, cerebellum, and medulla. Additionally, the CCFv3 omits key cerebellar layers, and its corresponding Nissl-stained reference volume is not precisely aligned, limiting its utilisability. To address these issues, we developed an extended atlas, the Blue Brain Project CCFv3 augmented (CCFv3aBBP), which includes a fully annotated mouse brain and an improved Nissl reference aligned in the CCFv3. This enhanced atlas also features the central nervous system annotation (CCFv3cBBP). Using this resource, we aligned 734 Nissl-stained brains to produce an average Nissl template, enabling an updated distribution of neuronal soma positions. These data are available as an open-source resource, broadening applications such as improved alignment precision, cell type mapping, and multimodal data integration.
... The objective of the challenge was to align tissue in the IHC WSIs to corresponding tissue in the H&E WSIs. WSI registration has previously been addressed in the ANHIR challenge 29 . While the ANHIR challenge made valuable contributions to the field of WSI registration, it was limited by the high quality of sections and WSIs, which is not representative of clinical material, as well as by the availability of both training and test data with 355 WSIs in total, albeit originating from a wide variety of organs and stains. ...
... The purpose of our study is to address the challenges associated with whole slide image matching to meet clinical needs [11], [12] including: 1) Large-size matching [13]. Matching large-size WSIs may enhance accuracy, although intuitive but with unknown gain. ...
Matching whole slide histopathology images to provide comprehensive information on homologous tissues is beneficial for cancer diagnosis. However, the challenge arises with the Giga-pixel whole slide images (WSIs) when aiming for high-accuracy matching. Learning-based methods are difficult to generalize well with large-size WSIs, necessitating the integration of traditional matching methods to enhance accuracy as the size increases. In this paper, we propose a multi-size guiding matching method applicable high-accuracy requirements. Specifically, we design learning multiscale texture to train deep descriptors, called TDescNet, that trains 64
256 and 256
128 size convolution layer as C64 and C256 descriptors to overcome staining variation and low visibility challenges. Furthermore, we develop the 3D-ring descriptor using sparse keypoints to support the description of large-size WSIs. Finally, we employ C64, C256, and 3D-ring descriptors to progressively guide refined local matching, utilizing geometric consistency to identify correct matching results. Experiments show that when matching WSIs of size 4096×4096 pixels, our average matching error is 123.48
\upmum
and the success rate is 93.02
in 43 cases. Notably, our method achieves an average improvement of 65.52
\upmum
in matching accuracy compared to recent state-of-the-art methods, with enhancements ranging from 36.27
\upmum
to 131.66
\upmum
. Therefore, we achieve high-fidelity whole-slice image matching, and overcome staining variation and low visibility challenges, enabling assistance in comprehensive cancer diagnosis through matched WSIs.
... Since images retrieved from CCD-type flatbed scanners show a distinct distortion perpendicular to the scan direction (Schubert, 2000), it is not possible to achieve a perfect alignment by rotation and translation alone ("rigid alignment") because of the parallax effect (stronger with higher relief). Our attempts to compensate for this effect by using an alignment algorithm that allows to locally deform the images ("elastic alignment", developed for histological sections; Borovec et al., 2020) did not lead to an improved alignment. This is especially unfortunate because this, aside from the optical properties of the scanner, appears to be the most limiting factor of using CCD-type flatbed scanners for recovering normal maps through photometric stereo. ...
... The registration of multimodal images in histology is a complex task as it involves addressing various challenges, including large image sizes, repetitive texture, non-linear elastic deformation, occlusions, missing sections, nonrigid deformation, contrast differences, appearance variations, and local structural disparities between slices. These challenges hinder the identification of unique landmarks for alignment, thus affecting the accuracy of registration [2,5]. It is important to highlight that the utilization of multimodal registration is needed for training data on photonic measurement techniques where the annotations are derived from the standard Hematoxylin and Eosin technique [6]. ...
Background
Multimodal histology image registration is a process that transforms into a common coordinate system two or more images obtained from different microscopy modalities. The combination of information from various modalities can contribute to a comprehensive understanding of tissue specimens, aiding in more accurate diagnoses, and improved research insights. Multimodal image registration in histology samples presents a significant challenge due to the inherent differences in characteristics and the need for tailored optimization algorithms for each modality.
Results
We developed MMIR a cloud-based system for multimodal histological image registration, which consists of three main modules: a project manager, an algorithm manager, and an image visualization system.
Conclusion
Our software solution aims to simplify image registration tasks with a user-friendly approach. It facilitates effective algorithm management, responsive web interfaces, supports multi-resolution images, and facilitates batch image registration. Moreover, its adaptable architecture allows for the integration of custom algorithms, ensuring that it aligns with the specific requirements of each modality combination. Beyond image registration, our software enables the conversion of segmented annotations from one modality to another.
... The Block Matching (BM) algorithm (Ourselin et al., 2001) was chosen as a robust strategy to register data from different modalities. This method was later included in the NiftyReg library (Modat et al., 2014) and is still well used in many applications (Niedworok et al., 2016;Iglesias et al., 2018;Balakrishnan et al., 2019;Borovec et al., 2020;Mancini et al., 2020). Normalized Mutual Information (NMI) . ...
Conventional histology of the brain remains the gold standard in the analysis of animal models. In most biological studies, standard protocols usually involve producing a limited number of histological slices to be analyzed. These slices are often selected into a specific anatomical region of interest or around a specific pathological lesion. Due to the lack of automated solutions to analyze such single slices, neurobiologists perform the segmentation of anatomical regions manually most of the time. Because the task is long, tedious, and operator-dependent, we propose an automated atlas segmentation method called giRAff, which combines rigid and affine registrations and is suitable for conventional histological protocols involving any number of single slices from a given mouse brain. In particular, the method has been tested on several routine experimental protocols involving different anatomical regions of different sizes and for several brains. For a given set of single slices, the method can automatically identify the corresponding slices in the mouse Allen atlas template with good accuracy and segmentations comparable to those of an expert. This versatile and generic method allows the segmentation of any single slice without additional anatomical context in about 1 min. Basically, our proposed giRAff method is an easy-to-use, rapid, and automated atlas segmentation tool compliant with a wide variety of standard histological protocols.
... ; https://doi.org/10.1101/2024.01.22.576608 doi: bioRxiv preprint make it impossible for regular users to apply those (Borovec et al., 2020). In all, results from image registration using BigWarp can be considered optimal, although several aspects should be improved. ...
Spinal cord injury (SCI) is a disabling disorder of the spinal cord resulting from trauma or disease. Neuronal death is a central event in the pathophysiology of spinal cord injury. Despite its importance and the large number of research studies carried out, we only have a fragmentary vision of the process focused on the specific targets of each study. It is our opinion that the research community has accumulated enough information which may be reanalyzed with novel tools to get a much more detailed, integrated vision of neuronal death after SCI. This work embeds this vision by creating NeuroCluedo, an open data repository to store and share images as well as the results from their analysis. We have employed this repository to upload the raw and processed images of spinal cord sections from a mouse model of moderate contusive SCI (Reigada et al., 2015) and used this information to: compare manual-, threshold-, and neuronal network-based neuron identifications; and to explore neuronal death at the injury penumbra 21 days after injury and the neuroprotective effects of the anti-apoptotic drug ucf-101. The results from these analyses i) indicate that the three identification methods yield coherent estimates of the total number of neurons per section; ii) identified the neural network as the optimal method, even in spinal sections with major artifacts and marked autofluorescence associated with spinal damage; iii) characterize neuronal distribution among Rexed laminae in the mice T11; iv) reveal that neuronal death distributes through all the gray matter in the penumbrae sections closer to the injury epicenter but concentrate in the intermediate region in sections located farther away; and that v) antiapoptotic effects of UCF-101 are highest in the intermediate region of the gray substance of the caudal segments closest to the injury epicenter. All methods and results, including raw and processed images, software, macros, and scripts, together with all data matrixes and results have been deposited and documented in the Open Science Framework (OSF) repository Neurocluedo (https://osf.io/n32z9/).
... We evaluated the proposed approach using the ACROBAT and ANHIR datasets [11], [12], preprocessed and curated by the dataset owners. ...
... Simultaneous detection of mRNA and proteins on the same tissue section reduces the computational burden associated with aligning spatially adjacent tissue sections and their latent structural variations in tissue architecture. 11,42 Using Visium-SPG, we generated transcriptome-scale tissue atlases of human ITC harboring AD-related neuropathology. The ITC is a cortical structure that succumbs to progressive accumulation of Ab and pTau in AD. 43 We focused on late-stage AD (Braak V-VI/CERAD Frequent) to define the biological consequences of severe pathology. ...
Recently, Deep Learning (DL) models have shown promising accuracy in analysis of medical images. Alzeheimer Disease (AD), a prevalent form of dementia, uses Magnetic Resonance Imaging (MRI) scans, which is then analysed via DL models. To address the model computational constraints, Cloud Computing (CC) is integrated to operate with the DL models. Recent articles on DL-based MRI have not discussed datasets specific to different diseases, which makes it difficult to build the specific DL model. Thus, the article systematically explores a tutorial approach, where we first discuss a classification taxonomy of medical imaging datasets. Next, we present a case-study on AD MRI classification using the DL methods. We analyse three distinct models-Convolutional Neural Networks (CNN), Visual Geometry Group 16 (VGG-16), and an ensemble approach-for classification and predictive outcomes. In addition, we designed a novel framework that offers insight into how various layers interact with the dataset. Our architecture comprises an input layer, a cloud-based layer responsible for preprocessing and model execution, and a diagnostic layer that issues alerts after successful classification and prediction. According to our simulations, CNN outperformed other models with a test accuracy of 99.285%, followed by VGG-16 with 85.113%, while the ensemble model lagged with a disappointing test accuracy of 79.192%. Our cloud Computing framework serves as an efficient mechanism for medical image processing while safeguarding patient confidentiality and data privacy.
Medical image alignment is an important research field in medical image processing, which is widely used in clinical diagnosis and treatment, such as surgical navigation, lesion tracking, and treatment evaluation. In this paper, an improved algorithm combining the Demons algorithm and SIFT algorithm is proposed, which uses the SIFT algorithm to represent the feature points in non-rigid medical images as a scale space sequence and normalize the descriptors in the scale space sequence. Then, the two-way alignment strategy and multi-resolution strategy are introduced to improve the accuracy of Demons algorithm in the alignment of non-rigid medical images with complex deformation. The study shows that the improved Demons algorithm can achieve better alignment results when the weights of the feature matching terms are taken as −1 and 1, which makes the improved Demons algorithm with the addition of SIFT feature terms perform optimally. Alignment simulation experiments found that the MSE value of this paper’s improved algorithm is only 0.077. The alignment effect of non-rigid medical images is much better than the comparison algorithm and can maintain a shorter running time. The algorithm in this paper can effectively realize the non-rigid alignment of medical images, which provides a reference method for medical diagnosis and the effective formulation of treatment plans.
The problem of artifacts in whole slide image acquisition, prevalent in both clinical workflows and research-oriented settings, necessitates human intervention and re-scanning. Overcoming this challenge requires developing quality control algorithms, that are hindered by the limited availability of relevant annotated data in histopathology. The manual annotation of ground-truth for artifact detection methods is expensive and time-consuming. This work addresses the issue by proposing a method dedicated to augmenting whole slide images with artifacts. The tool seamlessly generates and blends artifacts from an external library to a given histopathology dataset. The augmented datasets are then utilized to train artifact classification methods. The evaluation shows their usefulness in classification of the artifacts, where they show an improvement from 0.10 to 0.01 AUROC depending on the artifact type. The framework, model, weights, and ground-truth annotations are freely released to facilitate open science and reproducible research.
Spatially resolved transcriptomics has been dramatically transforming biological and medical research in various fields. It enables transcriptome profiling at single‐cell, multi‐cellular, or sub‐cellular resolution, while retaining the information of geometric localizations of cells in complex tissues. The coupling of cell spatial information and its molecular characteristics generates a novel multi‐modal high‐throughput data source, which poses new challenges for the development of analytical methods for data‐mining. Spatial transcriptomic data are often highly complex, noisy, and biased, presenting a series of difficulties, many unresolved, for data analysis and generation of biological insights. In addition, to keep pace with the ever‐evolving spatial transcriptomic experimental technologies, the existing analytical theories and tools need to be updated and reformed accordingly. In this review, we provide an overview and discussion of the current computational approaches for mining of spatial transcriptomics data. Future directions and perspectives of methodology design are proposed to stimulate further discussions and advances in new analytical models and algorithms.
This article is categorized under: RNA Methods > RNA Analyses in Cells
RNA Evolution and Genomics > Computational Analyses of RNA
RNA Export and Localization > RNA Localization
The problem of artifacts in whole slide image acquisition, prevalent in both clinical workflows and research-oriented settings, necessitates human intervention and re-scanning. Overcoming this challenge requires developing quality control algorithms, that are hindered by the limited availability of relevant annotated data in histopathology. The manual annotation of ground-truth for artifact detection methods is expensive and time-consuming. This work addresses the issue by proposing a method dedicated to augmenting whole slide images with artifacts. The tool seamlessly generates and blends artifacts from an external library to a given histopathology dataset. The augmented datasets are then utilized to train artifact classification methods. The evaluation shows their usefulness in classification of the artifacts, where they show an improvement from 0.10 to 0.01 AUROC depending on the artifact type. The framework, model, weights, and ground-truth annotations are freely released to facilitate open science and reproducible research.
Computational Pathology (CPath) is an interdisciplinary science that augments developments of computational approaches to analyze and model medical histopathology images. The main objective for CPath is to develop infrastructure and workflows of digital diagnostics as an assistive CAD system for clinical pathology, facilitating transformational changes in the diagnosis and treatment of cancer that are mainly address by CPath tools. With evergrowing developments in deep learning and computer vision algorithms, and the ease of the data flow from digital pathology, currently CPath is witnessing a paradigm shift. Despite the sheer volume of engineering and scientific works being introduced for cancer image analysis, there is still a considerable gap of adopting and integrating these algorithms in clinical practice. This raises a significant question regarding the direction and trends that are undertaken in CPath. In this article we provide a comprehensive review of more than 800 papers to address the challenges faced in problem design all-the-way to the application and implementation viewpoints. We have catalogued each paper into a model-card by examining the key works and challenges faced to layout the current landscape in CPath. We hope this helps the community to locate relevant works and facilitate understanding of the field’s future directions. In a nutshell, we oversee the CPath developments in cycle of stages which are required to be cohesively linked together to address the challenges associated with such multidisciplinary science. We overview this cycle from different perspectives of data-centric, model-centric, and application-centric problems. We finally sketch remaining challenges and provide directions for future technical developments and clinical integration of CPath. For updated information on this survey review paper and accessing to the original model cards repository, please refer to GitHub. Updated version of this draft can also be found from arXiv.
In histopathology, the tissue slides are usually stained by common H&E stain or special stains (MAS, PAS, and PASM, etc.) to clearly show specific tissue structures. The rapid development of deep learning provides a good solution to generate virtual staining images to significantly reduce the time and labor costs associated with histochemical staining. However, most existing methods need to train a special model for every two stains, which consumes a lot of computing resources with the increasing of staining types. To address this problem, we propose an unsupervised multi-domain stain transfer method, GramGAN, which realizes the progressive transfer through cascaded Style-Guided blocks. For each Style-Guided block, we design a style encoding dictionary to characterize and store all the staining style information. In addition, we propose a Rényi entropy-based regularization term to improve the discrimination ability of different styles. The experimental results show that our method can realize accurate transferring among multiple staining styles with better performance. Furthermore, we build and publish a special stained image dataset suitable for glomeruli segmentation (including H&E staining), where the accuracy of glomeruli detection and segmentation can be significantly improved after transferring H&E-stained images to PAS-stained and PASM-stained ones by our method. The code is publicly available at:
https://github.com/xianchaoguan/GramGAN
.
Feature-based registration has become increasingly popular in digital pathology for achieving initial global alignment between image pairs. However, the selection of algorithms used in this approach is often not well-justified. Specifically, the choice of local feature descriptor is rarely, if ever, discussed in the context of digital pathology. The majority of feature-based whole-slide image registration methods rely on the SIFT descriptor. In this study, we demonstrate that the choice of descriptor significantly influences the quality of registration results and that the BRIEF descriptor captures more optimal information for histological image registration.
Lossless and near-lossless image compression is of paramount importance to professional users in many technical fields, such as medicine, remote sensing, precision engineering and scientific research. But despite rapidly growing research interests in learning-based image compression, no published method offers both lossless and near-lossless modes. In this paper, we propose a unified and powerful deep lossy plus residual (DLPR) coding framework for both lossless and near-lossless image compression. In the lossless mode, the DLPR coding system first performs lossy compression and then lossless coding of residuals. We solve the joint lossy and residual compression problem in the approach of VAEs, and add autoregressive context modeling of the residuals to enhance lossless compression performance. In the near-lossless mode, we quantize the original residuals to satisfy a given
error bound, and propose a scalable near-lossless compression scheme that works for variable
bounds instead of training multiple networks. To expedite the DLPR coding, we increase the degree of algorithm parallelization by a novel design of coding context, and accelerate the entropy coding with adaptive residual interval. Experimental results demonstrate that the DLPR coding system achieves both the state-of-the-art lossless and near-lossless image compression performance with competitive coding speed.
The use of medical data for machine learning, including unsupervised methods such as clustering, is often restricted by privacy regulations such as the Health Insurance Portability and Accountability Act (HIPAA). Medical data is sensitive and highly regulated and anonymization is often insufficient to protect a patient’s identity. Traditional clustering algorithms are also unsuitable for longitudinal behavioral health trials, which often have missing data and observe individual behaviors over varying time periods. In this work, we develop a new decentralized federated multiple imputation-based fuzzy clustering algorithm for complex longitudinal behavioral trial data collected from multisite randomized controlled trials over different time periods. Federated learning (FL) preserves privacy by aggregating model parameters instead of data. Unlike previous FL methods, this proposed algorithm requires only two rounds of communication and handles clients with varying numbers of time points for incomplete longitudinal data. The model is evaluated on both empirical longitudinal dietary health data and simulated clusters with different numbers of clients, effect sizes, correlations, and sample sizes. The proposed algorithm converges rapidly and achieves desirable performance on multiple clustering metrics. This new method allows for targeted treatments for various patient groups while preserving their data privacy and enables the potential for broader applications in the Internet of Medical Things.
Simple Summary
Three-dimensional models of tumor vascular networks are of significant importance for in vitro and in silico investigations of, for example, the efficiency of anti-cancer drugs in an early stage of clinical transition and can be potentially used for the development of in vitro systems as 3D-printable vascular networks to facilitate personalized medicine and randomized controlled clinical trials. In this work, histologic slices of a human pancreatic tumor are used as examples to establish an algorithm-based method that enables the reconstruction of a 3D vascular network model. The advantages of this method are high resolution and accuracy concerning the characteristics of the vascular network (e.g., density, trajectory of vessels).
Abstract
For reliable in silico or in vitro investigations in, for example, biosensing and drug delivery applications, accurate models of tumor vascular networks down to the capillary size are essential. Compared to images acquired with conventional medical imaging techniques, digitalized histological tumor slices have a higher resolution, enabling the delineation of capillaries. Volume rendering procedures can then be used to generate a 3D model. However, the preparation of such slices leads to misalignments in relative slice orientation between consecutive slices. Thus, image registration algorithms are necessary to re-align the slices. Here, we present an algorithm for the registration and reconstruction of a vascular network from histologic slices applied to 169 tumor slices. The registration includes two steps. First, consecutive images are incrementally pre-aligned using feature- and area-based transformations. Second, using the previous transformations, parallel registration for all images is enabled. Combining intensity- and color-based thresholds along with heuristic analysis, vascular structures are segmented. A 3D interpolation technique is used for volume rendering. This results in a 3D vascular network with approximately 400–450 vessels with diameters down to 25–30 µm. A delineation of vessel structures with close distance was limited in areas of high structural density. Improvement can be achieved by using images with higher resolution and or machine learning techniques.
Cross-modality image synthesis is an active research topic with multiple medical clinically relevant applications. Recently, methods allowing training with paired but misaligned data have started to emerge. However, no robust and well-performing methods applicable to a wide range of real world data sets exist. In this work, we propose a generic solution to the problem of cross-modality image synthesis with paired but non-aligned data by introducing new deformation equivariance encouraging loss functions. The method consists of joint training of an image synthesis network together with separate registration networks and allows adversarial training conditioned on the input even with misaligned data. The work lowers the bar for new clinical applications by allowing effortless training of cross-modality image synthesis networks for more difficult data sets.
Background and objective:
Histopathological image registration is an essential component in digital pathology and biomedical image analysis. Deep-learning-based algorithms have been proposed to achieve fast and accurate affine registration. Some previous studies assume that the pairs are free from sizeable initial position misalignment and large rotation angles before performing the affine transformation. However, large-rotation angles are often introduced into image pairs during the production process in real-world pathology images. Reliable initial alignment is important for registration performance. The existing deep-learning-based approaches often use a two-step affine registration pipeline because convolutional neural networks (CNNs) cannot correct large-angle rotations.
Methods:
In this manuscript, a general framework ARoNet is developed to achieve end-to-end affine registration for histopathological images. We use CNNs to extract global features of images and fuse them to construct correspondent information for affine transformation. In ARoNet, a rotation recognition network is implemented to eliminate great rotation misalignment. In addition, a self-supervised learning task is proposed to assist the learning of image representations in an unsupervised manner.
Results:
We applied our model to four datasets, and the results indicate that ARoNet surpasses existing affine registration algorithms in alignment accuracy when large angular misalignments (e.g., 180 rotation) are present, providing accurate affine initialization for subsequent non-rigid alignments. Besides, ARoNet shows advantages in execution time (0.05 per pair), registration accuracy, and robustness.
Conclusion:
We believe that the proposed general framework promises to simplify and speed up the registration process and has the potential for clinical applications.
The astounding success made by artificial intelligence in healthcare and other fields proves that AI can achieve human-like performance. However, success always comes with challenges. Deep learning algorithms are data-dependent and require large datasets for training. Many junior researchers faced a lack of data, because of a variety of reasons. Medical image acquisition, annotation, and analysis are costly, and their usage is constrained by ethical restrictions. They also require several other resources, such as professional equipment and expertise. That makes it difficult for novice and non-medical researchers to have access to medical data. Thus, as comprehensive as possible, this paper provides a collection of medical image datasets with their associated challenges for deep learning research. We have collected the information of around three hundred datasets and challenges mainly reported between 2007 and 2020 and categorized them into four categories: head & neck, chest & abdomen, pathology & blood, and “others”. The purpose of our paper is to provide a list, as up-to-date and complete as possible, that can be used as a reference to easily find the datasets for medical image analysis and the information related to these datasets.
Histopathologic assessment routinely provides rich microscopic information about tissue structure and disease process. However, the sections used are very thin, and essentially capture only 2D representations of a certain tissue sample. Accurate and robust alignment of sequentially cut 2D slices should contribute to more comprehensive assessment accounting for surrounding 3D information. Towards this end, we here propose a two-step diffeomorphic registration approach that aligns differently stained histology slides to each other, starting with an initial affine step followed by estimating a deformation field. It was quantitatively evaluated on ample (n = 481) and diverse data from the automatic non-rigid histological image registration challenge, where it was awarded the second rank. The obtained results demonstrate the ability of the proposed approach to robustly (average robustness = 0.9898) and accurately (average relative target registration error = 0.2%) align differently stained histology slices of various anatomical sites while maintaining reasonable computational efficiency (<1 min per registration). The method was developed by adapting a general-purpose registration algorithm designed for 3D radiographic scans and achieved consistently accurate results for aligning high-resolution 2D histologic images. Accurate alignment of histologic images can contribute to a better understanding of the spatial arrangement and growth patterns of cells, vessels, matrix, nerves, and immune cell interactions.
3D medical image registration is of great clinical importance. However, supervised learning methods require a large amount of accurately annotated corresponding control points (or morphing), which are very difficult to obtain. Unsupervised learning methods ease the burden of manual annotation by exploiting unlabeled data without supervision. In this paper, we propose a new unsupervised learning method using convolutional neural networks under an end-to-end framework, Volume Tweening Network (VTN), for 3D medical image registration. We propose three innovative technical components: (1) An end-to-end cascading scheme that resolves large displacement; (2) An efficient integration of affine registration network; and (3) An additional invertibility loss that encourages backward consistency. Experiments demonstrate that our algorithm is 880x faster (or 3.3x faster without GPU acceleration) than traditional optimization-based methods and achieves state-of-theart performance in medical image registration.
Our research [Team Name: URAN] of 'Neuromorphic Neural Network for Multimodal Brain Tumor Segmentation and Survival analysis' demonstrates its performance on 'Overall Survival Prediction', without any prior training using multi-modal MRI Image segmentation by medical doctors (though provided by BraTS 2018 challenge data). Two segmentation categories are adopted instead of three segmentations , which is beyond the ordinary scope of BraTS 2018's segmentation challenge. The test results(the worst blind test, 51%: the nominal test, 71%) of 'survival prediction' challenge have proved the early feasibility of our neuromorphic neural network for helping the medical doctors, considering the human clinical accuracy of 50%~70%.
Image registration is a common task for many biomedical analysis applications. The present work focuses on the benchmarking of registration methods on differently stained histological slides. This is a challenging task due to the differences in the appearance model, the repetitive texture of the details and the large image size, between other issues. Our benchmarking data is composed of 616 image pairs at two different scales — average image diagonal 2.4k and 5k pixels. We compare eleven fully automatic registration methods covering the widely used similarity measures. For each method, the best parameter configuration is found and subsequently applied to all the image pairs. The performance of the algorithms is evaluated from several perspectives — the registrations (in)accuracy on manually annotated landmarks, the method robustness and its computation time.
Image registration is a common task for many biomedical analysis applications. The present work focuses on the benchmarking of registration methods on differently stained histological slides. This is a challenging task due to the differences in the appearance model, the repetitive texture of the details and the large image size, between other issues. Our benchmarking data is composed of 616 image pairs at two different scales - average image diagonal 2.4k and 5k pixels. We compare eleven fully automatic registration methods covering the widely used similarity measures (and optimization strategies with both linear and elastic transformation). For each method, the best parameter configuration is found and subsequently applied to all the image pairs. The performance of the algorithms is evaluated from several perspectives - the registrations (in)accuracy on manually annotated landmarks, the method robustness and its processing computation time.
Motivation:
Digital pathology enables new approaches that expand beyond storage, visualization or analysis of histological samples in digital format. One novel opportunity is 3D histology, where a three-dimensional reconstruction of the sample is formed computationally based on serial tissue sections. This allows examining tissue architecture in 3D, for example, for diagnostic purposes. Importantly, 3D histology enables joint mapping of cellular morphology with spatially resolved omics data in the true 3D context of the tissue at microscopic resolution. Several algorithms have been proposed for the reconstruction task, but a quantitative comparison of their accuracy is lacking.
Results:
We developed a benchmarking framework to evaluate the accuracy of several free and commercial 3D reconstruction methods using two whole slide image datasets. The results provide a solid basis for further development and application of 3D histology algorithms and indicate that methods capable of compensating for local tissue deformation are superior to simpler approaches.
Availability:
Code: https://github.com/BioimageInformaticsTampere/RegBenchmark. Whole slide image datasets: http://urn.fi/urn:nbn:fi:csc-kata20170705131652639702.
Contact:
pekka.ruusuvuori@tut.fi.
Supplementary information:
Supplementary data are available at Bioinformatics online.
Histology permits the observation of otherwise invisible structures of the internal topography of a specimen. Although it enables the investigation of tissues at a cellular level, it is invasive and breaks topology due to cutting. Three-dimensional (3D) reconstruction was thus introduced to overcome the limitations of single-section studies in a dimensional scope. 3D reconstruction finds its roots in embryology, where it enabled the visualisation of spatial relationships of developing systems and organs, and extended to biomedicine, where the observation of individual, stained sections provided only partial understanding of normal and abnormal tissues. However, despite bringing visual awareness, recovering realistic reconstructions is elusive without prior knowledge about the tissue shape.
3D medical imaging made such structural ground truths available. In addition, combining non-invasive imaging with histology unveiled invaluable opportunities to relate macroscopic information to the underlying microscopic properties of tissues through the establishment of spatial correspondences; image registration is one technique that permits the automation of such a process and we describe reconstruction methods that rely on it. It is thereby possible to recover the original topology of histology and lost relationships, gain insight into what affects the signals used to construct medical images (and characterise them), or build high resolution anatomical atlases.
This paper reviews almost three decades of methods for 3D histology reconstruction from serial sections, used in the study of many different types of tissue. We first summarise the process that produces digitised sections from a tissue specimen in order to understand the peculiarity of the data, the associated artefacts and some possible ways to minimise them. We then describe methods for 3D histology reconstruction with and without the help of 3D medical imaging, along with methods of validation and some applications. We finally attempt to identify the trends and challenges that the field is facing, many of which are derived from the cross-disciplinary nature of the problem as it involves the collaboration between physicists, histopathologists, computer scientists and physicians.
Delineation of the cardiac right ventricle is essential in generating clinical measurements such as ejection fraction and stroke volume. Given manual segmentation on the first frame, one approach to segment right ventricle from all of the magnetic resonance images is to find point correspondence between the sequence of images. Finding the point correspondence with non-rigid transformation requires a deformable image registration algorithm which often involves computationally expensive optimization. The central processing unit (CPU) based implementation of point correspondence algorithm has been shown to be accurate in delineating organs from a sequence of images in recent studies. The purpose of this study is to develop computationally efficient approaches for deformable image registration. We propose a graphics processing unit (GPU) accelerated approach to improve the efficiency. The proposed approach consists of two parallelization components: Parallel Compute Unified Device Architecture (CUDA) version of the deformable registration algorithm; and the application of an image concatenation approach to further parallelize the algorithm. Three versions of the algorithm were implemented: 1) CPU; 2) GPU with only intra-image parallelization (Sequential image registration); and 3) GPU with inter and intra-image parallelization (Concatenated image registration). The proposed methods were evaluated over a data set of 16 subjects. CPU, GPU sequential image, and GPU concatenated image methods took an average of 113.13, 16.50 and 5.96 seconds to segment a sequence of 20 images, respectively. The proposed parallelization approach offered a computational performance improvement of around 19× in comparison to the CPU implementation while retaining the same level of segmentation accuracy. This study demonstrated that the GPU computing could be utilized for improving the computational performance of a non-rigid image registration algorithm without compromising the accuracy.
Objective:
To develop a flexible method of separation and quantification of immunohistochemical staining by means of color image analysis.
Study design:
An algorithm was developed to deconvolve the color information acquired with red-green-blue (RGB) cameras and to calculate the contribution of each of the applied stains based on stain-specific RGB absorption. The algorithm was tested using different combinations of diaminobenzidine, hematoxylin and eosin at different staining levels.
Results:
Quantification of the different stains was not significantly influenced by the combination of multiple stains in a single sample. The color deconvolution algorithm resulted in comparable quantification independent of the stain combinations as long as the histochemical procedures did not influence the amount of stain in the sample due to bleaching because of stain solubility and saturation of staining was prevented.
Conclusion:
This image analysis algorithm provides a robust and flexible method for objective immunohistochemical analysis of samples stained with up to three different stains using a laboratory microscope, standard RGB camera setup and the public domain program NIH Image.
In this paper, we present a fast method for registration of multiple large, digitised whole-slide images (WSIs) of serial histology sections. Through cross-slide WSI registration, it becomes possible to select and analyse a common visual field across images of several serial section stained with different protein markers. It is, therefore, a critical first step for any downstream co-localised cross-slide analysis. The proposed registration method uses a two-stage approach, first estimating a fast initial alignment using the tissue sections’ external boundaries, followed by an efficient refinement process guided by key biological structures within the visual field. We show that this method is able to produce a high quality alignment in a variety of circumstances, and demonstrate that the refinement is able to quantitatively improve registration quality. In addition, we provide a case study that demonstrates how the proposed method for cross-slide WSI registration could be used as part of a specific co-expression analysis framework.
We present an image processing pipeline which accepts a large number of images, containing spatial expression information for thousands of genes in Drosophila imaginal discs. We assume that the gene activations are binary and can be expressed as a union of a small set of non-overlapping spatial patterns, yielding a compact representation of the spatial activation of each gene. This lends itself well to further automatic analysis, with the hope of discovering new biological relationships. Traditionally, the images were labeled manually, which was very time consuming. The key part of our work is a binary pattern dictionary learning algorithm, that takes a set of binary images and determines a set of patterns, which can be used to represent the input images with a small error. We also describe the preprocessing phase, where input images are segmented to recover the activation images and spatially aligned to a common reference. We compare binary pattern dictionary learning to existing alternative methods on synthetic data and also show results of the algorithm on real microscopy images of the Drosophila imaginal discs.
The form and exact function of the blood vessel network in some human organs, like spleen and bone marrow, are still open research questions in medicine. In this paper we propose a method to register the immunohistological stainings of serial sections of spleen and bone marrow specimens to enable the visualization and visual inspection of blood vessels. As these vary much in caliber, from mesoscopic (millimeter-range) to microscopic (few micrometers, comparable to a single erythrocyte), we need to utilize a multi-resolution approach.
Our method is fully automatic; it is based on feature detection and sparse matching. We utilize a rigid alignment and then a non-rigid deformation, iteratively dealing with increasingly smaller features. Our tool pipeline can already deal with series of complete scans at extremely high resolution, up to 620 megapixels. The improvement presented increases the range of represented details up to smallest capillaries. This paper provides details on the multi-resolution non-rigid registration approach we use. Our application is novel in the way the alignment and subsequent deformations are computed (using features, i.e. “sparse”). The deformations are based on all images in the stack (“global”).
We also present volume renderings and a 3D reconstruction of the vascular network in human spleen and bone marrow on a level not possible before. Our registration makes easy tracking of even smallest blood vessels possible, thus granting experts a better comprehension.
A quantitative evaluation of our method and related state of the art approaches with seven different quality measures shows the efficiency of our method. We also provide z-profiles and enlarged volume renderings from three different registrations for visual inspection.
This paper presents a method for correcting erratic pairwise registrations when reconstructing a volume from 2D histology slices. Due to complex and unpredictable alterations of the content of histology images, a pairwise rigid registration between two adjacent slices may fail systematically. Conversely, a neighbouring registration, which potentially involves one of these two slices, will work. This grounds our approach: using correct spatial correspondences established through neighbouring registrations to account for direct failures. We propose to search the best alignment of every couple of adjacent slices from a finite set of transformations that involve neighbouring slices in a transitive fashion. Using the proposed method, we obtained reconstructed volumes with increased coherence compared to the classical pairwise approach, both in synthetic and real data.
Robust and fully automatic 3D registration of serial-section microscopic images is critical for detailed anatomical reconstruction of large biological specimens, such as reconstructions of dense neuronal tissues or 3D histology reconstruction to gain new structural insights. However, robust and fully automatic 3D image registration for biological data is difficult due to complex deformations, unbalanced staining and variations on data appearance. This study presents a fully automatic and robust 3D registration technique for microscopic image reconstruction, and we demonstrate our method on two ssTEM datasets of drosophila brain neural tissues, serial confocal laser scanning microscopic images of a drosophila brain, serial histopathological images of renal cortical tissues and a synthetic test case. The results show that the presented fully automatic method is promising to reassemble continuous volumes and minimize artificial deformations for all data and outperforms four state-of-the-art 3D registration techniques to consistently produce solid 3D reconstructed anatomies with less discontinuities and deformations.
Most medical image registration algorithms suffer from a directionality bias that has been shown to largely impact subsequent analyses. Several approaches have been proposed in the literature to address this bias in the context of nonlinear registration, but little work has been done for global registration. We propose a symmetric approach based on a block-matching technique and least-trimmed square regression. The proposed method is suitable for multimodal registration and is robust to outliers in the input images. The symmetric framework is compared with the original asymmetric block-matching technique and is shown to outperform it in terms of accuracy and robustness. The methodology presented in this article has been made available to the community as part of the NiftyReg open-source package.
It is known that image registration is mostly driven by image edges. We have taken this idea to the extreme. In segmented images, we ignore the interior of the components and focus on their boundaries only. Furthermore, by assuming spatial compactness of the components, the similarity criterion can be approximated by sampling only a small number of points on the normals passing through a sparse set of keypoints. This leads to an order-of-magnitude speed advantage in comparison with classical registration algorithms. Surprisingly, despite the crude approximation, the accuracy is comparable. By virtue of the segmentation and by using a suitable similarity criterion such as mutual information on labels, the method can handle large appearance differences and large variability in the segmentations. The segmentation does not need not be perfectly coherent between images and over-segmentation is acceptable. We demonstrate the performance of the method on a range of different datasets, including histological slices and Drosophila imaginal discs, using rigid transformations.
Histology is the microscopic inspection of plant or animal tissue. It is a critical component in diagnostic medicine and a tool for studying the pathogenesis and biology of processes such as cancer and embryogenesis. Tissue processing for histology has become increasingly automated, drastically increasing the speed at which histology labs can produce tissue slides for viewing. Another trend is the digitization of these slides, allowing them to be viewed on a computer rather than through a microscope. Despite these changes, much of the routine analysis of tissue sections remains a painstaking, manual task that can only be completed by highly trained pathologists at a high cost per hour. There is, therefore, a niche for image analysis methods that can automate some aspects of this analysis. These methods could also automate tasks that are prohibitively time-consuming for humans, e.g., discovering new disease markers from hundreds of whole-slide images (WSIs) or precisely quantifying tissues within a tumor.
Most registration algorithms suffer from a directionality bias that has been shown to largely impact on subsequent analyses. Several approaches have been proposed in the literature to address this bias in the context of non-linear registration but little work has been done in the context of global registration. We propose a symmetric approach based on a block-matching technique and least trimmed square regression. The proposed method is suitable for multi-modal registration and is robust to outliers in the input images. The symmetric framework is compared to the original asymmetric block-matching technique, outperforming it in terms accuracy and robustness.
Image registration of biological data is challenging as complex deformation problems are common. Possible deformation effects can be caused in individual data preparation processes, involving morphological deformations, stain variations, stain artifacts, rotation, translation, and missing tissues. The combining deformation effects tend to make existing automatic registration methods perform poor. In our experiments on serial histopathological images, the six state of the art image registration techniques, including TrakEM2, SURF + affine transformation, UnwarpJ, bUnwarpJ, CLAHE + bUnwarpJ and BrainAligner, achieve no greater than 70% averaged accuracies, while the proposed method achieves 91.49% averaged accuracy. The proposed method has also been demonstrated to be significantly better in alignment of laser scanning microscope brain images and serial ssTEM images than the benchmark automatic approaches (p < 0.001). The contribution of this study is to introduce a fully automatic, robust and fast image registration method for 2D image registration.
We describe an automatic method for fast registration of images with very different appearances. The images are jointly segmented into a small number of classes, the segmented images are registered, and the process is repeated. The segmentation calculates feature vectors on superpixels and then it finds a softmax classifier maximizing mu-tual information between class labels in the two images. For speed, the registration considers a sparse set of rectangular neighborhoods on the interfaces between classes. A triangulation is created with spatial regularization handled by pairwise spring-like terms on the edges. The optimal transformation is found globally using loopy belief propagation. Multiresolution helps to improve speed and ro-bustness. Our main application is registering stained histological slices, which are large and differ both in the local and global appear-ance. We show that our method has comparable accuracy to standard pixel-based registration, while being faster and more general.
Evaluating various algorithms for the inter-subject registration of brain magnetic resonance images (MRI) is a necessary topic receiving growing attention. Existing studies evaluated image registration algorithms in specific tasks or using specific databases (e. g., only for skull-stripped images, only for single-site images, etc.). Consequently, the choice of registration algorithms seems task-and usage/parameter-dependent. Nevertheless, recent large-scale, often multi-institutional imaging-related studies create the need and raise the question whether some registration algorithms can 1) generally apply to various tasks/databases posing various challenges; 2) perform consistently well, and while doing so, 3) require minimal or ideally no parameter tuning. In seeking answers to this question, we evaluated 12 general-purpose registration algorithms, for their generality, accuracy and robustness. We fixed their parameters at values suggested by algorithm developers as reported in the literature. We tested them in 7 databases/tasks, which present one or more of 4 commonly-encountered challenges: 1) inter-subject anatomical variability in skull-stripped images; 2) intensity homogeneity, noise and large structural differences in raw images; 3) imaging protocol and field-of-view (FOV) differences in multi-site data; and 4) missing correspondences in pathology-bearing images. Totally 7,562 registrations were performed. Registration accuracies were measured by (multi-) expert-annotated landmarks or regions of interest (ROIs). To ensure reproducibility, we used public software tools, public databases (whenever possible), and we fully disclose the parameter settings. We show evaluation results, and discuss the performances in light of algorithms' similarity metrics, transformation models and optimization strategies. We also discuss future directions for the algorithm development and evaluations.
The analysis of protein-level multigene expression signature maps computed from the fusion of differently stained immunohistochemistry images is an emerging tool in cancer management. Creating these maps requires registering sets of histological images, a challenging task due to their large size, the non-linear distortions existing between consecutive sections and to the fact that the images correspond to different histological stains and thus, may have very different appearance. In this manuscript, we present a novel segmentation-based registration algorithm that exploits a multi-class pyramid and optimizes a fuzzy class assignment specially designed for this task. Compared to a standard nonrigid registration, the proposed method achieves an improved matching on both synthetic as well as real histological images of cancer lesions.
Registration of histopathology images of consecutive tissue sections stained with different histochemical or immunohistochemical stains is an important step in a number of application areas, such as the investigation of the pathology of a disease, validation of MRI sequences against tissue images, multi-scale physical modelling etc. In each case information from each stain needs to be spatially aligned and combined to ascertain physical or functional properties of the tissue. However, in addition to the gigabyte size images and non-rigid distortions present in the tissue, a major challenge for registering differently stained histology image pairs is the dissimilar structural appearance due to different stains highlighting different substances in tissues. In this paper, we address this challenge by developing an unsupervised content classification method which generates multi-channel probability images from a roughly aligned image pair. Each channel corresponds to one automatically identified content class. The probability images enhance the structural similarity between image pairs. By integrating the classification method into a multi-resolution block matching based non-rigid registration scheme (our previous work [9]), we improve the performance of registering multi-stained histology images. Evaluation was conducted on 77 histological image pairs taken from three liver specimens and one intervertebral disc specimen. In total six types of histochemical stains were tested. We evaluated our method against the same registration method implemented without applying the classification algorithm (intensity based registration) and the state-of-art mutual information based registration. Superior results are obtained with the proposed method.
The aim of this paper is to validate an image registration pipeline used for histology image alignment. In this work a set of histology images are registered to their correspondent optical blockface images to make a histology volume. Then multi-modality fiducial markers are used to validate the alignment of histology images. The fiducial markers are catheters perfused with a mixture of cuttlefish ink and flour. Based on our previous investigations this fiducial marker is visible in medical images, optical blockface images and it can also be localized in histology images. The properties of this fiducial marker make it suitable for validation of the registration techniques used for histology image alignment. This paper reports on the accuracy of a histology image registration approach by calculation of target registration error using these fiducial markers.
Three dimensional (3D) tissue reconstructions from the histology images with different stains allows the spatial alignment of structural and functional elements highlighted by different stains for quantitative study of many physiological and pathological phenomena. This has significant potential to improve the understanding of the growth patterns and the spatial arrangement of diseased cells, and enhance the study of biomechanical behavior of the tissue structures towards better treatments (e.g. tissue-engineering applications).
This paper evaluates three strategies for 3D reconstruction from sets of two dimensional (2D) histological sections with different stains, by combining methods of 2D multi-stain registration and 3D volumetric reconstruction from same stain sections.
The different strategies have been evaluated on two liver specimens (80 sections in total) stained with Hematoxylin and Eosin (H and E), Sirius Red, and Cytokeratin (CK) 7.
A strategy of using multi-stain registration to align images of a second stain to a volume reconstructed by same-stain registration results in the lowest overall error, although an interlaced image registration approach may be more robust to poor section quality.
Deformable image registration is a fundamental task in medical image processing. Among its most important applications, one may cite: i) multi-modality fusion, where information acquired by different imaging devices or protocols is fused to facilitate diagnosis and treatment planning; ii) longitudinal studies, where temporal structural or anatomical changes are investigated; and iii) population modeling and statistical atlases used to study normal anatomical variability. In this paper, we attempt to give an overview of deformable registration methods, putting emphasis on the most recent advances in the domain. Additional emphasis has been given to techniques applied to medical images. In order to study image registration methods in depth, their main components are identified and studied independently. The most recent techniques are presented in a systematic fashion. The contribution of this paper is to provide an extensive account of registration techniques in a systematic manner.
For the past 25 years NIH Image and ImageJ software have been pioneers as open tools for the
analysis of scientific images. We discuss the origins, challenges and solutions of these two programs,
and how their history can serve to advise and inform other software projects.
Anatomy of large biological specimens is often reconstructed from serially sectioned volumes imaged by high-resolution microscopy. We developed a method to reassemble a continuous volume from such large section series that explicitly minimizes artificial deformation by applying a global elastic constraint. We demonstrate our method on a series of transmission electron microscopy sections covering the entire 558-cell Caenorhabditis elegans embryo and a segment of the Drosophila melanogaster larval ventral nerve cord.
This paper aims to present a review of recent as well as classic image registration methods. Image registration is the process of overlaying images (two or more) of the same scene taken at different times, from different viewpoints, and/or by different sensors. The registration geometrically align two images (the reference and sensed images). The reviewed approaches are classified according to their nature (area-based and feature-based) and according to four basic steps of image registration procedure: feature detection, feature matching, mapping function design, and image transformation and resampling. Main contributions, advantages, and drawbacks of the methods are mentioned in the paper. Problematic issues of image registration and outlook for the future research are discussed too. The major goal of the paper is to provide a comprehensive reference source for the researchers involved in image registration, regardless of particular application areas.
In this paper, we present an easy-to-follow procedure for the analysis of tissue sections from 3D cell cultures (spheroids) by matrix-assisted laser desorption/ionization mass spectrometry imaging (MALDI MSI) and laser scanning confocal microscopy (LSCM). MALDI MSI was chosen to detect the distribution of the drug of interest, while fluorescence immunohistochemistry (IHC) followed by LSCM was used to localize the cells featuring specific markers of viability, proliferation, apoptosis and metastasis. The overlay of the mass spectrometry (MS) and IHC spheroid images, typically without any morphological features, required fiducial-based coregistration. The MALDI MSI protocol was optimized in terms of fiducial composition and antigen epitope preservation to allow MALDI MSI to be performed and directly followed by IHC analysis on exactly the same spheroid section. Once MS and IHC images were coregistered, the quantification of the MS and IHC signals was performed by an algorithm evaluating signal intensities along equidistant layers from the spheroid boundary to its center. This accurate colocalization of MS and IHC signals showed limited penetration of the clinically tested drug perifosine into spheroids during a 24 h period, revealing the fraction of proliferating and promigratory/proinvasive cells present in the perifosine-free areas, decrease of their abundance in the perifosine-positive regions and distinguishing between apoptosis resulting from hypoxia/nutrient deprivation and drug exposure.
Consecutive thin sections of tissue samples make it possible to study local variation in e.g. protein expression and tumor heterogeneity by staining for a new protein in each section. In order to compare and correlate patterns of different proteins, the images have to be registered with high accuracy. The problem we want to solve is registration of gigapixel whole slide images (WSI). This presents 3 challenges: (i) Images are very large; (ii) Thin sections result in artifacts that make global affine registration prone to very large local errors; (iii) Local affine registration is required to preserve correct tissue morphology (local size, shape and texture). In our approach we compare WSI registration based on automatic and manual feature selection on either the full image or natural sub-regions (as opposed to square tiles). Working with natural sub-regions, in an interactive tool makes it possible to exclude regions containing scientifically irrelevant information. We also present a new way to visualize local registration quality by a Registration Confidence Map (RCM). With this method, intra-tumor heterogeneity and characteristics of the tumor microenvironment can be observed and quantified.
The analysis of the pure motion of subnuclear structures without influence of the cell nucleus motion and deformation is essential in live cell imaging. In this work, we propose a 2D contour-based image registration approach for compensation of nucleus motion and deformation in fluorescence microscopy time-lapse sequences. The proposed approach extends our previous approach which uses a static elasticity model to register cell images. Compared to that scheme, the new approach employs a dynamic elasticity model for forward simulation of nucleus motion and deformation based on the motion of its contours. The contour matching process is embedded as a constraint into the system of equations describing the elastic behavior of the nucleus. This results in better performance in terms of the registration accuracy. Our approach was successfully applied to real live cell microscopy image sequences of different types of cells including image data that was specifically designed and acquired for evaluation of cell image registration methods. An experimental comparison with existing contour-based registration methods and an intensity-based registration method has been performed. We also studied the dependence of the results on the choice of method parameters.
Cardiac magnetic resonance perfusion examinations enable noninvasive quantification of myocardial blood flow. However, motion between frames due to breathing must be corrected for quantitative analysis. Although several methods have been proposed, there is a lack of widely available benchmarks to compare different algorithms. We sought to compare many algorithms from several groups in an open benchmark challenge. Nine clinical studies from two different centers comprising normal and diseased myocardium at both rest and stress were made available for this study. The primary validation measure was regional myocardial blood flow based on the transfer coefficient
, which was computed using a compartment model and the myocardial perfusion reserve (MPR) index. The ground truth was calculated using contours drawn manually on all frames by a single observer, and visually inspected by a second observer. Six groups participated and 19 different motion correction algorithms were compared. Each method used one of three different motion models: rigid, global affine, or local deformation. The similarity metric also varied with methods employing either sum-of-squared differences, mutual information, or cross correlation. There were no significant differences in
or MPR compared across different motion models or similarity metrics. Compared with the ground truth, only
for the sum-of-squared differences metric, and for local deformation motion models, had significant bias. In conclusion, the open benchmark enabled evaluation of clinical perfusion indices over a wide range of methods. In particular, there was no benefit of nonrigid registration techniques over the other methods evaluated in this study. The benchmark data and results are available from the Cardiac Atlas Project (
www.cardiacatlas.org
).
