ArticlePDF Available

Abstract and Figures

Inactivation of the retinoblastoma gene in mouse embryos results in morphological changes in the placenta, which has been shown to affect fetal survivability. The construction of a 3D virtual placenta aids in accurately quantifying structural changes using image analysis. The placenta dataset consisted of 786 images totaling 550 GB in size, which were registered into a volumetric dataset using ITK's registration framework. The registration process faces many challenges arising from the large image sizes, damages during sectioning, staining gradients both within and across sections, and background noise leading to local solutions. In this work, we implement a rigorous ITK-based preprocessing pipeline for removing noise and employ a novel 2-level optimization strategy for enhanced registration in ITK. We provide 3D visualizations and numerical results to demonstrate our improvements.
Content may be subject to copyright.
ITK-based Registration of Large Images from
Light Microscopy: A Biomedical Application
K. Mosaliganti1, T. Pan2, R. Machiraju1, K. Huang2, and J. Saltz2
Department of 1Computer Science and Engineering, 2Biomedical Informatics
The Ohio State University, Columbus, Ohio, USA
Abstract. Inactivation of the retinoblastoma gene in mouse embryos
results in morphological changes in the placenta, which has been shown
to affect fetal survivability. The construction of a 3D virtual placenta
aids in accurately quantifying structural changes using image analysis.
The placenta dataset consisted of 786 images totaling 550 GB in size,
which were registered into a volumetric dataset using ITK’s registra-
tion framework. The registration process faces many challenges arising
from the large image sizes, damages during sectioning, staining gradients
both within and across sections, and background noise leading to local
solutions. In this work, we implement a rigorous ITK-based preprocess-
ing pipeline for removing noise and employ a novel 2-level optimization
strategy for enhanced registration in ITK. We provide 3D visualizations
and numerical results to demonstrate our improvements.
1 Introduction
Many human diseases have genetic bases that manifest phenotypically as func-
tional and/or structural deficiencies. The retinoblastoma (Rb) gene is one of the
first to be associated with a specific cancer, and has been studied extensively
in both human cells and mouse models [1]. Recently, it has been shown that
inactivation of Rb (Rb) in mouse embryo results in morphological changes in
the placenta, including reduction in vascularity of the labyrinth layer. Figure
1 shows a sample slide with the approximate labyrinth,spongiotrophoblast and
glycogen layers. The labyrinth layer serves the critical function of gas, nutrient,
and waste exchange between the mother and the fetus and hence understanding
its interaction with neighboring tissue layers is important. A decrease in vas-
cularity is thought to contributes to fetal death at 14.5 days of gestation [2].
Image analysis of the structural changes in mouse placenta provides an oppor-
tunity for quantitative correlation of genetic changes with phenotype expressions.
Specifically, the goals of image analysis are segmentation of the tissue layers, vi-
sualization of the finger-like infiltrations into the labyrinth, and quantitative
measurement of volume and surface area of the layers. Registration of the tissue
sections is therefore, an important pre-processing step for various algorithms as
shown in Fig. 1 (Right), and its accuracy directly affects the quantification and
Fig. 1. Left: Placenta slice showing different tissue types. A small region magnified to
show labyrinth layer (within red boundaries), spongiotrophoblast layer (between the
red and green boundaries) and spongiotrophoblast and glycogen infiltrations (yellow
circles). Right: Overall image analysis framework
interpretation of the results. Mutant (Rb) placenta was harvested at 13.5 days
of gestation and prepared using a standard histological protocol. It was fixed in
paraffin and sectioned at 3µm thickness using a microtome. Serial sections were
mounted on glass slides and stained with hematoxylin and eosin. The slides are
scanned in an Aperio ScanScope slide digitizer at 200x magnification. The image
dimensions on average were (15K x 15K x 3). The entire placenta produced 786
color images, ranging in size from 500 MB to 1 GB each. The manual nature
of the tissue preparation process required extensive preprocessing steps as well
as modifications to the registration process. Our pipeline was implemented in in
the National Library of Medicine’s (NIH/NLM) Insight Segmentation and Reg-
istration Toolkit (ITK) [3]. The following artifacts were observed in the images:
Fig. 2. A: background noise from acquisition. B: tissue tear and shear. C,D: consecutive
sections have non-linear intensity variations.
1. Orientation differences: Sections are mounted in different orientations on glass
slides due to the manual nature of the process.
2. Luminance gradient: Sections mounted close to the edge of the glass slide
produce images with significant luminance gradients (Fig. 2A).
3. Non-tissue noise: Dust and air bubbles in the slide may cause artifacts.
4. Damaged and missing sections: During sectioning and mounting, sections are
occasionally torn, folded, or discarded entirely. (Fig. 2B).
5. Staining variations: Differences in section thickness, staining duration, and
Fig. 3. Left: Pre-processing pipeline. Right: ITK-based registration framework. Trans-
forms are scaled when passed from a lower to higher resolution.
stain concentration result in color variations (Fig. 2C,D).
In what follows, Section 2 explains the ITK framework for noise removal
with the details of our registration framework components in Section 3. Section
4 provides details on our new optimization strategy with experimental and vi-
sualization results. Finally, we report our conclusions and outline our plans for
the future in Section 5.
2 Preprocessing Framework
Defective Section Exclusion: Defective sections are identified using manual in-
spection and excluded from the pipeline. They are later accounted for in the 3D
reconstruction and visualization process.
Tissue Detection: The first step separates the tissue (foreground) from the back-
ground (noise). We use the k-BCC clustering approach [4] with multi-resolution
kd-tree optimizations for tissue detection, as it performs well in the presence of
luminance gradient, background noise, and staining variations. It is a variation
of the popular k-means algorithm adapted to the light microscopy images. It is
validated for the placenta slices and compared with the Expectation Maximiza-
tion (EM) and k-means clustering in [4]. Tissue pixels are identified and stored
as binary masks which are used in the level-set based edge detection and the
PCA alignment of sections.
Level-Sets for edge detection The binary mask obtained from the clustering al-
gorithm has a poor description of the placenta boundary as shown in Figure 4.
Moreover, parts of the placenta interior that were initially occupied by tissues of
low luminance (such as vasculature) are clustered as background. These hollow
regions also contribute to noise when evaluating the registration match across
images since they are usually not well-correlated. Therefore, by using level-sets
[5], an accurate boundary description is obtained by growing a contour inwards
from the edges of the image boundary.
PCA Alignment: The possible translations and rotations in a plane define the
transform space for the placenta images. Using a priori knowledge that mouse
placenta sections are typically elongated in shape, we use principle component
analysis (PCA) to estimate tissue orientation. The orientations are used to ini-
tialize registration, thus restricting the transform search space. PCA is performed
Fig. 4. Left: An example binary mask obtained from the KBCC algorithm. Center:An
example tissue mask image from level-sets. Right: Top sequence shows the initial ran-
dom orientation of the placenta images. The bottom sequence shows PCA-aligned
for each tissue mask image. The gleaned orientation angle is used to rotate and
align the original histology images. Figure 4 shows the same two sequences of
images before and after PCA alignment.
3 Mutual Information Based Registration
While the placenta images are acquired using the same staining protocol, they
have multimodal characteristics due to non-linear variations in pixel intensities.
This may be attributed to different stain absorptions, tissue thickness and lu-
minance gradients. Under these conditions, it is difficult to detect appropriate
landmarks for point or surface-based registration [6]. Intramodal registration
methods that rely on linear correlation of pixel values are also inadequate [7].
Mutual information (MI) [8] based methods are effective in registering multi-
modal images with non-linear pixel intensity correlations.
We perform registration as an optimization process that searches for an im-
age transform that allows the closest similarity between consecutive images. The
input is a pair of stationary (S) and moving (M) images. The framework itera-
tively passes Mthrough 4 stages: transform,metric,optimizer, and interpolator
as shown in Figure 3 (Right). At a given iteration, the transform stage applies
transform Tnon Mwhich is then resampled onto a grid in the interpolator
stage to yield Tn(M). The metric stage computes the goodness of fit between S
and Tn(M). Based on the similarity metric values from current and prior itera-
tions, the optimizer produces a refinement in transform Tn+1. At convergence,
Sand Tn(M) have the optimal metric value thereby aligning the images. For
the placenta dataset, we register each image to its predecessor, and the pairwise
transforms were merged sequentially to obtain each image’s global transform.
We adopt regular step gradient descent optimizer [9] in the interpolator and
optimizer stages. We model the transform using rigid 2D transform, which allows
2D rotation and translation. This choice is based on the physical conditions that
the tissue does not elastically deform during sectioning or mounting, that the
microtome produces parallel sections, and that all images are acquired at the
same magnification. The only variable components of the transform are in-plane
rotation and translation.
Fig. 5. Application of the regular step gradient descent (A) and the two-level opti-
mizer (B) on PCA-aligned slides. Note that the regular step gradient descent settles
in local solutions. Right: Plot showing the metric values after convergence (blue). The
improvement in metric values using the two-level versus the standard optimizer is also
shown (red). Note that the negative of mutual information is plotted.
We apply the multiresolution approach, using 2-level image pyramids as
shown in Fig. 3 (Right). Optimal transforms obtained from a lower magnifica-
tion (Xn) are scaled and used as initialization for registration of the next higher
magnification (Xn+1). The process is repeated for each magnification level to
obtain the final transforms. We note that at magnifications higher than 20x, the
computational cost for registration outweighs the improvements in accuracy.
4 Two-level Optimization
Tissue detection and PCA-based image alignment significantly improved reg-
istration performance by reducing the search space for the optimal transform.
However, 3D visualization of the registered placenta revealed that the transforms
remained suboptimal. Experiments with different optimizers such as conjugate-
gradient and gradient descent did not affect performance significantly due to
overtly noisy metrics. We propose the use of a novel two-level optimization
scheme by searching the transform space neighborhood with a random walk
Optimizers such as regular-step gradient descent and conjugate gradient ([9])
increase the rate of convergence by refining the transform based on their current
learning rates from observed MI gradients. However, their performance degrades
rapidly since the learning rates are sensitive to data noise. In the placenta data,
we observe that ITK optimizers converge to local solutions as shown in Fig. 5(A)
for most image pairs (better MI values were later observed with the two-level
We use a two-level optimization strategy, which perturbs converged solutions
via random walks and then restarts the normal registration process. In essence,
the perturbation introduces a step in the transform space that is larger than
the specified learning rate of the regular step gradient descent optimizer. The
step is a restricted translation and rotation around the PCA initialization. If the
previous result was a local minimum, then the perturbation may result in a better
solution with higher MI (lower -MI) as shown in Fig. 5 (B) and a new solution
is realized. The process is repeated until the MI does not improve even after an
user-defined Nperturbations. Similar to standard optimizers, convergence to a
global solution is not guaranteed. However, the registration is likely to perform
better since this approach allows hill climb against the gradient direction.
Fig. 6. Left: 3D visualization of the labyrinth layer within the placenta slices, shown as
white boundaries. Right: A single finger like infiltration from the spongiotrophoblast
into the labyrinth is zoomed and shown.
In the placenta images, we determined that maximum transform deviation
of PCA aligned images is ±80 pixels in translation and ±10 degrees in rotation
around the tissue centroid. The hybrid optimization strategy lead to global solu-
tions in 99% of the images within N=20 iterations. The remaining 1% converges
within N=50 iterations. Figure 5 (Right) shows the plots of the final MI values
of all the images (blue) using the two-level optimizer. The improvement of the
two-level optimizer compared to standard optimizer is shown as a difference in
the MI values at convergence (red). The average negative MI of all images using
the standard and two-level optimizers are -0.122 and -0.142 respectively. Fig-
ure 6 (Left) shows the 3D visualization of the labyrinth layer and the placenta
contours. Figure 6 (Right) show the zoomed cross-section of a single finger like
infiltration from the spongiotrophoblast into the labyrinth layer. Figure 7 is yet
another example of finger visualization.
5 Conclusion and Future Work
In this paper, we described our experience in registering large serial sections
of a histology sample. Our contribution involves methods to adapt the ITK
registration process for real-world data such as the mouse placenta. We utilized
pre-processing techniques to account for acquisition artifacts, defective histology
sections, and large parameter search space. We also employed a multi-resolution
implementation of MI registration algorithm. We propose a new optimization
Fig. 7. Both images are cropped volumes from the original dataset focusing on infiltra-
tions into the labyrinth layer. In both images the labyrinth is shown in blue. In the top
image glycogen is shown in yellow and placenta boundary in white, while the bottom
image highlights glycogen in red. The cross-hairs indicate the center of the infiltration
into the labyrinth layer.
strategy that has a higher probability of convergence to global solutions albeit
with longer processing times. Our results show an improvement in the final 3D
reconstructions and overall lower MI metric values.
1. Mouse Genome Sequencing Consortium: Initial sequencing and comparative analysis
of the mouse genome. Nature 420 (2002) 520–562
2. Wu, L., de Bruin, A., Saavedra, H.I., Starovic, M., Trimboli, A., Yang, Y., Opavska,
J., Wilson, P., Thompson, J.C., Ostrowski, M.C., Rosol, T.J., Woollett, L.A., Wein-
stein, M., Cross, J.C., Robinson, M.L., Leone, G.: Extra-embryonic function of Rb
is essential for embryonic development and viability. Nature 421 (2003) 942–947
3. Ib´a˜nez, L., Schroeder, W.: The ITK Software Guide. The Insight and Registration
Toolkit [], Kitware, Inc. (2003)
4. Rao, K., Machiraju, R., Heverhagen, J., Saltz, J., Knopp, M.: Exploratory segmen-
tation using geometric tessellations (in presss). In: Proceedings of 10th International
Fall Workshop on Vision, Modeling and Visualization. (2005)
5. Malladi, R., Sethian, J.A., Vermuri, B.C.: Shape modeling with front propagation:
A level set approach. IEEE Trans. on Pattern Analysis and Machine Intelligence
17(2) (1995) 158–174
6. Maintz, J.A.B.: Retrospective registration of tomographic brain images. Ph.D.
dissertation, Univ. Utrecht, Utrecht, The Netherlands (1996)
7. Maes, F., Vandermeulen, D., Suetens, P.: Medical image registration using mutual
information. Proceedings of the IEEE 91(10) (1996) 1699–1722
8. Maes, F., Collignon, A., Vandermeulen, D., Marchal, G., Suetens, P.: Multimodality
image registration by maximization of mutual information. IEEE Trans. Medical
Imaging 16(2) (1997) 187–198
9. Maes, F., Vandermeulen, D., Suetens, P.: Comparative evaluation of multiresolution
optimization strategies for multimodality image registration by maximization of
mutual information. Medical Image Analysis 3(4) (1999) 373–386
ResearchGate has not been able to resolve any citations for this publication.
Full-text available
User Guide for developer who use ITK in their applications. Everything you need to install, use, and extend the Insight Segmentation and Registration Toolikit ITK. Includes detailed examples, installation procedures, and system overview for ITK version 2.4. (The included examples are taken directly from the ITK source code repository and are designed to demonstrate the essential features of the software.) The book comes with a CD-ROM that contains a complete hyperlinked version of the book plus ITK source code, data, Windows binaries, and extensive class documentation. Also includes CMake binaries for managing the ITK build process on a variety of compiler and operating system configurations. U.S. National Library of Medicine (NLM).
Full-text available
The retinoblastoma (Rb) gene was the first tumour suppressor identified1. Inactivation of Rb in mice results in unscheduled cell proliferation, apoptosis and widespread developmental defects, leading to embryonic death by day 14.5 (refs 2-4). However, the actual cause of the embryonic lethality has not been fully investigated. Here we show that loss of Rb leads to excessive proliferation of trophoblast cells and a severe disruption of the normal labyrinth architecture in the placenta. This is accompanied by a decrease in vascularization and a reduction in placental transport function. We used two complementary techniques - tetraploid aggregation and conditional knockout strategies - to demonstrate that Rb-deficient embryos supplied with a wild-type placenta can be carried to term, but die soon after birth. Most of the neurological and erythroid abnormalities thought to be responsible for the embryonic lethality of Rb-null animals were virtually absent in rescued Rb-null pups. These findings identify and define a key function of Rb in extraembryonic cell lineages that is required for embryonic development and viability, and provide a mechanism for the cell autonomous versus non-cell autonomous roles of Rb in development.
Conference Paper
Full-text available
Mutual information of image intensities has been proposed as a new matching criterion for automated multi-modality image registration. In this paper the authors give experimental evidence of the power and the generality of the mutual information criterion by showing results for various applications involving CT, MR and PET images. The authors' results illustrate the large applicability of the approach and demonstrate its high suitability for routine use in clinical practice
Full-text available
Shape modeling is an important constituent of computer vision as well as computer graphics research. Shape models aid the tasks of object representation and recognition. This paper presents a new approach to shape modeling which retains some of the attractive features of existing methods and overcomes some of their limitations. The authors' techniques can be applied to model arbitrarily complex shapes, which include shapes with significant protrusions, and to situations where no a priori assumption about the object's topology is made. A single instance of the authors' model, when presented with an image having more than one object of interest, has the ability to split freely to represent each object. This method is based on the ideas developed by Osher and Sethian (1988) to model propagating solid/liquid interfaces with curvature-dependent speeds. The interface (front) is a closed, nonintersecting, hypersurface flowing along its gradient field with constant speed or a speed that depends on the curvature. It is moved by solving a “Hamilton-Jacobi” type equation written for a function in which the interface is a particular level set. A speed term synthesized from the image is used to stop the interface in the vicinity of object boundaries. The resulting equation of motion is solved by employing entropy-satisfying upwind finite difference schemes. The authors present a variety of ways of computing the evolving front, including narrow bands, reinitializations, and different stopping criteria. The efficacy of the scheme is demonstrated with numerical experiments on some synthesized images and some low contrast medical images
Full-text available
Analysis of multispectral or multitemporal images requires proper geometric alignment of the images to compare corresponding regions in each image volume. Retrospective three-dimensional alignment or registration of multimodal medical images based on features intrinsic to the image data itself is complicated by their different photometric properties, by the complexity of the anatomical objects in the scene and by the large variety of clinical applications in which registration is involved. While the accuracy of registration approaches based on matching of anatomical landmarks or object surfaces suffers from segmentation errors, voxel-based approaches consider all voxels in the image without the need for segmentation. The recent introduction of the criterion of maximization of mutual information, a basic concept from information theory, has proven to be a breakthrough in the field. While solutions for intrapatient affine registration based on this concept are already commercially available, current research in the field focuses on interpatient nonrigid matching.
In modern clinical practice, the clinician can make use of a vast array of specialized imaging techniques supporting diagnosis and treatment. For various reasons, the same anatomy of one patient is sometimes imaged more than once, either using the same imaging apparatus (monomodal acquisition ), or different ones (multimodal acquisition). To make simultaneous use of the acquired images, it is often necessary to bring these images in registration, i.e., to align their anatomical coordinate systems. The problem of medical image registration as concerns human brain images is addressed in this thesis. The specific chapters include a survey of recent literature, CT/MR registration using mathematical image features (edges and ridges), monomodal SPECT registration, and CT/MR/SPECT/PET registration using image features extracted by the use of mathematical morphology.
A new approach to the problem of multimodality medical image registration is proposed, using a basic concept from information theory, mutual information (MI), or relative entropy, as a new matching criterion. The method presented in this paper applies MI to measure the statistical dependence or information redundancy between the image intensities of corresponding voxels in both images, which is assumed to be maximal if the images are geometrically aligned. Maximization of MI is a very general and powerful criterion, because no assumptions are made regarding the nature of this dependence and no limiting constraints are imposed on the image content of the modalities involved. The accuracy of the MI criterion is validated for rigid body registration of computed tomography (CT), magnetic resonance (MR), and photon emission tomography (PET) images by comparison with the stereotactic registration solution, while robustness is evaluated with respect to implementation issues, such as interpolation and optimization, and image content, including partial overlap and image degradation. Our results demonstrate that subvoxel accuracy with respect to the stereotactic reference solution can be achieved completely automatically and without any prior segmentation, feature extraction, or other preprocessing steps which makes this method very well suited for clinical applications.
Maximization of mutual information of voxel intensities has been demonstrated to be a very powerful criterion for three-dimensional medical image registration, allowing robust and accurate fully automated affine registration of multimodal images in a variety of applications, without the need for segmentation or other preprocessing of the images. In this paper, we investigate the performance of various optimization methods and multiresolution strategies for maximization of mutual information, aiming at increasing registration speed when matching large high-resolution images. We show that mutual information is a continuous function of the affine registration parameters when appropriate interpolation is used and we derive analytic expressions of its derivatives that allow numerically exact evaluation of its gradient. Various multiresolution gradient- and non-gradient-based optimization strategies, such as Powell, simplex, steepest-descent, conjugate-gradient, quasi-Newton and Levenberg-Marquardt methods, are evaluated for registration of computed tomography (CT) and magnetic resonance images of the brain. Speed-ups of a factor of 3 on average compared to Powell's method at full resolution are achieved with similar precision and without a loss of robustness with the simplex, conjugate-gradient and Levenberg-Marquardt method using a two-level multiresolution scheme. Large data sets such as 256(2) x 128 MR and 512(2) x 48 CT images can be registered with subvoxel precision in <5 min CPU time on current workstations.
Exploratory segmentation using geometric tessellations (in presss)
  • K Rao
  • R Machiraju
  • J Heverhagen
  • J Saltz
  • M Knopp
Rao, K., Machiraju, R., Heverhagen, J., Saltz, J., Knopp, M.: Exploratory segmentation using geometric tessellations (in presss). In: Proceedings of 10th International Fall Workshop on Vision, Modeling and Visualization. (2005)