IMPROVED REGISTRATION FOR LARGE ELECTRON MICROSCOPY IMAGES
Ayelet Akselrod-Ballin1, Davi Bock2, R.Clay Reid2, Simon K. Warfield1
1Computational Radiology Laboratory, Children’s Hospital, Harvard Medical School, Boston, USA
2Department of Neurobiology, Harvard Medical School, USA
In this paper we introduce a novel algorithm for alignment of
Electron Microscopy images for 3D reconstruction. The algo-
rithm extends the Expectation Maximization - Iterative Clos-
est Points (EM-ICP) registration algorithm to go from point
matching to patch matching. We utilize local patch charac-
teristics to achieve improved registration. The method is ap-
plied to enable 3D reconstruction of Transmission Electron
Microscopy (TEM) images. We demonstrate results on large
TEM images and show the increased alignment accuracy of
IndexTerms— Registration, Reconstruction, Microscopy
The problem of image registration is defined as finding the
transformation that aligns images into one frame of reference.
This allows integration and comparison of information across
images. In this work we focus on reconstruction of three
dimensional (3D) Transmission Electron Microscopy (TEM)
images of neural tissue. Determining the detailed connections
in brain circuits is an essential unsolved problem in neuro-
science . Electron microscopy which provides resolutions
on the order of a nanometer is the primary tool for resolving
the 3D structure and connectivity of neurons. The difficulties
in reconstruction of neural circuitry from series of EM images
are due to the high resolution and the large size of the im-
ages, and the large amount of details of the relevant features
(,). Additionally, the deformation induced by both the
acquisition process and the intrinsic deformation of the slices
prevents using classical registration approaches developed for
conventional clinical imaging modalities.
Existing approaches for registration broadly divide either
into approaches that directly operate on image intensities or
to feature based alignments seeking to identify features that
should be aligned and an optimal transformation that brings
them into alignment . A recent study  addressed the
section to section matching as part of a complete algorithm
This investigation was supported in part by a research grant from CIMIT,
grants RG 3478A2/2 and RG 4032A1/1 from the NMSS, and by NIH grants
R03 CA126466, R01 RR021885, R01 GM074068, R01 EB008015 and P30
for assembling 3D volumes from EM data. This approach
first identifies feature descriptors based on a gradient vector
pyramid and then exploits these features to match adjacent
slices. In this work, we are motivated by a number of re-
cent registration methods that have used a set of small im-
age patches also called blocks, windows, fragments, or tem-
plates for alignment of microscopy images.  presented an
approach for 3D reconstruction based on a block matching
strategy where the local displacements were utilized to ro-
bustly estimate a global rigid transformation.  developed
a solution for 3D reconstruction of a series of TEM images
based on the finite support properties of the cubic B-splines,
where the initial estimate for the affine registration was based
on the block matching technique described in .
The main motivation for using image patches is that even
in situations of many artifacts and deformations, small areas
of overlap across the images can suffice for achieving accu-
rate alignment of microscopic data. However, in contrast with
previous approaches our new algorithm is not limited to a lo-
cal search neighborhood, an outlier rejection scheme is not
employed explicitly and the approach utilizes correspondence
probabilities allowing multiple matches instead of a restricted
number of exact correspondences. This enables us to consider
more regions in the images and to consider all of the potential
correspondences, ensuring that the search does not overlook
the correct set of correspondences, and thus dramatically im-
proving the robustness and accuracy of the registration.
Our unified approach is closely related to set of methods
designed to work with features such as point sets, line seg-
ments and surfaces. A popular approach in this context is the
Expectation Maximization Iterative Closest Points (EM-ICP)
introduced by . The algorithm determines the transforma-
tion that matches a set of model and scene points. The ap-
proach exploits the EM approach to optimize simultaneously
for correspondences and the registration transformation. The
algorithm was shown to be robust, precise and fast yet it has
an intrinsic limitation since by considering only the points co-
ordinates, only point spatial information is taken into account.
In this work we propose a novel algorithm for alignment
of features between cross-sectional slices by extending the
EM-ICP to include a normalized correlation (NC) similar-
ity measure of image patches. The probabilistic formulation
for transform estimation is modified to integrate the NC in
the definition of the prior probability for alignment. This re-
sults in a unified EM-ICP-NC approach which to our knowl-
edge has not been used for image registration before. The
main idea in this approach is to match local image patches
across the successive slices. By combining the benefits of
the EM-ICP with image patches we avoid the dependence on
points etc.) and operate directly on intensities. The transfor-
mation search is based on maximizing the similarity in inten-
sity between corresponding pixels. This is done without prior
segmentation of structures and without prior detection of cor-
responding features across the slices. The method is suited
for parallel distributed computation on a networked cluster,
permitting scalability to arbitrarily large images.
The remainder paper is organized as follow: Section 2 de-
scribes the algorithm; Section 3 presents experimental results,
and in Section 4 we discuss our conclusion and future work.
The data set on which we tested our algorithm consists of a
series of 200 TEM images of the lateral geniculate nucleus
of a ferret. Each image is about 10000 × 10000 pixels large
with a pixel resolution of 3nm and a slice thickness of 60nm.
Blendmont, a utility that is part of the IMOD package ,
was used to reconstruct the large field of view image from the
5 × 5 mosaics of smaller images coming from the camera.
The volume reconstruction is obtained by pairwise alignment
of consecutive slices and then composing them by taking as
reference the middle of the stack. Thus we focus on the prob-
lem of aligning a pair of successive sections. The input in-
cludes the fixed scene image ISand a moving model image
IM. OuraimistofindthetransformationT aligningthescene
with the model.
The algorithm consists of two main steps. In the first step,
pute the Normalized Correlation (NC) between each scene
patch and each model patch. Consequently, given a scene
patch psand model patch pm, their NC is estimated as
We use the NC similarity measure since NC is invariant to
linear intensity transformation and it is assumed that for small
corresponding image patches in two successive slices, the in-
tensities are locally related by some linear intensity trans-
formation. The images produced can sometime be severely
rotated or flipped; hence a brute-force searching scheme is
applied to account for different orientations. Namely, the
patches extracted are rotated by multiple angles and the al-
gorithm is performed with these different starting conditions.
Table 1. Outline of the Algorithm
Given a pair of successive slices a fixed (scene) and mov-
step 1: Extraction of features:
-A1. Extraction: Small image patches are extracted at differ-
ent locations for both images.
-A2. Evaluation: The scene features extracted are matched
against the model features, based on the NC similarity
measure between the patches.
Step 2: Transformation computations:
-B1. Select top percent of corresponding patches between
the images, and represent the patch by its center point.
-B2. Apply extended EM-ICP utilizing the NC for the ini-
tialization and as a prior during the iterations.
Then the angle that performs best between each pair of im-
ages is selected to determine the transformation for this pair.
In the second step, we construct two set of points based on
the center coordinates of the patches. Utilizing the patch pairs
with high NC similarity we compute the transformation with
the EM-ICP-NC algorithm outlined below. Table 1 presents
an outline of the registration algorithm proposed.
Let sibe the points of the scene set S ∈ R2and mithe points
of the model patch set M ∈ R2, with nsand nmdetermin-
ing the number of points respectively. T represents the rigid
transformation from the scene to the model. The probability
of a point sito correspond to the model points miis modeled
by a Gaussian probability distribution. In the case of homo-
geneous isotropic Gaussian noise
p(si|mj,T) = exp(−||T ∗ si− mj||2/2σ2)
where σ represents the noise in the measurement.
The idea is to maximize the log-likelihood of the data
distribution logp(S,A|M,T) where the unknown correspon-
dences are considered as a hidden random variables A ∈
RnS×nM. The algorithm starts by initialization of the trans-
formation (T), and repeats until convergence of the two EM
steps. In the E step, T is fixed and the probability of the
matches (A) are estimated.
In the classic EM-ICP the prior probability of the matches is
based on the uniform law: πij =
prior to account for the NC similarity of the patches taking
nM. Here we extend the
advantage of our use of patches rather than points. Thus the
prior is based on the normalized NC defined in 1.
Therefore we compute E(Aij) as follows:
πijexp(−||T ∗ si− mj||2/2σ2)
This same NC prior is used in the initialization to determine
the initial transformation T.
In the M-step, A is fixed and likelihood is optimized w.r.t
kπikexp(−||T ∗ si− mk||2/2σ2)
Using the EM-ICP-NC yields:
2.3. Computational complexity
The algorithm involves four parameters; including the size of
the patch K, the spacing ∆ between two following patches,
the number of patch pairs nppused and σ (in our implemen-
tation we used K = 100 × 100 and ∆ = 30, npp = 200,
σ = 1). Thus the computational complexity is determined
by the number of images T in the data base (T = 200) and
their size Q in pixels. The images were downsampled using
Gaussian smoothing plus bi-linear interpolation to a size of
≈ 1000 × 1000. The number of patches is determined by
their size and spacing and is proportional to TQ. Each patch
is compared to all the patches in the neighboring image, by
convolution, requiring time also proportional to Q. We as-
sume that the EM-ICP can be performed efficiently and its
computation time is small compared with the first step. Con-
sequently the overall complexity is O(TQ2).
3. EXPERIMENTS AND RESULTS
We validated our algorithm compared to manual human regis-
trations on a set of 43 pairs of slices. The registration was per-
formed by manually selecting corresponding points in a pair
of consecutive images and computing the pairwise transfor-
mation, which we denote as (T∗) based on Horn’s method .
Given a set of corresponding points in two systems, Horn’s
lem relating these points. Table 2 compares the results ob-
tained by three automatic approaches including the EM-ICP-
NC approach, the classic EM-ICP  and Horn’s  method.
Mean 50 ± 41
155 ± 276
862 ± 817
Table 2. Comparing our results to to the ground truth manual
transformation and to two other automatic approaches.Our
algorithm has higher accuracy and precision than previously
The L2difference between the transformations of all three au-
tomatic approaches are compared to the ground truth transfor-
mation computed based on manual selection of points. Figure
1 illustrates the alignment results obtained by our approach.
We present a novel algorithm for alignment of large TEM
images. To our knowledge this is the first attempt to inte-
grate EM-ICP together with a patch matching strategy and to
incorporate a similarity based prior into the EM-ICP model.
Our algorithm obtains accurate results and shows robustness
to noise and perturbations such as large rotations. The algo-
rithm can contribute to various applications both in TEM and
other medical imaging modalities. Future work will increase
the efficiency by utilizing a multiscale model and generalize
the approach to non-rigid registration and image mosaicing.
 S. J. Smith, “Circuit reconstruction tools today,” Curr Opin
Neurobiol, vol. 17 (5), pp. 601–608, 2007.
 N. Blow, “Following the wires,” Nature Methods, vol. 4 (11),
pp. 975–980, 2007.
 P. Koshevoy, T. Tasdizen, R. Whitaker, B. Jones, and R. Marc,
“Assembly of large three-dimensional volumes from serial-
section transmission electron microscopy,” MICCAI, 2006.
 J.B.A. Maintz and M.A. Viergever, “A survey of medical image
registration,” Medical image analysis, vol. 2(1), pp. 1–16, 1998.
 S. Ourselin, A. Roche, G. Subsol, X. Pennec, and N. Ayache,
“Reconstructinga 3d structure from serial histological sections,”
Image and Vision Computing, vol. 19, pp. 25–31, 2000.
 J. Dauguet, D. Bock, C. R. Reid, and S. K. Warfield, “Alignment
of large image series using cubic b-splines tessellation: Applica-
tion to transmission electron microscopy data,” MICCAI, 2006.
 S. Granger and X. Pennec, “Multi-scale em-icp: A fast and
robustapproachforsurfaceregistration,” ECCV,vol.4, pp.418–
 J. Kremer, D. Mastronarde, and J. McIntosh, “Computer visual-
ization of three dimensional image data using IMOD,” J. Struct.
Biol., vol. 116, pp. 71–76, 1996.
 B. K. P. Horn, “Closed-form solution of absolute orientation us-
ing unit quaternions,” Journal of the Optical Society of America,
vol. 4, pp. 629–642, 1987.
(1a) Fixed Slice(1b) Moving Slice - Before(1c) Checkerboard composite - Before
(1d) Fixed Slice(1e) Moving Slice - After(1f) Checkerboard composite - After
Successive pair (#1) of slices before and after alignment.
(2a) Fixed Slice(2b) Moving Slice - Before (2c) Checkerboard composite - Before
(2d) Fixed Slice(2e) Moving Slice - After (2f) Checkerboard composite - After
Successive pair (#2) of slices before and after alignment.
Fig. 1. Illustration of alignment results.