Page 1

IMPROVED REGISTRATION FOR LARGE ELECTRON MICROSCOPY IMAGES

Ayelet Akselrod-Ballin1, Davi Bock2, R.Clay Reid2, Simon K. Warfield1

1Computational Radiology Laboratory, Children’s Hospital, Harvard Medical School, Boston, USA

2Department of Neurobiology, Harvard Medical School, USA

ABSTRACT

In this paper we introduce a novel algorithm for alignment of

Electron Microscopy images for 3D reconstruction. The algo-

rithm extends the Expectation Maximization - Iterative Clos-

est Points (EM-ICP) registration algorithm to go from point

matching to patch matching. We utilize local patch charac-

teristics to achieve improved registration. The method is ap-

plied to enable 3D reconstruction of Transmission Electron

Microscopy (TEM) images. We demonstrate results on large

TEM images and show the increased alignment accuracy of

our approach.

IndexTerms— Registration, Reconstruction, Microscopy

1. INTRODUCTION

The problem of image registration is defined as finding the

transformation that aligns images into one frame of reference.

This allows integration and comparison of information across

images. In this work we focus on reconstruction of three

dimensional (3D) Transmission Electron Microscopy (TEM)

images of neural tissue. Determining the detailed connections

in brain circuits is an essential unsolved problem in neuro-

science [1]. Electron microscopy which provides resolutions

on the order of a nanometer is the primary tool for resolving

the 3D structure and connectivity of neurons. The difficulties

in reconstruction of neural circuitry from series of EM images

are due to the high resolution and the large size of the im-

ages, and the large amount of details of the relevant features

([2],[3]). Additionally, the deformation induced by both the

acquisition process and the intrinsic deformation of the slices

prevents using classical registration approaches developed for

conventional clinical imaging modalities.

Existing approaches for registration broadly divide either

into approaches that directly operate on image intensities or

to feature based alignments seeking to identify features that

should be aligned and an optimal transformation that brings

them into alignment [4]. A recent study [3] addressed the

section to section matching as part of a complete algorithm

This investigation was supported in part by a research grant from CIMIT,

grants RG 3478A2/2 and RG 4032A1/1 from the NMSS, and by NIH grants

R03 CA126466, R01 RR021885, R01 GM074068, R01 EB008015 and P30

HD018655.

for assembling 3D volumes from EM data. This approach

first identifies feature descriptors based on a gradient vector

pyramid and then exploits these features to match adjacent

slices. In this work, we are motivated by a number of re-

cent registration methods that have used a set of small im-

age patches also called blocks, windows, fragments, or tem-

plates for alignment of microscopy images. [5] presented an

approach for 3D reconstruction based on a block matching

strategy where the local displacements were utilized to ro-

bustly estimate a global rigid transformation. [6] developed

a solution for 3D reconstruction of a series of TEM images

based on the finite support properties of the cubic B-splines,

where the initial estimate for the affine registration was based

on the block matching technique described in [5].

The main motivation for using image patches is that even

in situations of many artifacts and deformations, small areas

of overlap across the images can suffice for achieving accu-

rate alignment of microscopic data. However, in contrast with

previous approaches our new algorithm is not limited to a lo-

cal search neighborhood, an outlier rejection scheme is not

employed explicitly and the approach utilizes correspondence

probabilities allowing multiple matches instead of a restricted

number of exact correspondences. This enables us to consider

more regions in the images and to consider all of the potential

correspondences, ensuring that the search does not overlook

the correct set of correspondences, and thus dramatically im-

proving the robustness and accuracy of the registration.

Our unified approach is closely related to set of methods

designed to work with features such as point sets, line seg-

ments and surfaces. A popular approach in this context is the

Expectation Maximization Iterative Closest Points (EM-ICP)

introduced by [7]. The algorithm determines the transforma-

tion that matches a set of model and scene points. The ap-

proach exploits the EM approach to optimize simultaneously

for correspondences and the registration transformation. The

algorithm was shown to be robust, precise and fast yet it has

an intrinsic limitation since by considering only the points co-

ordinates, only point spatial information is taken into account.

In this work we propose a novel algorithm for alignment

of features between cross-sectional slices by extending the

EM-ICP to include a normalized correlation (NC) similar-

ity measure of image patches. The probabilistic formulation

for transform estimation is modified to integrate the NC in

Page 2

the definition of the prior probability for alignment. This re-

sults in a unified EM-ICP-NC approach which to our knowl-

edge has not been used for image registration before. The

main idea in this approach is to match local image patches

across the successive slices. By combining the benefits of

the EM-ICP with image patches we avoid the dependence on

representation(identificationofcorrespondingsurfaces, lines,

points etc.) and operate directly on intensities. The transfor-

mation search is based on maximizing the similarity in inten-

sity between corresponding pixels. This is done without prior

segmentation of structures and without prior detection of cor-

responding features across the slices. The method is suited

for parallel distributed computation on a networked cluster,

permitting scalability to arbitrarily large images.

The remainder paper is organized as follow: Section 2 de-

scribes the algorithm; Section 3 presents experimental results,

and in Section 4 we discuss our conclusion and future work.

2. METHODS

The data set on which we tested our algorithm consists of a

series of 200 TEM images of the lateral geniculate nucleus

of a ferret. Each image is about 10000 × 10000 pixels large

with a pixel resolution of 3nm and a slice thickness of 60nm.

Blendmont, a utility that is part of the IMOD package [8],

was used to reconstruct the large field of view image from the

5 × 5 mosaics of smaller images coming from the camera.

2.1. Algorithm

The volume reconstruction is obtained by pairwise alignment

of consecutive slices and then composing them by taking as

reference the middle of the stack. Thus we focus on the prob-

lem of aligning a pair of successive sections. The input in-

cludes the fixed scene image ISand a moving model image

IM. OuraimistofindthetransformationT aligningthescene

with the model.

The algorithm consists of two main steps. In the first step,

weextractpatchesfromthesceneandmodelimagesandcom-

pute the Normalized Correlation (NC) between each scene

patch and each model patch. Consequently, given a scene

patch psand model patch pm, their NC is estimated as

NC(ps,pm) =

cov(ps,pm)

?var(ps)var(pm)

(1)

We use the NC similarity measure since NC is invariant to

linear intensity transformation and it is assumed that for small

corresponding image patches in two successive slices, the in-

tensities are locally related by some linear intensity trans-

formation. The images produced can sometime be severely

rotated or flipped; hence a brute-force searching scheme is

applied to account for different orientations. Namely, the

patches extracted are rotated by multiple angles and the al-

gorithm is performed with these different starting conditions.

Table 1. Outline of the Algorithm

Given a pair of successive slices a fixed (scene) and mov-

ing(model) image:

step 1: Extraction of features:

-A1. Extraction: Small image patches are extracted at differ-

ent locations for both images.

-A2. Evaluation: The scene features extracted are matched

against the model features, based on the NC similarity

measure between the patches.

Step 2: Transformation computations:

-B1. Select top percent of corresponding patches between

the images, and represent the patch by its center point.

-B2. Apply extended EM-ICP utilizing the NC for the ini-

tialization and as a prior during the iterations.

Then the angle that performs best between each pair of im-

ages is selected to determine the transformation for this pair.

In the second step, we construct two set of points based on

the center coordinates of the patches. Utilizing the patch pairs

with high NC similarity we compute the transformation with

the EM-ICP-NC algorithm outlined below. Table 1 presents

an outline of the registration algorithm proposed.

2.2. EM-ICP-NC

Let sibe the points of the scene set S ∈ R2and mithe points

of the model patch set M ∈ R2, with nsand nmdetermin-

ing the number of points respectively. T represents the rigid

transformation from the scene to the model. The probability

of a point sito correspond to the model points miis modeled

by a Gaussian probability distribution. In the case of homo-

geneous isotropic Gaussian noise

p(si|mj,T) = exp(−||T ∗ si− mj||2/2σ2)

where σ represents the noise in the measurement.

The idea is to maximize the log-likelihood of the data

distribution logp(S,A|M,T) where the unknown correspon-

dences are considered as a hidden random variables A ∈

RnS×nM. The algorithm starts by initialization of the trans-

formation (T), and repeats until convergence of the two EM

steps. In the E step, T is fixed and the probability of the

matches (A) are estimated.

(2)

p(A|S,M,T) =

πijp(si|mj,T)

?

k

πikp(si|mk,T)

(3)

In the classic EM-ICP the prior probability of the matches is

based on the uniform law: πij =

prior to account for the NC similarity of the patches taking

1

nM. Here we extend the

Page 3

advantage of our use of patches rather than points. Thus the

prior is based on the normalized NC defined in 1.

πij=

NC(pi,pj)

?

k

NC(pi,pk)

(4)

Therefore we compute E(Aij) as follows:

(AT)ij=

πijexp(−||T ∗ si− mj||2/2σ2)

?

This same NC prior is used in the initialization to determine

the initial transformation T.

In the M-step, A is fixed and likelihood is optimized w.r.t

to T.

kπikexp(−||T ∗ si− mk||2/2σ2)

(5)

Tn+1= argmax

T

(EA[logP(S,A|M,T)])

(6)

Using the EM-ICP-NC yields:

CAT(T) =

1

nm

ns

?

i

nm

?

j

(AT)ijlogp(si|mj,T)

(7)

2.3. Computational complexity

The algorithm involves four parameters; including the size of

the patch K, the spacing ∆ between two following patches,

the number of patch pairs nppused and σ (in our implemen-

tation we used K = 100 × 100 and ∆ = 30, npp = 200,

σ = 1). Thus the computational complexity is determined

by the number of images T in the data base (T = 200) and

their size Q in pixels. The images were downsampled using

Gaussian smoothing plus bi-linear interpolation to a size of

≈ 1000 × 1000. The number of patches is determined by

their size and spacing and is proportional to TQ. Each patch

is compared to all the patches in the neighboring image, by

convolution, requiring time also proportional to Q. We as-

sume that the EM-ICP can be performed efficiently and its

computation time is small compared with the first step. Con-

sequently the overall complexity is O(TQ2).

3. EXPERIMENTS AND RESULTS

We validated our algorithm compared to manual human regis-

trations on a set of 43 pairs of slices. The registration was per-

formed by manually selecting corresponding points in a pair

of consecutive images and computing the pairwise transfor-

mation, which we denote as (T∗) based on Horn’s method [9].

Given a set of corresponding points in two systems, Horn’s

methodfindstheclosedformsolutiontotheleastsquareprob-

lem relating these points. Table 2 compares the results ob-

tained by three automatic approaches including the EM-ICP-

NC approach, the classic EM-ICP [7] and Horn’s [9] method.

EM-ICP-NC

Mean 50 ± 41

EM-ICP

155 ± 276

Horn

862 ± 817

||Tauto− T∗||:

Table 2. Comparing our results to to the ground truth manual

transformation and to two other automatic approaches.Our

algorithm has higher accuracy and precision than previously

described methods.

The L2difference between the transformations of all three au-

tomatic approaches are compared to the ground truth transfor-

mation computed based on manual selection of points. Figure

1 illustrates the alignment results obtained by our approach.

4. SUMMARY

We present a novel algorithm for alignment of large TEM

images. To our knowledge this is the first attempt to inte-

grate EM-ICP together with a patch matching strategy and to

incorporate a similarity based prior into the EM-ICP model.

Our algorithm obtains accurate results and shows robustness

to noise and perturbations such as large rotations. The algo-

rithm can contribute to various applications both in TEM and

other medical imaging modalities. Future work will increase

the efficiency by utilizing a multiscale model and generalize

the approach to non-rigid registration and image mosaicing.

5. REFERENCES

[1] S. J. Smith, “Circuit reconstruction tools today,” Curr Opin

Neurobiol, vol. 17 (5), pp. 601–608, 2007.

[2] N. Blow, “Following the wires,” Nature Methods, vol. 4 (11),

pp. 975–980, 2007.

[3] P. Koshevoy, T. Tasdizen, R. Whitaker, B. Jones, and R. Marc,

“Assembly of large three-dimensional volumes from serial-

section transmission electron microscopy,” MICCAI, 2006.

[4] J.B.A. Maintz and M.A. Viergever, “A survey of medical image

registration,” Medical image analysis, vol. 2(1), pp. 1–16, 1998.

[5] S. Ourselin, A. Roche, G. Subsol, X. Pennec, and N. Ayache,

“Reconstructinga 3d structure from serial histological sections,”

Image and Vision Computing, vol. 19, pp. 25–31, 2000.

[6] J. Dauguet, D. Bock, C. R. Reid, and S. K. Warfield, “Alignment

of large image series using cubic b-splines tessellation: Applica-

tion to transmission electron microscopy data,” MICCAI, 2006.

[7] S. Granger and X. Pennec, “Multi-scale em-icp: A fast and

robustapproachforsurfaceregistration,” ECCV,vol.4, pp.418–

432, 2002.

[8] J. Kremer, D. Mastronarde, and J. McIntosh, “Computer visual-

ization of three dimensional image data using IMOD,” J. Struct.

Biol., vol. 116, pp. 71–76, 1996.

[9] B. K. P. Horn, “Closed-form solution of absolute orientation us-

ing unit quaternions,” Journal of the Optical Society of America,

vol. 4, pp. 629–642, 1987.

Page 4

(1a) Fixed Slice(1b) Moving Slice - Before(1c) Checkerboard composite - Before

(1d) Fixed Slice(1e) Moving Slice - After(1f) Checkerboard composite - After

Successive pair (#1) of slices before and after alignment.

(2a) Fixed Slice(2b) Moving Slice - Before (2c) Checkerboard composite - Before

(2d) Fixed Slice(2e) Moving Slice - After (2f) Checkerboard composite - After

Successive pair (#2) of slices before and after alignment.

Fig. 1. Illustration of alignment results.