Content uploaded by Thomas Polzin
Author content
All content in this area was uploaded by Thomas Polzin on Nov 11, 2016
Content may be subject to copyright.
Content uploaded by Thomas Polzin
Author content
All content in this area was uploaded by Thomas Polzin on Dec 07, 2015
Content may be subject to copyright.
This is a preprint of
@ARTICLE{LotzEtAl2015,
author = {J. Lotz and J. Olesch and B. Müller and T. Polzin and P. Galuschka and J. M.
Lotz and S. Heldmann and H. Laue and M. González-Vallinas and A. Warth and B.
Lahrmann and N. Grabe and O. Sedlaczek and K. Breuhahn and J. Modersitzki},
title = {{Patch-Based Nonlinear Image Registration for Gigapixel Whole Slide
Images}},
journal={IEEE Transactions on Biomedical Engineering},
year = {2015},
volume = {63},
number = {9},
pages = {1812 - 1819},
keywords={biomedical optical imaging;cancer;image registration;image
segmentation;interpolation;lung;medical image processing;NGF distance
measurement;antibody;cell level;gigapixel whole slide images;histology images;human
lung cancer data;immunohistochemical stain;interpolation;manual
segmentation;nonlinear deformation model;patch-based nonlinear image
registration;smooth nonlinear deformation;Cancer;Deformable models;Image
reconstruction;Image registration;Image resolution;Interpolation;Three-dimensional
displays;Computer-aided diagnosis;digital pathology;histopathology;image
registration},
doi={10.1109/TBME.2015.2503122},
}
The final publication is available at IEEE via
http://dx.doi.org/10.1109/TBME.2015.2503122
1
Patch-Based Nonlinear Image Registration for
Gigapixel Whole Slide Images
J. Lotz, J. Olesch, B. Müller, T. Polzin, P. Galuschka, J. M. Lotz, S. Heldmann, H. Laue, M.
González-Vallinas, A. Warth, B. Lahrmann, N. Grabe, O. Sedlaczek, K. Breuhahn, J. Modersitzki
Abstract—Objective: Image Registration of whole
slide histology images allows the fusion of ne-grained
information - like dierent immunohistochemical stains
- from neighboring tissue slides. Traditionally, pathol-
ogists fuse this information by looking subsequently
at one slide at a time. If the slides are digitized and
accurately aligned at cell-level, automatic analysis can
be used to ease the pathologist’s work. However, the
size of those images exceeds the memory capacity of
regular computers. Methods: We address the challenge
to combine a global motion model that takes the
physical cutting process of the tissue into account with
image data that is not simultaneously globally available.
Typical approaches either reduce the amount of data to
be processed or partition the data into smaller chunks
to be processed separately. Our novel method rst
registers the complete images on a low resolution with
a nonlinear deformation model and later renes this
result on patches by using a second nonlinear registra-
tion on each patch. Finally the deformations computed
on all patches are combined by interpolation to form
one globally smooth nonlinear deformation. The NGF
distance measure is used to handle multi-stain images.
Results: The method is applied to ten whole slide image
pairs of human lung cancer data. The alignment of
85 corresponding structures is measured by comparing
manual segmentations from neighboring slides. Their
oset improves signicantly, by at least 15 %, compared
to the low-resolution nonlinear registration. Conclu-
sion/Signicance: The proposed method signicantly
improves the accuracy of multi-stain registration which
allows to compare dierent anti-bodies at cell-level.
Keywords: computer-aided diagnosis, digital pathol-
ogy, histopathology, image registration
J. Lotz, J. Olesch, P. Galuschka, J. M. Lotz, S. Held-
mann, H. Laue and J. Modersitzki are with Fraunhofer
MEVIS, Lübeck/Bremen, Germany
T. Polzin and J. Modersitzki are with the Institute of
Mathematics and Image Computing, University of Lübeck,
Germany
B. Müller, M. González-Vallinas, A. Warth and K.
Breuhahn are with the Institute of Pathology, University
Hospital Heidelberg, Germany
B. Lahrmann and N. Grabe are with the Tissue Imaging
and Analysis Center, University of Heidelberg, Germany
O. Sedlaczek is with the Department of Radiology, Ger-
man Cancer Research Center and with Diagnostic and
Interventional Radiology, University Medical Center Hei-
delberg, Germany
Corresponding author: johannes.lotz@mevis.fraunhofer.de
I. Introduction: High resolution histological
whole slide imaging
IN cancer diagnostics and histology-related basic re-
search, much insight into molecular and cellular inter-
actions, tissue growth, and tissue organization is gained by
analyzing consecutive and dierently stained histological
sections. For this procedure, a xed tissue is transferred in
a paran block and cut into 2-5 µm thin slices, stained by
e.g. immunohistochemistry, and subsequently examined
by a scientist or physician using conventional or virtual
microscopy.
In order to correlate the staining intensity, staining pat-
terns, and even subcellular localization of dierent proteins
or antigens, co-staining is frequently required. However,
the detection of dierent antigens is usually dicult due to
cross-reactivity of primary and secondary antibodies used
for the staining process [1]. Adjacent serial sections can
be used to separate cross-reacting chemicals, by staining
them separately, resulting in two or more images, one for
each antibody. To recombine the information from the
separate stains, a precise, multi-modal image registration
is essential.
When dealing with histological whole slide images, an
important challenge is the size of these images. At its
maximum resolution, a whole slide image often exceeds the
size of 100,000 ×100,000 pixels. Established registration
methods cannot process data at this resolution without
being adapted for special high performance computing
hardware.
However, high power magnication and especially the
adjustment of staining information derived from dierent
slides are of central importance for basic research and for
medical diagnostics. For example, an integrated picture
containing morphological and partly subcellular features
(e.g., the nuclear shape) together with the expression of
specic tumor cell markers is necessary for reliable diag-
nosis of some solid tumor. Moreover, combining staining
patterns from adjacent sections may allow the mapping
of protein expression to specic cell populations in tissues
consisting of multiple cell populations.
Next to the accurate alignment of corresponding tissue
structures, the regularity and reliability of the deformation
Copyright (c) 2015 IEEE. Personal use of this material is permit-
ted. However, permission to use this material for any other purposes
must be obtained from the IEEE by sending an email to pubs-
permissions@ieee.org.
2
is crucial to the quality of a registration result. In the
process of cutting sections from a block, dierent artifacts
can occur [2]. Some of these deformations, such as tissue
compression, have an inuence on large parts of the tissue
slide. To undo such a deformation, a model that globally
couples all parts of the tissue seems appropriate. Common
choices are diusive [3], elastic [4] or curvature-based
deformation models [5].
The global coupling of the deformation leads to a
dilemma when facing patch-based registration methods as
the image information is not globally available at a high
resolution.
Previous work dealing with the registration of such
images focuses either on a nonlinear registration of low-
resolution images [6, 7, 8] or approaches the problem
with patch-based methods where smaller patches of the
image are registered anely [9, 7, 10]. While being con-
siderably faster, low resolution approaches cannot take
local deformations into account. These deformations are
invisible without using the full-resolution data. Patch-
based methods that rely on a combination of ane or
rigid registrations are limited in the number of degrees
of freedom.
We present a novel method that rst registers the
complete images on a low resolution and later renes this
result on patches by using a second nonlinear registration
on each patch. Finally the deformations computed on all
patches are combined by interpolation to form one globally
smooth nonlinear deformation. This approach combines
global deformation information on a coarse level with a
local correction.
We organize the rest of this paper as follows: Related
work dealing with the registration of histology images
will be discussed in Section II. We then demonstrate the
challenges a registration approach has to deal with when
it comes to histology images in Section III. The core of the
presented method is the nonlinear, variational registration
approach [11]. The relevant parts with respect to the
computational challenges on large images will be discussed
in Section IV. Our extension, a patch-based nonlinear
registration method, will be presented in Section IV-E.
We apply the new method to human lung cancer data, as
described in Section V, and present an evaluation of the
results in Section VI. In the end, we discuss merits and
shortcomings of the presented method.
II. Related work
Digital pathology is an active topic of research. Most
of the work in this eld dealing with image registration
focuses on 3D reconstruction which usually includes mul-
tiple image registrations of consecutive histologic slides.
This paper focuses on the core of these methods, the
registration of two consecutive slides.
Starting back in the 90s, the rst methods were es-
tablished to reconstruct digitized histological data to 3D
volumes mostly for a better anatomical understanding
of specic organs. Many papers formulate the goal to
reconstruct 2D histological images to 3D volumes and fuse
them to corresponding 3D volumes of another modality,
e.g. MRI or PET scans [12, 13], block-face images [14, 15,
16, 17] or both [18, 19]. For this aim every 2D histological
slide is aligned with a corresponding slide of the reference
volume. As the reference volumes are of limited image
resolution, the used resolutions of the histological slide
images are also low.
A dierent kind of method solves the problem without
a reference volume. For low resolution 3D reconstructions
from histological data, an ane or rigid registration of two
or more consecutive slides is satisfying [20, 21, 22, 16, 23,
24]. More complex deformation models allow a more ac-
curate alignment. Examples include piecewise or weighted
ane deformations [25, 10, 26], b-spline deformations [27,
6, 7], a moving least squares approach applied to SIFT
points [28] and elastic registration [29].
Today, advanced imaging technology results on the one
hand in much higher amounts of data and on the other
hand in a shift of reconstruction tasks. One example
is the growing interest in the reconstruction of global
or functional entities such as micro-vasculature or im-
munohistochemical markers [7]. The new challenge is to
reconstruct and fuse the data on a completely dierent
level: smaller structures and the comparison of dierent
functional markers across slides are of increasing interest,
resulting in the need of reconstructions ideally on the scale
of cell nuclei. For this task, ane or rigid solutions are
not sucient. Non-linear deformations that occur in the
cutting process have to be corrected to achieve satisfying
results.
Even though the performance of the technology to
compute histological reconstructions has advanced signif-
icantly, the high amount of histological data cannot be
handled with the classical established methods on common
workstation computers. This even holds true for the data
needed to fulll only a subset of the 3D reconstruction
task, the registration of two successive slides. There are
dierent approaches to address the challenge of the large
image dimensions.
In the following, we distinguish between global methods
that compute a solution based on extracted features or
another subset of the data on the one hand and those
methods that implement a divide and conquer approach on
the other. Global detection of cell nuclei is used by Weiss
et al. [30] to reduce the tissue data to nuclei densities that
can be stored eciently due to the sparsity of the nuclei.
These densities are independent of a particular staining
and are used to compute a global deformation of whole
slide images.
Schwier et al. [8] reduce the image data to segmented
vessel structures to steer their two-step approach. First a
rigid, iterative best-t matching of the segmented vessel
structures is calculated which is rened by an elastic
registration step on a low-resolution image. The resulting
deformation is then applied to the original slice data.
By matching SIFT features, Cardona et al. [28] register
images from transmission electron microscopy (TEM).
Patches captured from TEM are stitched in plane while
3
they are reconstructed in 3D at the same time. Using SIFT
point correspondences as distance measure, they combine
rigid and nonlinear deformation components by a moving
least squares approach [31].
Instead of reducing the amount of data and possibly
losing important detail information, local methods divide
the image into smaller parts and process these parts
independently. One advantage of such an approach is that
well established registration methods can be used. One
big interest in these methods is the way the individually
computed results are combined into one nal deformation.
The idea of transforming selected image regions anely
is followed in [25]. Arsigny et al. compute a global, poly-
ane registration by combining multiple ane transfor-
mations while maintaining smoothness at the tile borders.
However, because of the simultaneous computation of mul-
tiple regions, the method is not meant to work with large
images. Using non-rectangular patches, Pitiot et al. [10]
propose a registration framework, where automatically
segmented partitions of the images are generated based on
tissue structure such as the gyri of the brain. The regions
are transformed independently by an ane registration. A
global transformation is then found by interpolating the
transformation between the registered partitions.
Closest to our new method is the work of Song et al.
[9] on three dimensional tissue reconstruction of histo-
logical sections that are dierently stained. The authors
propose a tile based approach previously published by
Roberts et al. [7] that rst computes a rough globally
rigid transformation which is then rened by calculating
rigid transformations on smaller patches of the image with
higher resolution. Multi-modal registration between dier-
ently stained sections is achieved by an automated content
classication. A global nonlinear deformation is computed
by interpolating between rigidly transformed points on
individual patches using b-spline transformations.
In the sectioning process, physical forces are exerted and
propagated globally through the tissue. Compared to the
above mentioned approaches we use a physically motivated
nonlinear transformation model (such as diusive, elastic
or curvature registration) on the entire domain of the
whole slide image. In previous experiments [32], we used a
zooming strategy to compute a high resolution registration
of a successively decreasing image area. We extend this
strategy by switching completely to nonlinear registrations
and by performing multiple registrations on overlapping
image regions. The resulting deformation vector elds are
nally combined to produce one large, smooth nonlinear
deformation.
III. Computational challenges in digital
pathology
The challenges in the registration of large images be-
come apparent in the following example. To compute the
transformation y∗that aligns two images Rand Twe
consider a registration framework that implements the
variational scheme [11] such as described in the pseudo-
code below:
y0= affine_pre_registration(R, T )
J(R, T , y)= dist anc e (R, T (y))
+ regularizer(y)
lo op unti l s topp ingC rite riaM e t :
co mpu te J(R, T , yi),∇J(R, T, y i)
yi+1 = compute_update(J, ∇J, yi)
i=i+1
end
y∗=yi
The details will be covered in Section IV.
The loop is usually embedded in a multi-level or coarse-
to-ne approach in order to convexify the registration
problem and to prevent the registration from converging
to a local minimum. In order to exploit the complete
image information, gray values at every pixel are accessed
multiple times during the optimization to compute the
image gradient and the distance measure. As loading this
data from the disk is slow, the images, the image gradient
and the transformation are usually kept in the computer’s
main memory.
The sizes of image and image gradient provide a lower
bound to the main memory required by the registration
algorithm. Typical whole slide images in digital pathol-
ogy are scanned with a magnication referred to as 40x
(typically 0.228 µm ×0.228 µm per pixel) and have
dimensions of 100000 ×100000 = 1010 pixels or even more.
If converted to grayscale images and stored in double
precision (64 bit / pixel = 8 byte / pixel), one such image
requires 8 ·1010 byte = 76294 MB = 74.51 GB of main
memory. Even at the slightly lower magnication of 20x
(0.455 µm ×0.455 µm per pixel), which seems sucient
for registration purposes, one image still sums up to 18.63
GB. Considering both images and the derivative of the
template image, the total memory requirement for the
registration is at least 75 GB. Often, the deformation
information can be stored on a lower resolution, therefore
we neglect it in this calculation. Further, we do not include
any other variables such as intermediate deformed images
or other temporary computation results.
With the intended use on a regular desktop workstation
or laptop computer in mind, these requirements excess the
available resources of rarely more than 32 GB of main
memory.
This problem has been addressed by processing the
image in patches as noted in Section II. We extend this
idea and propose a novel method where a physically moti-
vated, nonlinear registration is computed rst globally and
then corrected locally on each patch. After all patches are
registered, the resulting deformation elds are combined
into one smooth deformation.
IV. Methods: Nonlinear Image registration
The main methodological contribution of this paper is
in the patch-based nonlinear registration. This registration
is preceded by an initial alignment that will be discussed
rst. Fig. 1 shows an overview of the dierent components
of the proposed automatic registration scheme.
4
foreground
detection
PCA-based
pre-alignment
manual
pre-alignment
OR
low-resolution
registration
(affine + nonlinear)
per-patch
nonlinear
registration
fusion of
transformation
for each
patch
Fig. 1. Registration scheme for two histology whole slide images. We
focus on patch-based registration and fusion of the deformation.
A. Pre-alignment
To initialize the registration, the slices are pre-aligned
by the following steps:
1. normalize the image intensities
2. identify the foreground of the image by variance
ltering and thresholding
3. align the image by their principal axes [33]
4. rene the alignment by an ane image registration
of the masks computed in 2.) using the SSD distance
measure [11].
A similar strategy has recently been used by [6] and
others. The result of the pre-alignment is used as an initial
guess to start the actual registration.
B. Nonlinear Image Registration
The core of the patch-wise registration is the image
registration method described in [11]. We understand
image registration as the computation of a deformation
y:R2→R2that maps from a reference coordinate
frame, dened on a reference image Ronto the coordinate
frame of the template image T. Using an ecient, matrix-
free implementation [34] of the variational approach, the
functional
J[y] = D[T, R, y] + S[y]−→
ymin
is optimized with respect to a deformation y.Dand S
represent a distance measure and a regularizer.
To cope with the multimodality of dierently stained
image sections, we choose the Normalized Gradient Field
(NGF) [35] distance measure
D[T, R, y] = NGF[T , R, y] =
Ω
1−∇T(y(x))T∇R(x)
||∇T(y(x))||ε||∇R(x)||ε2
dx.
where ||x||2
ε=||x||2
2+ε2to assess image similarity.
By aligning normalized image gradients, NGF not only
allows the registration of dierently stained images but
also copes nicely with dierent staining intensities in same-
stain (monomodal) registration.
The histological cutting process exerts forces to the
tissue block that we want to model physically. As a trade-
o between accurate modeling and computation speed, we
choose a diusive regularizer [3]
S[y] = α
2Ω
⟨∇y(x),∇y(x)⟩dx
which can be interpreted as a special case of the linear
elasticity [4].
C. Eciently discretizing the deformation
We represent the deformation on the reference image’s
domain by a deformed grid y∈Rm×n×2of size m×n
where
y=x+u
is a combination of a regular, cell-centered grid x=
h·[0.5,1.5, ..., m −h/2] ×h·[0.5,1.5, ..., n −h/2] with
spacing hand a displacement urelative to the grid. By
using regularization, we can assume that the deformation
is smooth in the sense that local variations are small.
For this reason, its resolution can be much lower than
the number of pixels in the image without loosing much
information. Intermediate positions are interpolated. We
chose the deformation resolution m×nto be one to two
orders of magnitude lower (in each dimension) than the
number of pixels in the image. The low amount of data
needed to store the deformation signicantly lowers the
memory requirements and runtime of the registration and
makes it possible to handle the global deformation for the
whole slide image. This is reected in the implementation
as shown in the following paragraphs.
The image Tis a representation of the underlying
image data which is obtained by linear interpolation. The
reference image Ris dened on its regular pixel grid
X∈RM×N×2of size M×N(> m ×n).
In this context, the expression ˆx∈Xin (1) is meant
as the sum over all grid points ˆx∈R2of the grid Xand
can be thought of as a for-loop. Furthermore, in a discrete
setting, ¯
∇is meant as a nite dierence operator.
5
The objective function with respect to the discretized
deformation can then be written as
min
uJ[T, R,y] = D[T, R, P y] + S[u],where
u=y−x,
D[T, R, P y] = NGF[T , R, P y]
=h2·
ˆx∈X
1−¯
∇T(Py[ˆx])T¯
∇R(ˆx)
|| ¯
∇T(Py[ˆx])||ε|| ¯
∇R(ˆx)||ε2
(1)
S[u] = α
2
ˆu∈u
⟨¯
∇ˆu, ¯
∇ˆu⟩
and Pis the prolongation operator which interpolates
the low resolution deformation onto the image grid X. The
square brackets in the expression y[x]denote the bilinear
interpolation of ybased on the four neighboring pixels of
xon the grid of y.
D. Optimization
The optimization of the objective function is embedded
in a multi-level, coarse-to-ne approach that avoids local
minima by starting the registration with a smoothed im-
age. The image’s resolution is then subsequently increased
to account for details in the images. This method has been
described multiple times, see e.g. [11] for more details.
As optimizer, an L-BFGS [36] implementation is used
that is initialized with the analytic Hessian of the regular-
izer.
An ane pre-registration is used to compute a rough
alignment of the two images that is used as an initial guess
for the deformation y.
E. Patch based image registration
As described in Section III, it is not practical to use
the above framework to compute a registration of two
whole slide images due to the large amount of data. In
contrast to earlier patch-based linear registration methods,
our method computes the registration in two steps which
are both nonlinear and which allows a free choice of the
deformation model.
By computing a rst nonlinear registration on low-
resolution data, global, large-scale deformations occurring
in the tissue are corrected. The result of this nonlinear
registration is then used as initial guess for a patch-wise
registration scheme.
1) Patchwise elastic registration: After computing the
registration on the low-resolution data, the image is parti-
tioned into patches. The patches are allowed to overlap.
A high-resolution correction of the rst deformation is
computed independently on each patch by means of a
second nonlinear registration. At this high resolution,
smaller structures are visible and drive the registration
process such that local deformations of the tissue slides
are compensated. Each local registration returns a vector
eld with the computed transformation.
The algorithm is described formally in the following
pseudo-code. The global domain Ωof the reference im-
age is dened as a rectangular region Ω = [ω11, ω12]×
[ω21, ω22]⊂R2in the world coordinate system of the ref-
erence object slide. The image deformation in this domain
is represented by a discrete deformation eld ywhich is
dened as an array of dimensions m×nand is coupled to
a world matrix W. Multiplication of Wwith homogeneous
pixel coordinates transforms these coordinates into the
world coordinate system of Ω. Each patch is dened on
a domain Ωj,k ⊂Ωand a deformation yj,k is computed:
co mpu te Ωj,k ,Wj,k fo r al l p atc hes i n Ω
for j = 1:Mpatches :
for k = 1:Npatches :
yj,k = minimize J(R, T , y)|Ωj,k
end
end
This results in the computation of Mpatches ·Npatches
deformations.
2) Fusion of deformations yj,k:In order to obtain a
global smooth deformation yΩ(x)on the original image
domain Ω, bilinear interpolation is used. We rst consider
the one-dimensional problem of fusing the patches in one
column. The following step will be repeated for all columns
k,k= 1, ..., Npatches.
If the coordinate xin the global deformation yΩis
only covered by one patch, say patch (j, k), the value
at this point can be obtained from the deformation yj,k
by interpolation. Note that the square brackets in the
expression y[x]again denote bilinear interpolation.
yΩ(x) = y(j,k)[W−1
j,k WΩx]
if WΩx∈Ωj,k
If a point in yis covered by two patches, we interpolate
linearly and obtain yΩ(x)by
yΩ(x) = d(x)·y(j,k)[W−1
j,k WΩx]
+ (1 −d(x)) ·y(j+1,k)[W−1
j+1,kWΩx]
if WΩx∈Ωj,k ∩Ωj+1,k
where d(x) = (x(1) −ω11
j+1,k)/(ω12
j,k −ω11
j+1,k)is the
relative distance of xto the border of patch j+ 1, k. Note
that ω12
j,k −ω11
j+1,k >0because the patches are overlapping.
The global smoothness of the fused deformation, is
assured by comparing the two-norm of the dierences
of the deformation vectors in the overlapping region
||y(j,k)[W−1
j,k WΩx]−y(j+1,k)[W−1
j+1,kWΩx]||2
2. In the course
of the experiments, the dierence between neighboring
patches was always lower than 5 %.
All patches are aligned in rows and columns such that
we can rst apply the above method to fuse each column
of patches and then use the same method again on the
resulting row.
The core registration component is implemented in a
C++ library with focus on eciency and shared-memory
parallelization [34]. The preprocessing and the patch-
based registration are assembled in the image processing
framework MeVisLab. The algorithm used to fuse the
deformation was implemented in the Julia programming
language and will be made publicly available.
6
V. Application to human lung cancer data
As a proof of principle, we applied the patch-based non-
linear registration method to a clinically relevant question:
human lung cancer. Non-small cell lung cancer (NSCLC)
with its two subtypes adenocarcinoma and squamous cell
carcinoma (SCC) is the most common cause of cancer-
related death worldwide [37]. Morphological features such
as the nuclear morphology as well as the expression of
marker gene panel (e.g. cytokeratins) are informative for
the characterization of tumor cell dedierentiation. We
therefore decided to use a primary SCC isolated from a
human lung cancer patient1. See the appendix for detailed
information about the staining process.
The algorithm was run on 10 independent slide pairs
with stains CD31 - H&E (2 pairs), H&E - Factor VIII,
Factor VIII - KL1, KL1 - CD31, CD146 - KL1 (4 pairs),
CD146 - AFOG (13 dierent slides in total, see Table I).
The patch size can be chosen depending on the memory
capacity of the computer at hand. We found a patch size
of 4096 ×4096 pixels to work well on a laptop computer
equipped with 16 GB of RAM and an Intel i7 processor.
Patches were overlapping by 20 % on each border. The
number of patches per image depend on the image’s size
at the desired magnication level. To trade-of visible detail
and computation time, we choose a magnication of 20x
(0.455 µm ×0.455 µm per pixel) for all images. This
results in a number of patches between 3 ×7 = 21 and 13
×17 = 221 patches per image. See Table I for an overview
of the data used for the evaluation.
The NGF distance measure was parametrized with ε=
10000, the regularization parameter was set to α= 0.1.
VI. Validation
The evaluation of the accuracy of a registration in
general is a dicult task and it is even more dicult
if no ground truth or gold standard is available. In ap-
plication to histology data, such as for example in 3D
reconstruction, an exact match of corresponding structures
is usually not even desired, as it would annihilate the
structural dierences present in two neighboring slides and
thus destroy the three-dimensional structure. In the case
of virtual double staining, three-dimensional structure is
not of primary interest, still, an objective ground truth
is not available. In [24], this problem is addressed by
comparing automatically detected nuclei-landmarks. How-
ever, detection of such nuclei correspondences is dicult
in multi-stain data. Accepting a possible bias in favor of an
intensity-based registration, we chose to manually segment
larger structures that are identiable in both slides.
The accuracy of the registration is evaluated by comput-
ing the dierences of manual segmentations of correspond-
ing structures after registration. A similar evaluation has
1Tissue samples were provided by the tissue bank of the National
Center of Tumor Diseases (NCT, Heidelberg, Germany) in accor-
dance with the regulations of the tissue bank and the approval of the
Ethics Committee of the Medical Faculty of Heidelberg University.
The experiments were ethically approved by the “Ethikkommission
der Medizinischen Fakultät der Universität Heidelberg” with the
approval number S-249/2010 and 207/2005.
TABLE I
Overview of the image data used for evaluation. The
algorithm has been evaluated on ten slide pairs and six
different stains. All slides were registered with the spatial
resolution of 0.455 µm ×0.455 µm per pixel. A plus (+)
denotes that the image resolution is given after
downsampling to 0.455 µm by a factor of 2 in each dimension.
dataset ID stainings image dimensions # patches
L0-1 CD31 - H&E R: 55680 ×46592 + 63
T: 59520 ×45568
L0-2 H&E - F. VIII R: 59520 ×45568 198
T: 55552 ×46720 +
L0-3 F. VIII - KL1 R: 55552 ×46720 + 221
T: 57536 ×44672 +
L0-4 KL1 - CD31 R: 57536 ×44672 + 206
T: 55552 ×51982 +
L0-5 CD31 - H&E R: 55552 ×51982 + 209
T: 56580 ×46592
L1-1 CD146 - KL1 R: 35712 ×29344 + 28
T: 31744 ×24960 +
L1-2 KL1 - CD146 R: 31744 ×24960 + 21
T: 35712 ×31744 +
L1-3 CD146 - KL1 R: 35712 ×31744 + 35
T: 23558 ×17280 +
L1-4 KL1 - CD146 R: 23558 ×17280 + 28
T: 23808 ×17280 +
K1-1 CD146 - AFOG R: 43648 ×43136 + 49
T: 41664 ×40192 +
inter-observer error registration error
Fig. 2. LEFT: Two independent annotations of the same structure
with representative inaccuracies, dmax = 11.4µm, davg = 1.8µm.
RIGHT: Registration result, template image with transformed con-
tour from reference image, dmax = 10.1µm, davg = 2.5µm. The
black bar has a length of 200 µm.
been used in [9]. In each slide pair, 5-12 structures were
segmented manually without knowledge of the registration
result. Each segmentation is represented by approximately
200-400 points.
For each two segmentations, represented by point sets
A={a1, ..., aN}and B={b1, ..., bM}, the maximum dmax
and mean oset davg between the corresponding structures
was computed with
dmax =max max
a∈Amin
b∈B∥a−b∥2,max
b∈Bmin
a∈A∥b−a∥2
and
davg =max avg
a∈A
min
b∈B∥a−b∥2,avg
b∈B
min
a∈A∥b−a∥2.
The maximum distance is known as the discrete Haus-
dor distance [38] and has been previously used to evaluate
7
histology registrations [9]. The mean distance between the
annotations is less sensitive towards outliers in the manual
annotations and is therefore included in the validation.
To estimate the inter-observer error while drawing the
segmentations, 12 annotations of one slide have been
drawn twice. The measured values for maximum and
mean distance between the segmentations is avg(dmax) =
15.1±11.9µm and avg(davg)=2.3±0.7µm which serves
as an approximation of the lowest measurable registration
error. Fig. 2 shows one of these segmentations (left) and
also a pair of segmentations after registration (right).
Distances were computed after PCA-pre-alignment, af-
ter low-resolution nonlinear registration and after the
patch-based registration. The results for each slide pair
are reported in Table II. The results show that the non-
linear pre-registration is an ecient method to obtain a
relatively accurate result if processing time is the priority.
However, after the additional correction using the patch-
based method, the distances between the structures are
lower in all cases. The overall reduction is 15 % for the
Hausdor distance and 36 % for the mean segmentation
distance. The nal registration error is in the order of
magnitude of the accuracy of the manual segmentations.
A one-sided, paired t-test was computed with the null-
hypothesis that the new method is not better than the
coarse-level registration. The hypothesis was rejected,
both improvements are statistically signicant (p<0.025).
As two metrics were used to compare the registration
accuracy, the signicance level was adjusted using the
Bonferroni correction.
The additional quality becomes also apparent if the
results are evaluated visually. One example is shown in
Fig. 3, where the alignment of the structures using the
patch-based method is almost at cell-level accuracy while
a signicantly larger registration error is visible in the
low-resolution registration. See the attached movie for an
illustration of the virtual double staining process based on
the registration result.
Comparing the registration accuracy to other methods
such as [9, 7, 10] is dicult due to the lack of a common
benchmark and freely available data. This remains true,
even in the cases where a comparable error measure has
been used. While the registration error shown above seems
to be lower than in the results reported by [9], tissue
properties such as slice thickness and tissue deformations
have a big inuence on the quality of the registration
and a fair comparison is not possible. To facilitate future
comparisons, the data and the segmentations used for the
evaluation in this paper have been made available2.
The present implementation is meant as a proof of
concept and has not been optimized for performance. Nat-
urally, the large amount of data in the whole slide images
increases the computation time. The average runtime of
the algorithm is 49 s for the nonlinear pre-registration (ex-
clusive of pre-alignment) and between 22 min (24 patches)
and 288 min (206 patches) for the patch-based correction.
2http://s.fhg.de/histo-registration-data
Fig. 3. Comparison of low-resolution (LEFT) and patch-based
(RIGHT) registration results on tissue stained with H&E and CD31.
Smoother structure correspondence is seen in the results generated
with the new method.
TABLE II
Maximum segmentation offset (discrete Hausdorff distance,
top) and Mean segmentation offset (bottom) after
PCA-based pre-alignment, after low-resolution nonlinear
registration and after patch-based registration on 10
evaluated slide pairs.
Maximum segmentation oset dmax
(discrete Hausdor distance)
dataset ID pre low patch
(# segmentations) aligned resolution based
L0-1 (5) 126.4 µm 21.1 µm 15.9 µm
L0-2 (6) 178.5 µm 20.8 µm 12.6 µm
L0-3 (6) 126.4 µm 18.6 µm 11.0 µm
L0-4 (6) 186.1 µm 21.0 µm 13.0 µm
L0-5 (6) 543.8 µm 22.8 µm 12.7 µm
L1-1 (12) 44.7 µm 18.7 µm 16.4 µm
L1-2 (11) 42.2 µm 18.6 µm 18.0 µm
L1-3 (11) 64.3 µm 19.9 µm 19.1 µm
L1-4 (10) 94.4 µm 35.2 µm 33.6 µm
K1-1 (12) 39.0 µm 26.8 µm 26.4 µm
average (85) 122 µm 22.7 µm 19.3 µm
Mean segmentation oset davg
dataset ID pre low patch
(# segmentations) aligned resolution based
L0-1 (5) 47.1 µm 8.7 µm 3.4 µm
L0-2 (6) 118.4 µm 9.1 µm 4.9 µm
L0-3 (6) 47.1 µm 6.3 µm 3.6 µm
L0-4 (6) 129.4 µm 7.8 µm 3.5 µm
L0-5 (6) 455.8 µm 10.7 µm 3.4 µm
L1-1 (12) 15.4 µm 4.9 µm 3.6 µm
L1-2 (11) 19.1 µm 4.1 µm 3.5 µm
L1-3 (11) 26.0 µm 5.2 µm 3.6 µm
L1-4 (10) 49.3 µm 4.8 µm 3.5 µm
K1-1 (12) 14.6 µm 5.8 µm 5.8 µm
average (85) 79.0 µm 6.1 µm 3.9 µm
The fusion of the deformation of 206 patches is computed
in less than 30 s.
VII. Conclusion
In registration of histology data, image size is an im-
portant issue as one slide can have an amount of data
REFERENCES 8
surpassing the equivalent of 30 CT images. When deal-
ing with nonlinear registration, a global deformation is
modeled and the deformation in one point of the image
domain has a global inuence on the image. This leads
to the dilemma where a global deformation needs to be
computed but the data cannot be handled globally at the
necessary resolution.
We propose a two-stage solution to this problem. First
a low-resolution global nonlinear registration is computed
that accounts for the low-frequency, global deformation
of the tissue. This registration is later corrected for the
high-frequency, local parts of the deformation which are
invisible at low resolution representations of the images.
We chose a simple approach computing the registration
independently for each patch. This approach already re-
sults in a signicant improvement compared to the low-
resolution registration. One downside is the signicantly
longer runtime which is unavoidable due to the larger
amount of data that is processed. However, the method
has the potential to be easily parallelization on multiple
machines as no communication between the processes that
align the patches is necessary. An interesting extension of
the method is in the use of the information from those
patches that are already computed when computing the
high-resolution patches. This however, poses new ques-
tions on the order in which patches should be computed
and is postponed to future work.
Attached multimedia file
As a proof of concept, the attached movie3demonstrates
the virtual double staining using the proposed patch-based
registration method. The initially shown AFOG stain is
registered to a slide stained with CD146. Stained epithelial
structures are highlighted in orange and transfered to the
AFOG stain where the two stains can now be analyzed
simultaneously.
Acknowledgment
Part of this work was supported by the MedSys-Network
LungSys which is funded by the German Federal Min-
istry of Education and Research, grant number 0316042J,
0316042B. Tissue samples were provided by the tissue
bank of the National Center of Tumor Diseases (NCT,
Heidelberg, Germany) and the biobank platform of the
German Center for Lung Research (DZL) in accordance
with the regulations of the tissue bank and the approval
of the Ethics Committee of the Heidelberg University.
M. González-Vallinas was supported by the Alfonso
Martin Escudero Foundation.
Appendix
Histology Staining Protocol
In brief, after xation of a tissue slice (about 7cm ×
5cm ×0.5cm) in 4 % buered formalin over night, the
tissue was cut in smaller pieces (about 1cm ×1cm),
3The movie is also available here: http://s.fhg.de/rg-dbl-stn
transferred in paran, and systematically cut in 1-2 µm
thick sections using a conventional microtome. After-
wards, ve consecutive sections were stained using hema-
toxylin/eosin (H&E) and acid fuchsin orange G (AFOG)
standard protocols. In addition, the following antibodies
were used for epitope-specic stains: anti-CD31/PECAM1
(clone MEC13.3, BD Biosciences, Heidelberg, Germany),
anti-CD146/MCAM (polyclonal, Atlas Antibodies, Stock-
holm, Sweden), Factor VIII light chain antibody (clone
H-100, Santa Cruz Biotechnology, Heidelberg, Germany,
and an anti-pan cytokeratin antibody (clone KL1, Ab-
cam, Cambridge, UK). Staining was performed using the
Dako Autostainer (Hamburg, Germany) and the following
protocol: tissue slides were air-dried in an incubator at
42° C over night and deparanized in xylene (2×10
min). After rehydration in graded ethanol, the slides were
pretreated in 0.01 M sodium citrate (pH 6.0) in a pressure
cooker for 10 min. Afterwards, primary antibodies in
PBS/Tween were added for 30 min at room temperature
and slides were washed with PBS/Tween for 5 min before
the secondary antibody was applied for 20 min (1:1.000 in
PBS/Tween). The samples were then incubated with 1 %
hydrogen peroxide diluted in PBS/Tween (5 min). After
signal detection using amino-ethyl-carbazol (AEC, 2×7
min) nuclei were stained using haematoxylin.
References
[1] I. B. Buchwalow and W. Böcker, Immunohistochem-
istry: Basics and Methods. Springer Science & Busi-
ness Media, 2010.
[2] V. Rastogi, “Artefacts: a diagnostic dilemma – a re-
view,” Journal of Clinical and Diagnostic Research,
vol. 7, no. 10, pp. 2408–2413, 2013.
[3] J. Modersitzki and B. Fischer, “Fast diusion reg-
istration,” in Contemporary Mathematics, vol. 313,
AMS, 2000, pp. 117–127.
[4] C. Broit, “Optimal registration of deformed images,”
PhD thesis, University of Pennsylvania, Philadel-
phia, PA, USA, 1981.
[5] B. Fischer and J. Modersitzki, “Curvature based im-
age registration,” Journal of Mathematical Imaging
and Vision, pp. 81–85, 2003.
[6] M. Feuerstein et al., “Reconstruction of 3-D his-
tology images by simultaneous deformable registra-
tion,” in Proceedings of MICCAI, G. Fichtinger et
al., Eds., vol. 14, Springer, 2011, pp. 582–589.
[7] N. Roberts et al., “Toward routine use of 3D
histopathology as a research tool,” The American
Journal of Pathology, vol. 180, no. 5, pp. 1835–1842,
2012.
[8] M. Schwier et al., “Registration of histological whole
slide images guided by vessel structures,” Journal of
Pathology Informatics, vol. 4, no. 10, 2013.
[9] Y. Song et al., “Unsupervised content classication
based nonrigid registration of dierently stained
histology images,” IEEE Transactions on Biomedical
Engineering, vol. 61, no. 1, pp. 96–108, 2014.
9
[10] A. Pitiot et al., “Piecewise ane registration of bi-
ological images for volume reconstruction.,” Medical
Image Analysis, vol. 10, no. 3, pp. 465–483, 2006.
[11] J. Modersitzki, FAIR: Flexible Algorithms for Image
Registration. SIAM, 2009.
[12] M. S. Mega et al., “Mapping histology to
metabolism: coregistration of stained whole-brain
sections to premortem PET in Alzheimer’s disease,”
NeuroImage, vol. 5, no. 2, pp. 147–153, 1997.
[13] M. M. Chakravarty et al., “The creation of a brain
atlas for image guided neurosurgery using serial his-
tological data,” NeuroImage, vol. 30, no. 2, pp. 359–
376, 2006.
[14] J. Dauguet et al., “Three-dimensional reconstruction
of stained histological slices and 3D non-linear reg-
istration with in-vivo MRI for whole baboon brain,”
Journal of Neuroscience Methods, vol. 164, no. 1,
pp. 191–204, 2007.
[15] S. Gefen et al., “Elastic 3-D alignment of rat brain
histological images,” IEEE Transactions on Medical
Imaging, vol. 22, no. 11, pp. 1480–1489, 2003.
[16] E. Bardinet et al., “Co-registration of histological,
optical and MR data of the human brain,” in Pro-
ceedings of MICCAI, ser. LNCS, vol. 2488, Tokyo:
Springer, 2002, pp. 548–555.
[17] B. Kim et al., “Mutual information for automated
unwarping of rat brain autoradiographs,” NeuroIm-
age, vol. 5, no. 1, pp. 31–40, 1997.
[18] T. Schormann and K. Zilles, “Three-dimensional
linear and nonlinear transformations: an integration
of light microscopical and MRI data,” Human brain
mapping, vol. 6, no. 5-6, pp. 339–347, 1998.
[19] E. Bardinet et al., “Three dimensional functional
cartography of the human basal ganglia by regis-
tration of optical and histological serial sections,”
in Proceedings of IEEE ISBI, Washington, 2002,
pp. 329–332.
[20] L. Hibbard and R. Hawkins, “Objective image align-
ment for three-dimensional reconstruction of digital
autoradiograms,” Journal of Neuroscience Methods,
vol. 26, no. 1, pp. 55–74, 1988.
[21] A. Andreasen et al., “Computer-assisted alignment
of standard serial sections without use of articial
landmarks. a practical approach to the utilization
of incomplete information in 3-D reconstruction of
the hippocampal region,” Journal of Neuroscience
Methods, vol. 45, no. 3, pp. 199–207, 1992.
[22] S. Ourselin et al., “Reconstructing a 3D structure
from serial histological sections,” Image and Vision
Computing, vol. 19, no. 1-2, pp. 25–31, 2001.
[23] G. Malandain et al., “Fusion of autoradiographs with
an mr volume using 2-D and 3-D linear transfor-
mations.,” NeuroImage, vol. 23, no. 1, pp. 111–127,
2004.
[24] Y. Xu et al., “A method for 3D histopathology
reconstruction supporting mouse microvasculature
analysis.,” PloS one, vol. 10, no. 5, 2015.
[25] V. Arsigny et al., “Polyrigid and polyane transfor-
mations: a novel geometrical tool to deal with non-
rigid deformations - application to the registration
of histological slices.,” Medical Image Analysis, vol.
9, no. 6, pp. 507–523, 2005.
[26] K. Huang et al., “Fast automatic registration algo-
rithm for large microscopy images,” in Proceedings
of Life Science Systems and Applications Workshop,
IEEE/NLM, 2006, pp. 1–2.
[27] T. Yunhao et al., “Feature curve-guided volume
reconstruction from 2D images,” in Proceedings of
IEEE ISBI, 2007, pp. 716–719.
[28] A. Cardona et al., “An integrated micro- and
macroarchitectural analysis of the drosophila brain
by computer-assisted serial section electron mi-
croscopy.,” PLoS biology, vol. 8, no. 10, 2010.
[29] O. Schmitt et al., “Image registration of sectioned
brains,” International Journal of Computer Vision,
vol. 73, no. 1, pp. 5–39, 2006.
[30] N. Weiss et al., “Multimodal image registration
in digital pathology using cell nuclei densities,” in
Proceedings of BVM 2015, H. Handels et al., Eds.,
Berlin, Heidelberg: Springer Vieweg, 2015, pp. 245–
250.
[31] S. Schaefer et al., “Image deformation using moving
least squares,” ACM Transactions on Graphics, vol.
25, no. 3, pp. 533–540, 2006.
[32] J. Lotz et al., “Zooming in: High Resolution 3D
Reconstruction of Dierently Stained Histological
Whole Slide Images,” in Proceedings of SPIE Medi-
cal Imaging: Digital Pathology, 2014, pp. 904104-1–
904104-7.
[33] N. M. Alpert et al., “The principal axes transfor-
mation –a method for image registration,” Journal
of Nuclear Medicine, vol. 31, no. 10, pp. 1717–1722,
1990.
[34] L. König and J. Rühaak, “A fast and accurate
parallel algorithm for non-linear image registration
using normalized gradient elds,” in Proceedings of
IEEE ISBI, Beijing, China, 2014.
[35] E. Haber and J. Modersitzki, “Intensity gradient
based registration and fusion of multi-modal im-
ages.,” Methods of Information in Medicine, vol. 46,
no. 3, pp. 292–299, 2006.
[36] D. C. Liu and J. Nocedal, “On the limited memory
BFGS method for large scale optimization,” Mathe-
matical Programming, vol. 45, no. 1-3, pp. 503–528,
1989.
[37] J. R. Molina et al., “Non-small cell lung cancer:
epidemiology, risk factors, treatment, and survivor-
ship,” Mayo Clinic Proceedings, vol. 83, no. 5,
pp. 584–594, 2008.
[38] D. Huttenlocher et al., “Comparing images using the
Hausdor distance,” IEEE Transactions on Pattern
Analysis and Machine Intelligence, vol. 15, no. 9,
pp. 850–863, 1993.
