Threedimensional reconstruction and quantification of cervical carcinoma invasion fronts from histological serial sections.
ABSTRACT The analysis of the threedimensional (3D) structure of tumoral invasion fronts of carcinoma of the uterine cervix is the prerequisite for understanding their architecturalfunctional relationship. The variation range of the invasion patterns known so far reaches from a smooth tumorhost boundary surface to more diffusely spreading patterns, which all are supposed to have a different prognostic relevance. As a very decisive limitation of previous studies, all morphological assessments just could be done verbally referring to single histological sections. Therefore, the intention of this paper is to get an objective quantification of tumor invasion based on 3D reconstructed tumoral tissue data. The image processing chain introduced here is capable to reconstruct selected parts of tumor invasion fronts from histological serial sections of remarkable extent (90500 slices). While potentially gaining good accuracy and reasonably high resolution, microtome cutting of large serial sections especially may induce severe artifacts like distortions, folds, fissures or gaps. Starting from stacks of digitized transmitted light color images, an overall of three registration steps are the main parts of the presented algorithm. By this, we achieved the most detailed 3D reconstruction of the invasion of solid tumors so far. Once reconstructed, the invasion front of the segmented tumor is quantified using discrete compactness.

Article: A generic classificationbased method for segmentation of nuclei in 3D images of early embryos.
[Show abstract] [Hide abstract]
ABSTRACT: Studying how individual cells spatially and temporally organize within the embryo is a fundamental issue in modern developmental biology to better understand the first stages of embryogenesis. In order to perform highthroughput analyses in threedimensional microscopic images, it is essential to be able to automatically segment, classify and track cell nuclei. Many 3D/4D segmentation and tracking algorithms have been reported in the literature. Most of them are specific to particular models or acquisition systems and often require the fine tuning of parameters. We present a new automatic algorithm to segment and simultaneously classify cell nuclei in 3D/4D images. Segmentation relies on training samples that are interactively provided by the user and on an iterative thresholding process. This algorithm can correctly segment nuclei even when they are touching, and remains effective under temporal and spatial intensity variations. The segmentation is coupled to a classification of nuclei according to cell cycle phases, allowing biologists to quantify the effect of genetic perturbations and drug treatments. Robust 3D geometrical shape descriptors are used as training features for classification. Segmentation and classification results of three complete datasets are presented. In our working dataset of the Caenorhabditis elegans embryo, only 21 nuclei out of 3,585 were not detected, the overall Fscore for segmentation reached 0.99, and more than 95% of the nuclei were classified in the correct cell cycle phase. No merging of nuclei was found. We developed a novel generic algorithm for segmentation and classification in 3D images. The method, referred to as Adaptive Generic Iterative Thresholding Algorithm (AGITA), is freely available as an ImageJ plugin.BMC Bioinformatics 01/2014; 15(1):9. · 3.02 Impact Factor  SourceAvailable from: Giuseppe Lippolis[Show abstract] [Hide abstract]
ABSTRACT: Prostate cancer is one of the leading causes of cancer related deaths. For diagnosis, predicting the outcome of the disease, and for assessing potential new biomarkers, pathologists and researchers routinely analyze histological samples. Morphological and molecular information may be integrated by aligning microscopic histological images in a multiplex fashion. This process is usually timeconsuming and results in intra and interuser variability. The aim of this study is to investigate the feasibility of using modern image analysis methods for automated alignment of microscopic images from differently stained adjacent paraffin sections from prostatic tissue specimens. Tissue samples, obtained from biopsy or radical prostatectomy, were sectioned and stained with either hematoxylin & eosin (H&E), immunohistochemistry for p63 and AMACR or Time Resolved Fluorescence (TRF) for androgen receptor (AR).Image pairs were aligned allowing for translation, rotation and scaling. The registration was performed automatically by first detecting landmarks in both images, using the scale invariant image transform (SIFT), followed by the wellknown RANSAC protocol for finding point correspondences and finally aligned by Procrustes fit. The Registration results were evaluated using both visual and quantitative criteria as defined in the text. Three experiments were carried out. First, images of consecutive tissue sections stained with H&E and p63/AMACR were successfully aligned in 85 of 88 cases (96.6%). The failures occurred in 3 out of 13 cores with highly aggressive cancer (Gleason score >= 8). Second, TRF and H&E image pairs were aligned correctly in 103 out of 106 cases (97%).The third experiment considered the alignment of image pairs with the same staining (H&E) coming from a stack of 4 sections. The success rate for alignment dropped from 93.8% in adjacent sections to 22% for sections furthest away. The proposed method is both reliable and fast and therefore well suited for automatic segmentation and analysis of specific areas of interest, combining morphological information with protein expression data from three consecutive tissue sections. Finally, the performance of the algorithm seems to be largely unaffected by the Gleason grade of the prostate tissue samples examined, at least up to Gleason score 7.BMC Cancer 09/2013; 13(1):408. · 3.33 Impact Factor  Satoshi Maruyama, Yoshihito Shimazu, Tomoo Kudo, Kaori Sato, Manabu Yamazaki, Tatsuya Abé, Hamzah Babkair, Jun Cheng, Takaaki Aoba, Takashi Saku[Show abstract] [Hide abstract]
ABSTRACT: Background We have demonstrated the induction of perlecanrich stroma of oral squamous cell carcinoma (SCC) on and after its start of invasion. However, it remains unknown how such a neoplastic stroma is actually arranged in tumor tissues.Methods To this end, tissue microarray samples, in which keratin and perlecan were contrastively labeled by immunohistochemistry, were threedimensionally analyzed using digital images and image analysis software to demonstrate the relationship between SCC foci and the perlecanpositive stromal space or that between carcinoma in situ (CIS) and invasive SCC foci.ResultsThe threedimensional (3D) reconstruction demonstrated three kinds of perlecan profiles for inside (I) and outside (O) areas of the carcinoma cell focus: mode 1, I+/O−; mode 2, I+/O+; and mode 3, I−/O+. Mode 1 was seen in CIS as well as SCC tumor massifs in the surface part. Mode 2 was seen in small SCC foci, which seemed isolated in 2D sections but were mostly continuous with the tumor massif in 3D reconstructions. Mode 3 was limited to small SCC foci, which were truly segregated from the tumor massif.Conclusions The results indicated that the 2D SCC focus isolation could not be regarded as invasion but that the SCC foci surrounded by perlecanpositive stroma (modes 2 and 3) could be regarded as a more objective measure for invasion of SCC. This is the first 3D tissuelevel demonstration of the neoplastic stroma space induced with oral SCC invasion, the presence of which we have predicted based on our previous 2D and tissue culture evidence.Journal of Oral Pathology and Medicine 04/2014; · 2.06 Impact Factor
Page 1
1286 IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 24, NO. 10, OCTOBER 2005
ThreeDimensionalReconstructionandQuantification
of Cervical Carcinoma Invasion Fronts From
Histological Serial Sections
UlfDietrich Braumann, Member, IEEE, JensPeer Kuska, Member, IEEE, Jens Einenkel, LarsChristian Horn,
Markus Löffler, and Michael Höckel
Abstract—The analysis of the threedimensional (3D) structure
of tumoral invasion fronts of carcinoma of the uterine cervix is
the prerequisite for understanding their architecturalfunctional
relationship. The variation range of the invasion patterns known
so far reaches from a smooth tumorhost boundary surface to
more diffusely spreading patterns, which all are supposed to have
a different prognostic relevance. As a very decisive limitation
of previous studies, all morphological assessments just could be
done verbally referring to single histological sections. Therefore,
the intention of this paper is to get an objective quantification
of tumor invasion based on 3D reconstructed tumoral tissue
data. The image processing chain introduced here is capable to
reconstruct selected parts of tumor invasion fronts from histo
logical serial sections of remarkable extent (90–500 slices). While
potentially gaining good accuracy and reasonably high resolution,
microtome cutting of large serial sections especially may induce
severe artifacts like distortions, folds, fissures or gaps. Starting
from stacks of digitized transmitted light color images, an overall
of three registration steps are the main parts of the presented
algorithm. By this, we achieved the most detailed 3D reconstruc
tion of the invasion of solid tumors so far. Once reconstructed, the
invasion front of the segmented tumor is quantified using discrete
compactness.
Index Terms—Biological tissues, image color analysis, image
processing, image registration, image segmentation, image shape
analysis, scientific visualization, tumors.
I. INTRODUCTION
T
views of tissue organization, e.g., tumor morphology and tumor
O our understanding, to really consider volumes but not
just single slices is essential in order to get new insight
Manuscript received December 28, 2004; revised July 12, 2005. This work
was supported by the German Research Foundation (DFG) under Grant BIZ6
1/13.TheAssociateEditorresponsibleforcoordinatingthereviewofthispaper
and recommending its publication was N. Ayache. Asterisk indicates corre
sponding author.
*U.D. Braumann is with the Interdisciplinary Center for Bioinformatics,
University Leipzig, Härtelstraße 1618, 04107 Leipzig, Germany (email: brau
mann@izbi.unileipzig.de).
J.P.Kuska iswiththeInterdisciplinaryCenterforBioinformatics,University
Leipzig, 04107 Leipzig, Germany (email: kuska@informatik.unileipzig.de).
J. Einenkel and M. Höckel are with the Department of Gynecology
and Obstetrics, University Leipzig, 04103 Leipzig, Germany (email:
jens.einenkel@medizin.unileipzig.de,
leipzig.de).
L.C. Horn is with the Institute of Pathology, University Leipzig, 04103
Leipzig, Germany (email: larschristian.horn@medizin.unileipzig.de).
M. Löffler is with the Institute for Medical Informatics, Statistics and Epi
demiology, the Coordination Center for Clinical Trials, and the Interdiscipli
nary Center for Bioinformatics, University Leipzig, 04107 Leipzig, Germany
(email: markus.loeffler@imise.unileipzig.de).
Digital Object Identifier 10.1109/TMI.2005.855437
michael.hoeckel@medizin.uni
growth. The threedimensional (3D) characterization of inva
sion patterns of squamos cell carcinoma of the uterine cervix
using histological serial sections is a current clinical question.
This gives demand for both high level image processing and
analysis. Properties of the hitherto observed twodimensional
(2D) tumor invasion fronts are supposed to have relevance for
the further prognosis of the respective patient [1]–[4]. Doing
those quantitative analyses in 3D, a new quality for the struc
tural and morphological assessment of the considered tumors
can be expected.
Threedimensional imaging modalities like computed to
mography (CT), cone beam computed tomography (CBCT),
nuclear magnetic resonance imaging (MRI), positron emis
sion tomography (PET), single photon emission computed
tomography (SPECT), etc. have become state of the art in
many fields of medical diagnostics and research. Besides
those macroscopic in vivo 3D techniques, for more detailed
analyses nondestructive 3D microscopy is available for in vitro
(partially applicable in vivo), as e.g., scanning transmission
ion microscopy (STIM) or particle induced Xray emission
(PIXE), scanning force microscopy (SFM), 3D electron mi
croscopy (3DEM), miniaturized computed tomography (µ
miniaturized nuclear magnetic resonance imaging (µ
confocal LASER scanning microscopy (CLSM), etc. For a
concise review on current highresolution imaging techniques
of (living) tissue, see [5].
CLSM could be successfully applied on precancerous cer
vical epithelial lesions [6] both ex vivo on biopsies in 3D, as
well as in vivo in 2D using a CLSMvariant referred to as “con
focal microendoscope.” The limited range of CLSM of about
100,
, 200 µm is acceptable for these epithelial lesions. The
whole epithelium’s thickness is 200,
ysis of cervical tumors, the CLSM’s penetration range unfortu
nately is too short.
Even though other in vivo techniques would be desirable for
detailed uterine cervix analyses, spatial resolutions of
as achieved e.g., in MRIs from the pelvic region unfortunately
do not appear sufficient for conclusions on tumor invasion and
infiltration which necessitates resolutions
far beyond typical cell diameters of
clinical diagnostics of cervical cancer (e.g., tumor staging),
MRI and partially CT are indispensable [7], [8] and partially
can be used to “predict” histopathologic features. The “ground
truth” for tumor typing, however, only can be obtained by
histopathology both using visual inspection and especially
),
) or
, 300 µm. For the anal
mm
mm, but not
10 µm. However, for
02780062/$20.00 © 2005 IEEE
Page 2
BRAUMANN et al.: 3DRECONSTRUCTION AND QUANTIFICATION OF CERVICAL CARCINOMA INVASION FRONTS1287
transmitted light microscopy once the tumor was surgically
resected.
Among the microscopic techniques, apart from those with
typically very limited fields of view (FOVs) and/or far subcel
lular spatial resolutions there is no well established 3D proce
dure or protocol for tumor imaging providing appropriate con
trasts/spatialranges/resolutions.Therefore,wedecidedtodothe
data acquisition using conventional transmittedlight imaging
based on HEstained histological serial sections. The reasons
for taking this modality are twofold: at this stage of our work
we wanted to stay as close as possible to the procedure usually
applied by the pathologist when assessing single slices in rou
tine. Moreover, both the achievable spatial resolution (
voxel edge length) and the effective FOV (typ. 0.1 cm ) can be
considered acceptable.
However, this necessarily requires solving the 3D tissue
reconstruction problem based on huge registered serial sections
without having a reference data set available to coregister. On
principle, thiskind ofproblem has a long history and infact was
alreadytackleddecadesago,e.g.,in[9]and[10]proposingsolu
tions using manually digitised line drawings. More recent work
essentially could benefit from much improved computational
power (especially concerning CPU and RAM) but also from
(analogue, later digital) imaging improvements. Consequently,
ambitiousalgorithmicsolutionshavebeendeveloped[11]–[13],
partially treated as a coregistration problem [14]–[17]. Even
though a completely different level of quality was reached
meanwhile, due to its nature such 3D reconstruction from
serial sections remain complex and timeconsuming. What is
the central problem we are facing is the absence of some hard
quality criterion to refer to. This means, we cannot utilize some
reference data set since unfortunately there is none. This kind
of dilemma has motivated us to strictly follow a coarsetofine
strategy, i.e., we do the reconstruction in a stepwise manner
and apply registration schemes with an increasing order of
complexity, first a rigid one, then a polynomial nonlinear one,
and finally a curvaturebased nonlinear one.
The ninecriteria based classification of the required registra
tion method(s) according to [18] would comprise: 2D/2D (ad
jacent image pairs), intrinsicdirect (pixel/voxel property based
only), curved (to compensate for nonlinear distortions, how
ever, rigid registrations might be required in addition), global
(affect entire images), automatic (no user interaction), with pa
rametersobtainedusinganoptimizationprocedure(eventhough
procedures using explicitly computable parameters might be
additionally applied), monomodal (histological sections only),
intrasubject (no pair of data sets), and refers to pelvic organs
(specimen of cervical tissue). What is crucial is the illposed
ness of the required underlying registrations [19], [20]. That
means, registration results might be decisively affected from
small changes in the images. And, for this work, some appli
cable algorithmic solution has to cope with a broad range of
different tumor invasion patterns—without knowing their char
acteristics a priori. This paper introduces our newly developed
dedicated processing chain. For an overview of the processing
chain, see Fig. 4. It further elucidates quantitative results as
sessing the tumor growth based on 3D data, and also discusses
the above mentioned related work.
10 µm
Fig. 1.
considered in this work. TNM nomenclature tumor stage T1b (see left)
is defined as a lesion greater than a microinvasive cancer, which has a
microscopically measured invasion of stroma 5.0 mm or less in depth and no
wider than 7.0 mm (as for T1a1 or T1a2), and as a tumor confined to cervix.
Stage T1b is subdivided into tumors of 4.0 cm or less (T1b1) and more than 4.0
cm in size (T1b2). Stage T2 (see right) is defined as tumor invasion beyond the
uterus but not to the pelvic wall or to the lower third of vagina. It is subdivided
into cases without (T2a) and with (T2b) parametrial invasion. C: cervix uteri
(neck of uterus), Co: corpus uteri (body of uterus), I: isthmus uteri (constricted
part of the uterus between neck and body), Ca: cavum uteri (uterine cavity), F:
openings of the uterine tubes (fallopian tube), V: vagina.
Sketch of cut sections of the uterus depicting all tumor stages
Main objective of this paper is to provide an automated algo
rithm objectively assessing the cervical tumor invasion based
on 3D reconstructed tissue volumes using serial sections of
cervical specimen of resected uteri. Papers focussing on the
histopathologicalandclinicaloutcomeofthisworkareinprepa
ration.
II. THE TISSUE RECONSTRUCTION PROCESS
A. Tissue Samples and Digitization
This paper comprises an overall of thirteen specimens of
squamous cell carcinoma of the uterine cervix (anatomic
tumor stages T1b1, T1b2, T2a, and T2b, according to TNM
nomenclature [7], [21], see Fig. 1), surgically managed by total
mesometrial resection [22]. Three selected samples basically
exemplify the different tumor invasion patterns observed in
pathology routine (see Fig. 2).
The resected and formalinfixed cervix was radially cut into
specimens (thickness: 6–8 mm) which were paraffinembedded
(see Fig. 3), then serially sliced using a microtome HM355S by
MICROM GmbH, Germany [Fig. 2(a): 500 @ 5 µm, Fig. 2(b):
100 @ 10 µm, Fig. 2(c): 230 @ 10 µm], and finally stained
with hematoxylineosin (HE) using a staining machine. Sec
tions as parallel planes starting from one of the radial cutting
planes typically have a rough extent of 2.5 cm
raw digitization area is 1300
10.45 mm
8.28 mm0.865 cm at a nominal pixelsize of
8.04 µm . The digitization of the serial sections was carried
out manually using the AxioVision 3.1 controlling PC software
directly reading from a digital 2/3 one chip CCDcamera Ax
ioCam MRc mounted on an Axioskop 2 plus transmitted light
1.5 cm. The
1030 pixels corresponding to
Page 3
1288IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 24, NO. 10, OCTOBER 2005
Fig. 2.
based on single HEstained slices.
Typical threetiered verbal quantification of tumor invasion patterns
microscopeequippedwithaPlanNEOFLUAR1.25
and the Video adapter 60C 2/3 0.63
made by Carl Zeiss, AG, Germany).
objective
(all mentioned products
Fig. 3.
paraffin ready for slicing on a microtome.
View onto a typical unstained cervix specimen embedded within
Under these conditions, due to the still limited FOV the dig
itization practically can be considered as a rough selection of a
region of interest (ROI) within the tumor invasion front. There
fore, one also could refer to the placement of the microscope
slides as a zeroorder registration step in order to maximize the
effectively reconstructible volume of interest (VOI) along the
tumor invasion front.
Some fiducials would be difficult to apply in our framework.
Since it remains unclear where the ROI/VOI within the spec
imen is located (without staining one cannot reasonably locate
the tumor invasion front) one would need to have some stained
reference section for “navigation.” Then one would have to in
directly place some fiducials (e.g., four—one nearby each ROI
corner). The choice of material is crucial (soft, but not too soft
rods). However, we expect the drawbacks are greater than the
benefits. Even if one would succeed placing the fiducials, one
unavoidablyintroducesdamages(direct,andalsoindirect,since
themicrotome blade could be much more worn).The benefit re
mains very limited, one could not spare any of the registrations
which are detailed in the following.
B. Rigid Registration
In the first stage, a serial section undergoes a successive pair
wise rigid coregistration of all slices. By this, the data set is
restricted to an effectively captured VOI. The method is com
puted on scalar (grayleveled) images obtained based on the
luminance
of the original
the (old) International Telecommunication Union’s recommen
dation ITUR BT.6015 as linear combination of the color pri
maries
color images following
(1)
The applied AxioCam MRc provides linear primaries (i.e.,
without gamma correction).
Page 4
BRAUMANN et al.: 3DRECONSTRUCTION AND QUANTIFICATION OF CERVICAL CARCINOMA INVASION FRONTS 1289
Fig. 4. An overview of the processing chain towards 3D tumor invasion front reconstruction at the example of specimen 8. Just starting from the unregistered
image stack, finally an appropriate basis for a subsequent automated 3D invasion front quantification is provided. The second column consists of three orthogonal
planes (two reconstructions: the xz planes above and the ?–? planes at right). In the third column, cutouts of the right half of the ?–? planes (second column) are
magnified, while in the fourth column even further zoomed cutouts are depicted (bottomleft quarters of the third column).
Page 5
1290 IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 24, NO. 10, OCTOBER 2005
What has tobe solvedis thefollowing transformation consid
ering the parameters
(rotation) and
tween the two scalar images
,(translation) be
and
(2)
In fact, since
sake of simplicity in the notation any real existing differences
of the images are neglected.
The approach we are using is a noniterative twostep al
gorithm consisting of a combination of the polarlogarithmic
FourierMellin invariant (FMI) descriptor, [23] and phaseonly
matched filtering (POMF) [24]. Variants thereof have suc
cessfully been applied in [25] and [26]. FMI basically is
a polarlogarithmic transform accomplished on the Fourier
transformed images converting both rotations and scalings
into shifts. POMF is a matching technique representing an
extension of the matched filtering approach. However, the latter
is highly depending on the image energy rather than the spatial
structures within. A solution, therefore, is to take a transfer
function equal to the spectral phase as done by POMF. While
the pure crosscorrelation technique as, e.g., applied in [16],
tends to result in quite broad/flat maxima, POMF will yield
much narrower maxima. This method provides a reasonable
compromise utilizing both energy and phase of the Fourier
transformed images
and . Experiments using a symmetric
POMF as proposed as SPOMF in [25] resulted in even more
narrow, sharper fits, however, with our histological data with
a variety of slight damages (fissures, missing parts, folds)
SPOMF turned out to be too susceptible compared with POMF.
The first part of the FMIPOMFbased scheme treats the ro
tational registration, while the second part takes this determined
angle and computes the translation by means of another POMF.
Hence, the goal of the very first step is to determine the angle
by which the image
andare images of adjacent sections, for the
is rotated with respect to image
(3)
Herein,
unit. While the spectral phase
closely depending on both translation and scaling, the spectral
magnitude is translation invariant
denotes the Fourier transform andthe imaginary
of the imageis
(4)
Now, since what is of interest is a rotational angle, the spectral
carthesiancoordinates
andare replacedbythespectralpolar
coordinates
(orientation) and(wave number)
(5)
which is abbreviated in the following as:
(6)
andare referred to as the FMIs of the images
(2). By Fourier transforming (6) one obtains
and ,
as phase shift
(7)
We determine this phase shift under the constraint that no
scaling is assumed
by the following POMF (the star
denotes the complex conjugate)
(8)
Finishing the first part, the intermediate result is
(9)
For this part, the choice of the used rotational centre is arbitrary,
however, since the images are naturally of limited extend, it is
recommended to always take the physical centre of
minimize boundary effects.
While the rotational part of the rigid registration is finished,
the principle for solving the second part
in order to
(10)
is related
(11)
Now, applying (2) by inserting the results of (8) and (11) this
rigid registration part is formally solved.
Further, for the implementation the following was applied.
•The images need to have an appropriate contrast. We take
the following method for local contrast enhancement [27]
(12)
Herein,
respectively, while
“Local” refers to a squared vicinity centered around
with a side length of 55 which is about the max
imum width of fissures in pixels.
obtain a strong but not maximum effect.
The images should be windowed in order to reduce
leakage artifacts of the fast Fourier transform (FFT) [28].
We used a Hann window
anddepict the global and local mean,
is the local standard deviation.
was set to 0.9 to
•
(13)
andare the image extents.
Page 6
BRAUMANN et al.: 3DRECONSTRUCTION AND QUANTIFICATION OF CERVICAL CARCINOMA INVASION FRONTS1291
•For all resampling steps throughout this paper at least
first order interpolation is recommended. We use higher
order polynomial interpolation (cubic splines).
In (5), only the half of the spectrum has to be considered,
assuming that
and are of real data, so that their Fourier
transforms are mirrorsymmetric w.r.t.
window is, therefore, halved as well.
The resolution of the angle
tation of (5); we take an angular scale of 720 for
sulting in a quarterdegree resolution; this does not imply
any iterative angle search, (8) requires one single max
imum search, where the index of the maximum is a direct
measure for the angle.
Due to the applied FFTs, the time complexity of the rigid reg
istration is
withdenoting the number of pixels,
the memory complexity is
rithm runs about 1 s on a standard PC for image sizes around
megapixels.
For an illustrative view on this processing step, for a typ
ical slicetoslice transition we have depicted the displacement
vector field using line integral convolution [29], [30], see Fig. 5.
This kind of visualization of directions and strenghs in vector
fields is considered more illustrative than any direct plot of the
vectors even though the respective sense of directions cannot be
shown.
•
or ; the Hann
•
depends on the implemen
re
. For one image pair the algo
C. Color Adaptation
Once the first registration step is carried out, the effectively
available tissue volume is more or less restricted to a core re
gion depending on the accuracy of the slice placement during
the digitization. Since the staining is going to have further im
portanceforthetumorassessment,itisnecessarytoconsiderthe
achieved staining wrt. to its constancy. Even though applying a
staining machine, the number of sections per series by far ex
ceeds its capacity, so that series only can be stained by stages,
thus unfortunately introducing fluctuations. Another reason for
a similar effect are some very slight thickness variations which
also can appear as fluctuations mainly affecting the color satu
ration.
This adaptation procedure is accomplished as second step,
since the completely unregistered data set is inappropriate for
doing a sectionwise adaptation. Once the data has passed a
first rough registration, for every section we can assume to have
corresponding ROIs for all images which is not the case just
after initial digitization. Hence, in this second step within the
reconstruction procedure we are going to treat possible fluctu
ations of the staining along the serial sections. The idea behind
the simple but effective procedure is as follows: the concerned
sample image’s staining is subsequently adapted using a linear
color transform based on statistical distribution parameters.
So, the essence of the scheme we are proposing is just to
force all sample images to have the same mean and covariance
matrix applying a linear transform. In principle, what has to be
computed is
(14)
Fig. 5.
the displacement vector field to adapt (b) onto (a) according to the obtained
parameters ? , ?
and ?
is depicted using line integral convolution. The
(cyclic) color codes the absolute value of the underlying displacements from
purplered (high) via blue, cyan, green, yellow to orangered (low). The
maximum displacement is 1184.3 µm (lower right), the minimum is 254.1 µm
(upper left) which is located closely to the “rotational center” (outside the
image).
Rigid registration examples: For a pair of adjacent slices (a) and (b)
Page 7
1292 IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 24, NO. 10, OCTOBER 2005
(in homogeneous coordinates), where
and
denote the transform matrix, the sample section
image, the transformed sample section image and the reference
section image,respectively,whilethelatteris manuallyselected
for each series.
where
values of the red, green and blue image channel, respectively,
on
: We consider allpixels belonging to the finite set
onlyconsistingofpixelswithanintensityofatmost
90% of the maximum intensity as well as with a minimum in
tensity above 0. This restricts the transform to nonbright image
regions, since one can assume to have a background consisting
of either bright or black image regions due to fissures or artifi
cially filled black margins, respectively.
The transform
in (14) consists of the following matrix
product:
,,,
,, and are the mean
(15)
Its individual factors are obtained as follows.
anddenote the offsets (referring to
tively) and are determined as
and , respec
(16)
and in analogy for
of rank 4.
denotestherotationmatrix.Itisobtainedasmatrixproduct
, withrepresenting the identity matrix
(17)
wherein
(wrt. decreasing order of their corresponding eigenvalues) of
the covariance matrix
of the centered color value data (in
analogy for
and)
is the matrix of sorted eigenvectors
(18)
This real symmetric and orthonormal matrix and the mean
vector represent an estimated multivariate distribution of the
color values in
space. Supposed
by solving the following eigenvalue problem:
has the (full) rank 3,
(19)
one obtains the three eigenvalues
eigenvectors
reduction to get a tridiagonal form and then based upon this
using the QL algorithm (with implicit shifts).
and their corresponding
. The problem was solved using Householder
denotes the scalings along the principal axes, which are
given as
(20)
The time complexity of the color adaptation is
denoting the number of pixels, the memory complexity is also
. For one image the algorithm runs less than 1 s on a stan
dard PC for image sizes around megapixels.
Two examples are given in Fig. 6. Although the method is
simple, the results can be considered adequate for our purposes
since mainly stainingrelated outliers with small fluctuations
are targeted. To the knowledge of the authors, even if simple,
this method is not implemented in standard image manipulation
software.
with
D. Polynomial Nonlinear Registration
This third stage basically does the compensation for
sliceglobal distortions using polynomial warping [31] based
on sparsely populated displacement vector fields taken from
automatically determined control points. Its basic form is
(21)
while
represents an undistorted reference image and
the already rigidly registered but still distorted coun
terpart. The unknown coefficients
and
of the
(
th degree for each independent variable)
respectively, can offhand be found once displacement vectors
are available. Those displacement vectors rely on the pairwise
correlate of partially overlapping image tiles (i.e., subimages).
These tiles are sized 128
128 pixels and overlap 96 pixels
in both directions. 128 was taken as the most appropriate
power of 2 (lots of FFTs have to be computed). The overlap
results in a density of control points of one per 32 pixels which
results in reasonable numbers of control points in the order of
1000. To prepare the tiling, we ensure the image dimensions
to be multiples of the nonoverlapping tile size by adding an
appropriate black margin. Again, we use POMF applied to all
tile pairs
computing the correspondencies to the control points
,
th degree polynomials
and,
and for
(22)
Page 8
BRAUMANN et al.: 3DRECONSTRUCTION AND QUANTIFICATION OF CERVICAL CARCINOMA INVASION FRONTS1293
Fig. 6.
registration. Insomecases,appearingasintenselystained as(c)amore obviousimprovementcan be achieved(d).Note thatforthiscomparison(c/d) isnotadjacent
to (a/b), there was just one slice skipped in between in order to be more illustrative. The reference slice for this serial section is depicted in (e).
Color adaptation examples: Usually the adaptation only leads to minor changes as from (a) to (b), whereas (a) is corresponding to Fig. 5(a) after rigid
So, for a tile pair
vector is, e.g.,
Toget
andthe corresponding displacement
.
estimates of
and
properthecoefficients
a multivariate linear regression using a leastsquares (LS) error
minimization is done. The multivariate model is
(23)
with
(24)
representing the matrix of displacement vector end points and
(25)
an arranged matrix of all coefficients. For compactness reasons
of the derivation, we have further introduced the matrix
ferred to as design matrix which is build up from
products of combinations of the control point coordinates (i.e.,
the displacement vector start points)
re
......
...
......
...
...
(26)
where
wise linearly independent row vectors are supposedto represent
istheerrormatrixinwhichtheassumedpair
Page 9
1294IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 24, NO. 10, OCTOBER 2005
a2Dmultivariatenormaldistributionwithanexpectationvalue
vector of
as well as a covariance matrix
The general form of the sum of squared errors is
.
(27)
which has to be minimized.
differentiating (27) with respect to
zero in order to find an extremum one achieves
denotes an estimate of
and setting this expression
. Now,
(28)
Reforming (28), one easily can isolate
full rank) and gets
(supposedhas the
(29)
Thissolutionisrepresentingaminimumifthesecondderivative
of (27)
(30)
is positive definite which holds true if
ready assumed above.Thus, the result of (29) can be considered
valid. The two columns
und
and , respectively, which now can be inserted in (21).
For the implementation of (29), even with a simple
Gaußelimination method and
lems throughout the matrix inversion have occurred. Unwanted
warping effects as reported in [32] have occasionally occured,
but could be managed as we have introduced the following
extension:Supposeda numberof
blocks) we artificially haveadded some
control points (associated with zerolength displacement vec
tors) and placed them just along the image margin as a “frame”
around the existing grid of control points. Under the condition
that the rigid registration step was successful, this kind of
“framing” is warrantable. This extension did decisively im
prove the method so that no “collapsing” or other unwanted
warpings thereupon did occur. Further, differing to the previous
rigid registration, this could be accomplished based on the
luminance of the color adapted images.
The time complexity of the polynomial registration is
due to the FFTs with
pixels, the memory complexity is
the algorithm runs about 3 s on a standard PC for image sizes
around megapixels.
Corresponding to Fig. 5 also for this polynomial registration
step we give an illustration of the resulting displacement vector
field for the same typical slicetoslice transition, see Fig. 7
using line integral convolution.
has the full rank as al
ofare the estimates of
no singularity prob
tiles(matching
denoting the number of
. For one image pair
E. StainingBased Tumor Probability
Now, while two registration steps are done, the serial section
is fairly realigned. Most of the slicetoslice transitions can be
consideredsmoothandmisalignmentsappearmainlylocally.To
Fig. 7.
pair of adjacent images (a) and (b) according to the estimated polynomial
coefficients the resulting displacement vector fields is visualised (c). The
(cyclic) color codes the absolute value of the underlying displacements from
purplered (high) via blue, cyan, green, yellow to orangered (low). The
maximum displacement is 84.4 µm, the minimum is 0 µm. What is visible
at the first glance is the inhomogeneity of the field, whereas on the right and
left there are two distinct local maxima of the displacement, in this case the
registrations leads to some contraction from the left/right/upper part toward a
region around below the slice center. Two vortices (upper right and lower left)
occur outside the physical slice and have very small strenghts.
Polynomial registration examples: Corresponding to Fig. 5, for a
treat those remaining registration errors, we subsequently need
to apply yet another registration step. Just like for the previous
registration steps, this one also applies to scalar data. Despite
of taking some luminancerelated images, we, therefore, want
to use scalar images highlighting the tumor regions. We gen
erate such images simply by computing stainingbased tumor
probability maps relying on the HE staining applied to all slices
short after sectioning. The probability maps are necessarily re
quired for thresholdbased tumor segmentation. The reason for
swapping these two steps is mainly that by this we can further
Page 10
BRAUMANN et al.: 3DRECONSTRUCTION AND QUANTIFICATION OF CERVICAL CARCINOMA INVASION FRONTS1295
Fig. 8.
and polynomially registered images (a) and (c) and have determined the respective tumor probability maps (b) and (d). This example also illustrates the usefulness
of the color correction by what the two probability maps appear qualitatively equal. Note that the suddenly emerging margin was artificially introduced in order to
get image dimensions as multiples of primes up to 5 which is useful for the many FFTs accomplished in the following curvature registration.
Stainingbased tumor probability computation examples: Corresponding to the image pairs of Fig. 6 we have taken the same two but now color corrected
attenuate artifacts mainly occurring outside the tumor regions,
which facilitates the final registration step.
Basically, it is required to manually obtain representa
tive tumor color samples from the respective serial section.
Precisely, we arbitrarily select a number of ten slices equidis
tantly along the series and let the pathologist draw in the
tumor boundary which provides us both with
triple samples
for tumor and nontumor. Now,
for the tumor probability we adopt normalized color values
color
leading to a projec
tion onto a sphere sector fitting within the RGB cube, where
. One of the normalized components is
redundant, so we can restrict to use
assume that for the tumor as well as for the nontumor
components follow multivariate normal distributions. So, we
estimate the multivariate densities for both sets
. We further
these
(31)
with
means, respectively. Finally, the probability for a pixel to
exhibit the color of tumor at
and denoting the covariance matrices and
is
(32)
with
.
Fig. 8 illustrates the tumor probability computation for two
images. The results indicate that the previous coloradaptation
(compare Fig. 6) is justifiable.
The time complexity of the stainingbased tumor probability
computation is
withdenoting the number of pixels, the
memory complexity is
as well. For one image the algo
rithm runs about 5 s on a standard PC for image sizes around
megapixels.
F. CurvatureBased Nonlinear Registration
In this processing stage, remaining local registration errors
are diminished and the imagetoimage transitions are further
smoothed. What generally has to be computed is a local dis
placement field as vector function
considered as representation of the mis
alignments. The nonparametric procedure we are applying for
this nonlinear registration uses a regularization term approxi
mating local curvature, which was introduced in [33] and re
cently studied in [20], [34]. While the authors state that the al
gorithm would include an automatic rigid alignment, with our
data we in fact could not benefit from this effect. This can be
explained as follows: their images strictly cover some complete
object(s)infrontofahomogeneousbackground,sothatitcanbe
more or less assumed to successfully find all correspondences
onebyone. However, in our image material we do not have
isolated objects with some delimited boundary. In fact, since
the digitized regions usually do not comprise any background
with
Page 11
1296 IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 24, NO. 10, OCTOBER 2005
but generally tissue, around the image margin for a consider
able portion of the images pairwise correspondences will be
missing. Therefore, from the viewpoint of this algorithm the
previous registration steps can be considered as preprocessing
in order to drastically improve a priori imagetoimage corre
spondence.
Basically, the distance measure to be minimized for this reg
istrationstepisthesumofsquareddifferencesoftheimage’sin
tensities [here, taken as tumor probabilities using (32)]. Again,
assume
represents an undistorted reference image and
the already both rigidly as well as polynomially registered but
still distorted counterpart, while the registration should do the
mapping
(33)
With
(34)
we define a joint registration criterion consisting of the sum of
squared differences
(35)
and the smoothing term
(36)
From the calculus of variations we know that a function
imizing (34) necessarily should be a solution for the EulerLa
grange equation
min
(37)
with
(38)
Forthecoupledsystemof4thorderpartialdifferentialequations
[see (37)] an artificial time parameter is introduced as
(39)
with the boundary condition of
image.
To solve (39) the time dependence is discretized using an
implicit midpoint rule for the linear operator
. For the integration over a single time step
gets
being periodic across the
one
(40)
for the propagation from
Green function
to. Defining the
(41)
the solutionfor the next time step is found by
(42)
Denoting the discrete Fourier components
(43)
and using [35, equation (25.3.33)] for the discrete version of the
biharmonic operator, (42) in the Fourier domain is given by
(44)
with
whereand andfor a
grid. When
is used to update . Every time integration step needs a
totaloffourFouriertransformsforthetwocomponentsof
the two components of
. We have chosen periodic boundary
conditions for the nonlinear registration. Other boundary con
ditions, e.g., with zero displacement on the boundary or zero
normal derivative for
can also be used. It shoud be noted that
we always could find a single transformation without the need
of restarts.
Fig. 9 illustrates the curvature registration for an image pair
using line integral convolution. Concerning the parametrization
of the algorithm, for all specimen we have applied a fixed max
imum number of 32 iteration steps, whereas
with an iteration time step
of 2.0. The solution is computed
in the Fourier space. We have iterated some certain fix number
of steps and applied both a fixed stepwidth and smoothing co
efficient. Remember, referencefree registration is an illposed
problem, so the difficulty is especially to avoid removing all
differences between two adjacent sections/images. The specific
choice of these mentioned parameters was made as follows: we
is computed the backward Fourier trans
form
and
was set to 5.0
Page 12
BRAUMANN et al.: 3DRECONSTRUCTION AND QUANTIFICATION OF CERVICAL CARCINOMA INVASION FRONTS1297
Fig. 9.
to Figs. 5 and 7, for a pair of adjacent images (a) and (b) the determined
displacement vector field is visualised (c) using line integral convolution.
The (cyclic) color codes the absolute value of the underlying displacements
from purplered (high) via blue, cyan, green, yellow to orangered (low).
The maximum displacement is 36.2 µm, the minimum is 0 µm. Compared
to Fig. 7 this displacement vector field is even more inhomogeneous and the
displacements are at most half as far as for the polynomial step.
Curvaturebased nonlinear registration example: Corresponding
wanted to have the curvature term get weighted as five times as
the squared differences. The time step and iteration steps were
selected based on visual inspections of tests making sure that no
eyecatching unwanted warpings occur. At the present stage of
our work we did not implement any time dependency of these
mentioned parameters.
To further improve the performance of those nonlinear
schemes, [36], [37] have proposed algorithms what they have
called PairandSmooth registrations which combine geometric
matching with intensitybased registration. This together with a
multigrid implementation will be a future direction of our work.
Because of the FFTbased implementation, the time com
plexity of the curvaturebased nonlinear registration step is
, with denoting the number of pixels, the memory
complexity is
. For one image pair the algorithm runs
about four minutes on a standard PC for image sizes around
megapixels. Comparing this CPU time with the time neces
sary to do the microtome sectioning, the staining, the manual
digitization, and a number of previous processing steps, we
still consider some minutes acceptable for one imagetoimage
transition. With the above mentioned multigrid implementation
(this is ongoing work) we expect a decisive computational
speedup.
G. Total Variation Filtering
Due to the pixel based color segmentation typically the data
is affected by a significant amount of noise. While this noise
is not essentially affecting the previous registration step, we
consider the necessity for an intermediate processing on the
reconstructed 3D data step in order to facilitate the succeeding
thresholdingbased segmentation. Nonlinear filters are in
general much better in preserving image structures compared
to linear ones. So, e.g., median filtering perfoming a ranking
operation will keep edges but remove outliers while, on the
other hand, linear binomial filtering will damp both. However,
while the median filter [27] is appropriate in case of simple
saltandpepper noise, its homogenizing properties remain
limited. More sophisticated schemes like nonlinear diffusion
filtering [38], [39] have been proposed, however these basically
require some certain stopping criterion, otherwise the image
structures get lost.
Instead, we have decided to apply nonlinear total variation
filtering [40]. This filter minimizes the functional
(45)
with
the probability that at
found. Let
white noise
, for the scalar 3D imagethat contains
a cancer voxel can be
be the original noisy image with Gaussian
exhibiting the following properties:
That
the EulerLagrange equation
minimizing (45) generally can be obtained solving
(46)
For this extremely nonlinear equation several solution methods
are known. Since the considered volume data are very large,
memory intensive methods are not feasible, because a nonlinear
solver would require several gigabytes of temporary memory.
Page 13
1298 IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 24, NO. 10, OCTOBER 2005
Fig. 10.
transformed by the curvatureregistration, and now were TV filtered attaining an edgepreserving smoothing as depicted in (b) and (d). Note that the TV filtering
is applied as 3D operation, which is why we show this adjacent pair here (the 3D effect can be best seen in the nontumour background).
Total variation filtering examples: This example again refers to the same pair as in Fig. 9. The two images of the adjacent slices (a) and (c) were
However, an appropriate solution method with low memory de
mands was proposed by Osher et. al. [40]. It transforms the
problem into a time dependent problem for
. So, instead of (46) we apply
with
(47)
with
ishes, the equation becomes singular so that the gradient must
be regularized as
. In regions where the gradient van
(48)
(49)
considering
. For the discrete solution
(50)
the summation runs over all next neighbors of
noted as
. For inner pointsin thevolume a 6neighborhood
stencil is used. Boundary points cover a reduced neighborhood
as respective grid points exist. This yields a nonlinear filter for
every mesh point
which is de
(51)
with
Thisfilteractsasanedgepreservinglowpass.Duringthecalcu
lation one simultaneously needs to store at least three data sets
, and . The main advantage of the filter is its rel
atively quick convergence toward the denoised result. The only
free parameter is . Its choice is of importance for the denoising
quality. Following [40] we use
(52)
withthestandarddeviationofthewhitenoise
of mesh points
. After 5–10 iterations a new
and used for the update of the
The time complexity of the total variation filtering is
with denoting the number of voxels, and also the memory
complexity is
. For one image series with typically 300
sections the algorithm runs about half an hour on a standard PC
for imagesizes around megapixels.Thisstep is a 3Doperation,
so it may have exorbitant RAM requirements, since
andhave to be accessible simultaneously.
andthenumber
is computed
.
,
Page 14
BRAUMANN et al.: 3DRECONSTRUCTION AND QUANTIFICATION OF CERVICAL CARCINOMA INVASION FRONTS1299
Fig. 11.Tumor segmentation example: The two images (a) and (b) show the results after thresholding the TV filtered images in Fig. 10(b) and (d), respectively
Fig. 12.(left) Multiplanar reconstruction and (right) 3D surface rendering of the invasion front for specimen 8.
Fig. 10 illustrates the 3D TV filtering effect on a pair of ad
jacent images.
H. Tumor Segmentation
In this final reconstruction step, just the TV filtered volume
data is binarized. According to (32) an illustrative criterion for
thresholding is where the two estimated densities for tumor and
nontumor exhibit the same magnitude so that the tumor proba
bility is 0.5. Let
represent the previously TV filtered scalar
image, the binarized result
is obtained as
for
for.
(53)
Both time and memory complexity of thresholding of course is
withdenoting the number of voxels. It takes a fraction
of 1 s on standard PC hardware.
Fig.11givesexamplescorrespondingtotheresultsofFig.10.
III. INVASION FRONT QUANTIFICATION
Once the smoothing by means of total variation filtering and
the segmentation was accomplished, following the 3D recon
struction process the tumor invasion front within the volume
data is going to be assessed. Hence, the invasion front firstly
is visualized and subsequently quantified.
A. ThreeDimensional Tumor Visualization
What is of basic interest is the topology of the invasion front.
One of the interesting questions at hand is how the tumor inva
sion front is shaped. Another question is the presence of pos
sibly separated tumor islets apart from the main tumor. Some
virtually have occured but turned out to have direct contact to
the data set outskirt. These were sorted out since it cannot be
decided if separated or not. The rest, however, was not straight
forward to be verified or falsified, due to the limitations of the
HE staining. HE in fact is just enhancing image contrast with
respect to the averaged local cell kernel density. In tumor cells,
thekernelsarebasicallyenlarged.Inexceptionalcases,misclas
sifications might occur as, e.g., for smaller inflamational cells
or some other dense tissue parts. We have let the pathologist
check all suspected tumor islets using a much larger magnifica
tion (40
instead of 1.25) but got none of them verified to
consist of tumor. What has remained for all our specimen was
one large connected tumor segment, a kind of “massif” VOI of
the tumor invasion front.
Therefore, in order to give a 3D illustration of the recon
structed tumor invasion, we do a surface rendering applying the
Page 15
1300 IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 24, NO. 10, OCTOBER 2005
Fig. 13.Views onto the 3D surface renderings of tumor invasion fronts, part I.
thresholdingcriterionfrom(53).Therenderingofthetumorsur
faces uses the wellknown algorithm from [41] with the mesh
displacement modification [42], [43]. A detailed discussion of
rendering algorithms can be found in [44].
The gallery of tumor invasions of our 13 specimens is shown
in the Figs. 12–14. These are the firstever visualizations of
a solid tumor’s invasion front with a resolution of
The renderings have been generated using MathGL3d [45],
an OpenGLbased interactive viewer for Mathematica’s 3D
graphics. Typical numbers of (potentially nonconnected) sur
face polygons occur from
series with 300 sections and image sizes around megapixels
10 µm.
. For a typical image