Page 1

1286 IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 24, NO. 10, OCTOBER 2005

Three-DimensionalReconstructionandQuantification

of Cervical Carcinoma Invasion Fronts From

Histological Serial Sections

Ulf-Dietrich Braumann, Member, IEEE, Jens-Peer Kuska, Member, IEEE, Jens Einenkel, Lars-Christian Horn,

Markus Löffler, and Michael Höckel

Abstract—The analysis of the three-dimensional (3-D) structure

of tumoral invasion fronts of carcinoma of the uterine cervix is

the prerequisite for understanding their architectural-functional

relationship. The variation range of the invasion patterns known

so far reaches from a smooth tumor-host boundary surface to

more diffusely spreading patterns, which all are supposed to have

a different prognostic relevance. As a very decisive limitation

of previous studies, all morphological assessments just could be

done verbally referring to single histological sections. Therefore,

the intention of this paper is to get an objective quantification

of tumor invasion based on 3-D reconstructed tumoral tissue

data. The image processing chain introduced here is capable to

reconstruct selected parts of tumor invasion fronts from histo-

logical serial sections of remarkable extent (90–500 slices). While

potentially gaining good accuracy and reasonably high resolution,

microtome cutting of large serial sections especially may induce

severe artifacts like distortions, folds, fissures or gaps. Starting

from stacks of digitized transmitted light color images, an overall

of three registration steps are the main parts of the presented

algorithm. By this, we achieved the most detailed 3-D reconstruc-

tion of the invasion of solid tumors so far. Once reconstructed, the

invasion front of the segmented tumor is quantified using discrete

compactness.

Index Terms—Biological tissues, image color analysis, image

processing, image registration, image segmentation, image shape

analysis, scientific visualization, tumors.

I. INTRODUCTION

T

views of tissue organization, e.g., tumor morphology and tumor

O our understanding, to really consider volumes but not

just single slices is essential in order to get new insight

Manuscript received December 28, 2004; revised July 12, 2005. This work

was supported by the German Research Foundation (DFG) under Grant BIZ-6

1/1-3.TheAssociateEditorresponsibleforcoordinatingthereviewofthispaper

and recommending its publication was N. Ayache. Asterisk indicates corre-

sponding author.

*U.-D. Braumann is with the Interdisciplinary Center for Bioinformatics,

University Leipzig, Härtelstraße 16-18, 04107 Leipzig, Germany (e-mail: brau-

mann@izbi.uni-leipzig.de).

J.-P.Kuska iswiththeInterdisciplinaryCenterforBioinformatics,University

Leipzig, 04107 Leipzig, Germany (e-mail: kuska@informatik.uni-leipzig.de).

J. Einenkel and M. Höckel are with the Department of Gynecology

and Obstetrics, University Leipzig, 04103 Leipzig, Germany (e-mail:

jens.einenkel@medizin.uni-leipzig.de,

leipzig.de).

L.-C. Horn is with the Institute of Pathology, University Leipzig, 04103

Leipzig, Germany (e-mail: lars-christian.horn@medizin.uni-leipzig.de).

M. Löffler is with the Institute for Medical Informatics, Statistics and Epi-

demiology, the Coordination Center for Clinical Trials, and the Interdiscipli-

nary Center for Bioinformatics, University Leipzig, 04107 Leipzig, Germany

(e-mail: markus.loeffler@imise.uni-leipzig.de).

Digital Object Identifier 10.1109/TMI.2005.855437

michael.hoeckel@medizin.uni-

growth. The three-dimensional (3-D) characterization of inva-

sion patterns of squamos cell carcinoma of the uterine cervix

using histological serial sections is a current clinical question.

This gives demand for both high level image processing and

analysis. Properties of the hitherto observed two-dimensional

(2-D) tumor invasion fronts are supposed to have relevance for

the further prognosis of the respective patient [1]–[4]. Doing

those quantitative analyses in 3-D, a new quality for the struc-

tural and morphological assessment of the considered tumors

can be expected.

Three-dimensional imaging modalities like computed to-

mography (CT), cone beam computed tomography (CBCT),

nuclear magnetic resonance imaging (MRI), positron emis-

sion tomography (PET), single photon emission computed

tomography (SPECT), etc. have become state of the art in

many fields of medical diagnostics and research. Besides

those macroscopic in vivo 3-D techniques, for more detailed

analyses nondestructive 3-D microscopy is available for in vitro

(partially applicable in vivo), as e.g., scanning transmission

ion microscopy (STIM) or particle induced X-ray emission

(PIXE), scanning force microscopy (SFM), 3-D electron mi-

croscopy (3DEM), miniaturized computed tomography (µ

miniaturized nuclear magnetic resonance imaging (µ

confocal LASER scanning microscopy (CLSM), etc. For a

concise review on current high-resolution imaging techniques

of (living) tissue, see [5].

CLSM could be successfully applied on precancerous cer-

vical epithelial lesions [6] both ex vivo on biopsies in 3-D, as

well as in vivo in 2-D using a CLSM-variant referred to as “con-

focal microendoscope.” The limited range of CLSM of about

100,

, 200 µm is acceptable for these epithelial lesions. The

whole epithelium’s thickness is 200,

ysis of cervical tumors, the CLSM’s penetration range unfortu-

nately is too short.

Even though other in vivo techniques would be desirable for

detailed uterine cervix analyses, spatial resolutions of

as achieved e.g., in MRIs from the pelvic region unfortunately

do not appear sufficient for conclusions on tumor invasion and

infiltration which necessitates resolutions

far beyond typical cell diameters of

clinical diagnostics of cervical cancer (e.g., tumor staging),

MRI and partially CT are indispensable [7], [8] and partially

can be used to “predict” histopathologic features. The “ground

truth” for tumor typing, however, only can be obtained by

histopathology both using visual inspection and especially

),

) or

, 300 µm. For the anal-

mm

mm, but not

10 µm. However, for

0278-0062/$20.00 © 2005 IEEE

Page 2

BRAUMANN et al.: 3-D-RECONSTRUCTION AND QUANTIFICATION OF CERVICAL CARCINOMA INVASION FRONTS1287

transmitted light microscopy once the tumor was surgically

resected.

Among the microscopic techniques, apart from those with

typically very limited fields of view (FOVs) and/or far sub-cel-

lular spatial resolutions there is no well established 3-D proce-

dure or protocol for tumor imaging providing appropriate con-

trasts/spatialranges/resolutions.Therefore,wedecidedtodothe

data acquisition using conventional transmitted-light imaging

based on HE-stained histological serial sections. The reasons

for taking this modality are twofold: at this stage of our work

we wanted to stay as close as possible to the procedure usually

applied by the pathologist when assessing single slices in rou-

tine. Moreover, both the achievable spatial resolution (

voxel edge length) and the effective FOV (typ. 0.1 cm ) can be

considered acceptable.

However, this necessarily requires solving the 3-D tissue

reconstruction problem based on huge registered serial sections

without having a reference data set available to co-register. On

principle, thiskind ofproblem has a long history and infact was

alreadytackleddecadesago,e.g.,in[9]and[10]proposingsolu-

tions using manually digitised line drawings. More recent work

essentially could benefit from much improved computational

power (especially concerning CPU and RAM) but also from

(analogue, later digital) imaging improvements. Consequently,

ambitiousalgorithmicsolutionshavebeendeveloped[11]–[13],

partially treated as a co-registration problem [14]–[17]. Even

though a completely different level of quality was reached

meanwhile, due to its nature such 3-D reconstruction from

serial sections remain complex and time-consuming. What is

the central problem we are facing is the absence of some hard

quality criterion to refer to. This means, we cannot utilize some

reference data set since unfortunately there is none. This kind

of dilemma has motivated us to strictly follow a coarse-to-fine

strategy, i.e., we do the reconstruction in a stepwise manner

and apply registration schemes with an increasing order of

complexity, first a rigid one, then a polynomial nonlinear one,

and finally a curvature-based nonlinear one.

The nine-criteria based classification of the required registra-

tion method(s) according to [18] would comprise: 2-D/2-D (ad-

jacent image pairs), intrinsic-direct (pixel/voxel property based

only), curved (to compensate for nonlinear distortions, how-

ever, rigid registrations might be required in addition), global

(affect entire images), automatic (no user interaction), with pa-

rametersobtainedusinganoptimizationprocedure(eventhough

procedures using explicitly computable parameters might be

additionally applied), monomodal (histological sections only),

intra-subject (no pair of data sets), and refers to pelvic organs

(specimen of cervical tissue). What is crucial is the ill-posed-

ness of the required underlying registrations [19], [20]. That

means, registration results might be decisively affected from

small changes in the images. And, for this work, some appli-

cable algorithmic solution has to cope with a broad range of

different tumor invasion patterns—without knowing their char-

acteristics a priori. This paper introduces our newly developed

dedicated processing chain. For an overview of the processing

chain, see Fig. 4. It further elucidates quantitative results as-

sessing the tumor growth based on 3-D data, and also discusses

the above mentioned related work.

10 µm

Fig. 1.

considered in this work. TNM nomenclature tumor stage T1b (see left)

is defined as a lesion greater than a micro-invasive cancer, which has a

microscopically measured invasion of stroma 5.0 mm or less in depth and no

wider than 7.0 mm (as for T1a1 or T1a2), and as a tumor confined to cervix.

Stage T1b is subdivided into tumors of 4.0 cm or less (T1b1) and more than 4.0

cm in size (T1b2). Stage T2 (see right) is defined as tumor invasion beyond the

uterus but not to the pelvic wall or to the lower third of vagina. It is subdivided

into cases without (T2a) and with (T2b) parametrial invasion. C: cervix uteri

(neck of uterus), Co: corpus uteri (body of uterus), I: isthmus uteri (constricted

part of the uterus between neck and body), Ca: cavum uteri (uterine cavity), F:

openings of the uterine tubes (fallopian tube), V: vagina.

Sketch of cut sections of the uterus depicting all tumor stages

Main objective of this paper is to provide an automated algo-

rithm objectively assessing the cervical tumor invasion based

on 3-D reconstructed tissue volumes using serial sections of

cervical specimen of resected uteri. Papers focussing on the

histopathologicalandclinicaloutcomeofthisworkareinprepa-

ration.

II. THE TISSUE RECONSTRUCTION PROCESS

A. Tissue Samples and Digitization

This paper comprises an overall of thirteen specimens of

squamous cell carcinoma of the uterine cervix (anatomic

tumor stages T1b1, T1b2, T2a, and T2b, according to TNM

nomenclature [7], [21], see Fig. 1), surgically managed by total

mesometrial resection [22]. Three selected samples basically

exemplify the different tumor invasion patterns observed in

pathology routine (see Fig. 2).

The resected and formalin-fixed cervix was radially cut into

specimens (thickness: 6–8 mm) which were paraffin-embedded

(see Fig. 3), then serially sliced using a microtome HM355S by

MICROM GmbH, Germany [Fig. 2(a): 500 @ 5 µm, Fig. 2(b):

100 @ 10 µm, Fig. 2(c): 230 @ 10 µm], and finally stained

with hematoxylin-eosin (HE) using a staining machine. Sec-

tions as parallel planes starting from one of the radial cutting

planes typically have a rough extent of 2.5 cm

raw digitization area is 1300

10.45 mm

8.28 mm0.865 cm at a nominal pixelsize of

8.04 µm . The digitization of the serial sections was carried

out manually using the AxioVision 3.1 controlling PC software

directly reading from a digital 2/3 one chip CCD-camera Ax-

ioCam MRc mounted on an Axioskop 2 plus transmitted light

1.5 cm. The

1030 pixels corresponding to

Page 3

1288IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 24, NO. 10, OCTOBER 2005

Fig. 2.

based on single HE-stained slices.

Typical three-tiered verbal quantification of tumor invasion patterns

microscopeequippedwithaPlan-NEOFLUAR1.25

and the Video adapter 60C 2/3 0.63

made by Carl Zeiss, AG, Germany).

objective

(all mentioned products

Fig. 3.

paraffin ready for slicing on a microtome.

View onto a typical unstained cervix specimen embedded within

Under these conditions, due to the still limited FOV the dig-

itization practically can be considered as a rough selection of a

region of interest (ROI) within the tumor invasion front. There-

fore, one also could refer to the placement of the microscope

slides as a zero-order registration step in order to maximize the

effectively reconstructible volume of interest (VOI) along the

tumor invasion front.

Some fiducials would be difficult to apply in our framework.

Since it remains unclear where the ROI/VOI within the spec-

imen is located (without staining one cannot reasonably locate

the tumor invasion front) one would need to have some stained

reference section for “navigation.” Then one would have to in-

directly place some fiducials (e.g., four—one nearby each ROI

corner). The choice of material is crucial (soft, but not too soft

rods). However, we expect the drawbacks are greater than the

benefits. Even if one would succeed placing the fiducials, one

unavoidablyintroducesdamages(direct,andalsoindirect,since

themicrotome blade could be much more worn).The benefit re-

mains very limited, one could not spare any of the registrations

which are detailed in the following.

B. Rigid Registration

In the first stage, a serial section undergoes a successive pair-

wise rigid co-registration of all slices. By this, the data set is

restricted to an effectively captured VOI. The method is com-

puted on scalar (gray-leveled) images obtained based on the

luminance

of the original

the (old) International Telecommunication Union’s recommen-

dation ITU-R BT.601-5 as linear combination of the color pri-

maries

color images following

(1)

The applied AxioCam MRc provides linear primaries (i.e.,

without gamma correction).

Page 4

BRAUMANN et al.: 3-D-RECONSTRUCTION AND QUANTIFICATION OF CERVICAL CARCINOMA INVASION FRONTS 1289

Fig. 4. An overview of the processing chain towards 3-D tumor invasion front reconstruction at the example of specimen 8. Just starting from the unregistered

image stack, finally an appropriate basis for a subsequent automated 3-D invasion front quantification is provided. The second column consists of three orthogonal

planes (two reconstructions: the x-z planes above and the ?–? planes at right). In the third column, cutouts of the right half of the ?–? planes (second column) are

magnified, while in the fourth column even further zoomed cutouts are depicted (bottom-left quarters of the third column).

Page 5

1290 IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 24, NO. 10, OCTOBER 2005

What has tobe solvedis thefollowing transformation consid-

ering the parameters

(rotation) and

tween the two scalar images

,(translation) be-

and

(2)

In fact, since

sake of simplicity in the notation any real existing differences

of the images are neglected.

The approach we are using is a noniterative two-step al-

gorithm consisting of a combination of the polar-logarithmic

Fourier-Mellin invariant (FMI) descriptor, [23] and phase-only

matched filtering (POMF) [24]. Variants thereof have suc-

cessfully been applied in [25] and [26]. FMI basically is

a polar-logarithmic transform accomplished on the Fourier

transformed images converting both rotations and scalings

into shifts. POMF is a matching technique representing an

extension of the matched filtering approach. However, the latter

is highly depending on the image energy rather than the spatial

structures within. A solution, therefore, is to take a transfer

function equal to the spectral phase as done by POMF. While

the pure cross-correlation technique as, e.g., applied in [16],

tends to result in quite broad/flat maxima, POMF will yield

much narrower maxima. This method provides a reasonable

compromise utilizing both energy and phase of the Fourier

transformed images

and . Experiments using a symmetric

POMF as proposed as SPOMF in [25] resulted in even more

narrow, sharper fits, however, with our histological data with

a variety of slight damages (fissures, missing parts, folds)

SPOMF turned out to be too susceptible compared with POMF.

The first part of the FMI-POMF-based scheme treats the ro-

tational registration, while the second part takes this determined

angle and computes the translation by means of another POMF.

Hence, the goal of the very first step is to determine the angle

by which the image

andare images of adjacent sections, for the

is rotated with respect to image

(3)

Herein,

unit. While the spectral phase

closely depending on both translation and scaling, the spectral

magnitude is translation invariant

denotes the Fourier transform andthe imaginary

of the imageis

(4)

Now, since what is of interest is a rotational angle, the spectral

carthesiancoordinates

andare replacedbythespectralpolar

coordinates

(orientation) and(wave number)

(5)

which is abbreviated in the following as:

(6)

andare referred to as the FMIs of the images

(2). By Fourier transforming (6) one obtains

and ,

as phase shift

(7)

We determine this phase shift under the constraint that no

scaling is assumed

by the following POMF (the star

denotes the complex conjugate)

(8)

Finishing the first part, the intermediate result is

(9)

For this part, the choice of the used rotational centre is arbitrary,

however, since the images are naturally of limited extend, it is

recommended to always take the physical centre of

minimize boundary effects.

While the rotational part of the rigid registration is finished,

the principle for solving the second part

in order to

(10)

is related

(11)

Now, applying (2) by inserting the results of (8) and (11) this

rigid registration part is formally solved.

Further, for the implementation the following was applied.

•The images need to have an appropriate contrast. We take

the following method for local contrast enhancement [27]

(12)

Herein,

respectively, while

“Local” refers to a squared vicinity centered around

with a side length of 55 which is about the max-

imum width of fissures in pixels.

obtain a strong but not maximum effect.

The images should be windowed in order to reduce

leakage artifacts of the fast Fourier transform (FFT) [28].

We used a Hann window

anddepict the global and local mean,

is the local standard deviation.

was set to 0.9 to

•

(13)

andare the image extents.

Page 6

BRAUMANN et al.: 3-D-RECONSTRUCTION AND QUANTIFICATION OF CERVICAL CARCINOMA INVASION FRONTS1291

•For all re-sampling steps throughout this paper at least

first order interpolation is recommended. We use higher

order polynomial interpolation (cubic splines).

In (5), only the half of the spectrum has to be considered,

assuming that

and are of real data, so that their Fourier

transforms are mirror-symmetric w.r.t.

window is, therefore, halved as well.

The resolution of the angle

tation of (5); we take an angular scale of 720 for

sulting in a quarter-degree resolution; this does not imply

any iterative angle search, (8) requires one single max-

imum search, where the index of the maximum is a direct

measure for the angle.

Due to the applied FFTs, the time complexity of the rigid reg-

istration is

withdenoting the number of pixels,

the memory complexity is

rithm runs about 1 s on a standard PC for image sizes around

megapixels.

For an illustrative view on this processing step, for a typ-

ical slice-to-slice transition we have depicted the displacement

vector field using line integral convolution [29], [30], see Fig. 5.

This kind of visualization of directions and strenghs in vector

fields is considered more illustrative than any direct plot of the

vectors even though the respective sense of directions cannot be

shown.

•

or ; the Hann

•

depends on the implemen-

re-

. For one image pair the algo-

C. Color Adaptation

Once the first registration step is carried out, the effectively

available tissue volume is more or less restricted to a core re-

gion depending on the accuracy of the slice placement during

the digitization. Since the staining is going to have further im-

portanceforthetumorassessment,itisnecessarytoconsiderthe

achieved staining wrt. to its constancy. Even though applying a

staining machine, the number of sections per series by far ex-

ceeds its capacity, so that series only can be stained by stages,

thus unfortunately introducing fluctuations. Another reason for

a similar effect are some very slight thickness variations which

also can appear as fluctuations mainly affecting the color satu-

ration.

This adaptation procedure is accomplished as second step,

since the completely unregistered data set is inappropriate for

doing a section-wise adaptation. Once the data has passed a

first rough registration, for every section we can assume to have

corresponding ROIs for all images which is not the case just

after initial digitization. Hence, in this second step within the

reconstruction procedure we are going to treat possible fluctu-

ations of the staining along the serial sections. The idea behind

the simple but effective procedure is as follows: the concerned

sample image’s staining is subsequently adapted using a linear

color transform based on statistical distribution parameters.

So, the essence of the scheme we are proposing is just to

force all sample images to have the same mean and covariance

matrix applying a linear transform. In principle, what has to be

computed is

(14)

Fig. 5.

the displacement vector field to adapt (b) onto (a) according to the obtained

parameters ? , ?

and ?

is depicted using line integral convolution. The

(cyclic) color codes the absolute value of the underlying displacements from

purple-red (high) via blue, cyan, green, yellow to orange-red (low). The

maximum displacement is 1184.3 µm (lower right), the minimum is 254.1 µm

(upper left) which is located closely to the “rotational center” (outside the

image).

Rigid registration examples: For a pair of adjacent slices (a) and (b)

Page 7

1292 IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 24, NO. 10, OCTOBER 2005

(in homogeneous coordinates), where

and

denote the transform matrix, the sample section

image, the transformed sample section image and the reference

section image,respectively,whilethelatteris manuallyselected

for each series.

where

values of the red, green and blue image channel, respectively,

on

: We consider allpixels belonging to the finite set

onlyconsistingofpixelswithanintensityofatmost

90% of the maximum intensity as well as with a minimum in-

tensity above 0. This restricts the transform to nonbright image

regions, since one can assume to have a background consisting

of either bright or black image regions due to fissures or artifi-

cially filled black margins, respectively.

The transform

in (14) consists of the following matrix

product:

,,,

,, and are the mean

(15)

Its individual factors are obtained as follows.

anddenote the offsets (referring to

tively) and are determined as

and , respec-

(16)

and in analogy for

of rank 4.

denotestherotationmatrix.Itisobtainedasmatrixproduct

, withrepresenting the identity matrix

(17)

wherein

(wrt. decreasing order of their corresponding eigenvalues) of

the covariance matrix

of the centered color value data (in

analogy for

and)

is the matrix of sorted eigenvectors

(18)

This real symmetric and orthonormal matrix and the mean

vector represent an estimated multivariate distribution of the

color values in

-space. Supposed

by solving the following eigenvalue problem:

has the (full) rank 3,

(19)

one obtains the three eigenvalues

eigenvectors

reduction to get a tridiagonal form and then based upon this

using the QL algorithm (with implicit shifts).

and their corresponding

. The problem was solved using Householder

denotes the scalings along the principal axes, which are

given as

(20)

The time complexity of the color adaptation is

denoting the number of pixels, the memory complexity is also

. For one image the algorithm runs less than 1 s on a stan-

dard PC for image sizes around megapixels.

Two examples are given in Fig. 6. Although the method is

simple, the results can be considered adequate for our purposes

since mainly staining-related outliers with small fluctuations

are targeted. To the knowledge of the authors, even if simple,

this method is not implemented in standard image manipulation

software.

with

D. Polynomial Nonlinear Registration

This third stage basically does the compensation for

slice-global distortions using polynomial warping [31] based

on sparsely populated displacement vector fields taken from

automatically determined control points. Its basic form is

(21)

while

represents an undistorted reference image and

the already rigidly registered but still distorted coun-

terpart. The unknown coefficients

and

of the

(

th degree for each independent variable)

respectively, can offhand be found once displacement vectors

are available. Those displacement vectors rely on the pair-wise

correlate of partially overlapping image tiles (i.e., subimages).

These tiles are sized 128

128 pixels and overlap 96 pixels

in both directions. 128 was taken as the most appropriate

power of 2 (lots of FFTs have to be computed). The overlap

results in a density of control points of one per 32 pixels which

results in reasonable numbers of control points in the order of

1000. To prepare the tiling, we ensure the image dimensions

to be multiples of the nonoverlapping tile size by adding an

appropriate black margin. Again, we use POMF applied to all

tile pairs

computing the correspondencies to the control points

,

th degree polynomials

and,

and for

(22)

Page 8

BRAUMANN et al.: 3-D-RECONSTRUCTION AND QUANTIFICATION OF CERVICAL CARCINOMA INVASION FRONTS1293

Fig. 6.

registration. Insomecases,appearingasintenselystained as(c)amore obviousimprovementcan be achieved(d).Note thatforthiscomparison(c/d) isnotadjacent

to (a/b), there was just one slice skipped in between in order to be more illustrative. The reference slice for this serial section is depicted in (e).

Color adaptation examples: Usually the adaptation only leads to minor changes as from (a) to (b), whereas (a) is corresponding to Fig. 5(a) after rigid

So, for a tile pair

vector is, e.g.,

Toget

andthe corresponding displacement

.

estimates of

and

properthecoefficients

a multivariate linear regression using a least-squares (LS) error

minimization is done. The multivariate model is

(23)

with

(24)

representing the matrix of displacement vector end points and

(25)

an arranged matrix of all coefficients. For compactness reasons

of the derivation, we have further introduced the matrix

ferred to as design matrix which is build up from

products of combinations of the control point coordinates (i.e.,

the displacement vector start points)

re-

......

...

......

...

...

(26)

where

wise linearly independent row vectors are supposedto represent

istheerrormatrixinwhichtheassumedpair-

Page 9

1294IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 24, NO. 10, OCTOBER 2005

a2-Dmultivariatenormaldistributionwithanexpectationvalue

vector of

as well as a covariance matrix

The general form of the sum of squared errors is

.

(27)

which has to be minimized.

differentiating (27) with respect to

zero in order to find an extremum one achieves

denotes an estimate of

and setting this expression

. Now,

(28)

Reforming (28), one easily can isolate

full rank) and gets

(supposedhas the

(29)

Thissolutionisrepresentingaminimumifthesecondderivative

of (27)

(30)

is positive definite which holds true if

ready assumed above.Thus, the result of (29) can be considered

valid. The two columns

und

and , respectively, which now can be inserted in (21).

For the implementation of (29), even with a simple

Gauß-elimination method and

lems throughout the matrix inversion have occurred. Unwanted

warping effects as reported in [32] have occasionally occured,

but could be managed as we have introduced the following

extension:Supposeda numberof

blocks) we artificially haveadded some

control points (associated with zero-length displacement vec-

tors) and placed them just along the image margin as a “frame”

around the existing grid of control points. Under the condition

that the rigid registration step was successful, this kind of

“framing” is warrantable. This extension did decisively im-

prove the method so that no “collapsing” or other unwanted

warpings thereupon did occur. Further, differing to the previous

rigid registration, this could be accomplished based on the

luminance of the color adapted images.

The time complexity of the polynomial registration is

due to the FFTs with

pixels, the memory complexity is

the algorithm runs about 3 s on a standard PC for image sizes

around megapixels.

Corresponding to Fig. 5 also for this polynomial registration

step we give an illustration of the resulting displacement vector

field for the same typical slice-to-slice transition, see Fig. 7

using line integral convolution.

has the full rank as al-

ofare the estimates of

no singularity prob-

tiles(matching

denoting the number of

. For one image pair

E. Staining-Based Tumor Probability

Now, while two registration steps are done, the serial section

is fairly re-aligned. Most of the slice-to-slice transitions can be

consideredsmoothandmisalignmentsappearmainlylocally.To

Fig. 7.

pair of adjacent images (a) and (b) according to the estimated polynomial

coefficients the resulting displacement vector fields is visualised (c). The

(cyclic) color codes the absolute value of the underlying displacements from

purple-red (high) via blue, cyan, green, yellow to orange-red (low). The

maximum displacement is 84.4 µm, the minimum is 0 µm. What is visible

at the first glance is the inhomogeneity of the field, whereas on the right and

left there are two distinct local maxima of the displacement, in this case the

registrations leads to some contraction from the left/right/upper part toward a

region around below the slice center. Two vortices (upper right and lower left)

occur outside the physical slice and have very small strenghts.

Polynomial registration examples: Corresponding to Fig. 5, for a

treat those remaining registration errors, we subsequently need

to apply yet another registration step. Just like for the previous

registration steps, this one also applies to scalar data. Despite

of taking some luminance-related images, we, therefore, want

to use scalar images highlighting the tumor regions. We gen-

erate such images simply by computing staining-based tumor

probability maps relying on the HE staining applied to all slices

short after sectioning. The probability maps are necessarily re-

quired for threshold-based tumor segmentation. The reason for

swapping these two steps is mainly that by this we can further

Page 10

BRAUMANN et al.: 3-D-RECONSTRUCTION AND QUANTIFICATION OF CERVICAL CARCINOMA INVASION FRONTS1295

Fig. 8.

and polynomially registered images (a) and (c) and have determined the respective tumor probability maps (b) and (d). This example also illustrates the usefulness

of the color correction by what the two probability maps appear qualitatively equal. Note that the suddenly emerging margin was artificially introduced in order to

get image dimensions as multiples of primes up to 5 which is useful for the many FFTs accomplished in the following curvature registration.

Staining-based tumor probability computation examples: Corresponding to the image pairs of Fig. 6 we have taken the same two but now color corrected

attenuate artifacts mainly occurring outside the tumor regions,

which facilitates the final registration step.

Basically, it is required to manually obtain representa-

tive tumor color samples from the respective serial section.

Precisely, we arbitrarily select a number of ten slices equidis-

tantly along the series and let the pathologist draw in the

tumor boundary which provides us both with

triple samples

for tumor and nontumor. Now,

for the tumor probability we adopt normalized color values

color

leading to a projec-

tion onto a sphere sector fitting within the RGB cube, where

. One of the normalized components is

redundant, so we can restrict to use

assume that for the tumor as well as for the nontumor

components follow multivariate normal distributions. So, we

estimate the multivariate densities for both sets

. We further

these

(31)

with

means, respectively. Finally, the probability for a pixel to

exhibit the color of tumor at

and denoting the covariance matrices and

is

(32)

with

.

Fig. 8 illustrates the tumor probability computation for two

images. The results indicate that the previous color-adaptation

(compare Fig. 6) is justifiable.

The time complexity of the staining-based tumor probability

computation is

withdenoting the number of pixels, the

memory complexity is

as well. For one image the algo-

rithm runs about 5 s on a standard PC for image sizes around

megapixels.

F. Curvature-Based Nonlinear Registration

In this processing stage, remaining local registration errors

are diminished and the image-to-image transitions are further

smoothed. What generally has to be computed is a local dis-

placement field as vector function

considered as representation of the mis-

alignments. The nonparametric procedure we are applying for

this nonlinear registration uses a regularization term approxi-

mating local curvature, which was introduced in [33] and re-

cently studied in [20], [34]. While the authors state that the al-

gorithm would include an automatic rigid alignment, with our

data we in fact could not benefit from this effect. This can be

explained as follows: their images strictly cover some complete

object(s)infrontofahomogeneousbackground,sothatitcanbe

more or less assumed to successfully find all correspondences

one-by-one. However, in our image material we do not have

isolated objects with some delimited boundary. In fact, since

the digitized regions usually do not comprise any background

with

Page 11

1296 IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 24, NO. 10, OCTOBER 2005

but generally tissue, around the image margin for a consider-

able portion of the images pair-wise correspondences will be

missing. Therefore, from the viewpoint of this algorithm the

previous registration steps can be considered as preprocessing

in order to drastically improve a priori image-to-image corre-

spondence.

Basically, the distance measure to be minimized for this reg-

istrationstepisthesumofsquareddifferencesoftheimage’sin-

tensities [here, taken as tumor probabilities using (32)]. Again,

assume

represents an undistorted reference image and

the already both rigidly as well as polynomially registered but

still distorted counterpart, while the registration should do the

mapping

(33)

With

(34)

we define a joint registration criterion consisting of the sum of

squared differences

(35)

and the smoothing term

(36)

From the calculus of variations we know that a function

imizing (34) necessarily should be a solution for the Euler-La-

grange equation

min-

(37)

with

(38)

Forthecoupledsystemof4thorderpartialdifferentialequations

[see (37)] an artificial time parameter is introduced as

(39)

with the boundary condition of

image.

To solve (39) the time dependence is discretized using an

implicit midpoint rule for the linear operator

. For the integration over a single time step

gets

being periodic across the

one

(40)

for the propagation from

Green function

to. Defining the

(41)

the solutionfor the next time step is found by

(42)

Denoting the discrete Fourier components

(43)

and using [35, equation (25.3.33)] for the discrete version of the

biharmonic operator, (42) in the Fourier domain is given by

(44)

with

whereand andfor a

grid. When

is used to update . Every time integration step needs a

totaloffourFouriertransformsforthetwocomponentsof

the two components of

. We have chosen periodic boundary

conditions for the nonlinear registration. Other boundary con-

ditions, e.g., with zero displacement on the boundary or zero

normal derivative for

can also be used. It shoud be noted that

we always could find a single transformation without the need

of restarts.

Fig. 9 illustrates the curvature registration for an image pair

using line integral convolution. Concerning the parametrization

of the algorithm, for all specimen we have applied a fixed max-

imum number of 32 iteration steps, whereas

with an iteration time step

of 2.0. The solution is computed

in the Fourier space. We have iterated some certain fix number

of steps and applied both a fixed stepwidth and smoothing co-

efficient. Remember, reference-free registration is an ill-posed

problem, so the difficulty is especially to avoid removing all

differences between two adjacent sections/images. The specific

choice of these mentioned parameters was made as follows: we

is computed the backward Fourier trans-

form

and

was set to 5.0

Page 12

BRAUMANN et al.: 3-D-RECONSTRUCTION AND QUANTIFICATION OF CERVICAL CARCINOMA INVASION FRONTS1297

Fig. 9.

to Figs. 5 and 7, for a pair of adjacent images (a) and (b) the determined

displacement vector field is visualised (c) using line integral convolution.

The (cyclic) color codes the absolute value of the underlying displacements

from purple-red (high) via blue, cyan, green, yellow to orange-red (low).

The maximum displacement is 36.2 µm, the minimum is 0 µm. Compared

to Fig. 7 this displacement vector field is even more inhomogeneous and the

displacements are at most half as far as for the polynomial step.

Curvature-based nonlinear registration example: Corresponding

wanted to have the curvature term get weighted as five times as

the squared differences. The time step and iteration steps were

selected based on visual inspections of tests making sure that no

eye-catching unwanted warpings occur. At the present stage of

our work we did not implement any time dependency of these

mentioned parameters.

To further improve the performance of those nonlinear

schemes, [36], [37] have proposed algorithms what they have

called Pair-and-Smooth registrations which combine geometric

matching with intensity-based registration. This together with a

multigrid implementation will be a future direction of our work.

Because of the FFT-based implementation, the time com-

plexity of the curvature-based nonlinear registration step is

, with denoting the number of pixels, the memory

complexity is

. For one image pair the algorithm runs

about four minutes on a standard PC for image sizes around

megapixels. Comparing this CPU time with the time neces-

sary to do the microtome sectioning, the staining, the manual

digitization, and a number of previous processing steps, we

still consider some minutes acceptable for one image-to-image

transition. With the above mentioned multigrid implementation

(this is ongoing work) we expect a decisive computational

speed-up.

G. Total Variation Filtering

Due to the pixel based color segmentation typically the data

is affected by a significant amount of noise. While this noise

is not essentially affecting the previous registration step, we

consider the necessity for an intermediate processing on the

reconstructed 3-D data step in order to facilitate the succeeding

thresholding-based segmentation. Non-linear filters are in

general much better in preserving image structures compared

to linear ones. So, e.g., median filtering perfoming a ranking

operation will keep edges but remove outliers while, on the

other hand, linear binomial filtering will damp both. However,

while the median filter [27] is appropriate in case of simple

salt-and-pepper noise, its homogenizing properties remain

limited. More sophisticated schemes like nonlinear diffusion

filtering [38], [39] have been proposed, however these basically

require some certain stopping criterion, otherwise the image

structures get lost.

Instead, we have decided to apply nonlinear total variation

filtering [40]. This filter minimizes the functional

(45)

with

the probability that at

found. Let

white noise

, for the scalar 3-D imagethat contains

a cancer voxel can be

be the original noisy image with Gaussian

exhibiting the following properties:

That

the Euler-Lagrange equation

minimizing (45) generally can be obtained solving

(46)

For this extremely nonlinear equation several solution methods

are known. Since the considered volume data are very large,

memory intensive methods are not feasible, because a nonlinear

solver would require several gigabytes of temporary memory.

Page 13

1298 IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 24, NO. 10, OCTOBER 2005

Fig. 10.

transformed by the curvature-registration, and now were TV filtered attaining an edge-preserving smoothing as depicted in (b) and (d). Note that the TV filtering

is applied as 3-D operation, which is why we show this adjacent pair here (the 3-D effect can be best seen in the nontumour background).

Total variation filtering examples: This example again refers to the same pair as in Fig. 9. The two images of the adjacent slices (a) and (c) were

However, an appropriate solution method with low memory de-

mands was proposed by Osher et. al. [40]. It transforms the

problem into a time dependent problem for

. So, instead of (46) we apply

with

(47)

with

ishes, the equation becomes singular so that the gradient must

be regularized as

. In regions where the gradient van-

(48)

(49)

considering

. For the discrete solution

(50)

the summation runs over all next neighbors of

noted as

. For inner pointsin thevolume a 6-neighborhood

stencil is used. Boundary points cover a reduced neighborhood

as respective grid points exist. This yields a nonlinear filter for

every mesh point

which is de-

(51)

with

Thisfilteractsasanedgepreservinglowpass.Duringthecalcu-

lation one simultaneously needs to store at least three data sets

, and . The main advantage of the filter is its rel-

atively quick convergence toward the denoised result. The only

free parameter is . Its choice is of importance for the denoising

quality. Following [40] we use

(52)

withthestandarddeviationofthewhitenoise

of mesh points

. After 5–10 iterations a new

and used for the update of the

The time complexity of the total variation filtering is

with denoting the number of voxels, and also the memory

complexity is

. For one image series with typically 300

sections the algorithm runs about half an hour on a standard PC

for imagesizes around megapixels.Thisstep is a 3-Doperation,

so it may have exorbitant RAM requirements, since

andhave to be accessible simultaneously.

andthenumber

is computed

.

,

Page 14

BRAUMANN et al.: 3-D-RECONSTRUCTION AND QUANTIFICATION OF CERVICAL CARCINOMA INVASION FRONTS1299

Fig. 11.Tumor segmentation example: The two images (a) and (b) show the results after thresholding the TV filtered images in Fig. 10(b) and (d), respectively

Fig. 12.(left) Multiplanar reconstruction and (right) 3-D surface rendering of the invasion front for specimen 8.

Fig. 10 illustrates the 3-D TV filtering effect on a pair of ad-

jacent images.

H. Tumor Segmentation

In this final reconstruction step, just the TV filtered volume

data is binarized. According to (32) an illustrative criterion for

thresholding is where the two estimated densities for tumor and

nontumor exhibit the same magnitude so that the tumor proba-

bility is 0.5. Let

represent the previously TV filtered scalar

image, the binarized result

is obtained as

for

for.

(53)

Both time and memory complexity of thresholding of course is

withdenoting the number of voxels. It takes a fraction

of 1 s on standard PC hardware.

Fig.11givesexamplescorrespondingtotheresultsofFig.10.

III. INVASION FRONT QUANTIFICATION

Once the smoothing by means of total variation filtering and

the segmentation was accomplished, following the 3-D recon-

struction process the tumor invasion front within the volume

data is going to be assessed. Hence, the invasion front firstly

is visualized and subsequently quantified.

A. Three-Dimensional Tumor Visualization

What is of basic interest is the topology of the invasion front.

One of the interesting questions at hand is how the tumor inva-

sion front is shaped. Another question is the presence of pos-

sibly separated tumor islets apart from the main tumor. Some

virtually have occured but turned out to have direct contact to

the data set outskirt. These were sorted out since it cannot be

decided if separated or not. The rest, however, was not straight-

forward to be verified or falsified, due to the limitations of the

HE staining. HE in fact is just enhancing image contrast with

respect to the averaged local cell kernel density. In tumor cells,

thekernelsarebasicallyenlarged.Inexceptionalcases,misclas-

sifications might occur as, e.g., for smaller inflamational cells

or some other dense tissue parts. We have let the pathologist

check all suspected tumor islets using a much larger magnifica-

tion (40

instead of 1.25) but got none of them verified to

consist of tumor. What has remained for all our specimen was

one large connected tumor segment, a kind of “massif” VOI of

the tumor invasion front.

Therefore, in order to give a 3-D illustration of the recon-

structed tumor invasion, we do a surface rendering applying the

Page 15

1300 IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 24, NO. 10, OCTOBER 2005

Fig. 13.Views onto the 3-D surface renderings of tumor invasion fronts, part I.

thresholdingcriterionfrom(53).Therenderingofthetumorsur-

faces uses the well-known algorithm from [41] with the mesh

displacement modification [42], [43]. A detailed discussion of

rendering algorithms can be found in [44].

The gallery of tumor invasions of our 13 specimens is shown

in the Figs. 12–14. These are the first-ever visualizations of

a solid tumor’s invasion front with a resolution of

The renderings have been generated using MathGL3d [45],

an OpenGL-based interactive viewer for Mathematica’s 3-D

graphics. Typical numbers of (potentially nonconnected) sur-

face polygons occur from

series with 300 sections and image sizes around megapixels

10 µm.

. For a typical image