Regression-based label fusion for multi-atlas segmentation.

Abstract

Automatic segmentation using multi-atlas label fusion has been widely applied in medical image analysis. To simplify the label fusion problem, most methods implicitly make a strong assumption that the segmentation errors produced by different atlases are uncorrelated. We show that violating this assumption significantly reduces the efficiency of multi-atlas segmentation. To address this problem, we propose a regression-based approach for label fusion. Our experiments on segmenting the hippocampus in magnetic resonance images (MRI) show significant improvement over previous label fusion techniques.

Full text

Page 1

Regression-Based Label Fusion for Multi-Atlas Segmentation
Hongzhi Wang, Jung Wook Suh, Sandhitsu Das, John Pluta, Murat Altinay, Paul Yushkevich∗
PICSL, Department of Radiology, University of Pennsylvania
Abstract
Automatic segmentation using multi-atlas label fusion
has been widely applied in medical image analysis. To sim-
plify the label fusion problem, most methods implicitly make
a strong assumption that the segmentation errors produced
by different atlases are uncorrelated. We show that violat-
ing this assumption significantly reduces the efficiency of
multi-atlas segmentation. To address this problem, we pro-
pose a regression-based approach for label fusion. Our ex-
periments on segmenting the hippocampus in magnetic res-
onance images (MRI) show significant improvement over
previous label fusion techniques.
1. Introduction
Atlas-based segmentation methods label an unknown
image by referring to a labeled image through deformable
registration. Due to its wide applicability, i.e., one can ap-
ply it to segment any anatomical structures by registering
it to an atlas, and the wide availability of registration tools,
atlas-based segmentation has been one of the most popular
techniquesusedinmedicalimageanalysis. Asanextension,
multi-atlas based segmentation makes use of more than one
reference atlas to compensate potential errors imposed by
using any single atlas. As extensive empirical studies have
verified in the recent literature, e.g. [9, 2], it is more accu-
rate than single atlas based segmentation.
Errors in atlas-based segmentation can be attributed to
dissimilarity in anatomy and/or appearance between the at-
las and the target image. Recent research has been focusing
on addressing this problem. For instance, research has been
done on optimally constructing a single atlas from training
data that is the most representative of a population [11].
Constructing multiple representative atlases from training
data has been considered as well, and usually works better
than single-atlas based approaches. Multi-atlas construc-
tion can be done either by constructing one representative
atlas for each mode obtained from clustering training im-
ages [1] or by simply selecting the most relevant atlases for
∗This work was supported by the Penn-Pfizer Alliance grant 10295
(PY) and the NIH grants K25 AG027785 (PY) and R01 AG037376 (PY).
the unknown image on the fly [18]. Either way, one needs
to combine the segmentation results obtained by referring
to different atlases to produce the final solution.
To combine segmentations, most label fusion techniques
implicitly assume that different atlases produce uncorre-
lated errors with respect to the target segmentation [13].
Under this assumption, according to the law of large num-
bers, errors produced by using any single atlas can be ef-
ficiently reduced when multiple atlases are used. Further-
more, when different atlases produce different segmenta-
tion accuracies, more efficiency can be achieved by assign-
ing higher non-negative weights to atlases that are expected
to produce more accurate results. In this regard, image
similarity-based local weighting has been shown to be the
mostaccuratelabelfusionstrategy[2,21]. Althoughtheun-
correlated error assumption significantly simplifies the label
fusion problem, it is often invalid in practice [9]. To address
this problem, our main contribution is to propose a regres-
sion technique for label fusion.
We apply our method to segment the hippocampus from
MRI and show significant improvements over similarity-
based local weighting label fusion.
2. Label fusion based multi-atlas segmentation
Atlas-based segmentation is motivated by the observa-
tion that segmentation strongly correlates with image ap-
pearance. A target image can be segmented by referring
to labeled images that have similar image appearance. Ac-
cordingly, an atlas contains a reference image and its seg-
mentation. After registering an atlas’s reference image to
the target image via deformable registration, one can di-
rectly transfer labels from the atlas to the target image.
Segmentation errors produced by this method are mainly
due to registration errors, i.e. that the registration associates
wrong regions from an atlas to the target image. Under the
assumption that the segmentation errors produced by using
different atlases are uncorrelated, errors obtained from any
single atlas can be efficiently corrected when multiple at-
lases are used. For example, the majority voting method
[7, 12] simply counts the vote for each label from each reg-
istered atlas and chooses the label receiving the most votes.
Let 푇퐹be a target image and 퐴1= (퐴1
퐹,퐴1
푆),...,퐴푛=
1113

Page 2

(퐴푛
warped atlas image and the corresponding warped manual
segmentation. The majority voting algorithm produces the
final segmentationˆ푇푆by:
퐹,퐴푛
푆) be 푛 registered atlases. 퐴푖
퐹and 퐴푖
푆denote the 푖푡ℎ
ˆ푇푆(푥) = argmax푙∈{1...퐿}
푛
∑
푖=1
퐼(퐴푖
푆(푥) = 푙)
(1)
where 푙 indexes through labels and 퐿 is the number of la-
bels. 푥 indexes through image pixels. 퐼(⋅) is an indicator
function that outputs 1 if the input is true and 0 otherwise.
Majority voting makes a strong assumption that differ-
ent atlases produce equal quality registrations for the target
image. To address this problem, recent work focuses on de-
veloping more accurate registration quality estimations. In
a broader context, most of these label fusion techniques can
be modeled under the maximum a posterior (MAP) infer-
ence framework [21]. The posterior label probability given
the target image can be estimated by:
ˆ 푝(푙∣푇퐹,푥) ≈
푛
∑
푖=1
푝(퐴푖∣푇퐹,푥)푝(푙∣퐴푖,푥)
(2)
푝(푙∣퐴푖,푥) is the label posterior probability defined by atlas
퐴푖. Typically, for deterministic atlases that have one unique
label for every image location, 푝(푙∣퐴푖,푥) is 1 if 푙 = 퐴푖
and 0 otherwise. Continuous label posterior probability can
be used as well especially when probabilistic atlases are
involved. Even for deterministic atlases, continuous label
posterior probability still can be derived, see [21] for some
examples. 푝(퐴푖∣푇퐹,푥) is the probability that atlas 퐴푖has
the correct label for 푇퐹 at location 푥, which can be in-
terpreted as a weight assigned to the atlas. As mentioned
above, this term mainly reflects the quality of registration.
One way to estimate this probability is based on local im-
age similarity. When summed square distance (SSD) and a
Gaussian model is used [21], it can be estimated by:
∑
where 풩(푥) defines a neighborhood centered around 푥. In
our experiment, we use a cubic neighborhood definition,
specified by a radius 푟. The radius specifies the Manhattan
distance from the center of the cubic region, i.e. the voxel
being considered, to the neighborhood boundary. Hence, an
image patch contains (2푟 + 1)3voxels. Note that the reg-
istration quality is estimated locally, which accommodates
nonuniform registration qualities over the entire image. The
inverse distance weighting has been applied to estimate this
term as well [2]:
⎡
푦∈풩(푥)
푆(푥)
푝(퐴푖∣푇퐹,푥) ∼ 푒푥푝(−
푦∈풩(푥)
[푇퐹(푦) − 퐴푖
퐹(푦)]2/휎) (3)
푝(퐴푖∣푇퐹,푥) ∼
⎣
∑
(푇퐹(푦) − 퐴푖
퐹(푦))2
⎤
⎦
−훽
(4)
where 휎 and 훽 are model parameters.
To reduce the noise effect, one can spatially smooth the
weightsforeachatlas. Inourexperiment, weusemeanfilter
smoothing with the smoothing window 풩, the same neigh-
borhood used for computing the similarity-based weights.
After smoothing, the weights are re-normalized s.t. for any
푥,∑푛
3. Problems raised by violating the assumption
of uncorrelated errors
푖=1푝(퐴푖∣푇퐹,푥) = 1.
Given a certain registration accuracy at location 푥, the
label posterior probability produced by any single warped
atlas can be modeled as obtained from the true label poste-
rior probability plus some error [24], i.e.:
푝(푙∣퐴푖,푥) = 푝(푙∣푇퐹,푥) + ℬ(퐴푖,푥) + 휖(퐴푖,푥)
where ℬ(퐴푖,푥) is the bias and 휖(퐴푖,푥) is a zero-mean ran-
dom error. Averaging over all registrations that produce the
same error distribution, the error produced by 퐴푖at 푥 can
be quantified by:
[(푝(푙∣퐴푖,푥) − 푝(푙∣푇퐹,푥))2]
where 푉 is the variance of the random error. After combin-
ing multiple atlases, the combined error is:
[
=
푝(퐴푖∣푇퐹,푥)2퐸
(5)
퐸
= ℬ(퐴푖,푥)2+푉 (휖(퐴푖,푥))
(6)
퐸(ˆ 푝(푙∣푇퐹,푦) − 푝(푙∣푇퐹,푦))2]
∑
푛
∑
The second term measures the error correlation between at-
lases. Under the assumption that the errors produced by
different atlases are uncorrelated, the correlation terms dis-
appear. By properly assigning larger weights to better reg-
istered atlases, one can significantly reduce uncorrelated er-
rors through label fusion. For instance, if all atlases produce
equal quality label probabilities with the same error 푒, ap-
plying uniform weights, 푝(퐴푖∣푇퐹,푥) =
combined error1
In reality, segmentation errors produced by different at-
lases are often correlated. When the errors produced by two
atlases are negatively correlated, i.e. they tend to make op-
posite mistakes, their correlation term is negative. Com-
bining them is even more effective in reducing errors. The
problem arises when the errors are positively correlated, i.e.
different atlases tend to make similar errors. Intuitively,
when several atlases produce the same wrong label, it is
easier for the wrong label to accumulate higher posteriors
푛
푖=1
[(푝(푙∣퐴푖,푥) − 푝(푙∣푇퐹,푥))2]
+ 2
푖,푗=1
푝(퐴푖∣푇퐹,푥)푝(퐴푗∣푇퐹,푥)ℬ(퐴푖,푥)ℬ(퐴푗,푥) (7)
1
푛, produces the
푛푒.
1114

Page 3

than the correct label, therefore causing errors. Theoret-
ically, when different atlases tend to make similar errors,
their error correlation terms are positive, which increases
the combined error (see (7)). In the worst case, when all
error correlation terms are positive and are greater than the
error produced by the best single atlas, for the best label
fusion result, zero weights have to be applied to all atlases
except the best one. The multi-atlas method reduces to a
single-atlas approach.
4. Regression-based label fusion
When errors are positively correlated, simply combin-
ing atlases by assigning higher weights to better registered
atlases becomes inefficient. To address this problem, we
propose to estimate the posterior label probability through
regression.
Given the image appearance of a target image at a local
patch 풩(푥), our goal is to estimate the label posterior distri-
bution at 푥, i.e. 푝(푙∣푇퐹(풩(푥))). This is a high dimensional
function of local appearance. Each registered atlas provides
one observed value for this function 푝(푙∣퐴푖
푝(푙∣퐴푖,푥) at one data point 퐴푖
to estimate a function’s values at unobserved data points is
interpolation or regression analysis. Note that the image
similarity-based weighting methods can be interpreted as
applying the nearest neighbor interpolation [19]. However,
as we show above, nearest neighbor interpolation does not
properly handle correlated errors.
Sinceweworkonhighdimensionalregressionproblems,
over-fitting is a key issue to be addressed. Hence, we study
low order polynomial regression for label fusion in this pa-
per. To this end, we model the label posterior probability
as a second order polynomial function of image intensities
from the local patch:
∑
퐹(풩(푥))) =
퐹(풩(푥)). One common way
푝(푙∣푇퐹(풩(푥))) =
푦∈풩(푥)
[훽푥
1(푦)푇퐹(푦) + 훽푥
2(푦)푇2
퐹(푦)]
(8)
훽푥
respectively. Since the regression model is anatomy depen-
dent, the regression weights may be different for different
locations. For a simple notation, we rewrite the equation:
1(⋅) and 훽푥
2(⋅) are first order and second order weights,
푝(푙∣푇퐹(풩(푥))) = 훽푡푇
1(푦1);...;훽푥
[푇퐹(푦1);...;푇퐹(푦푚);푇퐹(푦1)2;...;푇퐹(푦푚)2], where 푦푖 ∈
풩(푥) for 푖 = 1,...,푚 and 푚 is the size of the local im-
age patch 풩(푥). 푡 stands for matrix transpose.
Using the observations obtained from the registered at-
lases, one can estimate 훽 by solving the following linear
equations:
(9)
훽 = [훽푥
1(푦푚);훽푥
2(푦1);...;훽푥
2(푦푚)] and 푇 =
훽푡[퐴퐹;(퐴퐹)2] = 훽푡퐴 = [푃푙]푡
(10)
퐴 = [퐴퐹;(퐴퐹)2] and 퐴퐹 = [푎1
the local appearances from the registered atlases. 푎푖
vector of size 푚 × 1 that stores the local image intensity
patch from 퐴푖
tor of the corresponding label posterior probabilities from
all atlases. When enough atlases are used, we can solve 훽
via linear least square fitting:
퐹,...,푎푛
퐹] is the matrix of
퐹is a
퐹. 푃푙= [푝(푙∣퐴1,푥);...;푝(푙∣퐴푛,푥)] is the vec-
훽 = (퐴퐴푡)−1퐴푃푙
(11)
Note that the matrix 퐴퐴푡has dimension 2푚×2푚. In prac-
tice, the number of atlases used are usually limited. For
instance, for small image patches with dimension 3×3×3
or 5 × 5 × 5, the patch size is usually greater than the num-
ber of atlases used for label fusion. In such cases, the matrix
does not have a full rank and the matrix inverse is not well
defined. To address this problem, we propose the following
approximation to implement the regression analysis.
푝(푙∣푇퐹(풩(푥))) =
≈
=
훽푡푇
훽푡퐴(퐴푡퐴)−1퐴푡푇
[푃푙]푡(퐴푡퐴)−1퐴푡푇
(12)
(13)
(14)
(14) is obtained by substituting (10).
퐴(퐴푡퐴)−1퐴푡is an identity matrix. (14) gives the accu-
rate solution. When 푛 < 2푚, (14) approximates the label
posterior probability via least square fitting. With this ap-
proximation, we transfer the problem of solving inverse of a
large matrix [퐴퐴푡]2푚×2푚into solving inverse of a smaller
matrix [퐴푡퐴]푛×푛. If this smaller matrix is still near singu-
lar, one can add an identity matrix weighted by a small pos-
itive number 휆 to it. Adding a weighted identity matrix can
be interpreted as penalizing large weights that exploit corre-
lations beyond some level of precision in the data sampling
process [20].
In (14), each registered atlas has one single weight com-
puted by 푤푛×1= (퐴푡퐴)−1퐴푡푇, i.e.:
When 푛 = 2푚,
푝(푙∣푇퐹(풩(푥))) ≈
푛
∑
푖=1
푤(푖)푝(푙∣퐴푖,푥)
(15)
Hence, as previous label fusion techniques, our regression-
based method applies weighted-voting as well. One main
difference is that the weights computed by our method
can be either positive or negative, while the weights used
by other similarity-based weighting approaches are non-
negative. When the errors produced by different atlases are
positively correlated, applying negative weights to some of
the atlases allows to cancel out the positively correlated er-
rors between these negatively weighted atlases and other
positively weighted atlases, which results in smaller com-
bined error (see equation (7)). Furthermore, most similarity
based weighting methods rely on some pre-selected weight-
ing models, e.g. (3). The optimal model parameter, e.g. the
1115

Page 4

standard deviation in the Gaussian model, is usually deter-
mined empirically. By contrast, our method automatically
determines the optimal weights.
To see that our approximation applies linear least square
fitting, note that 푤 is the linear least square fitting solution
of the following linear equations when 푛 < 2푚:
푛
∑
We compute weights such that both the original target im-
age and itssquared image can be linearlyinterpolated by the
warped atlas reference images and their squared images, re-
spectively. In other words, the weighted local appearance of
the warped atlases center around the local appearance of the
target image. Since atlas-based segmentation relies on the
assumption that segmentations are strongly correlated with
appearances, putting the local target image at the center of
the weighted warped local atlas images can be interpreted
as an effort of putting the local segmentation of the target
image at the center of the weighted warped local atlas seg-
mentations, therefore the weighted segmentation errors are
uncorrelated.
푖=1
푤(푖)[퐴푖
퐹(풩(푥));퐴푖
퐹(풩(푥))
2] = [푇퐹(풩(푥));푇퐹(풩(푥))2]
(16)
4.1. Error analysis
Since we attempt to model the relationship between im-
age appearance and segmentation labels, the errors pro-
duced by our method can be categorized into two classes:
1) the appearance-label model selection error, i.e., the er-
rors caused by employing the polynomial function to model
the appearance-label relationship and 2) the model fitting
error, i.e., the target image can not be perfectly represented
as a linear combination of the atlas reference images.
The model fitting errors usually can be alleviated by em-
ploying enough atlases for label fusion. The actual number
of atlases required for accurate model fitting depends on
the complexity of the appearance-label model. Typically,
more complex appearance-label models require more at-
lases. Hence, when limited atlases are used, to avoid signifi-
cant model fitting errors one should use simple appearance-
label models, like ours. However, segmentation labels usu-
ally correlate to image appearances in a more ambiguous
fashion. Due to the factors such as the limitations in the
imaging process and employing local image appearance for
label fusion, most local appearance patterns may have more
than one plausible segmentation and vice versa.
these circumstances, the performance of our method also
depends on how well the polynomial model captures the re-
lationship between appearances and segmentation labels.
Under
4.2. A toy example
Fig. 1 demonstrates applying our regression approach
to a toy example in 2D. In this binary segmentation prob-
lem, we have five atlases. After registration, suppose that
Figure 1. Illustration of label fusion on a toy example. The target
segmentation, 푇, is shown on the top left corner, followed by five
registered atlases, 퐴1to 퐴5. For simplicity, we use binary label
posteriors and assume that the images have the same appearance
patterns as the segmentations. Error correlations are all positive,
indicating a strong positive correlation between the label errors
produced by the atlases. As a result, majority voting (MV) pro-
duces a result biased towards atlas 퐴1and 퐴4. Similarity-based
Gaussian weighting label fusion (LW) reduces to single-atlas seg-
mentation, i.e. only the most similar atlas, 퐴3, has a non-zero
weight. To compensate the overall bias towards 퐴4, when linear
regression is applied, only 퐴4receives a negative weight to can-
cel out the consistent errors among all atlases. For LW and our
method, the atlas weights computed for the center pixel are also
used for non-center pixels in this example.
the foreground label patterns of the target image and the
warped atlases on a local patch with size 3 × 3 have the
structures shown in Fig. 1. For simplicity, we consider de-
terministic atlases. The label posterior produced by each
atlas is either 1 or 0. The registration quality is assumed to
be constant within the patch. Furthermore, suppose that the
image appearances have the same patterns as the foreground
labels such that image appearances indeed linearly correlate
to segmentation labels.
The pairwise correlations of label posterior errors be-
tweentheatlasesareallpositive, indicatingaconsistentbias
in the atlases. Also note that 퐴1and 퐴4have the largest
combined inter-atlas error correlations, indicating that they
contain the most common segmentation errors produced by
all atlases. As a result, the segmentation obtained from ma-
jority voting has a structure more similar to 퐴1and 퐴4, with
five mislabeled pixels. Due to the strong error correlation,
the optimal image similarity based weighting label fusion
approachreducestothesingle-atlasbasedsegmentation, i.e.
applying zero weights to all atlases except the most simi-
lar atlas 퐴3. Applying our regression technique, only atlas
퐴4receives a negative weight. This allows to cancel some
1116

Page 5

of the consistent errors among all the atlases. Correspond-
ingly, our method gives the best solution on this local patch
with two mislabeled pixels. Since the polynomial model ac-
curately captures the appearance-label relationship for this
example, the errors are purely due to the model fitting error.
4.3. Improving the robustness of model fitting
Since we face a high dimensional regression problem,
the computed weights may be sensitive to outliers/noises
caused by registration errors. In this section, we propose
two techniques to reduce the model fitting error.
4.3.1 Similarity-ranking based regularization
Note that our regression approach represents the target im-
age as a linear combination of the registered atlas images. It
is reasonable to expect that the atlases whose reference im-
ages are more similar to the target image will contribute dif-
ferently from those whose reference images are less similar
to the target image. Such similarity-ranking based weight-
ing strategy has been explored in the classifier combining
literature, [10, 16]. To alleviate the over-fitting problem, we
propose to accommodate this prior knowledge to regularize
our regression weights. To this end, we rewrite equation
(16) as follows:
푇 =
푛
∑
푖=1
푤(푖)퐴휋(푖)
(17)
where 휋(1),...,휋(푛) is a permutation of 1,...,푛. 퐴휋(푖)=
[퐴휋(푖)
퐹퐹
(풩(푥)))2] is the appearance vector
of the 푖푡ℎ most similar atlas to the target image 푇 =
[푇퐹(풩(푥));푇퐹(풩(푥))2], measured by SSD. Now, the re-
gression weights assigned to atlases are associated with
their similarity rankings.
To estimate these similarity-ranking based weights,
again we use the atlases. Since all atlases are registered
to the same target image, these registrations also estab-
lish the pairwise correspondence between the atlases via
the target image. Hence, the image patches 푇퐹(풩(푥)) and
퐴푖
structure. Followingthelogicthatonecanestimatethelabel
posterior probability for the target image using the atlases,
given the label posterior probability of the target image, we
should be able to regress the label posterior probability for
any atlas using the target image and the remaining atlases as
well. Forthispurpose, westillneedrepresentthewarpedat-
las reference image as a linear combination of the target im-
age and the remaining warped atlas reference images. For a
simpler notation, let the appearance vectors of the target im-
age and the warped atlases be represented by 퐹1,...,퐹푛+1.
Similar to equation (17), any image 퐹푗can be represented
(풩(푥));(퐴휋(푖)
퐹(풩(푥)) for 푖 = 1,...,푛 all represent the same anatomy
as a similarity-ranking based linear combination of the re-
maining images as follows:
퐹푗=
푛
∑
푖=1
푤(푖)퐹휋푗(푖)
(18)
where 퐹휋푗(푖)is the 푖푡ℎmost similar image to 퐹푗from the
remaining images and 휋푗(1),...,휋푗(푛) is a permutation of
1,...,푗 − 1,푗 + 1,...,푛 + 1.
Under this formulation, each image provides a set of
linear constraints for solving the weights 푤. Simultane-
ously solving all these linear equations produces the de-
sired weights. The main advantage of employing similarity-
ranking based weights is that it significantly increases the
training data through a cross validation process, therefore
the results are less sensitive to noises.
4.3.2 Reducing extrapolation via local searching
When the warped atlas images are scattered around the tar-
get image, the target image is located within the reference
range defined by the atlases. The posterior label distribu-
tion can be reliably estimated via interpolating label dis-
tributions obtained from the atlases. When the warped at-
lases are strongly biased, most warped atlases deviate from
the target segmentation in a consistent way and the tar-
get image may be completely outside the reference range.
Our method is more akin to extrapolation. Extrapolation
is based on a strong assumption that the fitted regression
model is still valid outside the reference range defined by
the training samples. Hence, extrapolation is usually more
prone to errors than interpolation. To alleviate such unreli-
ability caused by extrapolation, we propose to use the most
similar image patches from each atlas for label fusion.
Note that the goal of image-based registration is to cor-
respond the most similar image patches between the regis-
tered images. However, the correspondence obtained from
registration may not give the maximal similaritybetween all
corresponding regions. For instance, deformable image reg-
istration usually needs to balance the image matching con-
straint and the regularization prior on deformation fields.
A global regularization constraint on the deformation fields
is necessary to clarify the ambiguous appearance-label re-
lationship arising from employing small image patches for
matching. However, enforcing a global regularization con-
straint on the deformation fields may compromise the local
image matching constraint. In such cases, the correspon-
dence that maximizes the appearance similarity between the
warped atlas and the target image may be within a small
neighborhood of the registered correspondence.
Motivated by this observation, instead of using the orig-
inal registered correspondence, we propose to remedy the
risk of extrapolation by searching for the correspondence,
1117

Page 6

that gives the most similar appearance matching, i.e. min-
imal SSD distance, within a small neighborhood centered
around the registered correspondence in each atlas. The lo-
cally searched optimal correspondence is:
푥푖= argmin푥′∈풩′(푥)
[퐴푖
퐹(풩(푥′)) − 푇퐹(풩(푥))]2
(19)
푥푖is the location from 푖푡ℎwarped atlas with the best im-
age matching for location 푥 in the target image within the
local area 풩′(푥). Again, we use a cubic neighborhood def-
inition, specified by a radius 푟푠. 풩′and 풩 may represent
different neighborhoods and they are the only free parame-
ters in our method. Instead of the registered corresponding
patch 퐴푖(풩(푥)), we apply the searched patch 퐴푖(풩(푥푖))
to produce the fused label at 푥 for the target image, i.e.
(2) becomes ˆ 푝(푙∣푇퐹,푥) ≈
Note that a similar local searching technique was recently
proposed by [5] to reduce noise effects for similarity-based
local weighting label fusion.
With more similar data for regression, the local search-
ing can significantly reduce the risk of extrapolation. In
this regard, larger searching neighborhoods are more de-
sirable. However, using larger local searching windows
also compromises the regularization prior on the deforma-
tion fields. This drawback makes the appearance-label rela-
tionship more ambiguous on local patches, resulting greater
model selection errors. Hence, avoiding model-fitting er-
rors and avoiding model-selection errors can not be satis-
fied simultaneously. It is reasonable to expect an optimal
searching range that balances these two factors.
∑푛
푖=1푝(퐴푖∣푇퐹,푥푖)푝(푙∣퐴푖,푥푖).
5. Experiments
In this section, we apply our method to segment the
hippocampus using T1-weighted MRI. The hippocampus
plays an important role in memory function. Macroscopic
changes in brain anatomy, detected and quantified by mag-
netic resonance imaging (MRI), consistently have been
shown to be predictive of Alzheimer’s disease (AD) pathol-
ogy and sensitive to AD progression [22]. Accordingly, au-
tomatic hippocampus segmentation from MR images has
been widely studied.
We use the data in the Alzheimer’s Disease Neuroimag-
ing Initiative (ADNI, www.loni.ucla.edu/ADNI). Our study
is conducted using 3 T MRI and only includes data from
mild cognitive impairment (MCI) patients and controls.
Overall, the data set contains 139 images (57 controls and
82 MCI patients). The images are acquired sagitally, with
1 × 1 mm in-plane resolution and 1.2 mm slice thickness.
To obtain manual segmentation, we first apply a landmark-
guidedatlas-basedsegmentationmethod[17]toproducethe
initial segmentation for each image. Each fully-labeled hip-
pocampus is manually edited by one of the authors follow-
ing a previously validated protocol [8].
For cross-validation evaluation, we randomly select 20
images to be the atlases and another 20 images for testing.
Image guided registration is performed by the Symmetric
Normalization (SyN) algorithm implemented by ANTS [3],
which was a top performer in a recent evaluation study [14],
between each pair of the atlas reference image and the test-
ing image. The cross-validation experiment is repeated 10
times. In each cross-validation experiment, a different set
of atlases and testing images are randomly selected from
the ADNI dataset.
We focus on comparing with similarity-based local
weighting methods, which are shown to be the most ac-
curate label fusion methods in recent experimental studies,
e.g. [2, 21]. We use majority voting (MV) and the STA-
PLE algorithm [23] to define the baseline performance. For
each method, we use binary label posteriors obtained from
the deterministic atlases. For similarity-based label fusion,
we apply Gaussian weighting (3) (LWGau) and inverse dis-
tance weighting (4) (LWInv).
Our method has two parameters, 푟 for the local appear-
ance window used in regression analysis, 푟푠for the local
searching window used in reducing the risk of extrapola-
tion. For each cross-validation experiment, the parame-
ters are optimized by evaluating a range of values (푟 ∈
{1,2,3,4}; 푟푠 ∈ {0,1,2,3}) using the atlases in a leave-
one-out cross-validation strategy. We measure the average
overlap between the automatic segmentation of each atlas
obtained via the remaining atlases and the reference seg-
mentation of that atlas, and find the optimal parameters that
maximize this average overlap. Similarly, The optimal lo-
cal searching window and local appearance window are de-
termined for LWGau and LWInv as well. In addition, the
optimal model parameters are also determined for LWGau
and LWInv, with the searching range 휎 ∈ [0.05,0.1,...,1]
and 훽 ∈ [0.5,1,...,10], respectively.
For robust image matching, instead of using the raw im-
age intensities, we normalize the intensity vector obtained
from each local image intensity patch such that the normal-
ized vector has zero mean and unit variance. To reduce the
noise effect, we spatially smooth the weights computed by
each method for each atlas. We use mean filter smooth-
ing with the smoothing window 풩, the same neighborhood
used for local appearance patches.
Fig. 2 shows some parameter selection experiments for
LWGau in the first cross-validation experiment. The results
are quantified in terms of Dice overlaps [6] between auto-
matic and reference segmentations of the atlases. The Dice
overlap between two regions, 퐴 and 퐵, measures the vol-
ume consistency as
periment, the optimal parameters for LWGau are 휎 = 0.05,
푟=2, 푟푠 = 2. Note that local searching only slightly im-
proves the performance for LWGau. Similar results are ob-
served for LWInv as well.
2∣퐴∩퐵∣
∣퐴∣+∣퐵∣. For this cross-validation ex-
1118

Page 7

0.10.20.30.40.5 0.6 0.7 0.80.91
0.8
0.82
0.84
0.86
0.88
0.9
0.92
sigma
Dice overlap
Gaussian local weighting (r=2,rs=0)
123
0.82
0.83
0.84
0.85
0.86
0.87
0.88
0.89
0.9
0.91
0.92
Local appearance window radius (r)
Dice overlap
Gaussian local weighting (rs=0,sigma=0.05)
0123
0.82
0.83
0.84
0.85
0.86
0.87
0.88
0.89
0.9
0.91
0.92
Local searching radius (rs)
Dice overlap
Gaussian local weighting (r=2,sigma=0.05)
Figure 2. Visualizing some of the parameter selection experiments
for LWGau using leave-one-out on the atlases for the first cross-
validation experiment. The figures show the performance of LW-
Gau with respect to the Gaussian weighting function (left), local
appearance window (middle) and local searching window (right),
respectively when the other two parameters are fixed (the fixed
parameters are shown in the figure’s title).
0123
0.878
0.88
0.882
0.884
0.886
0.888
0.89
0.892
0.894
Local searching radius (rs)
Dice overlap


ours, r=1
ours, r=2
ours, r=3
ours, r=4
Figure 3. Leave-one-out performance by our method on the atlases
for the first cross-validation experiment when different appearance
and searching windows are used.
For our regression-based label fusion, we apply a condi-
tioning identity matrix with weight 휆=0.01. Fig. 3 shows
the performance of our method when applied on the atlases
in a leave-one-out fashion in the first cross-validation exper-
iment. Comparing to LWGau, local searching yields more
improvement for our method. Most improvement is ob-
tained by applying searching windows with 푟푠 = 1. This
result indicates that by using more similar patches for re-
gression, local searching significantly reduces the risk of
extrapolation. Overall, our method produces ∼ 1% Dice
improvement over LWGau and LWInv on the atlases in this
cross-validation experiment.
As discussed above, although using larger local search-
ing neighborhoods reduces the chance of extrapolation, it
also increases the ambiguity in the appearance-label re-
lationship. For small image patches with 푟 = 1, the
appearance-label relationship is the most ambiguous. Good
appearance matchings using small patches do not necessar-
ily indicate good label matchings. Applying larger local
searching windows makes the appearance-label relationship
even more ambiguous. Hence, using large local search-
ing windows with 푟푠 > 1 significantly reduces the per-
formance. Using larger image patches reduces ambiguity
in the appearance-label relationship, i.e., good appearance
matchings using large patches are more likely to indicate
good label matchings. Hence, using larger patches we can
method
MV
STAPLE
LWGau
LWInv
LWReg
left right
0.836±0.084
0.846±0.086
0.886±0.027
0.885±0.027
0.892±0.025
0.829±0.069
0.841±0.086
0.875±0.030
0.873±0.030
0.882±0.028
Table 1. Results in terms of Dice overlap produced by each
method.
afford larger local searching windows. Note that apply-
ing larger appearance windows yields smoother local ap-
pearance similarity variations, therefore results in smoother
local weights for label fusion. Over-smoothing the local
weights for label fusion by using larger local appearance
windows reduces the label fusion accuracy.
Table 1 shows the results produced by each method.
Overall, LWGau and LWInv produce similar results, both
significantly outperform majority voting and the STAPLE
algorithm. Our method outperforms similarity based lo-
cal weighting approaches by 0.7% of Dice overlap, which
is substantial because when segmenting from scratch, our
manual rater produces average intra-rater segmentation
overlap of 0.90. Our improvement is statistically signifi-
cant as well, with 푝 < 0.0001 on the paired Student’s t-test
for each cross-validation experiment. Fig. 4 shows some
segmentations produced by LWGau and our method.
imageLWGauLWReg
Figure 4. Sagittal views of hippocampus segmentations produced
by LWGau and our method. Red: manual; Blue: automatic; Pink:
overlap between manual and automatic segmentation.
Comparing to the state of the art
highest published hippocampus segmentation results. All
these methods are based on similarity-based local weight-
ing label fusion. The experiments in [4, 5] are conducted
in a leave-one-out strategy on data set containing 80 con-
trol subjects. They report average Dice overlaps of 0.887
and 0.884 respectively. For controls, we produce Dice over-
lap of 0.896 ± 0.021. [15] uses a template library of 55
[4, 5, 15] present the
1119

Page 8

atlases. However, for each test image, both the original
image and its flipped mirror image are used. Hence, [15]
effectively uses 110 atlases for label fusion. [15] reports
results in Jaccard index (퐽퐼(퐴,퐵) =
side hippocampus of 10 controls, 0.80 ± 0.03, and 10 MCI
patients, 0.81 ± 0.04. Our results for the left hippocampus
are 0.820 ± 0.033 for controls and 0.795 ± 0.042 for MCI
patients. Overall, using significantly fewer atlases, we pro-
duce results compare favorably to the state-of-the-art.
∣퐴∩퐵∣
∣퐴∪퐵∣) for the left
6. Conclusions
We proposed a regression-based label fusion technique.
Unlike previous image similarity based local weighting
techniques, our method does not assume that the segmen-
tation errors produced by different atlases are uncorrelated.
To ensure robust regression in high dimensional space, we
proposed a similarity-ranking based regularization tech-
nique and a local searching technique.
method, we conducted segmentation experiments on a hip-
pocampus segmentation problem. In our experiment, our
method significantly outperformed the state of the art label
fusion technique, the similarity-based local weighting label
fusion method. Using significantly fewer atlases, our hip-
pocampus segmentation results still compare favorably to
the state of the art in published work.
To validate our
References
[1] S. Allassonniere, Y. Amit, and A. Trouve. Towards a co-
herent statistical framework for dense deformable template
estimation. Journal of the Royal Statistical Society: Series
B, 69(1):3–29, 2007.
[2] X.Artaechevarria,A.Munoz-Barrutia,
de Solorzano. Combination strategies in multi-atlas
image segmentation: Application to brain MR data. IEEE
Tran. Medical Imaging, 28(8):1266–1277, 2009.
[3] B. Avants, C. Epstein, M. Grossman, and J. Gee. Symmet-
ric diffeomorphic image registration with cross-correlation:
Evaluating automated labeling of elderly and neurodegener-
ative brain. Medical Image Analysis, 12:26–41, 2008.
[4] D. Collins and J. Pruessner. Towards accurate, automatic
segmentation of the hippocampus and amygdala from MRI
by augmenting ANIMAL with a template library and label
fusion. NeuroImage, 52(4):1355–1366, 2010.
[5] P. Coupe, J. Manjon, V. Fonov, J. Pruessner, N. Robles, and
D. Collins. Nonlocal patch-based label fusion for hippocam-
pus segmentation. In MICCAI, 2010.
[6] L. Dice. Measure of the amount of ecological association
between species. Ecology, 26:297–302, 1945.
[7] L. K. Hansen and P. Salamon. Neural network ensembles.
IEEE Trans. on PAMI, 12(10):993–1001, 1990.
[8] D. Hasboun, M. Chantome, A. Zouaoui, M. Sahel, M. Dela-
doeuille, N. Sourour, M. Duymes, M. Baulac, C. Marsault,
and D. Dormont. MR determination of hippocampal volume:
andC.O.
Comparison of three methods. Am J Neuroradiol, 17:1091–
1098, 1996.
[9] R. Heckemann, J. Hajnal, P. Aljabar, D. Rueckert, and
A. Hammers. Automatic anatomical brain MRI segmenta-
tion combining label propagation and decision fusion. Neu-
roImage, 33:115–126, 2006.
[10] T. K. Ho, J. Hull, and S. Srihari. Decision combination in
multiple classifier systems. IEEE Trans. on Pattern Analysis
and Machine Intelligence, 16(1):66–75, 1994.
[11] S. Joshi, B. Davis, M. Jomier, and G. Gerig. Unbiased dif-
feomorphism atlas construction for computational anatomy.
NeuroImage, 23:151–160, 2004.
[12] J. Kittler. Combining classifiers: A theoretical framework.
Pattern Analysis and Application, 1:18–27, 1998.
[13] J. Kittler and F. Alkoot. Sum versus vote fusion in multiple
classifier systems. IEEE Trans. on PAMI, 25(1):110–115,
2003.
[14] A. Klein. Evaluation of 14 nonlinear deformation algo-
rithms applied to human brain MRI registration. NeuroIm-
age, 46(3):786–802, 2009.
[15] K. Leung, J. Barnes, G. Ridgway, J. Bartlett, M. Clarkson,
K. Macdonald, N. Schuff, N. Fox, and S. Ourselin. Auto-
mated cross-sectional and longitudinal hippocampal volume
measurement in mild cognitive impairment and Alzheimer’s
Disease. NeuroImage, 51:1345–1359, 2010.
[16] O. Melnik, Y. Vardi, and C.-H. Zhang. Mixed group ranks:
Preference and confidence in classifier combination. IEEE
Trans. on PAMI, 26(8):973–981, 2004.
[17] J. Pluta, B. Avants, S. Glynn, S. Awate, J. Gee, and J. Detre.
Appearance and incomplete label matching for diffeomor-
phic template based hippocampus segmentation. Hippocam-
pus, 19:565–571, 2009.
[18] T. Rohlfing, R. Brandt, R. Menzel, and C. Maurer. Evalu-
ation of atlas selection strategies for atlas-based image seg-
mentation with application to confocal microscopy images
of bee brains. NeuroImage, 21(4):1428–1442, 2004.
[19] T. Rohlfing, R. Brandt, R. Menzel, D. B. Russakoff, and
C. R. M. Jr. Quo vadis, atlas-based segmentation?
Handbook of Medical Image AnalysisVolume III: Registra-
tion Models, pages 435–486, 2005.
[20] S. Roweis and L. Saul. Nonlinear dimensionality reduction
bu locally linear embedding. Science, 290:2323–2326, 2000.
[21] M. Sabuncu, B. Yeo, K. V. Leemput, B. Fischl, and P. Gol-
land. A generative model for image segmentation based on
label fusion. IEEETrans.onMedical Imaging, 29(10):1714–
1720, 2010.
[22] R. Scahill, J. Schott, J. Stevens, and M. R. N. Fox. Map-
ping the evolution of regional atrophy in Alzheimer’s Dis-
ease: unbiased analysis of fluidregistered serial MRI. Proc.
Natl. Acad. Sci. U. S. A., 99(7):4703–4707, 2002.
[23] S. Warfield, K. Zou, and W. Wells. Simultaneous truth and
performance level estimation (STAPLE): an algorithm for
the validation of image segmentation. IEEE Trans. on Medi-
cal Imaging, 23(7):903–921, 2004.
[24] S. Warfield, K. Zou, and W. Wells.
segmentation by estimating rater bias and variance. Philo-
sophicalTransactionsoftheRoyalSociety, 366(1874):2361–
2375, 2008.
The
Validation of image
1120