Hierarchical segmentation of vegetation areas in high spatial resolution images by fusion of multispectral information
ABSTRACT A new region-based methodology for the automated extraction and hierarchical segmentation of vegetation areas into high spatial resolution images is proposed. This approach is based on the iterative and cooperative fusion of the independent segmentation results of equal or different resolution spectral bands, combined with an unsupervised classification into vegetation and no-vegetation regions. The result is a hierarchy of partitions with most relevant information at different levels of resolution of the vegetation areas. In addition, the high flexibility of the scheme allows different configurations depending on the final purpose. For instance, considering the size of the vegetation areas into the hierarchy, or prioritizing the information into the high resolution panchromatic band to improve the accuracy of both vegetation extraction and segmentation. This general tool for vegetation analysis is tested into high spatial resolution images from IKONOS and QuickBird satellites.
- [show abstract] [hide abstract]
ABSTRACT: Watershed transformation in mathematical morphology is a powerful morphological tool for image segmentation that is usually defined for greyscale images and applied to the gradient magnitude of an image. This paper presents an extension of the watershed algorithm for multispectral image segmentation. A vector‐based morphological approach is proposed to compute gradient magnitude from multispectral imagery, which is then input into watershed transformation for image segmentation. The gradient magnitude is obtained at multiple scales. After an automatic elimination of local irrelevant minima, a watershed transformation is applied to segment the image. The segmentation results were evaluated and compared with other multispectral image segmentation methods, in terms of visual inspection, and object‐based image classification using high resolution multispectral images. The experimental results indicate that the proposed method can produce accurate segmentation results and higher classification accuracy, if the scales and contrast parameter are appropriately selected in the gradient computation and subsequent local minima elimination. The proposed method shows encouraging results and can be used for segmentation of high resolution multispectral imagery and object based classification.International Journal of Remote Sensing 01/2007; 28(19):4429-4452. · 1.14 Impact Factor
HIERARCHICAL SEGMENTATION OF VEGETATION AREAS IN HIGH SPATIAL
RESOLUTION IMAGES BY FUSION OF MULTISPECTRAL INFORMATION
Felipe Calderero1, Ferran Marqués1, Javier Marcello2, Francisco Eugenio2
1Signal Theory and Communications Department. Technical University of Catalonia (UPC)
E-mail: email@example.com, firstname.lastname@example.org
2Signals and Communications Department. University of Las Palmas of Gran Canaria
E-mail: email@example.com, firstname.lastname@example.org
A new region-based methodology for the automated extraction and
hierarchical segmentation of vegetation areas into high spatial
resolution images is proposed. This approach is based on the
iterative and cooperative fusion of the independent segmentation
results of equal or different resolution spectral bands, combined
with an unsupervised classification into vegetation and no-
vegetation regions. The result is a hierarchy of partitions with most
relevant information at different levels of resolution of the
vegetation areas. In addition, the high flexibility of the scheme
allows different configurations depending on the final purpose. For
instance, considering the size of the vegetation areas into the
hierarchy, or prioritizing the information into the high resolution
panchromatic band to improve the accuracy of both vegetation
extraction and segmentation. This general tool for vegetation
analysis is tested into high spatial resolution images from IKONOS
and QuickBird satellites.
Index Terms— Image segmentation, region merging,
multispectral images, information fusion
High spatial resolution imagery offers new opportunities for
potentially more accurate detection and classification than
traditional satellite imagery. Nevertheless, an increasingly smaller
spatial resolution causes that single pixels no longer capture the
characteristics of classification target. Hence, an increasingly
smaller spatial resolution does not necessarily benefit classification
performance and accuracy for traditional pixel-based classification
approaches . Thus, preliminary feature extraction techniques are
of great importance for the success of classification methodologies
when applied to high resolution imagery.
One important feature extraction approach is image segmentation.
It partitions the image into a set of homogeneous regions under a
certain criterion. Regions are a first level of abstraction, being
more robust and semantically meaningful than pixels. A large
number of algorithms have been proposed for image segmentation
 but there has been little progress in the unsupervised
segmentation of multispectral imagery [3-5]. These approaches do
not consider the fact that the required level of detail into the
segmentation depends on the final application, especially for
vegetation characterization. Hence, a general tool for vegetation
analysis should provide a hierarchical representation where,
instead of a unique partition, different region-based explanations at
different levels of detail are given. In addition, most of the
previous techniques assume equal resolution bands and do not
consider including higher resolution information, such as the
panchromatic band. In general, the higher resolution information is
included into the low resolution bands by an enhancement process
based on image fusion techniques  at pixel level. However, it is
not clear how image fusion influences the next steps into the
processing chain, such as segmentation and classification.
In this context we present a new region-based methodology for the
automated extraction and hierarchical segmentation of high spatial
resolution images. Although the proposed technique may be
applied to the study of different types of information, this work
focuses on the extraction and segmentation of the vegetation areas.
This approach is based on a high level iterative and cooperative
scheme that fuses the independent segmentation results of each
spectral band and an unsupervised vegetation and no-vegetation
classification. This strategy provides an unsupervised hierarchical
segmentation of the vegetation areas of the image. The high
flexibility of the proposed scheme allows obtaining:
An unsupervised hierarchical vegetation segmentation for
equal resolution bands (for instance, B, G, R, IR) including
or not scale information into the hierarchy.
An unsupervised hierarchical vegetation segmentation for
different resolution bands
panchromatic information), improving the accuracy and
resolution of both vegetation extraction and segmentation.
The proposed framework has been tested for high spatial resolution
images from commercial satellites, such as IKONOS and QuickBird.
The paper is structured as follows. Section 2 presents the general
cooperative region merging scheme applied to the hierarchical
multispectral segmentation. Section 3 presents the experimental
results for the segmentation of equal resolution bands (Section
3.1), and including the higher resolution panchromatic band
(Section 3.2). Finally, conclusions are outlined in Section 4.
(for instance, including
2. MULTISPECTRAL COOPERATIVE REGION MERGING
The general cooperative region merging strategy for multispectral
hierarchical segmentation is presented in Figure 1. It is formed by
five main steps: the region merging step, where the separated
segmentation for each spectral band is performed; the
segmentation fusion step, where the independent segmentation
results of each band are combined into a consensus partition; the
region-based classification step, where an unsupervised vegetation
classification of the regions is performed; the meet step, where the
classification and segmentation results are fused; and the scale
filter, where the scale consistency of the partitions is assured.
IV - 232 978-1-4244-3395-7/09/$25.00 ©2009 IEEE IGARSS 2009
The central idea is to let the system evolve by itself, starting with a
basic agreement (given by an initial partition) and searching for
partitions with decreasing number of regions by further consensus
iteration after iteration. This is done instead of finding a coarser
direct consensus partition for spectral band partitions (risking
introducing under-segmentation errors). It can be shown that this
iterative scheme provides with a partition hierarchy: iteration after
iteration, a partition coarser than the previous partition is obtained.
This scheme was successfully used into the joint segmentation of
other types of information, such as color and depth information .
2.1 Region Merging Step
A region merging step is associated with each criteria or
information source. Starting from an initial partition of the image
data (or directly all pixels at the first iteration), this step performs a
region merging process providing a merging sequence of the
regions in the initial partition according to the characteristics of the
spectral band considered (see Figure 1).
The region merging techniques used in this work are based on a
modified version of the general region merging techniques based
on information theory statistical measures proposed in  (here
using the ASH probability density function estimator ).
Precisely, a merging process formed by a Bhattacharyya merging
criterion and a scale-based merging order is chosen for its good
compromise between under- and over-segmentation errors, in the
context of both color homogeneous and texture region segmentation.
2.2Segmentation Fusion: Maximum Mutual Information Partition
The goal of this block is to determine a good consensus partition
among the independent segmentation result of each band. In other
words, observing the sequence of mergings we would like to
determine the set of region fusions where the independent
processes mostly agree. Hence, we search for the partition with
smallest number of regions where the segmentation results of each
band still have a large degree of agreement. This process is done in
two steps: (i) creating a fused merging sequence among the
different segmentation results; and (ii) determining the partition
within the fused merging sequence with smallest number of
regions that still shows a large degree of agreement with respect to
the segmentation of the individual spectral bands.
2.2.1 Creation of a fused merging sequence
To have a common reference, the merging sequences are written in
terms of the removal of the boundaries between regions of the
initial partition. For instance, removing boundary (A, B) is
equivalent to merging regions A and B of the initial partition. The
fused merging sequence is then a new boundary removal ordering
obtained after combining the orderings provided by the region
merging process of each band. Common combination functions are
the maximum (preventing
(preventing oversegmentation), or mean (compromise).
2.2.2 Determining the multispectral consensus partition
Once the fused merging sequence has been computed, the next step
is to determine the segment of mergings where segmentation
results in each band present a significant agreement. The partition
at this point will be considered as the consensus partition among
the different spectral bands.
To study the evolution of the agreement into the merging process,
the difference in priority given by each individual band and the
new fused sequence is computed and ordered according to the
fused sequence. In general, the differences in priority are small for
the first region mergings (merging decisions are clear at this stage).
Nevertheless, as the merging process continues, the differences
increase as regions present more complex models and depend on
previous decisions. Hence, the initial correlation of the different
sequences decreases and, at some point, they significantly diverge.
We model this behavior as follows. Consider the merging sequence
of a spectral band,
obtained combining the merging sequences of the
multispectral bands (as explained in Section 2.2.1). We expect that
both sequences have a significant correlation at an initial segment
that is not present after this point (both sequences can be
considered independent). Under these premises, their mutual
information is given by
( , )I X Y
conditional entropy can be written as:
, and the fused sequence,
( )( | )H XH X Y
, where the
( | ),,|,,,,
H X YH xxyy H xx
The first term on the right represents the conditional entropy in the
part where both sequences are correlated, and the second refers to
the segment where they significantly differ.
To compute the first term we assume that the difference between
the merging sequence of a band and the fused sequence in the part
where both sequences are similar can be considered as independent
samples of a certain random variable, e. In turn, the second term is
determined assuming that the probability of all mergings at a
certain stage is equally likely (which may be true assuming no
knowledge on the merging processes carried in each band), and
assuming that the order given to each element is independent of the
previous mergings (which is a simplification, since larger regions
will be formed by the merging of elementary regions, as thus, may
partially depend on previous fusions). Thus,
H X Yk H e
In this context, finding the initial segment where both sequences
mostly agree is equivalent to determine the value of k that maximizes
the mutual information. Using the same assumption to compute the
entropy of the merging sequence, H X , as before, we obtain: ( )
argmax (| )argmaxlog( )
kI X Ynjk H e
Figure 1. Cooperative region merging scheme for multispectral
hierarchical vegetation extraction and segmentation.
IV - 233
This expression can be extended to the set of merging sequences
obtained from each spectral band:
argmax (,, | )Y
k I XX
I XXYI X Y
considering merging sequences independent between them. Finally,
multispectral consensus partition is obtained, using a conservative
approach, by the removal of the first 0.75 k?
2.3 Region-Based Classification
This block performs an unsupervised classification into vegetation
and no-vegetation of the regions into the initial partition at the
current iteration, using the information provided by the normalized
differential vegetation index (NDVI) . For each region, the
NDVI mean value over all its pixels is determined. Then, regions
are classified based on their mean NVDI index into two classes
using k-means or fuzzy k-means algorithm. The vegetation
partition that is fused with the multispectral consensus partition (see
Section 2.5) is obtained by relabeling the vegetation classification.
2.4 Meet Step
The meet operation outputs the intersection of the input partitions,
that is, a partition that includes all boundaries present in the input
partitions. In other words, pixels with equal label vectors (each
component being the label at a particular band) belong to the same
region into the meet partition.
The goal of this block is to assure that the relevant information
about the vegetation and no-vegetation areas is preserved into the
consensus partition of the multispectral bands. In that sense, the
vegetation partition is included as prioritary information into the
system, preventing vegetation and no-vegetation areas to merge.
2.5 Scale Filter
The scale filter assures the scale consistency of all the obtained
partitions, that is, removes the set of regions that are too small to
be significant at the current level of resolution. For that purpose an
adaptive scale threshold is defined on the region areas, similarly to
the scale-based merging order used in :
Number of Regions
The ? parameter controls the minimum resolution at each scale. In
our experiments, we have chosen a low value for this parameter,
, to be sure that only clearly meaningless regions at that
scale are discarded. Out-of-scale regions are merged according to
the most reliable or relevant available information. For instance,
for equal resolution channels NVDI is used (see Section 3.1) and
so is the panchromatic band for the different channel resolution
configuration (see Section 3.2). As the scale-filtered partition is a
coarser partition that the previous one, the hierarchical structure of
the output partitions is maintained.
As shown in Figure 1, the position of the scale filter determines
whether the area of the vegetation is introduced or not into the
created hierarchy. If the filter is placed before the meet operation
(Position A), isolated vegetation areas are preserved into the
hierarchical segmentation, independently of their size. On the
contrary, if the scale filter is located after the meet operation
(Position B), only the vegetation areas that are large enough to be
significant at the given scale are preserved.
3. EXPERIMENTAL RESULTS
The performance of the proposed framework has been tested for
high spatial resolution images from commercial satellites, such as
IKONOS (four channels of multispectral data at 4 m resolution and
one panchromatic channel at 1 m resolution) and QuickBird (four
channels of multispectral data with 2.4 m resolution and one
panchromatic channel with 0.6 m resolution).
3.1 Configuration for Equal Resolution Bands
In this first case, four different bands with equal resolution (B, G,
R, IR) were considered for each multispectral image. As shown in
Figure 1, each band was injected to a different region merging
process. NDVI was computed and used as input for the region-
based classification block. Independently of its position, the scale
filter used the NDVI to merged out-of-scale regions.
In Figure 2, an example of the hierarchical segmentation results
obtained on a 221x261 subimage of a Quickbird multispectral
image is shown. In this case, the maximum of the merging
sequences was used to create the consensus partition among the
different segmentations of each band. We observed that the results
are similar for the maximum and mean functions, and significantly
worst for the minimum. In the presented example, the scale filter is
set in Position A (see Fig. 1) such that the vegetation area is not
included into the partition hierarchy. The same configuration but
including the area into the hierarchy is shown in Figure 3 for the
same subimage. Note that in this case, as the level of detail into the
hierarchy decreases, only the vegetation areas with enough area to
be relevant at the given scale are preserved.
Applications analyzing vegetation areas independently of their
size, and hence, interesting on preserving even small vegetation
areas will find into the first configuration a valuable solution.
Moreover, applications searching for vegetation areas of a certain
size, or interested on the granularity and distribution of the green
areas, will prefer the second configuration.
3.2 Configuration for Different Resolution Bands
In this case, in addition to the four equal resolution bands used in
the previous experiments (B, G, R, IR), the panchromatic band was
also used. To provide with partitions of equal resolution the four
bands of equal resolution were interpolated by a factor of four
using nearest neighbor interpolation algorithm. Instead of the
NDVI, the panchromatic band was used by the scale filter
(independently of its position) to merge out-of-scale regions, as it
is considered as the most accurate channels thanks to its higher
An example for a subset of 201x146 pixels into the low resolution
bands (804x584 pixels at the panchromatic band) image is shown
in Figure 4. The hierarchical segmentation using only the equal
resolution bands and including also the panchromatic band were
computed. In both cases, the maximum function was used for the
combination, and the vegetation area was not included into the
hierarchy. From the two levels of the partition hierarchy shown, it
can be seen that the use of the panchromatic information improves
the correct segmentation of the field structures (that is more
evident at the partition with higher level of detail), and also is able
to improve the classification into vegetation and no-vegetation
IV - 234
Figure 4. Hierarchical vegetation segmentation of Quickbird
subimage. The first row shows the color (R,G,B composition)
image (top-left), the NDVI index (bottom-left), and the
panchromatic band (right) (not at their proportional scale). Second
row shows the results for two different levels of the partition
hierarchy using B, G, R, and IR bands (equal resolution channels) in
the segmentation process. The third row shows partitions at similar
levels of the hierarchy for the scheme configuration including also
the panchromatic band.
areas (more evident at the partition with less level of detail).
Thus, the previous results show that including more accurate
spatial information (panchromatic band) not only improves the
segmentation of the multispectral images, but also the
classification performance into vegetation and no-vegetation areas.
This is a consequence of the cooperation between the segmentation
and classification stages proposed in our scheme.
The presented results show that the proposed hierarchical
multichannel segmentation approach provides an unsupervised and
flexible tool for the analysis and classification of vegetation into a
broad range of remote sensing applications.
 Q. Yu, P. Gong, N. Clinton, G. Biging, M. Kelly, and D. Schirokauer,
“Object based detailed vegetation classification with airborne high spatial
resolution remote sensing imagery,” Photogrammetric Engineering and
Remote Sensing, vol. 72, pp. 799–811, 2006.
 N. Pal and S. Pal, “A review on image segmentation techniques,”
Pattern Recognition, vol. 26, pp. 1277–1294, 1996.
 P. Li and X. Xiao, “Multispectral image segmentation by a
multichannel watershed-based approach,” International Journal of
Remote Sensing, vol. 28, no. 19, pp. 4429–4452, 2007.
 M. Sellami, F. Chaabane, C. Fetita, “Morphological Segmentation of
Multispectral Images for Land Cover Mapping,” Proc. of IGARSS 2008,
vol.3, pp. 326-329, 2008.
 Y. Zhang, X. Feng, and X. Le, “Segmentation on Multispectral
Remote Sensing Image Using Watershed Transformation,” Proc. of
CISP '08, vol.4, pp.773-777, 2008.
 F. Laporterie-Dejean, H. de Boissezon, G. Flouzat, M.-J. Lefevre-
Fonollosa, “Thematic and statistical
panchromatic/multispectral fusion methods on simulated PLEIADES-
HR images,” Information Fusion, vol. 6, no. 3, pp. 193-212, 2004.
 F. Calderero and F. Marques, “Hierarchical fusion of color and depth
information at partition level by cooperative region merging,” to appear
in Proceedings of ICASSP’09, 2009.
 F. Calderero and F. Marques, “General region merging approaches
based on information theory statistical measures,” Proceedings of
ICIP’08, pp. 3016–3019, 2008.
 D. W. Scott, “Averaged shifted histograms: Effective nonparametric
density estimators in several dimensions,” The Annals of Statistics, vol.
13, no. 3, pp. 1024–1040, 1985.
 R. A. Schowengerdt, Remote sensing models and methods for image
processing, Academic Press, 3rd edition, 2007.
evaluations of five
Figure 3. Example of different levels of segmentation obtained
including the vegetation area into the creation of the hierarchy
(Position B), using the same image as in Figure 2.
Figure 2. Hierarchical vegetation segmentation of a Quickbird
subimage. First row. Left: Portion of original multispectral image
(R,G,B composition). Righ: NDVI index. Second row: Two
different levels of the hierarchy of partitions. Example computed
using maximum fusion criteria and not including the vegetation area
into the hierarchy (Position A). No-vegetation areas are in white.
IV - 235