Urban-Area and Building Detection Using SIFT Keypoints and Graph Theory

Abstract

Very high resolution satellite images provide valuable information to researchers. Among these, urban-area boundaries and building locations play crucial roles. For a human expert, manually extracting this valuable information is tedious. One possible solution to extract this information is using automated techniques. Unfortunately, the solution is not straightforward if standard image processing and pattern recognition techniques are used. Therefore, to detect the urban area and buildings in satellite images, we propose the use of scale invariant feature transform (SIFT) and graph theoretical tools. SIFT keypoints are powerful in detecting objects under various imaging conditions. However, SIFT is not sufficient for detecting urban areas and buildings alone. Therefore, we formalize the problem in terms of graph theory. In forming the graph, we represent each keypoint as a vertex of the graph. The unary and binary relationships between these vertices (such as spatial distance and intensity values) lead to the edges of the graph. Based on this formalism, we extract the urban area using a novel multiple subgraph matching method. Then, we extract separate buildings in the urban area using a novel graph cut method. We form a diverse and representative test set using panchromatic 1-m-resolution Ikonos imagery. By extensive testings, we report very promising results on automatically detecting urban areas and buildings.

Full text

Page 1

1
Urban Area and Building Detection Using SIFT
Keypoints and Graph Theory
Beril Sırmac ¸ek, Student Member, IEEE, Cem¨Unsalan, Member, IEEE,
Abstract—Very high resolution satellite images provide valu-
able information to researchers. Among these, urban area
boundaries and building locations play crucial roles. For a
human expert, manually extracting this valuable information
is tedious. One possible solution to extract this information
is using automated techniques. Unfortunately, the solution is
not straightforward if standard image processing and pattern
recognition techniques are used. Therefore, to detect the urban
area and buildings in satellite images, we propose using scale
invariant feature transform (SIFT) and graph theoretical tools.
SIFT keypoints are powerful in detecting objects under various
imaging conditions. However, SIFT is not sufficient for detecting
urban areas and buildings alone. Therefore, we formalize the
problem in terms of graph theory. In forming the graph, we
represent each keypoint as a vertex of the graph. The unary
and binary relationship between these vertices (such as spatial
distance and intensity values) lead to edges of the graph. Based on
this formalism, we extract the urban area using a novel multiple
subgraph matching method. Then, we extract separate buildings
in the urban area using a novel graph cut method. We form a
diverse and representative test set using panchromatic one meter
resolution Ikonos imagery. By extensive testings, we report very
promising results on automatically detecting urban areas and
buildings.
Index Terms—SIFT; Multiple subgraph matching; Graph cut;
Urban area detection; Building detection.
I. INTRODUCTION
Satellite images offer valuable information to researchers.
The resolution of earlier satellite imagery (such as Landsat)
would not allow detecting separate man-made or natural
objects. Therefore, researchers mostly focused on extracting
the region properties from these imagery. As the advent of
very high resolution (VHR) satellite imagery (such as Ikonos
and Quickbird), it became possible to observe these objects.
Besides region properties, extracting man-made objects in very
high resolution satellite images may help researchers in various
ways, such as automated map making.
Among different man-made objects, buildings play an im-
portant role. Therefore, the robust detection of buildings
in very high resolution satellite images requires a specific
consideration. Unfortunately, it is still tedious for a human
expert to manually label buildings in a given satellite image.
One main reason is the total number of objects in the scene.
The other reason is the resolution of the satellite image.
Although the resolution of the satellite imagery has reached
an acceptable level, it is still not possible for a human expert
to extract information from it in a robust manner. To solve
The authors are with the Department of Electrical and Electronics Engi-
neering, Computer Vision Research Laboratory, Yeditepe University, Istanbul,
Turkey (e-mail: unsalan@yeditepe.edu.tr).
this problem, researchers introduced automated urban area and
building detection methods using very high resolution satellite
and aerial images.
Zerubia et al. [18] used texture information to detect the
urban area in both optical and radar images. They extracted
texture parameters using chain based Gaussian models. Then,
they used clustering with a Markovian segmentation step.
Their system is robust to sensor changes, but not very robust to
resolution changes in images. Rokos et al. [12] used building
density information to classify residential regions. Benedik-
tsson et al. [1] used mathematical morphological operations
to extract structural information to detect the urban area in
satellite images.¨Unsalan and Boyer [30], [32] extracted linear
features from satellite images to detect residential regions.
They benefit from spatial coherence and graph theory for
labeling urban or residential regions.¨Unsalan [29], in a related
study used graph theory to grade changes in urban regions.
Fonte et al. [7] considered corner detectors to obtain the
type of structure in a satellite image. They concluded that,
corner detectors may give distinctive information on the type
of structure in an image. Bhagavathy and Manjunath [2] used
texture motifs for modeling and detecting regions (such as golf
parks, harbors) in satellite images. Bruzzone and Carlin [3]
proposed a pixel based system to classify very high resolution
satellite images. They used support vector machines fed with
a novel feature extractor. Zhong and Wang [35] introduced
conditional random fields to learn dependencies in the image.
They fuse the multilevel structural information to obtain urban
areas in satellite images. A related problem in the literature is
the satellite image classification. The literature is vast on this
topic. Some related papers can be counted as [15], [17], [6],
[4], [20], [10], [16].
Kim and Muller [14] used graph theory to detect build-
ings in aerial images. They extracted linear features in the
given image and used them as vertices of a graph. Then,
they extracted buildings by applying subgraph matching with
their model building graph. Finally, they used intensity and
shadow information to verify the building appearance. This
study follows a similar strategy as we did. Different from
us, they used color aerial images and linear features. Segl
and Kaufmann [26] combined supervised shape classification
with unsupervised image segmentation in an iterative way.
Their method allows searching objects (like buildings) in high
resolution satellite images. ¨Unsalan and Boyer [31] studied
on multispectral satellite images to detect buildings and street
networks in residential regions. Their method depends on
vegetation indices, clustering, decomposing binary images,
and graph theory. They also offer a nice review on building

Page 2

2
detection algorithms in the literature. Mayunga et al. [21] used
polygons, formed by edge information, to detect buildings.
They proposed a novel snake algorithm starting from an
approximate polygon center. The snake grows radially until it
fits a closed polygon shape. Then, they used linear features to
verify the building appearance. Peng et al. [25] also proposed
an improved snake model to detect buildings in color aerial
images. They report good detection results with their system.
Huang et al. [11] considered fusion of multispectral Ikonos
imagery to classify objects (including buildings) in urban
regions. Molinier et al. [23] considered detecting man-made
structures in satellite images using PicSOM. Gamba et al. [8]
used boundary information to extract the map of an urban area.
They fed the boundary and non-boundary data to two different
classifiers. Then, they combined results of two classifiers. They
obtained satisfactory results to detect urban area buildings on
VHR imagery. Wei and Xin [33] introduced a method based on
level-sets to segment out man-made objects in aerial images.
Katartzis and Sahli [13] used a hierarchical stochastic model
based on perceptual organization to detect building rooftops in
color satellite images. Zhang et al. [34] used airborne LIDAR
data to detect buildings.
Detecting buildings in satellite images is a difficult task for
several reasons. Building may be imaged from different view-
points. The illumination and contrast in the image may not be
sufficient for detecting buildings. There may be several other
structures, such as nearby trees and street segments making
the building detection problem harder. In addition to these
difficulties, buildings do not have standard size and shape. All
these issues make building detection a hard problem.
Scale invariant feature transform (SIFT) is a good candidate
for urban area and building detection in satellite images.
SIFT is a local descriptor extraction method having valuable
properties such as invariance to illumination and viewpoint
[19]. In the literature, it is extensively used to match objects
represented by template images, in a given image. Unfortu-
nately, the standard SIFT implementation is not sufficient for
urban area and building detection from satellite images alone.
There are many similar and nearby buildings in the satellite
image. Therefore, standard feature matching does not work
properly.
In this study, we propose novel methods to detect the urban
area and buildings from panchromatic VHR Ikonos images.
Our detection methods are based on SIFT keypoints and graph
theoretical tools. We first upsample the image by six to help
SIFT keypoint extraction. Then, we apply a nonlinear bilateral
filtering operation to smooth out unwanted noise terms in
the image [28]. Afterwards, we extract local SIFT keypoints
(and associated features) from the processed satellite image.
Different from the original SIFT, we cast the urban region
detection problem as one of multiple subgraph matching. In
forming the graph, each SIFT keypoint is taken as a vertex.
The neighborhood between different vertices is summarized as
edges of the graph. In the same way, we formulate the building
detection problem in terms of graph cut. Gautama et al. [9]
in a related study used error-tolerant graph matching to find
correspondences between the detected image features and the
geospatial vector data.
We test our urban area and building detection methods on
a fairly diverse and representative image set formed from
panchromatic Ikonos images of 28 different urban sites in
Adana, Ankara, and Istanbul, Turkey. They contain various
building types, as well as various urban area characteristics.
Tests indicate the potential of our urban area and building
detection methods. To explain our novel detection methods in
detail, we start with preprocessing of our satellite images.
II. BILATERAL FILTERING FOR PREPROCESSING
Ikonos satellite images are not directly suitable for robust
SIFT keypoint extraction due to their resolution. The main
problem here is the minimum size of the filter scale used in
standard SIFT keypoint extraction. Unfortunately, this filter
scale is fairly large for detecting building properties. There-
fore, we first upsample the image by six in each coordinate
axis using bilinear interpolation. To note here, nearest neighbor
interpolation may also be used with a slight performance
change. Besides, upsampling itself is not sufficient. We should
also eliminate noise in the satellite image. Unfortunately, ob-
jects are so small in these images that, it is almost impossible
to discard noise terms without disturbing object boundaries
using standard linear filters. Therefore, we propose using
bilateral filtering proposed by Tomasi and Manduchi [28]. It
performs nonlinear smoothing on images by keeping the edge
information. Nonlinear smoothing is performed by combining
the geometric and intensity similarity of pixels. Elad [5] shows
that, bilateral filtering is closely related to edge preserving
smoothing operations such as anisotropic diffusion. We next
explain the bilateral filtering operation using Elad’s notation.
Let Ig(x,y) be a grayscale image having values in the range
[0,1]. Let Ib(x,y) be the bilateral filtered version of Ig(x,y).
The filtering operation can be represented as:
Ib(x,y) =
?N
n=−N
?N
m=−NW(x,y,n,m)Ig(x − n,y − m)
?N
This equation is simply a normalized weighted average of
a neighborhood of 2N + 1 by 2N + 1 pixels around the
pixel location (x,y). The weight W(x,y,n,m) is computed
by multiplying the following two factors:
n=−N
?N
m=−NW(x,y,n,m)
(1)
W(x,y,n,m) = Ws(x,y,n,m) × Wr(x,y,n,m)
where Ws(x,y,n,m) stands for the geometric weight factor.
It is based on the Euclidean distance between the center pixel
(x,y) and the (x − n,y − m) pixel as:
?
This way, nearby samples influence the final result more than
distant ones.
The second weight, Wr(x,y,n,m), is based on the
grayscale intensity distance between the values of the center
pixel (x,y) and the (x−n,y−m) pixel. Again it is based on
the Euclidean distance between intensity values as:
(2)
Ws(x,y,n,m) = exp
−(x − n)2+ (y − m)2
2σ2
s
?
(3)

Page 3

3
Wr(x,y,n,m) = exp
?
−(Ig(x,y) − Ig(x − n,y − m))2
2σ2
r
?
(4)
Thus, pixels with close grayscale intensity values tend to
influence the final result more than those having distant values.
The bilateral filter is controlled by three parameters. N
dictates the support of the filter. Larger support gives stronger
smoothing. The parameters σsand σrcontrol the decay of the
two weight factors. We pick N = 5, σs= 3 and σr= 0.1 for
implementation. These values are picked keeping in mind the
minimum geometric size and intensity variations of building
in the Ikonos image to be detected. For other satellite or aerial
image types, these parameters should be adjusted accordingly.
In this study, we use Adana8 image given in Fig. 1 to
illustrate our method step by step. In this test image, it is
hard to detect the urban area and the buildings due to the
low contrast between building rooftops and the background.
Besides, this is a typical test site.
Fig. 1.The Adana8test image
To provide a detailed explanation of our system, we focus
on a subpart of the Adana8image as in Fig. 2. Here, we only
consider two buildings. We first provide the bilateral filtering
results of this subpart image in the same figure. As can be
seen, the bilateral filtered image is fairly smooth, with the
edge information of buildings kept fairly well.
(a) Original (b) Bilateral filtered
Fig. 2. Subpart of the Adana8test image, and its bilateral filtering result
III. SCALE INVARIANT FEATURE TRANSFORM (SIFT)
Lowe [19] proposed the scale invariant feature transform
(SIFT) for object detection based on its template image. SIFT
leads to local image descriptors, invariant to translation, scal-
ing, and rotation. These descriptors are also partially invariant
to illumination changes and 3D projection. Mikolajczyk and
Schmid [22] compared SIFT descriptors with other invariant
feature descriptors. They concluded that SIFT performs best
under changes in scale, rotation, and illumination. Lowe pro-
posed detecting an object in an image by descriptor matching.
To do so, first a template image is obtained for the object
to be detected in the test image. Its descriptors are extracted.
Then, descriptors for the test image are extracted. A one to one
matching between template and test image descriptors lead to
detecting the object in the test image.
Buildings can be considered as objects, to be detected, in
the satellite image. In satellite images, buildings can have dif-
ferent illumination conditions, structural differences and size
changes. More importantly, they can be imaged from different
viewpoints. Invariance properties of SIFT can handle most
of these variations. Therefore, SIFT is suitable for building
detection from satellite images. Next, we briefly explain SIFT
and keypoint extraction. More detail on SIFT can be found in
[19].
The first stage of keypoint detection is to identify locations
and scales that can be repeatedly assigned under differing
views of the same object. Detecting locations that are invariant
to scale change of the image can be accomplished by searching
for stable features across all possible scales, using a continuous
function of scale known as scale space. The scale space of an
image is defined as a function L(x,y,σ) that is produced from
the convolution of a variable scale Gaussian, with the input
image Ib(x,y) as:
L(x,y,σ) =
1
2πσ2exp
?
−x2+ y2
2σ2
?
∗ Ib(x,y)
(5)
where ∗ is the convolution operation in x, y and σ is the
Gaussian scale.
To efficiently detect stable keypoint locations in scale space,
Lowe proposed using the scale-space extrema. It is calculated
from the difference of Gaussian images computed from the two
nearby scales separated by a constant multiplicative factor k
as:
D(x,y,σ) = L(x,y,kσ) − L(x,y,σ)
In order to detect the extrema of D(x,y,σ), each sample
point is compared to its eight neighbors in the current image
and nine neighbors in the scale above and below. The sample
point is selected only if it is larger than all of these neighbors
or smaller than all of them. Once a keypoint candidate has been
obtained by comparing a pixel to its neighbors, the next step is
to perform a detailed fit to the nearby data for location, scale,
and ratio of the principal curvatures. This information allows
points to be rejected that have low contrast (and are therefore
sensitive to noise) or are poorly localized along the edge. We
label the keypoint by its spatial coordinates as (xi,yi).
One or more orientations are assigned to each keypoint
location based on local image gradient directions. By assigning
a consistent orientation to each keypoint based on local image
properties, the keypoint descriptor can be represented relative
to this orientation. Therefore, it has an invariance to image
rotation.
(6)

Page 4

4
The keypoint descriptor generated by SIFT algorithm is
created by sampling the magnitudes and orientations of the
image gradient in the patch around the keypoint. Finally, a 128
element vector (represented as fi) is formed as a descriptor.
This vector is then normalized to unity magnitude to have an
invariance to illumination.
We extract keypoints and vector representations from the
Adana8 subpart image as in Fig. 3. In this figure, vectors
originate from keypoint locations. They represent the dominant
direction of the SIFT descriptor for each keypoint as well as
their strength. As can be seen, keypoints in this image are
located around building corners and other significant intensity
changes.
(a) Keypoints(b) Vectors
Fig. 3.
subpart image
SIFT keypoints and their vector representation on the Adana8
IV. DETECTING THE URBAN AREA AND BUILDINGS
In the original SIFT, the object to be detected is represented
by one or several template images. Then, keypoint descriptors
are obtained for each template. Keypoint descriptors for the
test image are also obtained. A one to one matching between
template keypoints and the test image keypoints is performed
using the Euclidean distance between keypoint descriptors.
SIFT descriptors are highly discriminative. Therefore, each
template keypoint matches with the closest (in the Euclidean
distance sense) test image keypoint. This is the strength of the
original SIFT based object detection. However, this property
of the SIFT is not suitable for our problem. First, we have
many buildings in the satellite image to be detected. Second,
it is not feasible to have templates for all types of buildings
to be detected. Therefore, we propose novel graph theoretical
methods to detect the urban area and buildings.
To detect buildings and the urban area, we use two template
building images as given in Fig. 4. The first template represents
a bright building, having a high contrast between background
and its rooftop. The second template represents a dark build-
ing, having relatively low contrast between the background
and its rooftop. These two templates cover a wide range of
building characteristics.
As mentioned above, each SIFT keypoint is described by a
vector vi = (xi,yi,fi). Here, (xi,yi) represents the spatial
coordinate of the keypoint. fi is the 128 element feature
vector for that keypoint. We first extract keypoints for the two
templates and the test image. Then, we represent them as va
i = 1,...,I for the dark building template; vr
the bright building template; and vt
test image respectively.
i
jj = 1,...,J for
mm = 1,...,M for the
(a) Bright building (b) Dark building
Fig. 4. Two building (bright and dark) templates used in this study.
A. Graph Representation
To detect the urban area and buildings, we cast the problem
in terms of graph theory. A graph G is represented as G =
(V,E), where V is the vertex set and E is the edge matrix
showing the relations between these vertices. For the urban
area and building detection, we represent keypoints extracted
from the dark building template (va), bright building template
(vr), and test image (vt) in a graph formation as Ga(Va,Ea),
Gr(Vr,Er), and Gt(Vt,Et) respectively. Let’s consider Ga.
Va= {va
?
0
where dij=?(xi− xj)2+ (yi− yj)2. We take ?1= 30 due
If this method is applied to higher resolution images, then the
value of ?1should also be increased accordingly. Ea(i,j) = 0
means that there is no edge between vertices viand vj. We
form Grand Gtin a similar way.
i} i = 1,...,I. Eais an I × I matrix defined as:
Ea(i,j) =
dij
if dij< ?1
otherwise
(7)
to the size of buildings we are detecting in the Ikonos imagery.
B. Multiple Subgraph Matching to Detect the Urban Area
Buildings in a region indicate the existence of an urban area.
Therefore, in order to detect the urban area, it is sufficient to
detect buildings. To detect all buildings in an image (without
selecting a special one), we apply multiple subgraph matching
between Ga,Gtand Gr,Gtseparately. Applying multiple
subgraph matching between the template and test images is
different from the original SIFT. As mentioned above, we have
many buildings in the same region and we want to detect
all at once without selecting a specific one. This formalism
simplifies our urban area detection problem since nearby
buildings effect each other’s detection probability in multiple
subgraph matching. Also, using this formalism, we relax the
graph matching condition.
From now on, we will explain our multiple subgraph
matching method on the Ga,Gtpair. The method is the
same for the Gr,Gtpair. Our multiple subgraph matching
method can be rephrased as a one to many matching between
two graph vertices both in unary and binary terms. For the
unary matching between Gaand Gt, we define multiple
vertex matching between two graphs Ga= (Va,Ea) and
Gt= (Vt,Et) as:
?
0
M1(va
i,vt
j) =
1
if ||fa
otherwise
i− ft
j|| < ?2
(8)

Page 5

5
M1(·,·) will be an I × M matrix. We check matching
∀va
tolerance value for unary matching. Since we have multiple
subgraph matching, such an adaptive parameter adjustment
strategy is necessary. We calculate ?2 as follows. We first
calculate the best match between Vaand Vt. We assume
this to be a valid match. In other saying, we assume that,
the matched vertex in the test image belongs to a building.
Then, we calculate the next best match comparing the match
distance with the first best match. We repeat this process as
long as the match distance is comparable with the best match
case. If there is a larger distance (such as two times the best
match), then we assume that the match is not valid. We pick
this distance value as ?2.
For urban area and building detection, matched keypoints
should also keep their structure. Therefore, we define binary
vertex matching between the two weighted graphs Ga=
(Va,Ea) and Gt= (Vt,Et) as:
i∈ Vaand ∀vt
j∈ Vt. In Eqn. 8, ?2stands for the adaptive
M2(Ea(i,j),Et(k,l)) =





1
if (M1(va
(M1(va
otherwise
i,vt
l) = 1) ∧ (γ < ?3)
k) = 1)∧
j,vt
0
(9)
where γ = |Ea(i,j)−Et(k,l)|. Here, ?3is the tolerance value
for binary matching. Based on the dimension of buildings
in our building template images, we set ?3 = 4. For other
building template images, ?3should be adjusted accordingly.
For example, if a larger building template is to be used, then ?3
should be increased. Similarly, if a smaller building template
is to be used, then ?3should be decreased.
We form a new graph, based on unary and binary matching
(matched vertices and edges of Gt). We call it as Gd=
(Vd,Ed) to indicate that it contains vertices and edges of
the test image graph matched with the dark building template
graph. vt
We form the edge matrix Edas:
m∈ Vdiff ∃va
i∈ Vasuch that M1(va
i,vt
m) = 1.
Ed(k,l) =





Et(k,l)
if ∃ i,j st.
M2(Ea(i,j),Et(k,l)) = 1
otherwise
0
(10)
We provide the constructed graph Gdon the Adana8
subpart image in Fig. 5 (a). As can be seen, all the matched
vertices of Gt(now the vertices of Gd) lie on the buildings in
the image. Due to multiple subgraph matching, most vertices
on different buildings also have edges connecting them.
(a) Graph, Gd
(b) Region, A(Gd)
Fig. 5.The graph and its region obtained from the Adana8subpart image
We apply the same procedure to form Gb= (Vb,Eb) in the
same way using the Gr,Gtpair. This graph indicates the unary
and binary matching between the bright building template and
the test image.
To locate the urban area containing buildings, we define
region of a graph. For a graph G(V,E) with vertices V = {vi}
having spatial coordinates vi= (xi,yi), we define its region
A(G) as:
i- vi∈ V ⇒ (xi,yi) ∈ A(G)
ii- Let lijbe the line segment joining vi,vjwhere E(i,j) ?=
0; (xa,ya) ∈ lij⇒ (xa,ya) ∈ A(G)
iii- Let tijkbe the triangle with corners vi,vj,vk∈ V where
E(i,j) ?= 0, E(i,k) ?= 0, E(j,k) ?= 0; (xa,ya) ∈ tijk ⇒
(xa,ya) ∈ A(G)
This region is as small as possible. It includes all vertices
and line segments joining them. To clarify this operation, we
formed the region of graph Gdon the Adana8subpart image.
We provide A(Gd) for this image in Fig. 5 (b).
There may be both bright and dark buildings in a given test
site. Therefore, we detect the final urban area, R using both
A(Gd) and A(Gb) as:
R = A(Gd)
?
A(Gb)
(11)
We provide the detected urban area from the Adana8test
image in Fig. 6. As can be seen, the urban area in this image
is correctly detected. We provide more urban area detection
examples, as well as qualitative results in the experiments
section.
Fig. 6.Detected urban area from the Adana8test image
C. Graph Cut Method to Detect Separate Buildings
Up to now, we benefit from the close proximity of buildings
in detecting the urban area. Now, the second step is detecting
each building alone. For separate building detection, we need
extra information. Therefore, we cut some edges of Gdand
Gbbased on the intensity criteria. We hypothesize that vertices
(keypoints) on the same building have similar intensity values
due to the building color. Therefore, to locate separate build-
ings, we cut edges between vertices having different intensity
values. We expect the cut edges to be the ones between
vertices on different buildings. We form two new graphs as
Gp= (Vp,Ep) and Gq= (Vq,Eq). Here, Vp= Vdand
Vq= Vb. Let’s consider Gp, we assign weights to its edges
as: