Content uploaded by Timo Bretschneider
Author content
All content in this area was uploaded by Timo Bretschneider on Apr 17, 2015
Content may be subject to copyright.
A Retrieval System for Remotely Sensed Imagery
T. Bretschneider, O. Kao
Department of Computer Science, Technical University of Clausthal,
Julius-Albert-Strasse 4, 38678 Clausthal-Zellerfeld, Germany
Tel: +495323727140; Fax: +495323727149
{bretschneider,okao}@informatik.tu-clausthal.de
Abstract: The retrieval of images in professional re-
mote sensing databases is based on world-oriented
information which is often not meaningful for users
who have a limited knowledge about remote sensing
but nevertheless have to work with satellite imagery.
Moreover many applications require a different query
formulation that cannot be performed with the pro-
vided standard mechanisms. Therefore this paper pre-
sents a content-based retrieval system that utilises the
spectral information content of multispectral satellite
imagery as well as the world-oriented features. The
actual content-based query operates on the query-by-
example approach – a powerful tool for both profes-
sional and non-professional remote sensing users. A
further improved flexibility for posing queries is
gained by not only supporting a-priori extracted fea-
tures for a fast retrieval but also a dynamic approach
which increases the quality of the retrieved images,
i.e. the degree of image similarity.
Keywords: Content-based retrieval, satellite im-
agery, multispectral feature extraction
1 Introduction
The development and application of orbital re-
mote sensing platforms result in the production of
huge amounts of image data – more than one
terabyte per satellite and day in the near future.
The obtained image data has to be systematically
collected, registered, organised, and classified in
a way that works completely unsupervised. Fur-
thermore, adequate search procedures and meth-
ods to formulate queries have to be provided to
access the information.
The retrieval of images in professional remote
sensing databases [1], [2], [3], [4], [5], [6] is
based on world-oriented information like the lo-
cation of the scene, the utilised scanner, and the
date of acquisition. However, these descriptions
are not meaningful for many users who have a
limited knowledge about remote sensing but nev-
ertheless have to work with satellite imagery. A
biologist who needs for a general assessment on
swampland in a large-scale region all corre-
sponding images depicts an example.
Therefore a content-based retrieval system
named (RS)2I (Retrieval System for Remotely
Sensed Imagery) was developed which uses an
example image as a starting point. Although the
idea is not new to the research community, none
of the published approaches [7], [8], [9], [10],
[11], [12] provides the required flexibility in gen-
eral. Either the systems are limited to a specific
satellite or need supervision. Both constraints
restrict the purpose of a general system beyond
the acceptable limit since the approaches are only
of use for an archive which is specialised on one
type of scanner or with enough man-power to
supervise the processing, respectively. This is
seldom true for users with a non-professional re-
mote sensing background. The proposed system
in this paper follows two different approaches
which can be combined, too. First all archived
satellite images are described by a selection of a-
priori extracted features, which are subsequently
stored in a relational database system. At query
time the user submits a sample image, which is
processed in the same way as the images in the
database. The features extracted from the sample
image can be compared with the corresponding
features of the archived images using a multidi-
mensional distance function. Based on the com-
puted distance a similarity ranking is created. The
second approach is based on dynamic feature ex-
traction. The main idea behind dynamic retrieval
is to extract the features after the query was for-
mulated and thus to adapt the extraction process
to the inherent requirements of the query image
and the preference of the user. Since the varia-
tions of the specific parameters are unlimited the
corresponding feature vectors cannot be stored
permanently. Finally, in either case a ranking is
presented to the user at the end. To get back to
the mentioned query on swampland, the most
sensitive approach is first to use the static world-
oriented features to restrict the further amount of
data to be processed. Then in the next step a-pri-
ori extracted features are used, e.g. contained
classes of ground cover which exhibit a high per-
centage of water. Since the features are most of
the time fairly abstract, the selected satellite im-
ages are dynamically post-processed to generate
the final query result. Note that each of the re-
trieval results can be used for a refinement of the
query parameters and feature values. The search
process is repeated iteratively, until the desired
image(s) occur(s) in the similarity ranking.
This paper is organised as follows: Section 2
describes the main system architecture of the
(RS)2I. Two of the required processing steps are
described in Section 3 in more detail. Finally the
conclusions and outlook complete the paper.
2 System Architecture
The architecture of the (RS)2I is similar to tradi-
tional image databases and consists of
• Graphical user interface supporting query-by-
example,
• Relational database system storing the meta-
information, i.e. the world-oriented features,
and the a-priori extracted features,
• Index structures for accelerated access to all a-
priori known features, and
• Parallel infrastructure for the efficient proc-
essing of the satellite imagery to enable dy-
namic feature extraction while guaranteeing a
reasonable response time.
The following sub-sections describe these com-
ponents in more detail.
2.1 Graphical User Interface
As already noted the graphical user interface al-
lows the user to initiate a similarity search using
a sample image. An illustration of the interface
layout is presented in Figure 1.
Figure 1: Graphical user interface of (RS)2I
The left hand side shows a vertical list of arbi-
trary user-defined query images while the right
hand side displays the ordered retrieval result
according to the chosen similarity measure. The
best match is displayed in the upper left corner of
the retrieval area and is certainly the query image
itself while the following sub-scenes are sections
of the same original satellite image with a quite
similar ground characteristic. On the later ranks
there are images of other scanners which exhibit
a comparable spectral information of related
ground cover.
The feature extraction and retrieval methods
were applied to a collection of multispectral
scenes of Landsat, SPOT, MOMS-02, and
IKONOS which offer spatial resolutions between
4m and 30m. The corresponding image archive
covers an area of more than 204,150 km2 and
contains a mixture of urban, agriculture, and for-
estry areas around the world. An in-depth analy-
sis of the retrieval results can be found in [13]
and is omitted here due to the limited space.
2.2 Database Features
The relational database stores the world-oriented
as well as the extracted content-based features.
The first set mainly consists of the type of scan-
ner, spectral characteristic of the system, acquisi-
tion time, scene location etc. which are supplied
by the respective satellite agency. The spectral
features are based on the outcome of a classifica-
tion process. i.e. the determined classes of ground
cover (Subsection 3.1). In total the feature vector
consists of 24 different elements whereby some
of them are related to the individual spectral band
of the scanner, thus extending the length of the
feature vector even further. The most dominant
features are the means and standard deviation of
the classes, spatial class neighbours, class com-
pactness in the image, size of classes etc. More-
over general features are stored which are not
related to a specific class but the whole classes
together, e.g. the mean over all class centres, av-
erage distance between classes. These enable the
database not just to search for individual classes
but also for groups of classes.
2.3 Index structures
The (RS)2I supports VP-trees (Vantage point) as
an index structure for the content-based retrieval
of the satellite imagery, which were introduced
by CHIUEH [14] and enable an efficient next
neighbour search. The advantage is obvious since
many queries in remote sensing are related to
geographical locations and the direct surround-
ing. The argumentation holds also true for fea-
tures extracted from spectral content since gener-
ally the focus is on the similarity of the spectral
characteristic.
Let M denote a set of n-dimensional vectors
with image features as components. In the inside
nodes of a VP-tree the vantage points are stored.
The leaf nodes contain the values of the extracted
features. The distribution of these values to the
vantage point can be easily demonstrated, if a
binary relation – as shown in Figure 2 – is given.
All values are sorted according to the distance to
the randomly chosen vantage point. Let
µ
be the
median of all distances, thus each sub tree, I> and
I≤ contains half of all points, i.e. features related
to the actual images. Both-sub trees have to be
searched if the distance of the search parameter p
to the vantage point v is in the interval [
µ
-
σ
,
µ
+
σ
].
The next neighbour search starts in the root
node by calculating the distance of the search
parameter q to the vantage point v and
σ
is the –
estimated – maximum distance. The next
neighbour is element of a subset, if the condition
µ
i-
σ
< d(q, v) <
µ
i+1+σ holds.
Figure 2: Sub-division of a-priori extracted
features according to a randomly chosen
vantage point v
All subsets fulfilling this criterion are examined
until the next neighbour is found. An unsuccess-
ful search occurs if the value
σ
is too small, thus
the whole process has to be repeated with a modi-
fied
σ
. This adaptation can be performed by ad-
dition or multiplication of constant value. More-
over, the original algorithm is modified in the
way, that a ranking of the k>1 next neighbours
can be generated.
2.4 Parallel Infrastructure
Dynamic retrieval of satellite imagery requires
the analysis of all image sections in the remote
sensing database and produces an enormous
computational load since the requirements grow
exponentially by using arbitrary sections instead
of pre-defined tiles (Subsection 3.2). Therefore,
the utilisation of parallel architectures is neces-
sary for the solution of the performance problem.
An analysis of the main demands of the (RS)2I
shows that not the computing power but the huge
data volume and the communication are the lim-
iting factors [15], [16]. Thus cluster architectures
are the most suitable choice due to the shared
data transfer effort and the reasonable price per
node.
The proposed architecture for the (RS)2I is
based on a Beowulf cluster which consist of a
master for receiving queries, broadcasting the
work schedule, and unifying the intermediate re-
sults. The actual processing is done by the com-
puting nodes which access a disjoint subset of the
images in the database. Those are content in-
dependently partitioned because every content re-
lated partitioning restricts the general flexibility.
Note that only the dynamic feature extraction
requires parallel computing resources to keep the
response times short. All query operations based
on a-priori information are executed on the mas-
ter node since the implemented VP-trees enables
efficient processing.
Usually a-priori extracted features and dy-
namic approaches are used together in a query,
e.g. in the mentioned query on swampland. In the
first processing step the world-oriented and static
features lead to a selection of images which are
arbitrary distributed over the entire cluster – in
the worst case on a single node. Only a redistri-
bution of the data that has to be processed in the
dynamic phase enables the full use of the avail-
able computing resources. A simple but efficient
strategy for this workload balancing for the
(RS)2I was described in [17].
3 Data Processing
The key factors for the system are the processing
of incoming data to extract the feature vectors
and the processing for the dynamic retrieval.
From the perspective of signal processing both
parts are considered as being identically and there
will be no distinction henceforth.
3.1 Feature Extraction
In the following a brief overview of the feature
extraction technique will be given which results
in the described feature vector mentioned in Sub-
section 2.2. The emphasis is on a stable technique
for a variety of different multispectral satellite
images and not on a highly precise classification
map since the focus for the retrieval is on meas-
uring the similarity of images rather than on the
exact description of the ground cover.
The algorithm consists of three stages: First
the involved number of classes is estimated
whereby a larger number guarantees that a good
classification result even for scenes with a mani-
fold ground cover can be obtained. Scenes with
less various ground cover will be post-processed
to avoid the break-up of otherwise consistent
classes. Subsequently the actual classification
process is performed based on a modified version
of the algorithm by LOONEY [18]. The advantage
of the technique using a fuzzy approach in com-
bination with a standard k-means classifier is its
stability and reliability. However, the algorithm
was not developed for remotely sensed imagery
and therefore leads to an inadequate number of
resulting classes. Thus, in a second iterative proc-
essing stage classes are combined which do not
fulfil certain criteria, e.g. their appearance is not
of significance for the feature extraction. Note
that this constraint is not necessarily related to the
size of a class but also to its distance towards the
next closest class. A third step in the processing
chain post-processes the data and satisfy physical
constraints. For a detailed description of the pro-
posed method refer to [13].
3.2 Sub-division of Imagery
The unsupervised classification of remotely
sensed imagery, i.e. the feature extraction, is a
computational burdensome process, particularly
due to the size of the data which can easily be
larger than 3000×3000 pixel per image. The re-
quired processing time depends highly on the
image size and grows, like shown in Figure 3(a),
exponentially for a pre-defined number of con-
tained classes. A smaller, but still significant,
influence has the number of classes, in which the
satellite image has to be classified. Figure 3(b)
depicts an overview for a fixed image size of
256×256 pixel. In summary it is reasonable to
separate the data in independent sections, which
can be processed by the cluster simultaneously.
Moreover a sub-division of the images in the
database is believed to be advisable since the
huge coverage in a scene results in a large num-
ber of arising classes. In most cases this number
is considerably larger than the one in the query
image since usually the user’s region of interest
contains less variety in ground cover due to its
smaller extent. Thus a similarity measure be-
tween two different scenes might be influenced in
an undesired way.
A well-known solution for the sub-division of
images is the quad-tree structure. The algorithm
works recursively and divides a given image in
four sub-images if the image fulfils a given crite-
rion, e.g. its variance is greater than a pre-defined
threshold. Figure 4(a) shows an example where
the solid lines indicate the first stage of the algo-
rithm, the dashed lines the second stage, and the
dotted lines the smallest desired division into
sub-images.
0
100
200
300
400
0 100 200 300 400 500
Image size
Processing time
(a)
0
2
4
6
8
10
12
14
3 4 5 6 7 8 9 10
Number of Classes
Processing time
(b)
Figure 3: Required processing time for classi-
fication, (a) variable image size and constant
number of classes, (b) variable number of
classes and constant image size
The approach has the advantages to enable pow-
erful indexing techniques in the final database
and to reduce the number of feature vectors
which have to be stored. However, the idea to
find homogenous areas is not always beneficial,
e.g. if a medium sized urban area along the lake-
shore is of interest. The quad-tree would separate
the water from the building area. A way out is not
just to classify the resulting tiles at the bottom
level of the tree structure but the intermediate
sub-images, too. This is not feasible in practice
due to the significant increase of the burdensome
computation for the larger sub-images, like indi-
cated in Figure 3(a). Therefore this paper follows
the approach to arrange the remotely sensed
image in tiles of the same size. A second set of
tiles, which overlaps the first set, is used to partly
compensate for the discontinuities along the
edges of the tiles. The process is shown in Figure
4(b) where the solid lines indicate the first set and
the dashed lines the second overlapping set.
(a)
(b)
Figure 4: Sub-division of an image: (a) quad-
tree approach, (b) overlapping tiles
Although the approach increases the storage re-
quirements for the feature vectors, it is compu-
tational more economically than the quad-tree in
this particular case. However, there are a few
drawbacks to this solution. The main disadvan-
tage is that the size of the tiles has to be deter-
mined in advanced and does not necessarily
match an appropriate choice with respect to the
size of the query image. One way out of this
problem is the application of the dynamic ap-
proach which increases the needed processing
time dramatically. Instead a modified quad-tree
approach is proposed that is not based on the
pixel values itself but on the next higher level of
abstraction – namely the extracted features. Thus
a new feature vector is created without process-
ing the image data. Although the results are dif-
ferent from the actual extracted features the de-
gree of similarity is sufficiently large due to the
immense reduction of the dimensionality from
the image to the describing vector which sup-
presses not required details. The fusion process is
straightforward and is based on statistics since
the classes themselves are expressed by statistical
measures.
4 Conclusion
A system for the content-based retrieval of mul-
tispectral satellite imagery was presented. The
approach combines the idea of using world-ori-
ented features as well as a-priori and dynamically
extracted feature to enable queries which were
not possible with standard image databases for
remotely sensed optical data. The results have
shown that the utilisation of the provided mecha-
nisms makes complex retrieval operation even for
persons without in-depth remote sensing back-
ground knowledge possibly.
The two main foci of the paper are on the
image database system architecture with its un-
derlying functionality and specific parts of the
data processing.
Future work includes the incorporation of ad-
ditional feature extraction mechanisms beyond
the usage of the spectral information. Particularly
the exploitation of spatial features is of interest
which is not only characterised by shape but also
by texture, i.e. the spatial distribution of spectral
information.
References
[1] J.T. Anderson, M. Stonebraker, “Sequoia 2000
metadata schema for satellite images”,
SIGMOD Record, vol. 23, no. 4, pp. 42–48,
1994.
[2] C.-S. Li, J.J. Turek, and E.Feig, “Progressive
template matching for content-based retrieval
of earth observing satellite image database”,
Proceedings of the SPIE, vol. 2606, pp. 134–
144, 1995.
[3] T. Stephenson, H. Voorhees, “IMACT: An
interactive, multiterabyte image archive”, Pro-
ceedings of the IEEE Symposium on Mass
Storage Systems, pp. 146–161, 1995.
[4] D. Chang, B. Moon, A. Acharya, and C. Shock,
“Titan: A high-performance remote sensing
database”, Proceedings of the Conference on
Data Engineering, pp. 375–383, 1997.
[5] C.T. Shock, C. Chang, B. Moon, A. Acharya,
L. Davis, J. Saltz, and A. Sussman, “The de-
sign and evaluation of a high-performance
earth science database”. Parallel Computing,
vol. 24, pp. 65–89, 1998.
[6] G.B. Marchisio, W.-H. Li, M. Sannella, and
J.R. Goldschneider, “GeoBrowser: An inte-
grated environment for satellite image retrieval
and mining”, Proceedings of the IEEE Interna-
tional Geoscience and Remote Sensing Sympo-
sium, vol. 2, pp. 669–673, 1998.
[7] A. Vellaikal, C. Kuo, and S. Dao, “Content-
based retrieval of remote sensed images using
vector quantization”, Proceedings of SPIE, vol.
2488, pp. 178–189, 1995.
[8] J. Barros, J. French, and W. Martin, “System
for indexing multi-spectral satellite images for
efficient content-based retrieval”, Proceedings
of the SPIE, vol. 2420, pp. 228–237, 1995.
[9] K. Seidel, M. Schröder, H. Rehrauer, G.
Schwarz, and M. Datcu, “Query by image
content from remote sensing archives”, Pro-
ceedings of the IEEE International Conference
on Geoscience and Remote Sensing, vol. 1, pp.
393–396, 1998.
[10] D.G. Healey, A. Jain, “Retrieving multispectral
satellite images using physics-based invariant
representations“, IEEE Transactions on Pattern
Analysis and Machine Intelligence, vol. 18, no.
8, pp. 842–848, 1996.
[11] G.B. Marchisio, J. Cornelison, “Content-based
search and clustering of remotely sensing im-
agery”, Proceedings of the IEEE International
Conference on Geoscience and Remote Sens-
ing, vol. 1, pp. 290–292, 1999.
[12] L. M. del Val Cura, N.J. Leite, and C.B.
Medeiros, “An architecture for content-based
retrieval of remote sensing images”. Proceed-
ings of the IEEE International Conference on
Multimedia and Expo, vol. 1, pp. 303–306,
2000.
[13] T. Bretschneider, R. Cavet, and O.Kao, “Re-
trieval of Remotely Sensed Imagery Using
Spectral Information Content”, to appear in:
Proceedings of the International Geoscience
and Remote Sensing Symposium 2002, Toronto,
Canada.
[14] T. Chiueh, “Content-based image indexing”,
Proceedings of the 20th Conference on Very
Large Databases, 582--593, 1994.
[15] T. Bretschneider, S. Geisler, and O. Kao,
“Simulation-based assessment of parallel ar-
chitectures for image databases”, to be pub-
lished in: Proceedings of the Conference on
Parallel Computing, 2001.
[16] S. Geisler, O. Kao, and T. Bretschneider,
“Analysis of cluster topologies for workload
balancing strategies in image databases”, Pro-
ceedings of the International Conference on
Parallel and Distributed Processing Tech-
niques and Applications, pp. 874–880, 2001.
[17] T. Bretschneider, O. Kao, “Retrieval of Multi-
spectral Satellite Imagery on Cluster Architec-
tures”, submitted to EuroPar 2002, Paderborn,
Germany.
[18] C. Looney, “A fuzzy clustering and fuzzy
merging algorithm”, Technical Report, CS-
UNR-101-1999, 1999.
