An Image Retrieval System Based on Colors and Shapes of Objects.

Full text

Page 1

An Image Retrieval System Based on Colors and Shapes of Objects

Kuo-Lung Hong1, Yung-Fu Chen2, Yung-Kuan Chan3, Chung-Chuan Cheng1
1Department of Management Information Systems
Chaoyang University of Technology, Taiwan
{klhung, s9314638}@cyut.edu.tw
2Department of Computer Science and Information Engineering
Dayeh University, Taiwan
yfbchen@mail.dyu.edu.tw
3Department of Management Information Systems
National Chung Hsing University, Taiwan
ykchan@nchu.edu.tw


Abstract

This paper proposes a context-based image
retrieval method, called color-shape based method
(CSBM), based on the colors and shapes of objects in
an image. The proposed method characterizes the
shape of an object by the object’s area and the
intercepted lengths obtained by intercepting the
object perimeter by eight lines with different
orientations passing through the object center. The
experimental results show that CSBM provides a
better performance than FCH and CCH. Besides, it
is insensitive to translation, rotation, distortion, scale,
and hue variations, but impressionable to contrast
and noise variations.


1 Introduction

A tremendous number of images have been
generated everyday in various applications. Hence, it
is necessary for developing an efficient image
retrieval technique to take care of images.
Context-based image retrieval has been studied for
more than two decades, which generally works with
features including color, texture, and shape [5]. Color
histogram is one of the most commonly adopted
features for designing image retrieval systems [7-12].
The advantages of color histogram are simple in
operation and easy for calculation. Additionally, the
feature is less affected by the rotation and translation
variations of the objects in an image. In order to
improve the efficacy of the conventional color
histogram (CCH) [17], a fuzzy-based technique,
namely fuzzy color histogram (FCH) [13], is
proposed for color histogram construction. The
method considers the degree of color similarity for
each pixel to be associated with all the histogram
bins using fuzzy memberships, in which the
memberships are evaluated based on the fuzzy
c-means algorithm [14-16]. It is demonstrated to be
more robust than the CCH in dealing with
quantization errors and changes in light intensity.
However, color histogram has a significant
drawback that it can delineate only the global
properties of an image because the local information
for individual objects is simply merged and diluted.
For example, a color histogram cannot express the
properties such as shapes and sizes of objects in an
image. To overcome the problem above, this paper
proposes an image retrieval method based on the
features including the colors as well as areas of the
objects, and the intercepted lengths obtained by
intercepting the object perimeter by eight lines with
different orientations passing through the object
center. The intercepted lengths are valuable in
discriminating shapes of the objects and are immune
to translation and rotation variations. In addition,
scaling invariance is achieved by normalizations that
the area of each object is divided by the image size
and the intercepted lengths by the sum of width and
height of the image. We call this method color-shape
based method (CSBM).
In CSBM, after inputting a query image Iq, the
features of Iq are compared with those of each
database image. The database images, most similar to
Iq, are delivered and ranked according to their
similarities to Iq. The experimental results show that
this method can efficiently distinguish two images by
the colors, areas, and shapes of the objects on both
images. Its retrieval accuracy is very robust and less
affected by translation, rotation, and scaling
variations


2 Related Works

Color histogram has been widely used as a feature
for image retrieval application [17]. It first classifies
colors of the database images as k clusters, known as
color bins, using K-mean algorithm [18], in which
the representative of each cluster is obtained by
calculating its centroid. Then, the color histogram of
an image can be obtained by assigning each pixel to
a color bin with smallest distance. By observing the
distribution of the color histogram, one can
~290~
The 23rd Workshop on Combinatorial Mathematics and Computation Theory

Page 2

distinguish how similar two images are. The measure
of similarity between two images is defined as:
∑ ∑ ∑ ∑
= = = =
j
1
where Hj(I) and Hj(I’) indicate the j-th bin of k color
histograms for image I and I'. The advantages for
using color histogram as a feature are that it is very
simple in operation and easy for calculation.
Additionally, it is immune to shift, rotation, and
scaling operations [7-12]. Although it has the
aforementioned advantages, there are still several
accompanied drawbacks, including: (1) the feature
can be determined only by the number of pixels in
each color bin; (2) the number of color bins should
be large enough to have better retrieval efficiency,
which in turn needs more memory space and
computation time; and (3) the feature only accounts
for global rather than local characteristics in an
image; hence it cannot be used for local image
retrieval.
Since CCH considers neither color similarity
across different bins nor color dissimilarity in the
same bin, it is sensitive to noisy interference coming
from illumination changes and quantization errors.
Hence it motivated Han and Ma [13] to develop an
image retrieval system based on the fuzzy color
histogram (FCH) which associates the color
similarity of each pixel to all the color bins through a
fuzzy-set membership function. The FCH, F(I) = [f1,
f2,… , fn], of an image I consisting N pixels can be
defined as:
∑ ∑ ∑ ∑
= = = =
j
1
where N is the number of color bins in the FCH, Pj
represents the probability of a pixel j selected from
image I, and µij (0≦
colors in CIELAB color space into n clusters using
the FCM clustering technique that each cluster
represents an FCH bin.
During image retrieval, the FCH of a query image
is extracted using the above approach, and then the
system compares the FCH with the FCHs of all the
database images. Dissimilarity between two images
with FCHs represented as FQ and FT which indicate
query and database image, respectively, can be




∑ ∑ ∑ ∑
= = = =
j
′ ′ ′ ′′ ′ ′ ′
= = = =
k
j
k
jj
IHIHIHs
1
)( ))(),(min(

∑ ∑ ∑ ∑
= = = =
j
= = = == = = =
NN
ijjiji
N
Pf
1
1
µ µ µ µµ µ µ µ

≦≦
≦µij
1)
≦
≦≦
≦ indicates the
membership degree of the j-th pixel to the i-th color
bin.
To speed up the computation, fine uniform
quantization is performed in the RGB color space by
mapping all the pixel colors into n' color histogram
bins, that n' must be large enough so that the color
difference between two adjacent bins is significantly
small. Then, the n' colors are transformed from RGB
to CIELAB color space. Finally, it classifies the n'
measured using the Euclidean distance, as defined
below:
[),(
QTQ
FFFd
− − − −= = = =
Fuzzy C-Means (FCM)
algorithm is used to compute the FCH. It classifies
the n' fine colors into n coarse colors and calculates
the membership matrix at the same time. The optimal
solution is obtained by minimizing an objective
function Jm, which is the weighted sum of squared
errors within each group:
∑ ∑ ∑ ∑∑ ∑ ∑ ∑
= = = =
ki
1
The algorithm stops when Jm is less then a given
ε. Here,
][]
T2
T
clustering
QT
FFF
− − − −

[14-16]
= = = =
− − − −= = = =
nc
ik
m
ikm
A
vxXVU
(
J
1
2
);,
µ µ µ µ

threshold
T
c21
]v ,..., v ,v[V = = = =

represents a vector of unknown cluster prototypes,
and the weighting exponent m controls the degree of
membership shared by c clusters. The value of µ µ µ µik
indicates the degree of membership for a data point
xk in set X = {x1, x2, …, xn} related to i-th cluster.
The membership for a data point to any cluster can
be delineated by a matrix U = [
ik
µ ],
][ 1 , 0
ik∈ ∈ ∈ ∈
µ µ µ µ
,
which satisfies the following constraints:
∑ ∑ ∑ ∑
= = = =
1k
The drawbacks of the FCH are that the feature can
only be used for accounting the global characteristics
and it needs a great computation time for calculating
the memberships.


3 Color-Shape Based Method

In a true color image, a pixel is generally
represented with a 24-bit number, such that there are
totally 224 different possible colors. If the possible
pixels’ colors in a full color image are reduced to a
few ones, the image usually becomes the image
consisting of several large regions, each of which is
made up of a set of pixels with the same color. In the
real world, there exist many such images, i.e.
trademark, cartoon, flag, traffic sign, and synthesized
images. Content in this sort of images is usually
composed of a group of objects. However,
segmenting the objects contained in an image is a
difficult task. Thus, this paper regards a region,
formed with identical color pixels, as one object.
Two similar images generally contain some bigger
objects that have similar colors and shapes. Figure
1(a) shows an image containing five objects A, B, C,
D and E. As shown in this figure, objects A and B are
usually treated as two different objects because
∈ ∈ ∈ ∈≤ ≤ ≤ ≤≤ ≤ ≤ ≤ = = = =
n
iik
Xxnk1, 1
and
µ µ µ µ

~291~
The 23rd Workshop on Combinatorial Mathematics and Computation Theory

Page 3

Figure 2: Objects having the same area but different shapes and PILs

they have different shapes even they have identical
color. Objects A and C are also different because
their colors are quite distinct although they have the
same shapes and sizes.

3.1 Feature Extraction

The color-shape based method first classifies the
pixels of all database images into K clusters using the
K-mean clustering algorithm [18]. The mean value of
each cluster is regarded as a representative color in a
color palette, namely common color palette (CCP),
for all images including database and query images.
For an image I, a color-reduced image I' containing
only K colors is generated by replacing each pixel in
I with a closest color in CCP. Hence, I' has much
less colors compared to I. The content of I' is
generally comprised of a group of big objects, each
with a uniform pixel color.
Every object has color, area, and shape. For an
object O with its area greater than a threshold value
ThA, the features including color, area, and perimeter
intercepted lengths (PIL) are computed and recorded;
otherwise, O is treated as a noise and ignored. The
area is featured as the number of pixels within O,
whereas the shape as the lengths of 8 perimeter
intercepts, L={l0, l1, …, l7}, obtained by 8 lines
intersecting the perimeter of O.
CSBM categorizes the directions from 0˚ to 360˚
into eight directions 0˚, 22.5˚, 45˚, 67.5˚, 90˚, 112.5˚,
135˚, and 157.5˚ signified by θ0, θ1,…, and θ7
respectively. The method gives a minimal bounding
rectangle (MBR) containing O where the rectangle is
oriented parallel to the X and Y axes, and it draws
eight lines through the central pixel of O’s MBR
where the included angles between the eight lines
and the X-axis are θ0, θ1,…, and θ7 respectively.
Figure 1 shows the eight lines. Let Li be the i-th one
of the eight lines, and P1, P2, …, Pe be the
intersections of Li and the boundary of O. We define
li, the maximal distance between any two of P1,
P2, …, Pe, as the length of Li. For object O, CSBM
writes down its color, area, and PILs L={l0, l1, …, l7}

into database.
The shape of an identified object is very crucial for
context-based approach to measure the similarity of
two images. The difference between the PILs of two
objects is demonstrated to be a good indicator of
shape variation. In case that they are identical in
areas but different in shape, their PILs are generally
different. As shown in Figure 2, although the areas of
all three objects are very close, their shapes are
greatly different; hence their PILs are quite different
too. To achieve rotation invariant, two objects are
compared in 8 different orientations that the smallest
Euclidean distance is used as an indicator for shape
differentiation.
When a tool, like a scanner, is used to input an
image, the image may be enlarged or shrunk because
of different scanner resolution setups. We call this
phenomenon scale variation. Let I be an image of
H×W pixels. To remedy the problem of scale
variations, this paper divides each value of PILs and
the area of each object in I by (H+W) and by (H×W),
respectively. CSBM is hence able to resist the impact
of scale variation.

3.2 Image Matching

q
1h
O
,
2h
O
, …,
O
Let
q
q
hnq
h
and
d
h1
O
,
d
h2
O
, …,
d
hnd
h
O
be all the objects with the h-th color of CCP
respectively in the color-reduced images
qI′ ′ ′ ′ and
dI′ ′ ′ ′ of the query image Iq and one database image Id.
q
h
n and
d
h
n are the number of objects, whose
colors are the h-th color in CCP, in
qI′ ′ ′ ′ and
dI′ ′ ′ ′ .
CSBM calculates the matching distance dij of two
q
hi
O
and
hj
O
follows:
dij =
ij
d + w×
objects
d
via
L
ij
d
and
A
ij
d
as
LA
ij
d
,
Figure 1: (a) An image and (b) The 8 perimeter intercepts of one object bounded by an MBR
A
B
C
D
E
(a)
(b)
The 23rd Workshop on Combinatorial Mathematics and Computation Theory
~292~

Page 4

Algorithm OBJ_Match( ):
Input:
q
h1
O
,
q
h2
O
, …,
q
hnq
h
O
and
d
h1
O
,
d
h2
O
, …,
d
hnd
h
O
.
Output:
h
MMD
h
0 0 ,
MMD
= 0;
for i = 0 to
q
h
n
h
0 , i
MMD
= Penalty(
q
hi
O )+
h
i0 , 1
MMD− − − −
;
for j = 0 to
d
h
n
h
0 j ,
MMD
= Penalty(
d
hj
O )+
h
01 j ,
MMD
− − − −;
for i = 1 to
q
h
n
for j = 2 to
d
h
n

h
j , i
MMD
=min(
h
i j , 1
MMD− − − −
+ Penalty(
q
hi
O ),
h
1 j , i
MMD
− − − −+ Penalty(
d
hj
O ),
h
1j1 , i
MMD
− − − −− − − −
+di,j)

where w is a given constant;
L
ij
d
and
A
ij
d
are
respectively the matching distances of PILs and areas
q
hi
O and
hj
O . Assume that (
of
dq
0l ,
q
1l , …,
q
7l )
and (
d
0l ,
d
1l , …,
d
7l ) are the PILs, and
q
i A as well
as
d
j A are the areas of
q
hi
O
and
d
hj
O
. The
color-shape based method defines
L
ij
d and
A
ij
d
as
follows, where z is a given constants.
z
7
0s
z
d
(
8modsr
q
s
7
0r
L
ij
llmin
= = = =
d
∑ ∑ ∑ ∑
= = = =
+ + + +
− − − −= = = =
)(
)
, and
d
j
q
i
A
ij
AAd
− − − −= = = =
.
CSBM adopts a dynamic programming algorithm
OBJ_Match( ) to calculate the minimal matching
distance between two sets of objects
q
h1
O
,
q
h2
O
, …,
q
hnq
h
O
and
d
h1
O
,
d
h2
O
, …,
d
hnd
h
O
. In this algorithm,
a two-dimension array MMDh, consisting of
q
h
n ×
h
n , is used to write down the minimal
d
matching distance of
q
h1
O
,
q
h2
O
, …,
q
hnq
h
O
and
d
h1
O
,
d
h2
O
, …,
d
hnd
h
O
. Entry
h
j , i
MMD
records
the minimal matching distance between
q
h1
O
,
q
h2
O
, …,
q
hi
O
and
d
h1
O
,
d
h2
O
, …,
d
hj
O
;
therefore, entry
h
n ,n
d
h
q
h
MMD
describes the final
minimal matching distance of
q
h1
O
,
q
h2
O
, …,
q
hnq
h
O
and
d
h1
O
,
d
h2
O
, …,
d
hnd
h
O
. In this algorithm,
each object in
q
h1
O
,
q
h2
O
, …,
q
hnq
h
O
may map into
at most one object in
d
h1
O
,
d
h2
O
, …,
d
hnd
h
O
, and
vice versa. For an object O which does not be
matched, the algorithm gives a penalty function
Penalty(O) which is defined as the following:
w,AlO
r
z
7
0r
z
r
× × × ×+ + + += = = = ∑ ∑ ∑ ∑
= = = =
)(
Penalty

where (l1, l2, …, l7) are the PILs and Ar is the area of
O.
Algorithm OBJ_Match is a function to compute
the minimal matching distance of the objects with the
h-th color of CCP in
color-shape based method defines the image
matching distance Dist of Iq and Id as
Dist =∑ ∑ ∑ ∑
= = = =
1h

3.3 System Evaluation

In this work, the average normalized modified
retrieval rank (ANMRR) proposed by MPEG-7
[19-20] are used as a benchmark for system
evaluation. The ANMRR not only reflects the recall
rate and precision information of the retrieved
images, but also shows the ranks of all the retrieved
images. Before defining ANMRR, several terms
must be defined beforehand.
For each query q, the averaged rank (AVR) can
be obtained by:
∑ ∑ ∑ ∑
= = = =
1
t
in which I(q) indicates the number of images in the
database which are most similar to the query image;
R(t) represents the rank of a returned image, which is
defined as:
   
+ + + +
T
, 1
qI′ ′ ′ ′ and
dI′ ′ ′ ′ . Finally, the
K
h
n ,n
d
h
q
h
MMD
.
′ ′ ′ ′
= = = =
)(
)()()(
qI
qItRqAVR
,
   
   
> > > >
≤ ≤ ≤ ≤
= = = =
′ ′ ′ ′
TtR
TtRtR
tR
)( if
)( if),(
)(
,

~293~
The 23rd Workshop on Combinatorial Mathematics and Computation Theory

Page 5

Table 1: Examples of classified query and database image pairs.
Category Rotation Scaling Translation
Query Image





Database Image




Category Noise Blurring Natural Image
Query Image



Database Image




where T is defined by min(4×NG(q), 2×GTM) with
GTM representing max{I(q)} for each query, and t
indicates the number of returned images which are
most similar to the query image q.
For each query image, the value of AVR depends
on I(q); it is necessary to minimize the influence of
I(q) on ranking. Hence, the modified retrieval rank
(MRR) can be defined as:
2
)(
- 0.5 -
qI
AVR(q)MRR(q)= = = =
.
As shown in the above equation, the value of MRR
depends on I(q), which can be normalized to the
value between 0 and 1. The normalized modified
retrieval rank (NMRR) was defined as:
)q ( I5 . 05 . 0T
)q
× × × ×
(MRR
− − − −
)q( NMRR
+ + + +
= = = =
.
For a perfect retrieval NMRR=0, which indicates
that the system can successfully retrieve all the
images that are similar to the query image. If the
system retrieves none of the images, then NMRR=1.
Finally, by considering all the queries, the
performance index can be obtained by calculating the
averaged normalized modified
(ANMRR), as shown below:
retrieval rank
Q)q(NMRRANMRR
Q
1q∑ ∑ ∑ ∑
= = = =
= = = =
,
where Q is the total number of queries.
For example, if a system which is requested to
return the most 15 similar images, then T=15.
Furthermore, if there are 6 images retrieved by the
system for each query image, then I(q) = 6. Suppose
the most four similar images are ranked as 1, 4, 9,
and 14, respectively, while the other two images are
ranked outside the top 15, the values of AVR(q) and
MRR(q) are 10 and 0.52, respectively.


4 Experiments and Discussions

This section investigates the performance of CSBM
and compares it to those of FCH and CCH by two
experiments. The first experiment is to test the
capacity for recognizing images, while the second
experiment explores its sensitivities to the rotation,
scaling, translation, noise, blurring, luminance,
contrast, hue, and distortion variations of images.
According to our other experiments, when given the
weighted parameters w and z to be 0.009 and 1.1
respectively, CSBM can give a better performance.
Hence, the following experiments will specify both
parameters to be above values.
The first experiment uses two sets of hybrid
images SetD={I
I
500} as testing images. Each of the two image sets
consists of 500 full color images including
synthesized images and natural images. The images
in SetD are used as the database images, while those
in SetQ as the query images. In this experiment, each
image pair (I
i, I
that is, the user obtaining the correct answer, if I
q
i as the query image. Table 1
shows a small part of images in both sets, which have
different variations. Table 2 shows the performances
of CSBM for CCP’s size K (= 16 and 27) with three
different area thresholds ThA (= 5, 10, and 20).
As shown in Table 2, CSBM provides better
performances than FCH and CCH whatever they are
measured by ANMRR.
d
1, I
d
2,…, I
d
500} and SetQ={I
q
1, I
q
2,…,
q
qd
i) are considered to be similar images,
d
i is
delivered for given I


~294~
The 23rd Workshop on Combinatorial Mathematics and Computation Theory