A new method for fuzzy information retrieval based on fuzzy hierarchical clustering and fuzzy inference techniques

Abstract

In this paper, we extend the work of Kraft et al. to present a new method for fuzzy information retrieval based on fuzzy hierarchical clustering and fuzzy inference techniques. First, we present a fuzzy agglomerative hierarchical clustering algorithm for clustering documents and to get the document cluster centers of document clusters. Then, we present a method to construct fuzzy logic rules based on the document clusters and their document cluster centers. Finally, we apply the constructed fuzzy logic rules to modify the user's query for query expansion and to guide the information retrieval system to retrieve documents relevant to the user's request. The fuzzy logic rules can represent three kinds of fuzzy relationships (i.e., fuzzy positive association relationship, fuzzy specialization relationship and fuzzy generalization relationship) between index terms. The proposed fuzzy information retrieval method is more flexible and more intelligent than the existing methods due to the fact that it can expand users' queries for fuzzy information retrieval in a more effective manner.

Full text

Page 1

216 IEEE TRANSACTIONS ON FUZZY SYSTEMS, VOL. 13, NO. 2, APRIL 2005
A New Method for Fuzzy Information Retrieval
Based on Fuzzy Hierarchical Clustering
and Fuzzy Inference Techniques
Yih-Jen Horng, Shyi-Ming Chen, Senior Member, IEEE, Yu-Chuan Chang, and Chia-Hoang Lee
Abstract—In this paper, we extend the work of Kraft et al.
to present a new method for fuzzy information retrieval based
on fuzzy hierarchical clustering and fuzzy inference techniques.
First, we present a fuzzy agglomerative hierarchical clustering al-
gorithm for clustering documents and to get the document cluster
centers of document clusters. Then, we present a method to con-
struct fuzzy logic rules based on the document clusters and their
document cluster centers. Finally, we apply the constructed fuzzy
logic rules to modify the user’s query for query expansion and
to guide the information retrieval system to retrieve documents
relevant to the user’s request. The fuzzy logic rules can represent
three kinds of fuzzy relationships (i.e., fuzzy positive association
relationship, fuzzy specialization relationship and fuzzy gener-
alization relationship) between index terms. The proposed fuzzy
information retrieval method is more flexible and more intelligent
than the existing methods due to the fact that it can expand users’
queries for fuzzy information retrieval in a more effective manner.
Index Terms—Fuzzy agglomerative hierarchical clustering,
fuzzy information retrieval systems, fuzzy logic rules, fuzzy rela-
tionships, query expansion.
I. INTRODUCTION
T
respect to a user’s queries and retrieve the documents with a
high degree of satisfaction to the user. In order to get good re-
trieval performance, the user’s query must be able to appropri-
ately describe the user’s requests. Currently, most of the com-
mercial information retrieval systems are based on the Boolean
logic model. They assume that a user’s queries can precisely
be characterized by the index terms. However, this assumption
is inappropriate due to the fact that the user’s queries may con-
tainfuzziness[22].Thereasonforthefuzzinesscontainedinthe
user’squeriesisthattheusermaynotknowmuchaboutthesub-
ject he/she is searching or may not be familiar with the informa-
tion retrieval system. Therefore, the query specified by the user
may not describe the information request properly. Since fuzzy
HE goal of an information retrieval system is to evaluate
the degrees of relevance of the collected documents with
Manuscript received October 7, 2002; revised November 6, 2003, March 2,
2004, and May 25, 2004. This work was supported in part by the National Sci-
ence Council, Republic of China, under Grant NSC 91-2213-E-011-052.
Y.-J. Horng and C.-H. Lee are with the Department of Computer and Infor-
mation Science, National Chiao Tung University, Hsinchu 300, Taiwan, R.O.C.
S.-M. Chen and Y.-C. Chang are with the Department of Computer Science
and Information Engineering, National Taiwan University of Science and Tech-
nology, Taipei 106, Taiwan, R.O.C.
Digital Object Identifier 10.1109/TFUZZ.2004.840134
set theory [29] can be used to describe imprecise or fuzzy infor-
mation, many researchers have applied the fuzzy set theory to
information retrieval systems [2], [3], [8], [9], [13], [18]–[22].
In [2], Bordogna et al. presented a relevance feedback model
based on associative neural networks to provide an association
mechanism in information retrieval systems. The purpose of the
association mechanism in information retrieval systems is to
build the association relationships between index terms and to
modify the user’s queries by adding or replacing index terms
associated with the queries. Generally speaking, the modified
user’s queries should find more relevant documents than that
of the original user’s queries and thus improve the retrieval
performance. Therefore, the study of the association mecha-
nism is very important in the field of information retrieval. In
[3], Chen et al. presented a fuzzy-based concept information
systemthatintegrateshumancategorizationandnumericalclus-
tering. In [8], Chen et al. presented a method for document re-
trieval using knowledge-based fuzzy information retrieval tech-
niques. In [9], Chen et al. presented fuzzy information retrieval
techniques based on multi-relationship fuzzy concept networks.
In [13], Horng et al. presented a fuzzy information retrieval
method based on document terms reweighting techniques,
In [20], Kraft et al. explored several ways of using fuzzy
clustering techniques in information retrieval systems, where
the most important one is to capture the relationships among
index terms. They use fuzzy logic rules to represent the associ-
ation relationships between index terms and to form the basis
of the association mechanism. After a user submits his/her
queries, the fuzzy logic rules are then applied under a fuzzy
logic system to modify the user’s original queries. Experi-
mental results show that the modified user’s queries can get a
better retrieval performance than the original queries. In [20],
Kraft et al. utilized the complete link clustering method and
the fuzzy c-means clustering method to partition documents
for information retrieval.
Inthispaper,weextendtheworkofKraftetal.[20]topresent
a new method to modify a user’s queries for fuzzy informa-
tion retrieval. First, we present a fuzzy agglomerative hierar-
chical clustering algorithm for clustering documents and to get
the document cluster center of each document cluster. Then, we
present a method to construct fuzzy logic rules based on the
document clusters and their document cluster centers. Finally,
we apply the constructed fuzzy logic rules to modify the user’s
queryfor queryexpansionandtoguidetheinformationretrieval
system to retrieve documents relevant to a user’s request. The
1063-6706/$20.00 © 2005 IEEE

Page 2

HORNG et al.: NEW METHOD FOR FUZZY INFORMATION RETRIEVAL217
fuzzy logic rules can represent three kinds of fuzzy relation-
ships (i.e., fuzzy positive association relationship, fuzzy spe-
cialization relationship, and fuzzy generalization relationship)
between index terms. The proposed document retrieval method
is more flexible and more intelligent than the existing methods
due to the fact that it can expand users’ queries for fuzzy infor-
mation retrieval in a more effective manner.
Therestofthispaperisorganizedasfollows.InSectionII,we
briefly review some clustering methods. In Section III, we pro-
pose a new fuzzy agglomerative hierarchical clustering method.
In Section IV, we compare the clustering performance of the
proposed fuzzy agglomerative hierarchical clustering method
with that of the complete link clustering method. In Section V,
we propose a new method for fuzzy logic rules discovery. In
Section VI, we propose a new method for query modification
for fuzzy information retrieval based on the constructed fuzzy
logic rules. The conclusions are discussed in Section VII.
II. A REVIEW OF CLUSTERING METHODS
The single link clustering (SLC) method is one of the hier-
archical agglomerative clustering methods (HACM). In [24],
Rohlf reviewed some algorithms of the single link clustering
method. The time complexity of these algorithms ranges from
to, where
of these algorithms are not suitable for information retrieval ap-
plications, when the data sets have large
sionality [23]. The SLINK algorithm [28] is one of the single
link clustering methods. In [28], Sibson pointed out that the
SLINK algorithm is an optimally efficient algorithm for the
single link clustering method. In [23], Rasmussen pointed out
that the SLINK algorithm generates the hierarchy in a form
known as the pointer representation. The SLINK method is one
of the SLC methods [24] and it also takes the major character-
istic of SLC, where the degree of similarity between two doc-
ument clusters is defined by the maximal degree between any
pairs of documents, each of which is in one of the two clusters.
In [20], Kraft et al. utilized the agglomerative hierarchical
clustering (AHC) method [23] and the fuzzy c-means clustering
method [1] to partition documents. In the following, we briefly
review these two clustering methods.
The agglomerative hierarchical clustering method for parti-
tioning the given documents is reviewed from [20] as follows.
Step 1)
Let each document be a document cluster.
Step 2)
Merge two document clusters which have the
largest degree of similarity into one document
cluster.
Step 3)
If the degree of similarity between any two docu-
ment clusters is smaller than a heuristic threshold
value
, where
else, go to Step 2).
In [20], Kraft et al. pointed out that the measurement of the
degreeof similaritybetweentwo documentclusters canbe done
by the CLC method [23], which takes the minimum degree of
similarity between any pairs of documents, each of which is in
one of the two clusters (i.e., one document from the first docu-
ment cluster and the other from the second document cluster).
is the number of items. Many
and high dimen-
, then Stop,
It is obvious that the agglomerative hierarchical clustering
method is a crisp clustering method. That is, in the resulting
document clusters, each document can only belong to exactly
one document cluster.
Theotherclusteringmethodusedin[20]isthefuzzyc-means
algorithm [1]. Assume that there is a set of
, where each document
a data point represented by a vector of dimension
follows:
documents,
can be regarded as
shown as
where
of term
thermore, assume that we want to partition the documents into
document clusters, where
rithm will get
document cluster centers and the membership
degreeofeachdocumentbelongingtoeachdocumentclusterby
minimizing the following objective function:
. Here,
, where
denotes the degree of significance
andin . Fur-
. Then, the fuzzy c-means algo-
(1)
Here,
respecttodocumentcluster
center of document cluster
tance from document
over,
and
is a control parameter, where
the degree of fuzziness of the clustering algorithm. The larger
the value of
, the higher the degree of fuzziness, i.e., the like-
lihood of a document belonging to multiple clusters at the same
time is higher. It should be noted that the membership degree
of each document
belonging to each document cluster
must be nonnegative, i.e.,
of the membership degrees of each document
each cluster must be equal to one, i.e.,
. The membership degree of document
to document cluster
is calculated as follows:
denotes the membership degree of document
,
, and
to document cluster center
are represented as vectors of dimension , and
with
denotesthedocumentcluster
denotes the dis-
. More-
. Note thatstands for
. Moreover, the summation
belonging to
, where
belonging
(2)
and the document cluster center
calculated as follows:
of document clusteris
(3)
Initially, the fuzzy c-means algorithm randomly assigns a value
for each
, whereand
Then, the following two steps are performed iteratively until the
difference of the values of
(and
and those in the previous iteration are smaller than some con-
vergencethreshold , where
in formula (3) to obtain the document cluster center
for each .
) in the current iteration
. First,it uses theexisting
. Then,

Page 3

218IEEE TRANSACTIONS ON FUZZY SYSTEMS, VOL. 13, NO. 2, APRIL 2005
it uses the newly derived document cluster center
formula (2) to update the value of
proven that the fuzzy c-means algorithm will converge.
In [20], Kraft et al. used several experts to evaluate the doc-
ument clusters produced by the CLC method [23] and the ones
produced by the fuzzy c-means clustering method [1]. Since
the fuzzy c-means algorithm will produce “fuzzy” document
clusters in the sense that each document may belong to mul-
tiple document clusters, it is in contrast to the “crisp” docu-
ment clusters produced by the complete link clustering method
in which each document only can belong to exactly one docu-
ment cluster. In order to compare the performance of these two
clustering methods, “hardening” is performed to the fuzzy doc-
ument clusters obtained by the fuzzy c-means algorithm. That
is, for each document
, they found a document cluster where
has a maximum membership degree among all the document
clusters. Then, they set the membership degree of document
in this document cluster to one and set the membership degrees
for the other document clusters to zero. From the experts’ opin-
ions [20],thecomplete linkclusteringmethodseemstoperform
better than the fuzzy c-means method. However, since Kraft et
al. performed the “hardening” operation to the fuzzy document
clusters,theexpertshaveonlycomparedtwosetsof“crisp”doc-
umentclustersderivedfromthetwoclusteringmethods,respec-
tively. Therefore, the advantage of the fuzzy clustering method
is not revealed in their experiments. In fact, the fuzzy clustering
method should perform better than the crisp clustering method
when the boundaries between documentclusters are not crisp or
when the document clusters are overlapping.
Because the traditional agglomerative hierarchical clustering
method is a crisp clustering method, where each document can
only belong to exactly one document, it lacks flexibility. There-
fore, in this paper, we extend the work of [20] to present a fuzzy
agglomerative hierarchical clustering method to overcome the
drawback of the traditional agglomerative hierarchical clus-
tering method, where a document can not belong to multiple
document clusters at the same time. Both the proposed fuzzy
agglomerative hierarchical clustering method and the fuzzy
c-means clustering method [1] have the advantage of flexibility
to allow a document to belong to multiple document clusters
at the same time. The difference between the fuzzy c-means
clustering method and the proposed fuzzy agglomerative hier-
archical clustering method is that the fuzzy c-means clustering
method needs to predefine the number of clusters by the user
andtheproposedmethodusesthe“similaritythresholdvalue”
and the “difference threshold value”
of clustering automatically, where
shown in
. In [1], Bezdek et al. have
to deal with the process
and.
III. FUZZY AGGLOMERATIVE HIERARCHICAL
CLUSTERING METHOD
In this section, we present a fuzzy agglomerative hierarchical
clustering (FAHC) for clustering documents. Let the member-
ship degree of document
belonging to cluster
by
, where
cluster
, then . The proposed fuzzy agglomerative
hierarchical clustering method is now presented as follows.
be denoted
. If is an element of
Fuzzy Agglomerative Hierarchical
Clustering Algorithm:
Input: The similarity threshold value
and the difference threshold value
where
and
Output: Document clusters.
Step 1: Let each document be a document
cluster and set the membership degree of
each document belonging to its document
cluster to 1.
Step 2: Find a pair of document clusters
andamong the set of document clus-
ters that has the largest degree of simi-
larity
andis not less than
.
Step 3: Find a set
ment clusters
and
the document clusters that have degrees
of similarity larger than or equal to
the similarity threshold value
.
Step 4: For each pair of document cluster
andin the set
begin
if
and the pair of document
clusters
andobtained in Step 2 and
the pair of document clusters
obtained in Step 3 share the same document
cluster
(i.e.,
begin
make a copy
of document cluster
merge document cluster
cluster
into a new cluster
for each element
cluster
, the membership degree
of document
to the newdocument cluster
is equal to
for each element
, the membership degree
document
belonging to the new
document cluster
;
merge document cluster
cluster
into a new cluster
for each element
, the membership degree
document
belonging to the new
document cluster
;
for each element
, the membership degree
document
belonging to the
new document cluster
,
.
, where
of pairs of docu-
among the rest of
, where
do
and
), then
;
with document
;
in document
belonging
;
in document cluster
of
is equal to
with document
;
in document cluster
of
is equal to
in document cluster
of
is equal to
end
else
begin

Page 4

HORNG et al.: NEW METHOD FOR FUZZY INFORMATION RETRIEVAL 219
merge document cluster
cluster
into a new cluster
for each element
, the membership degree
document
belonging to the new
document cluster
;
for each element
, the membership degree
document
belonging to the new
document cluster
end
end.
Step 5: Recalculate the degree of simi-
larity between each pair of document clus-
ters by taking the minimum of the degrees
of similarity between any pair of docu-
ments, where the documents in each pair
are taken from different document clus-
ters.
Step 6: If the degree of similarity be-
tween any two document clusters is smaller
than the threshold value
then Stop
else go to Step 2.
with document
;
in document cluster
of
is equal to
in document cluster
of
is equal to;
, where,
After partitioning documents into several document clusters,
the document cluster center
obtained by the following formula:
of document clustercan be
(4)
where
in document
ument
,denotes the weight of term
,denotes the membership degree of doc-
belonging to document cluster
, and. This holds for all
,,
.
IV. COMPARISON OF THE CLUSTERING RESULTS OF THE CLC
METHOD AND THE PROPOSED FAHC METHOD
We have implemented the SLINK method [23], the CLC
method [23] and the proposed FAHC method on a Pentium
IV PC using Delphi version 5.0. We have chosen 247 research
reports in the field of computer science [30] as the set of
documents for clustering, which are a subset of a collection
of research reports of the National Science Council (NSC),
Taiwan, R.O.C. Each report consists of several parts, including
a report identifier, a title, the researchers’ names, a Chinese ab-
stract, an English abstract,
, etc. Since the proposed method
intends to deal with English documents, we take the English
abstracts of the reports to represent the contents of the docu-
ments. However, since each selected document contains a large
amount of words, these documents should be preprocessed
to reduce the set of words into a manageable size before the
clustering algorithms are applied on the selected documents.
The selected documents are preprocessed in two steps. First,
words appearing with high frequencies in all documents are
eliminated. Then, the word extractor stems each remaining
word to its “root form” [10]. The collection of these root-for-
matted words forms a set of indexterms
The weight “
is calculated as follows:
for the document set.
” of termin document
(5)
where (5) is derived from [26],
of term
appearing in document
of documents containing term ,
index terms contained in document
number of documents in the corpus. The larger the value of
, the more important is the term
document
. Note that the value of
is normalized and is between zero and one.
After the weight of each index term in each document has
been calculated, we can represent each document
shown as follows:
denotes the frequency
denotes the number
denotes the number of
, and
,
denotes the
in
as a vector
(6)
where
term
of terms in the set of index terms.
The degree of similarity
ments
andis calculated by the following formula:
denotes the weight of
denotes the number in document,, and
between any two docu-
(7)
where
We use the 247 research reports [30] as the set of documents
forclustering. TableIshowstheSLINKmethodpartitioningthe
247 documents into document clusters for different “similarity
threshold values” , where the “similarity threshold value”
between 0.04 and 0.08. When the “similarity threshold value”
, there is only one cluster produced. When the “sim-
ilarity threshold value”
increases, there are more document
clusters which contain more than one document being produced
and there are more and more document clusters being produced
containingonlyonedocument. TableII showsthecomplete link
method [23] partitioning the 247 NSC documents into different
numbersofdocumentclustersfordifferent“similaritythreshold
values” , where the “similarity threshold value”
0 to 0.025, and there is no document cluster containing only one
document until
reaches or is equal to 0.025. From Tables I
and II, we can see that the SLINK method produced more doc-
ument clusters containing only one document than the complete
link method. In this case, the SLINK method has a poorer clus-
tering result than the complete link method due to the fact that it
produces too many document clusters containing only one doc-
ument compared to the complete link method.
.
is
ranges from

Page 5

220IEEE TRANSACTIONS ON FUZZY SYSTEMS, VOL. 13, NO. 2, APRIL 2005
TABLE I
DOCUMENT CLUSTERS PRODUCED BY THE SINGLE LINK METHOD [23] FOR
DIFFERENT SIMILARITY THRESHOLD VALUES ?
TABLE II
DOCUMENT CLUSTERS PRODUCED BY THE COMPLETE LINK METHOD [23] FOR
DIFFERENT SIMILARITY THRESHOLD VALUES ?
In the following, we compare the document clusters gener-
ated by the hierarchical clustering method [23] with the ones
generated by the proposed FAHC method. It is obvious that
the document clusters generated by a good clustering method
should have a maximum degree of within-cluster similarity
and a minimum degree of between-cluster similarity [3]. That
is, the degree of similarity between documents in the same
document clusters should be high and the degree of similarity
between documents in different document clusters should be
low. Thus, we adopt the heuristic measure, called the category
binding (CB) presented in [3] to evaluate the clustering results.
Assume that
obtained by the clustering methods, then the value of CB of the
set of document clusters is calculated by the following formula:
is a set of document clusters
(8)
where
within-cluster similarity and
the average degree of between-cluster similarity of the set of
document clusters. From (8), we can see that a good clustering
result should get a large value of
Theaveragedegree of
is calculated as follows:
denotes the average degree of
denotes
.
within-clustersimilarity
(9)
where
cluster
the page, where
ument cluster
document
notesthemembershipdegreeofdocument
ument cluster
,
between documents
notes a parameter representing the cohesion of documents in a
single-instance document cluster [3].
Theaveragedegreeof
is calculated as follows:
denotes the cohesion of documents in document
and is calculated by (10), as shown at the bottom of
denotes the number of documents in doc-
,denotes the membership degree of
belonging to document cluster,de-
belongingtodoc-
denotes the degree of similarity
calculated by (7), andandde-
between-clustersimilarity
(11)
Here,
document clusters
denotes the degree of similarity between
and and is calculated as follows:
(12)
where
,
,
longing to document cluster
ship degree of document
and
two documents
By observing the 247 NSC research reports [30], we can see
that it is appropriate to partition these documents into 23–30
document clusters. Therefore, we tune the “similarity threshold
value”
of the complete link method [23] to partition the 247
NSC research reports into an appropriate number of document
clusters and compute the CB value based on (8)–(12) for each
clustering result as shown in Table III. Moreover, we also tune
the “similarity threshold value”
glomerative hierarchical clustering method to partition the 247
NSC documents into an appropriate number of document clus-
ters and compute the CB value based on (8)–(12) for each clus-
teringresultasshowninTableIV(whenthedifferencethreshold
value
) and Table V (when the difference threshold
value
), where the values of
to 0.001 and 0.0005, respectively
The “difference threshold value”
glomerative hierarchical clustering method can be regarded as
denotes the number of documents in document cluster
denotes the number of documents in document cluster
denotes the membership degree of document
,
belonging to document cluster
denotes the degree of similarity between any
andcalculated by (7).
be-
denotes the member-
,
of the proposed fuzzy ag-
are heuristically set
of the proposed fuzzy ag-
if
otherwise
(10)