Lower Dimensional Representation of Text Data Based on Centroids and Least Squares

Abstract

Dimension reduction in today's vector space based information retrieval system is essential for improving computational efficiency in handling massive amounts of data. A mathematical framework for lower dimensional representation of text data in vector space based information retrieval is proposed using minimization and a matrix rank reduction formula. We illustrate how the commonly used Latent Semantic Indexing based on the Singular Value Decomposition (LSI/SVD) can be derived as a method for dimension reduction from our mathematical framework. Then two new methods for dimension reduction based on the centroids of data clusters are proposed and shown to be more efficient and effective than LSI/SVD when we have a priori information on the cluster structure of the data. Several advantages of the new methods in terms of computational efficiency and data representation in the reduced space, as well as their mathematical properties are discussed.Experimental results are presented to illustrate the effectiveness of our methods on certain classification problems in a reduced dimensional space. The results indicate that for a successful lower dimensional representation of the data, it is important to incorporate a priori knowledge in the dimension reduction algorithms.

Full text

Page 1

Lower Dimensional Representation of Text Data
based on Centroids and Least Squares
Haesun Park,
?Moongu Jeon,
?andJ. Ben Rosen
?
July, 2001
February, 2003, revised
Abstract
Dimension reduction in today’s vector space based information retrieval system is essential for improving
computational efficiency in handling massive amounts of data. A mathematical framework for lower dimen-
sional representation of text data in vector space based information retrieval is proposed using minimization
and a matrix rank reduction formula. We illustrate how the commonly used Latent Semantic Indexing based
on the Singular Value Decomposition (LSI/SVD) can be derived as a method for dimension reduction from
our mathematical framework. Then two new methods for dimension reduction based on the centroids of
data clusters are proposed and shown to be more efficient and effective than LSI/SVD when we have a priori
information on the cluster structure of the data. Several advantages of the new methods in terms of compu-
tational efficiency and data representation in the reduced space, as well as their mathematical properties are
discussed.
Experimental results are presented to illustrate the effectiveness of our methods on certain classification
problems in a reduced dimensional space. The results indicate that for a successful lower dimensional
representation of the data, it is important to incorporate a priori knowledge in the dimension reduction
algorithms.
?The work of all three authors was supported in part by the National Science Foundation grant CCR-9901992. Any opinions,
findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the
views of the National Science Foundation (NSF). Dept. of Computer Science and Engineering, Univ. of Minnesota, Minneapolis,
MN 55455, U.S.A., e-mail: hpark@cs.umn.edu.
?Dept. of Computer Science and Engineering, Univ. of California, Santa Barbara, Santa Barbara, CA 93106, U.S.A., e-mail:
jeon@cs.ucsb.edu.
?Dept. of Computer Science and Engineering, Univ. of Minnesota, Minneapolis, MN 55455 and Dept. of Computer Science
and Engineering, Univ. of California, San Diego, La Jolla, CA 92093, U.S.A. e-mail: jbrosen@cs.ucsd.edu.
1

Page 2

1Introduction
Today’s exponential growth of the internet and computing power make it possible to accumulate tremendous
amounts of data while users demand more efficient techniques to obtain useful information from this data.
In information retrieval systems, the data takes various forms such as text, image and multimedia. In this
paper we will concentrate on text or document data, and especially on vector space based methods. The
vector space based information retrieval system, originated by G. Salton [33, 34], represents documents as
vectors in a vector space.
Specifically, a term-document matrix
??
?
?
?
?
?
????
?
?
??
???
(1)
is formed based on the collection of documents, where
collection and
? is the total number of terms in the document
? is the number of documents. Each column of
? represents a document, and in the matrix
?
term
methods have been developed [12, 34]. For other related topics such as stemming and removing stop lists,
see [12, 34]. The SMART system [34] is one of the most influential test beds where the vector space based
method is successfully implemented.
One major advantage of a vector space based method is that the algebraic structures of the term-
document matrix can be exploited using the techniques developed in linear algebra. In particular, we believe
that incorporation of a priori knowledge in the data such as cluster structures in its vector space representa-
tion is important in building an effective information retrieval system.
For achieving higher efficiency and effectiveness in manipulating these data, it will be necessary to
find a lower dimensional representation of the data [9]. A vector space based information retrieval system
needs to solve the following three problems frequently: document retrieval, classification, and clustering.
Document retrieval is used to extract relevant documents from a text database given a query. Classification
is the process of assigning new data to its proper group. The group is also called class or category [1]. A
common classificationsystem is composed of data collection, featuregeneration, featureselection, classifier
design, and finally system evaluation and feedback [15, 29, 35]. Among them feature selection is of great
importance for the quality of classification and computational cost of the classifier. Dimension reduction
can be considered as a feature selection process in a vector space based method. Clustering is the process to
find homogeneous groups (clusters) of data based on the values of their vector components and a predefined
measure [20]. While the category structure is known in classification, in cluster analysis little or nothing is
known about the category structure. All that is available is a collection of data whose category memberships
are unknown. The objective is to discover a category structure in the data set [1].
In document retrieval, classification, and in some clustering algorithms, the major computation involves
comparison of two vectors, which will be affected by different weighting schemes and the similarity mea-
sures [22, 34]. With dimension reduction of the given text collection, the complexity of subsequent compu-
tationsinvolvedintheseproblemscanbesubstantiallyreduced. To achievehigherefficiencyincomputation,
often it is necessary to reduce the dimension severely, and in the process, we may lose much information
which was available in the original data. Therefore, it is important to achieve a better representation of data
in the lower dimensional space according to specific tasks to be performed after the dimension reduction,
such as classification, rather than simply reducing the dimension of the data to best approximate the full
term-document matrix. The significance of this has been recognized by Hubert, et.al. [19], for example.
The difficulty involved is that it is not easy to measure how well a certain dimension reduction method pro-
???
??
?,
?
??are weighted frequencies of each word in a specific document representing the importance of
? in document
?. The simplest
?
??is binary, but to improve the retrieval performance, various weighting
2

Page 3

vides a good representation of the original data. It seems that this can only be estimated using experimental
results.
The dimension reduction methods that we will discuss in this paper are based on the vector subspace
computation in linear algebra. Unlike other probability and frequency based methods, where a set of rep-
resentative words are chosen, the vector subspace computation will give reduction in the dimension of the
term space where for each dimension in the reduced space we cannot easily attach corresponding words or
a meaning. In Latent semantic indexing (LSI) [2, 3, 8, 13, 31], the singular value decomposition (SVD) of
the term-document matrix is utilized for conceptual retrieval and lower dimensional representation. The di-
mension reduction by the optimal lower rank approximation from the SVD has been successfully applied in
numerous applications, e.g. in signal processing. In these applications, often what the dimension reduction
achieves is the effect of removing noise in the data. In caseof informationretrieval, often the term-document
matrix has either full rank or close-to full rank. Also the meaning of noise in the text data collection is not
well understood, unlikein otherapplicationssuchas signalprocessing[32] orimageprocessing. In addition,
in information retrieval, the lower rank approximation is not only a tool for rephrasing a given problem into
anotherwhich is easierto solve, but the data representationin the lower dimensional space itself is important
[19] for further data processing.
In this paper, we propose a mathematical framework for lower dimensional representation of text data
using the tools of minimization and matrix rank reduction formula. In our method, a lower dimensional
representation of the original data is achieved by finding a lower rank approximate decomposition of the
data matrix. This approximation is realized as a minimization problem, or by using matrix rank reduction
formula. When the minimization is achieved using the matrix Frobenius norm, the lower dimensional repre-
sentation of the data becomes the projected representation. How successfully we choose the projection will
certainly influence the quality of the lower dimensional representation of the data. In particular, it will be
important to choose the projection so that a priori knowledge on the data collection is reflected as much as
possible. However, it is not always clearhow to represent a priori knowledge mathematicallyto obtain better
lower dimensional data. We attempt to present a general mathematical framework of dimension reduction in
vector space based information retrieval, propose two dimension reduction methods based on the centroids
and the least squares formulation, and illustrate the importance of incorporating a priori knowledge of the
cluster structure in the data.
2Lower Dimensional Representation of Term-Document Matrix
In a vector space based text retrieval system, each document is treated as a vector. Each term corresponds
to one component in the vector, therefore, occupies one dimension in the Cartesian coordinate system. This
means that implicitly, it is assumed that the terms are independent from each other. In fact, this assumption
is not necessarily true, for some words are more closely related. In addition, in handling the vast amounts
of data, allowing one extra dimension for each term will make the computation complexity extremely high
even after preprocessing with stemming and removing the stop lists [12, 24]. Therefore, the dimension
reduction in vector space based information retrieval is important for higher efficiency and effectiveness.
To mathematically understand the problem of lower dimensional representation of the given document
sets, we will first assume that the reduced dimension, which we will denote as
given or determined in advance. Then given a term-document matrix
problem is to find a transformation
??????????????, is
???
???and an integer
?, the
?
?
??
???that maps each vector
?
?in the
? dimensional space to a
3

Page 4

vector
?
?in the
? dimensional space :
?
?
??
?
??
???
??
?
??
???
????? ??
(2)
Once the transformation
?
?is found, any vector
???
???can find its
? dimensional representation as
?
as an approximation problem where the given matrix
???
?
???
???. Rather than looking for the mapping that achieves this explicitly, one can rephrase this
? has to be decomposed into two matrices
? and
? as
????
(3)
where both
approximate factorization is not unique since for any nonsingular matrix
???
???with rank(
?) =
? and
???
???with rank(
? ) =
? are to be found. This lower rank
???
???,
???????????
??
???
and
in two different but related ways. The first is in terms of matrix rank reduction formula and the second is
as a minimization problem. The matrix rank reduction formula has been studied extensively in numerical
linear algebra as well as psychometrics and applied statistics [6, 7, 16, 17, 19]. Here, we summarize the
results that are relevant to our problem of lower dimensional representation of the term-document matrix.
? ????????? and
? ?????
??
????. This problem of approximate decomposition (3) can be recast
THEOREM 1 (Matrix Rank Reduction Theorem) Let
the matrix
???
???be a given matrix with
? ????????. Then
?????????????
??
????
(4)
where
???
???and
???
???,
???, satisfies
??????????????????????????????
??
?????
(5)
if and only if
?????
???is nonsingular.
The only restriction on the premultiplier
product
and makes incorporation of a priori knowledge possible. For our purpose, we will concentrate mostly on
Eqn. (4), which we call matrix rank reduction formula. In fact, in [6] it is shown that many of the matrix
decompositions can be derived using the matrix rank reduction formula. It is easy to see that the rank
approximation from the truncated SVD of
We will discuss more on this in the next subsection.
Minimizing the error matrix
? and the postmultiplier
? is on their dimensions and that the
??? be nonsingular. It is this choice for
? and
? that makes the dimension reduction flexible
?
? provides a solution that minimizes
??? ??
?or
??? ??
?[4, 11, 14].
? in a certain norm
? is equivalent to solving a minimization problem
???
???
????????
?
(6)
where
???
???with rank(
?) =
? and
???
???with rank(
? ) =
?. Eqn. (4) and Eqn. (6) are related as
????????????
??
? ??
It is well known that with
used LSI exploits the SVD of the term-document matrix [2, 3, 8, 10]. We emphasized that for a successful
??? or
?, the best approximationis obtained from the SVD of
?. The commonly
4

Page 5

rankreductionscheme, it isimportanttoexploit a prioriknowledge. The incorporationof a prioriknowledge
can be translated to choosing the matrix
(6). However, mathematical formulation of the a priori knowledge as a constraint is not always easy or even
possible.
In the following, we discuss various ways to choose the matrices
some interesting test results that illustrate that although the SVD of
of minimizing the distance between
the clustering of the data matrix
documents. The choice of reduced dimension
the integer
made to facilitate the usage of the SVD by either parallelizing, preserving sparsity of the term-document
matrix, or finding a faster decomposition that can approximate the SVD. The SVD certainly gives the best
lower rank approximation. However, we show in this paper that the SVD may not give the ultimate solution
in the case of dimension reduction for the task of classification. We summarize the LSI/SVD in the next
subsection.
? and
? in (4), or adding a constraint in the minimization problem
? and
? . In Section 3, we also show
? gives the best approximation in terms
? and
?? for
??? or
?, some other choices of
? and
? based on
? may give a better reduced dimensional representation of the original
? will be discussed later, and for now we will assume that
? is given. The LSI/SVD has been a topic of active research [2, 3, 27] and the efforts have been
2.1Latent Semantic Indexing with SVD
It is well known that for any matrix
SVD is defined as
???
???, its singular value decomposition (SVD) exists [4, 14]. The
?????
?
(7)
where
???
???
????
???
????
???
?
?
???
?
??
?
???
?
?????????
?
????
?
?
with
left and right singular vectors, respectively. It is widely known that the noise filtering can be achieved via a
truncatedSVD.Ifwereplaceatrailingdiagonalcomponents of
left and right singular vectors, then the rank
??????????,
?
?
??
?
?????
?
?? which are the singular values, and the columns of
? and
? are
?withzerosandeliminatethecorresponding
? representation can be obtained as
???
?
??
?
?
?
?
??
?
?
?
?
?
?
?
?
?
?
?
?
?
?
??
??
?
?
?
?
?
?
?
??
?
?
?
?
?
?
?
(8)
where
when the matrix
or Frobenius norm. It can be easily shown that the minimum error is obtained with
which gives
?
?
??
???
??
?
??
???, and
?
?
??
???. The rank
? approximation (8) can also be obtained
? in the matrix rank reduction formula shown in (4) is minimized using matrix
?
?norm
???
?
?and
???
?,
??? ??? ???
??
????????
?
???
?
?
??
?
?
??
??
?
?
??
???
?
?
?
???
?
?
??
??
?
?
?
?
???
?
?
?
?
?
?
?
The LSI is based on the assumption that there is some underlying latent semantic structure in the data of
the term-document matrix that is corrupted by the wide variety of words used in documents and queries for
5

Page 6

the same objects (the problem of polysemy and synonymy, see [8]). It is claimed that the SVD and statistics
of the frequency of association terms make it possible that two documents of similar topic can belong to
the same cluster or be retrieved simultaneously for a query even if they do not share the same keywords
[2]. The basic idea of the LSI/SVD is that if two document vectors represent the same topic, they will share
many associating words with a keyword, and they will have very close semantic structures after dimension
reduction via the SVD.
Inclassification,clustering,anddocument retrieval, the fundamentaloperationistocomparea document
(or pseudo-document) to another document (or pseudo-document). The choice of similarity measure plays
an important role [22]. In the vector space based information retrieval, the most commonly used similarity
measures are,
inner product is used as a measure, the documents are compared as,
?
?norm (Euclidean distance), inner product, cosine, or variations of these [22]. When the
?
?
???
?
?
?
?
??
?
?
?
?
?
?
?
?
?
?
?
?
?
?
???
?
?
?
?
???
?
?
?
?
??
(9)
i.e., the inner product between a pair of columns of
a pair of columns of
vectors in the reduced dimension. This argument holds for the cosine similarity measure as well when the
vectors are properly normalized. In general the above derivation is valid only for the inner product measure.
Consider the inner product between a new vector
? can be approximated by the inner product between
?
?
?
?
?. Accordingly,
?
?
?
?
?
??
???is considered a representation of the document
???
???and the document vectors in
?:
?
?
???
?
?
?
???
?
???
?
?
?
?
?
?
????
?
?
?
???
?
?
?
??
(10)
Eqn. (10) shows that a new vector
???
???can be represented as
????
?
?
?
(11)
in the
space. In LSI/SVD, it has also been proposed that
? dimensional space, since the columns of
?
?
?
?
?represent the columns of
? in the
? dimensional
? be reduced to a vector in
?
???as
????
??
?
?
?
??
(12)
In Section 2.2.4, we give a different derivation and interpretation of the transformations (11) and (12) of
to
?
??. In fact, our derivation will clearly illustrate how one can obtain
????
???shown in Eqn. (12).
2.2 Dimension Reduction of Cluster Structured Data
Although the SVD provides the optimal approximation
?? of
? that gives the minimum distance when
?
its columns can be grouped into a number of clusters. In a cluster structured matrix, each column is more
closely related to a certain set of columns than to others. We will show that there are other approximation
schemes that are often superior to the SVD in producing better reduced dimensional representation of the
text data for the classification task when the data has a cluster structure.
?? or
? in Eqn. (6), the SVD does not take into account that the data matrix
? is cluster structured, i.e.,
2.2.1 Representation of Each Cluster
First, we will assume that the data set is cluster structured and already grouped into a number of clusters.
This assumption is not a restriction, since we can cluster the data set if it is not already clustered, using one
of the several existing clustering algorithms such as k-means [9, 20]. Also especially when the data set is
6

Page 7

huge, we can assume that the data has a cluster structure and it is often necessary to cluster the data first to
utilize the tremendous amount of information in an efficient way.
Suppose we are given a data matrix
the matrices
represented well in a reduced dimensional space. For this purpose, we want to choose each column of
that it represents the corresponding cluster. For any given scalar data set with the data items
a commonly used representative of the data set is the mean value
? whose columns are grouped into
? clusters. We want to find
? and
? with
? columns and
? rows, respectively, with
???? , so that the
? clusters are
? so
?
?
??
?
??????
?,
?
?
?
?
?
?
?
???
?
?
?
(13)
The mean gives the minimum variance
?
?
???
??
?
??
?
?
?
????
Æ??
?
?
???
??
?
?Æ?
?
????
Æ??
???
?
????
?
??Æ ????????
?
?
?
(14)
The mean value can be extended to data sets in a vector space. Suppose
centroid defined as
?
?
??
?
??????
?
??
???. Then its
??
?
?
?
?
???
?
?
?
?
?
????
???
(15)
where
centroid is the vector which achieves the minimum variance in the following sense:
?? ????????? ??
?
??
???, is commonly used as a vector that represents the vector data set. The
?
?
???
??
?
???
?
?
????
???
???
?
?
???
??
?
???
?
?
????
???
???
?????
?
?
?
?
?
(16)
Eqn. (16) shows that the centroid vector gives the smallest distance in Frobenius norm between the matrix
? and the rank one approximation
representatives, such as medoid, see [30]. In the following three subsections, we describe how we find the
approximation
is a special case of our model.
??
?where
? is to be determined. For other alternatives for cluster
?? that exploitstheclusterstructureusingthecentroids, andalsoillustratethat theLSI/SVD
2.2.2Minimization with Centroid Vectors
Assume that the data matrix
? is partitioned into
? clusters and define the centroid matrix
??
?
?
?
?
?
????
?
?
??
???
?
where the
we find the rank reducing decomposition
the
?th column is the centroid of the
?th cluster. In our Centroid algorithm for dimension reduction,
???? by first taking
? as the centroid matrix
? that represents
? clusters in the data, and then solving the least squares problem
???
???
???
????????
?
?
The solution matrix
tation of
???
???gives a
? dimensional representation of
?, which is the projected represen-
? in the range space of the matrix
?.
7

Page 8

Algorithm 1 : Centroid algorithm for Dimension Reduction
Given a data set
sional representation
???
???with
? clusters and a vector
???
???, this algorithms computes a
? dimen-
????
???of
?.
1. Compute the centroid
?
?of the
?th cluster,
?????
2. Set
??
?
?
?
?
?
????
?
?
3. Solve
???
??
???????
?
This can also be explained using the matrix rank reduction formula. Let
the columns of
?
?denote the set of indices of
otherwise ?
? that belong to the cluster
?. Defining a grouping matrix
???
???as
????????????????
?
? ???
??where
???
???
and
???????
?
?
if
???
?
?
(17)
the centroid matrix
? can be represented as
?????
(18)
In addition, the solution
? for
???
?
??????
?
obtained with
?? ?? is
????
?
??
??
?
?
?. This in turn, gives the matrix rank reduction expression
?and the postfactor
???????????????
?
??
??
?
?
?
(19)
?????????
?
?
?
???
??
??
?
?
?
???
(20)
with the prefactor
???
?
???? in Eqn. (4). After obtaining the decomposition
?
by solving the minimization problem
??? as discussed above, any new data
???
???can be transformed to the lower dimensional space
???
????
???
???????
?
?
(21)
This dimension reduction method is summarized in Algorithm 1, the Centroid algorithm for dimension
reduction.
In the Centroid-based classification algorithm which is summarized in Algorithm 2, the similarity be-
tween a vector
to belong to the cluster
minimization problem
? to be classified and the centroid vectors
?
?,
?????, is compared and
? is determined
? when the similarity between
? and
?
?is the highest. Since the solution for the
????????
?
?
?
???????
(22)
is the
fore, the centroids in the reduced space are orthogonal and the independence of the vectors that represent
?th column of the identity matrix
?
?
??
???, the centroids are represented as the unit vectors. There-
8

Page 9

Algorithm 2 : Centroid-based Classification
Given a data matrix
it finds the index
? with
? clusters and
? corresponding centroids,
?
?,
?????, and a vector
???
???,
? of the cluster in which the vector
? belongs.
? find the index
similarity measure between
we take the index with the minimum value. Using the cosine measure,
? such that
???????
?
?,
?????, is minimum (or maximum), where
???????
?
? is the
? and
?
?. (For example,
???????
?
??????
?
?
?using the
?
?norm, and
???????
?
?? ???????
?
??
?
?
?
?
???
?
??
?
?
?
?
and we take the index with the maximum value.)
the clusters is maximized as a result of projection on to the space spanned by the columns in the matrix
?. After obtaining the
Centroid-based classification algorithm
the centroids of the clusters in the reduced space. When the Euclidean distance is used as the similarity
measure, we look for
? dimensional representation
?? of the vector
? from the Centroid algorithm, in the
?? is simply compared to the unit vectors
?
?,
?????, which are
??????
?????
?????
?
?
?
?
(23)
Theminimum willbeachievedwhen
of
component of
measure is used:
?istheindexof thelargestcomponent of
??. Therefore,thecomputation
? of the 2-norms in Eqn. (23) becomes unnecessary, and we simply need to find out where the largest
? lies and it will be the cluster index where
? belongs. The situation is similar when the cosine
??????
?????
??
?
?
?
????
?
??
?
?
?
(24)
is obtained again when
will be the same whether we use 2-norm or cosine as similarity measures. The computational complexity is
lower since the computation of
components in
? is the index of the largest component of
??. Therefore, the result of classification
? of the 2-norm (or cosine) will be replaced by finding the largest of the
?
??.
2.2.3Minimization with an Orthogonal Basis of the Cluster Representatives
If the factor
well approximated with the correlation in
? has orthonormal columns in a rank
? approximation
???? , then the correlation in
? is
? since
?
?
???
?
?
?
????
?
??
In addition, most of the common similarity measures can directly be inherited from the full dimensional
space to the reduced dimensional space, since for any vector
???
???,
???
?
?????
?
????
?
?
where
? has orthonormal columns.
9

Page 10

Algorithm 3 : Orthogonal Centroid algorithm for Dimension Reduction
Given a data set
tation
???
???with
? clusters and a vector
???
???, it computes a
? dimensional represen-
?? of
?.
1. Compute the centroid
?
?of the
?th cluster,
?????
2. Set
??
?
?
?
?
?
????
?
?
3. Compute the reduced QR decomposition of
?, which is
???
?
?
4.
????
?
?
?
No matter how the matrix
sition of the matrix
triangular matrix
? is chosen, this can be achieved by computing the reduced
?? decompo-
?. The QR decomposition of
? gives an orthogonal matrix
???
???and an upper
???
???such that
???
?
?
?
?
?
(25)
With
reduced QR decomposition of
????
?
?
?
?,
?
?
??
???, and
?
?
??
???????, we have
???
?
?, which is often called the
?. Premultiplying
?
?
???
?
??
?
?
?to Eqn. (25) gives
?
?
?
???
and
?
?
?
????
(26)
InourOrthogonalCentroidalgorithmfordimensionreduction, thematrix
decomposition of the centroid matrix
?
?isfromthereducedorthogonal
?, and the least squares solution for
???
?
???
?
??? ??
?
(27)
gives the
3: Orthogonal Centroid algorithm for dimension reduction. The
Centroid algorithm and
the solution for
?-dimensional representation
???
???of
???
???. Our algorithm is summarized in Algorithm
? dimensional representations
? from the
? from the Centroid algorithm are related since
???
?
?
? and
???? where
? is
???
?
?????? ??
?
?
(28)
In both expressions
the data
???? and
???
?
????, since
????
?
?? , the error
?
?in approximation of
? is
?
?
????????????
?
?
?
???
??
?
?
?
?
??
and the error in correlation is
?
?
??
?
????
?
??
?
???
?
?
???
?
?
??
?
????
?
?? ???
?
?
?
???
??
??
?
?
?
???
10

Page 11

Although
in the Orthogonal Centroid method since
?
?normisinvariantunderorthogonaltransformation,thisdoesnot holdforthetransformation
?
?
?
?
?
???. However, we now show that the transformation by
?
space identical to those in the reduced space obtained by the Orthogonal Centroid algorithm [21].
Definition: Order of Similarity: Given any vector
similarity
columns of
?still has a certain invariance property which makes the Centroid-based classification results in the full
???
???and any matrix
???
???, the order of
?????? is an ordered indices of columns of
? according to the similarity between
? and the
?
?. For example, when
??
?
?
?
?
?
?
?
?
??
???and
????
?
?
?
?????
?
?
?
?????
?
?
?,
????
The following two theorems show that the order of similarity between any data item and the centroid
matrix is preserved after dimension reduction by the Orthogonal Centroid algorithm.
?
???????? ?? in
?
?norm similarity measure.
THEOREM 2 (ORDER PRESERVING IN
in the full dimensional space between any given vector
preserved in the reduced space obtained by the Orthogonal Centroid algorithm. That is,
where
?
?NORM) The order of similarity
?????? with
?
?norm measure
? and the centroid matrix
???
???is completely
????????????
?
??
????
?
?
? and
???
?
?
?, and the reduced QR decomposition of
? is
?
?
?.
Proof: Since
?
?
?
?
?
?? from Eqn. (26),
????
?
?
?
?
???
?
????
?
??
?
?
???
?
?
????
?
??
?
?
???
?
?
????
?
??
?
?
???
?
?
????
?
??
?
?
???
?
?
??
?
?
?
(29)
If
????
?
?
?
?????
?
?
?
?
then
??
?
?
????
?
??
?
?
???
?
?
??
?
?
???
?
?
????
?
??
?
?
???
?
?
??
?
?
and
??
?
?
????
?
??
?
???
?
?
????
?
??
?
?
This means that the Orthogonal Centroid reduction method preserves the order of the
between any vector and the centroids in the full dimensional space after dimension reduction.
?
?
?norm similarity
?
THEOREM 3 (ORDER PRESERVING IN COSINE MEASURE) Theorderofsimilarity
sure in the full dimensional space between any given vector
pletely preserved in the reduced space obtained by the Orthogonal Centroid algorithm. That is,
??????withcosinemea-
? and the centroid matrix
???
???is com-
???????
??????? where
????
?
?
? and
?
???
?
?
?, and the reduced QR decomposition of
? is
?
?
?.
Proof: Let
???????
?
? be cosine between vectors
? and
?
?. Since
?
?
?
?
?
?? from Eqn. (26),
???????
?
???????
?
???
?
?
?
??
??
?
??
?
?
?
?
?
??
?
??
?
??
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
??
?
??
?
??
?
?
?
?
?
?
and
?????????
?
?? ?????
?
?
???
?
?
?
?
??
?
?
?
?
?
?
?
?
?
??
?
?
??
?
??
?
?
?
?
?
?
?
(30)
If
???????
?
?????????
?
?, then
?
?
?
?
?
?
?
?
?
??
?
??
?
??
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
??
?
??
?
??
?
?
?
?
?
?
?
11

Page 12

Accordingly,
?
?
?
?
?
?
?
?
?
??
?
?
??
?
??
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
??
?
?
??
?
??
?
?
?
?
?
?
?
(31)
i.e.,
?????????
?
???????????
?
?.
?
The above two theorems show that we can completely recover the orders of both the
similarities between any vector and the centroids, when the original dimension is reduced to the number
of categories
exactly the same classification results using the reduced data from the Orthogonal Centroid algorithm as
with the full dimensional data. Note that the order preserving property of the dimension reduction obtained
by the Orthogonal Centroid algorithm holds regardless of the cluster quality. In fact, the order preserving
property in either
?
?and cosine
? by the Orthogonal Centroid algorithm. Therefore, Centroid-based classification produces
?
?norm or cosine measure does not depend on the fact that the columns of the matrix
? are the centroids of the clusters. As far as the dimension reduction is achieved by an orthogonal basis
?
reductionby
?for the range space of any matrix
?, the order between a datum and the columns of
? after dimension
?
?
?willbepreserved. However, intheabovetheorems, wespecificallydiscussedthecasewhen
? is the centroid matrix due to its relevance to the Orthogonal Centroid dimension reduction algorithm in
conjunction with the the centroid-based classification method. The savings in computational cost from
dimension reduction is obvious. In Section 3, some experimental results are given to show the performance
of our algorithms.
2.2.4 LSI/SVD using the Minimization Model
Now we show how the LSI/SVD can be explained using our least squares model and achieve the results
shown in Eqn. (11) and Eqn. (12). A solution for the minimization problem (6) can be obtained from the
SVD of
the least squares problem
? with
???
?and
???
?
?
?
?. In fact, this is the optimal solution when
??? or
?. Then solving
???
??
??
?
?????
?
(32)
we obtain
????
?
?
??
(33)
This produces the same result as Eqn. (11). In LSI/SVD,
reduced dimension, which are
would obtain from solving
?? is compared with the document vectors in the
?
?
?
?
?. Note that the columns of
?
?
?
?
?are the solution vectors
??
?which we
???
??
??
?
????
?
?
?
(34)
for
?
Using our minimization model, it can easily be shown that the representation
solution for
????.
?? in Eqn. (12) is the
???
?
??
?
?
?
?????
?
?
(35)
12

Page 13

Algorithm 4 : k Nearest Neighbor (knn) Classification
Given a data matrix
the vector
????
?
??????
?
℄ with
? clusters and a vector
???
???, it finds the cluster in which
? belongs.
1. From the similarity measure
to distinguish the algorithm parameter from the number of clusters.)
???????
?
? for
?????, find the
?
?nearest neighbors of
?. (We use
?
?
2. Among these
?
?vectors, count the number belonging to each cluster.
3. Assign
? to the cluster with the greatest count in the previous step.
therefore, in this case, the matrix
reduced dimension are the columns of
such as cosine and
only to the inner product for similarity measure as in the original derivation of the LSI/SVD.
It has been reported that the LSI/SVD gives the best performance in text retrieval when the reduced
dimension is between 100 and 300 [2]. Considering the size of the text documents these days, even this
range of 100 to 300 can be quite a dramatic reduction of the dimension. In fact, the reduced dimension will
rarely be the numerically estimated rank of the original term-document matrix, unlike in other applications
such as signal processing where the SVD is often utilized.
? is considered to be
?
?
?
?and accordingly, the document vectors in the
?
?
?. Our derivation also makes it possible for any similarity measure
?
?norm to be used in comparing the vectors in the reduced dimension. It is not restricted
3Experimental Results
In this section, we present several experimental test results that illustrate the performance of our dimension
reduction methods. We compare the classification results obtained using the data in the full space and the
reduceddimensionaldatafromourdimensionreductionmethods,theCentroidalgorithmandtheOrthogonal
Centroid algorithm. For classification, we use the centroid-based classification algorithm as well as the k
nearest neighbor (knn) algorithms presented in Algorithm 4. We also compare the classification results
obtained using the reduced dimensional data produced by our algorithms and by the SVD.
In preprocessingthe text data the stop lists such as “the” and “a” which appearin almost every document
are eliminated, since they are not considered as important words differentiating documents from each other.
Then the Porter’s stemming algorithm [12] was applied to process the words into their stems in order to
reduce the data dimension and increase the effectiveness of classification. For example, “computer”, “com-
putation”, and “computing” are processed into one stem word “comput”. We used the term frequencies in
generating the term-document matrix.
Test I: Inthefirsttest, thepurposeistoexaminetherelationship betweenthedataitemsandthecentroids
in the full space and the reduced dimensional space. We use a relatively low dimensional clustered data set
which is artificially generated by the algorithm presented in [20]. The program generated 200 data items
of three classes in a 10 dimensional space. There are 63, 79 and 58 data items in class 1, class 2, and
class 3, respectively, forming a data matrix
reduced to 3 from 10 using the Orthogonal Centroid algorithm and the classification was performed using
the Centroid-based classification algorithm in the full space as well as in the reduced space. Then we
compare classification in the full and reduced space. Table 1 shows the distances between each data item
and the centroids in the full dimensional space and the reduced space with the
???
??????. Then the dimension of the data vectors are
?
?norm measure and with
13

Page 14

Table 1: Test I,
space and in the full space.
?
?norm and cosine similarityvalues between the data and the three centroids, in the reduced
??
?
?
??
?
??
?
??
?
???
?
??
?
??
?
?????
?
?
?
?
??
?
?
?
?
??????
?
??
?
?
Data
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
0.67
0.17
1.04
0.60
0.84
1.36
0.86
1.09
1.18
1.57
0.94
1.22
0.61
0.73
1.20
1.23
1.46
0.66
1.34
1.86
0.33
0.54
0.57
0.52
0.87
1.40
1.62
1.52
1.30
1.53
1.12
0.88
1.02
1.53
1.15
1.48
1.17
1.25
1.34
1.38
0.44
0.28
0.47
0.38
0.67
1.42
0.88
1.35
0.83
1.24
1.49
1.43
1.31
1.18
1.85
1.32
1.50
0.92
0.84
1.40
1.75
1.70
1.09
1.50
2.07
0.68
1.26
0.93
0.80
1.31
1.68
1.84
1.70
1.36
4.81
1.68
1.24
1.34
1.63
1.46
1.59
1.63
1.45
1.47
1.70
1.03
0.92
0.89
0.57
1.69
0.97
1.00
0.88
0.97
0.92
0.77
0.92
0.89
0.91
0.73
0.90
0.83
0.96
0.94
0.84
0.89
0.74
0.92
0.85
0.46
0.98
0.96
0.96
0.97
0.93
0.67
0.65
0.74
0.75
0.57
0.89
0.92
0.89
0.79
0.84
0.73
0.84
0.83
0.79
0.80
1.00
0.99
0.98
0.99
0.99
0.87
0.92
0.78
0.95
0.83
0.73
0.78
0.78
0.84
0.67
0.79
0.76
0.91
0.92
0.75
0.80
0.69
0.81
0.83
0.42
0.94
0.81
0.85
0.89
0.85
0.59
0.60
0.70
0.74
0.51
0.80
0.85
0.78
0.77
0.76
0.70
0.71
0.73
0.72
0.73
0.88
0.91
0.93
0.97
0.89
?
?
?
?
?
?
?
?
?
??
?
??
?
??
?
??
?
??
?
???
?
???
?
???
?
???
?
???
cosine measure. Since the space is limited, we only show the results for the first five data items of each
cluster. The numbers with the underlines show the best similarity values (the smallest for the
and the largest for the cosine) and therefore the classification results from the Centroid-based classification.
The first column of the table contains the label of each data, the numerical values in the next three columns
are Euclidean distances between data
columns represent those in the reduced space. Then the next six columns show these values measured
using the cosine similarity. Note that
in both cases of
classification accuracy.
In Tests II-IV, three cases are tested using the
sion reduction methods by the Centroid algorithm and the Orthogonal Centroid algorithm. Then using the
Centroid-based classification algorithm, each item was classified in the reduced space, as well as in the full
space.
Test II: A data set with three clusters was formed using the data from the SMART system: 82 docu-
ments from computer and information science (ADI), 1400 documents from aerospace engineering (CRAN-
FIELD), and 1033 documents from medical science (MEDLINE). Input data and the classification results
from the Centroid-based classification in the full space and in the reduced dimensional space are presented
in Table 2. This is a relatively easy classification problem since the three classes are rather disjoint. The
classification results from the Centroid-based classification applied to the dimension reduced data of the
Centroid algorithm are identical when we used the
2.2.2. Also the Centroid-based classification results in the full space and the reduced space given by the Or-
thogonal Centroid method are identical, which corroborates Theorems 2 and 3. This phenomenon is shown
?
?norm,
?
?and centroids
?
?in the full dimensional space, and the next three
?
?is misclassified in the full space as well as in the reduced space
?
?and cosine similarity measures. In the next tests, we use larger data sets and show the
?
?norm and cosine measures, and using the dimen-
?
?norm and the cosine measure, as shown in Section
14

Page 15

Table 2: Test II, classification accuracy in the full space and in the reduced dimensional spaces. Classifica-
tion was performed using the Centroid-based classification algorithm.
Data from the SMART system
category
ADI
CRANFIELD
MEDLINE
Classification Accuracy (in %)
from Centroid-based Classification algorithm (in %)
FullCentroid
number of data items
82
1400
1033
1
2
3
Orthgonal Cent.
Dimension
????????????
96.9
96.9
??????
89.0
98.4
????
?
?
89.0
98.4Cosine
throughout our tests.
Test III: The data set for this test is constructed using the MEDLINE database available from the Na-
tional Institute of Health. We randomly selected five clusters from MEDLINE and obtained 500 documents
from each cluster, forming a data set with 2500 documents. The total number of terms are 22095 after
preprocessing of the stop list removal and stemming. The categories have many common words related
to cancer. Using the Centroid and Orthogonal Centroid algorithms, the dimension 22095 is dramatically
reduced to 5, the number of classes. Two different classification algorithms, the Centroid-based classifica-
tion algorithm and the knn algorithm with various values of the parameter
classification results obtained from the Centroid-based classification algorithm are identical in the full space
as well as in the 5-dimensional space given by the Orthogonal Centroid algorithm. The computational time
for the knn algorithm is dramatically reduced after the dimension reduction since the comparisons between
the data to be classified and all other data points are now made in the 5-dimensional space, instead of in the
full 22095-dimensional space.
Table 3 shows that the classification results with both measures of
in the full and reduced dimensional space obtained from the Orthogonal Centroid method. The results for
various values of
interesting that the classification accuracy of the knn algorithm was better overall after dimension reduction.
In thisparticulardataset, classificationaccuracyas well as theCPU timeof theCentroid-basedclassification
method was much better than those of the knn method.
Test IV: The data set here is an extension of the data set used in Test III. It has three extra categories
of MEDLINE data in addition to five from Test III. As in the second data set, 500 documents are chosen
from each category, making a data set of 4000 documents. After preprocessing, the total number of terms
was 29152. Again our Centroid and Orthogonal Centroid algorithms are tested and two different similarity
measures are examined to determine which measure is more appropriate for classifying the text data. The
test procedure is the same as that in the previous tests. The results are shown in Table 4.
In all of the above three tests, the overall classification accuracy with the cosine measure is better. This
is consistent with other substantial test results we obtained comparing several similarity measures in text
processing [23]. Since the Centroid-based classification results with the cosine and
?
?, were used in this test. The
?
?norm and cosine are identical
?
?with the
?
?norm in knn were much worse than those with the cosine measure. It is
?
?norm measures are
15