Content uploaded by James C. Bezdek
Author content
All content in this area was uploaded by James C. Bezdek on May 01, 2017
Content may be subject to copyright.
576 IEEE TRANSACTIONS ON FUZZY SYSTEMS, VOL. 8, NO. 5, OCTOBER 2000
Generalized Fuzzy c-Means Clustering Strategies
Using Norm Distances
Richard J. Hathaway, Member, IEEE, James C. Bezdek, Fellow, IEEE, and Yingkang Hu
Abstract—Fuzzy c-means (FCM) is a useful clustering tech-
nique. Recent modifications of FCM using norm distances
increase robustness to outliers. Object and relational data versions
of FCM clustering are defined for the more general case where
the norm or semi-norm is used as the
measure of dissimilarity. We give simple (though computationally
intensive) alternating optimization schemes for all object data
cases of in order to facilitate the empirical examination of
the object data models. Both object and relational approaches are
included in a numerical study.
Index Terms—Clustering, fuzzy c-means, norm, outlier.
I. INTRODUCTION
THE fuzzy c-means (FCM) algorithm [1] has successfully
been applied to a wide variety of clustering problems [2].
This approach partitions a set of object data
into c-(fuzzy) clusters based on a computed minimizer of the
fuzzy within-group least squares functional
(1)
where fuzzification parameter;
prototype (or mean) of
the th cluster;
degree to which datum
belongs to the th
cluster;
matrix of cluster proto-
types;
partition matrix;
Euclidean or 2-norm
squared.
For later notational convenience, we will array the object data
as columns in the object data matrix
. The partition matrix is a convenient
tool for representing cluster structure in the data ;
Manuscript received September 6, 1999; revised May 15, 2000. The work of
R. J. Hathaway was supported by the ONR under Grant 00014-96-1-0642 and in
partbytheFacultyResearchSubcommitteeofGeorgiaSouthernUniversity. The
work of J. C. Bezdek was supported by the ONR under Grant 00014-96-1-0642 .
The work of Y. Hu was supported in part by the Faculty Research Subcommittee
of Georgia Southern University.
R. J. Hathaway and Y. Hu are with the Mathematics and Computer Science
Department, Georgia Southern University, Statesboro, GA 30460 USA (e-mail:
hathaway@ieee.org).
J. C. Bezdek is with the Department of Computer Science, University of West
Florida, Pensacola, FL 32514 USA.
Publisher Item Identifier S 1063-6706(00)08467-8.
we define the set of all nondegenerate fuzzy partition
matrices for partitioning data into clusters as
(2)
The most popular and effective method of optimizing (1) is
the fuzzy c-means algorithm, which alternates between opti-
mizations of over with fixed and
over with fixed, producing a sequence .
Specifically, the st value of is computed
using the th value of in the right-hand side of
for
(3)
Thenthe updated st value of is used to calculate the st
value of via
where (4)
for and (5)
The FCM iteration is initialized using some (or
possibly ) and continues by alternating the updates in
(3) and (4) until the difference measured in any norm on
(or )in successive partition matrices (or matrices) is less
than some prescribed tolerance .
While FCM has proven itself to be very useful, the quality
of the computed cluster centers can sometimes
be degraded due to the effects of outliers in the data set. This
occurs because ,
the datum-to-prototype dissimilarity term in (1), can place con-
siderable weight on outlying data points, thus pulling cluster
prototypes away from the “center” or main distribution of the
(nonoutlying) cluster.
There are a number of useful approaches for controlling the
harmful effects of outlying data, including the possibilistic clus-
tering approach of Krishnapuram and Keller [3] and the fuzzy
noise-clustering approach of Dave [4]. Most important to this
note is the work of Kersten [5]–[7] and Miyamoto and Agusta
[8], who independently suggested replacing with
in the FCM functional in order
1063–6706/00$10.00 © 2000 IEEE
Authorized licensed use limited to: Richard Hathaway. Downloaded on February 1, 2009 at 13:19 from IEEE Xplore. Restrictions apply.
HATHAWAY et al.: GENERALIZED FUZZY c-MEANS CLUSTERING STRATEGIES 577
Fig. 1. An illustration of the possible shapes of .
to increase robustness against outlying data. Earlier work that
uses the norm or its square in FCM-based clustering ap-
pears in Bobrowski and Bezdek [9] and Jajuga [10]. Bobrowski
and Bezdek [9] also gave a method for optimizing when the
square of the sup norm is used in (1).
In this note, we examine FCM-based clustering using general
norm distances, where the norm of the -dimensional real
vector is defined as . In Sec-
tion II, we present an object-data strategy for using norms
due to Miyamoto and Agusta [11], [12] and Overstreet [13].
Additionally, we describe how a relational data approach can be
taken using the non-Euclidean fuzzy c-means (NERFCM) algo-
rithm of Hathaway and Bezdek [14]. The object-data approach
operates directly and solely, on the object datamatrix . There-
lational approach clusters indirectly through the
use of derived dissimilarity data , where is some
measure of the dissimilarity between and . The two strate-
gies will be compared using numerical examples in Section III.
The final section summarizes our findings.
II. EXTENSIONS OF FUZZY c-MEANS
A. Object-Data Strategy
This approach is based on a direct modification of the fuzzy
c-means functional . The generalization of (1) that we
consider here was originally proposed by Miyamoto and Agusta
[11] and later extended by Overstreet [13]. The objective func-
tion is
(6)
The optimization of (6) is relatively straightforward and the
choice of has considerable effect on the influence of outliers
and other properties of the representation of the clusters. We see
that isthe original FCM functional and is the more ro-
bust functional used by Kersten [5]–[7]. Miyamoto and Agusta
[11], [12] consider this model in general for and the
range of is extended to include in Overstreet [13].
The object-data approach considered in this correspondence is
based on iterative minimization of .
The optimization of (6) for general is more complicated and
costly than the optimization of the special case (FCM).
However, as with FCM, optimization can be done by alternating
separate optimizations over the and variables. The ( -vari-
able) minimizer of over (for a fixed )isgiven
by (4) using the datum-to-prototype dissimilarities
for and (7)
Appropriate methods of computing the -variable minimizer of
over (for a fixed ) depend on the value of
, but in all cases this optimization can be decoupled into
independent univariate minimizations of functions of the form
for and (8)
The geometric form of is nonconvex for , with
a cusp at each datum value .For is convex and
piecewise linear, with a corner at each . The function is
differentiable and strictly convex if . The three types
of shapes are illustrated in Fig. 1 using the function .
We choose to minimize in (8) in the simplest possible way
sinceour emphasis here is on understanding the properties ofthe
clusterings and not on computational efficiency. For ,we
note that takes its minimum value over for one (or more)
elements in the set .For , the minimizing
value of is simply taken to be the smallest of the values,
which globally minimize over . The mean of
the smallest and largest globally minimizing values is used
for the special case of .For , the computed value of
is taken to be a numerical approximation to the unique zero
of , obtained
here using the method of bisection.
We summarize the object-data strategy. It consists of alter-
nating optimizations of in (6) between the and vari-
ables. The optimization over the variable is accomplished
using(7) in (4) and the optimization overthe variableis decou-
pled into univariate optimizations of functions of the form
(8). The univariate optimizations are essentially done using ex-
haustive search over for and bisection on
for . (Exhaustive search, which is necessary for ,
is prohibitively expensive for sufficiently large data sets and we,
therefore, acknowledge a practical limitation to the usefulness
Authorized licensed use limited to: Richard Hathaway. Downloaded on February 1, 2009 at 13:19 from IEEE Xplore. Restrictions apply.
578 IEEE TRANSACTIONS ON FUZZY SYSTEMS, VOL. 8, NO. 5, OCTOBER 2000
of this model in some cases. We repeat that the emphasis here
is on studying the clustering solutions produced by the various
models.) The alternating optimization is continued until succes-
sive partitions are within of each other, as mea-
sured by the sup norm of - . It is important to under-
stand that the independence of the components of (for fixed
) in (6) allows the minimizing to be calculated using only
univariate optimizations; and the result so obtained does opti-
mize over for .
B. Relational-Data Strategy
The relational-data strategy uses the non-Euclidean re-
lational fuzzy c-means (NERFCM) in [14]. In essence,
NERFCM is a safeguarded version of RFCM, which is the
relational dual of fuzzy c-means. RFCM produces the FCM
clustering of indirectly using relational data
. Given some matrix of rela-
tional data , the NERFCM algorithm iteratively generates
a sequence of partition matrices according to the
following steps. The current matrix is used to calculate
vectors according to
(9)
These vectors are then used to calculate new dissimilarities
according to
for and (10)
If necessary, the dissimilarities are altered to guarantee posi-
tivity ([14]) and then they are used in (4) to generate the new
iterate. The iteration is continued until the sup-norm differ-
ence in successive matrices is sufficiently small.
The relational data approach for an extension of FCM
consists of applying NERFCM with the -based relational
data .
This approach will produce a terminal partition matrix ,
which attempts to represent the cluster memberships for
, but it does not directly provide cluster proto-
types . We recover meaningful prototypes by
using the terminal partition with and solving
(11)
We remark that the duality theory (in [14] and [15]) that guar-
antees that the same solution is found using the object and re-
lational versions of FCM holds only for . At other values
for , it is possible that the object data version and its relational
derivative yield different pairs for the same choices of
common algorithmic parameters. One purpose for trying a rela-
tional approach for is to discover any general similarities
between the object and relational approaches that extend past
the limited duality theory. Theoretical convergence of the object
data algorithms for is shown in [12]; existing con-
vergence theory ([14]) for the relational approach only covers
the case of .
Fig. 2. “Two cluster data” scatterplot and initial prototypes, .
III. NUMERICAL EXPERIMENTS
In all experiments we chose and stopped iteration as
soon as the absolute value of differences of all pairs of elements
(i.e., ) in a successive pair of matrices dif-
fered by less than 0.00001. The first experiment uses the (
point) data set in Fig. 2, which consists of three (25 point) ra-
dial clusters centered at and
and a varying number of outliers located at
the indicated positions. The purpose of this experiment is to in-
vestigate sensitivity to outliers of the object and relational ap-
proaches for various values of . The leftmost column of the
table gives the number of outliers included with the three clus-
tersin the sample.The number of outliers isevenand the outliers
are evenly divided between the two positions shown in Fig. 2.
All iteration sequences are initialized using a hard partition that
correctly partitions the three clusters and groups each outlier
with its nearest cluster. Fora computed set of terminal prototype
vectors and , we measure the sensitivity to the outliers
as the Frobenius norm distance between the true centers and
terminal prototype vectors: , where
is defined by .
Note the effects of increasing numbers of outliers as we move
down the rows of Table I. We see no deterioration in the quality
of the terminal prototype vectors for as many as 24 outliers for
both the object and relational approaches with and
. The last few rows indicate that this resistance to out-
liers is actually slightly greater for than for .
For any value of , the deviation of the computed proto-
types from the true cluster centers steadily increases with the
number of outliers. The object and relational data results are
quite comparable for and, as predicted by duality theory,
they produce identical results for .For , the object
and relational protoypes eventually vary from the true centers
for sufficiently large numbers of outliers, but fewer outliers are
required to cause substantial deviation for the relative data ap-
proach. Based on the results of this experiment, the object data
Authorized licensed use limited to: Richard Hathaway. Downloaded on February 1, 2009 at 13:19 from IEEE Xplore. Restrictions apply.
HATHAWAY et al.: GENERALIZED FUZZY c-MEANS CLUSTERING STRATEGIES 579
TABLE I
DEVIATION OF COMPUTED FROM TRUE CLUSTER CENTERS:
Fig. 3. “Two cluster data” scatterplot and initial prototypes .
approach for offers the greatest robustness and efficient
implementational approaches for it are discussed in [7].
The results of the first experiment show that in at least
some cases, the errors produced by the object and relational
approaches are of a similar magnitude. Is it also true that
the computed prototypes and partition matrices of the two
approaches are themselves very similar? The remaining nu-
merical experiments use other artificial two-dimensional data
sets that allow us to graphically depict the effect of on the
placement of the terminal prototype vectors. The two data
sets (and initial prototype values) are depicted in Figs. 3 and
4 and are, respectively, called the “two cluster data” and “no
cluster data.” Using identical data values, initializations,
and stopping criteria, we calculate and
using the relational and object data approaches, respectively.
We calculated the Frobenius norm difference in the terminal
partitions and prototypes produced by the two approaches as
and . These differences
are given for a range of values using the “two cluster data”
and “no cluster data” in Table II.
Duality theory for NERFCM ([11], [12]) guarantees that the
difference is zero when , but note that it is reasonably
Authorized licensed use limited to: Richard Hathaway. Downloaded on February 1, 2009 at 13:19 from IEEE Xplore. Restrictions apply.
580 IEEE TRANSACTIONS ON FUZZY SYSTEMS, VOL. 8, NO. 5, OCTOBER 2000
Fig. 4. “No cluster data” scatterplot and initial prototypes .
TABLE II
FROBENIUS NORM DIFFERENCE IN TERMINAL OBJECT AND RELATIONAL
VALUES
small for near two. Note also how the difference is some-
times much greater than the difference . We also see from
this table that the relational and object-based results can be very
different for the important case of .
The remaining figures in this section graphically depict the
position of the computed terminal prototypes for a range of
values. Fig. 5 shows the results for the “two cluster data”
obtained by the two approaches. Note that the outliers have
increasing effect as increases from to .As
continues to increase above two, the outlying data has an
even more powerful draw on the prototypes, which move ever
nearer to the approximate center of the figure. Note that for
, the terminal prototypes produced by the object-data
approach collapse into coincident clusters. Because the “two
cluster data” set has vertical symmetry, so do and from
either approach.
Fig. 6 shows a similar experiment for the relational-data ap-
proach applied to the “no cluster data.” While this is not a “clus-
tering” example, we used it to better understand the behavior of
the methods. Surprisingly, we observed coincident prototypes at
the center of the data for small values of such as and
nearly coincident prototypes for large values such as ;
the prototypes are most different and farthest from the center
when . The behavior of the object-data approach on this
example was similar in that gave the most separated pro-
totype values.
IV. DISCUSSION
We described relational and object-data approaches for gen-
erating norm extensions of FCM. Also, we examined the
behavior of the approaches for various values using artificial
data sets. We believe that the two most useful models are ob-
ject data based and correspond to and .For , the
fuzzy c-means algorithm in [1] offers the least expensive clus-
tering technique of all, and it works very well in most cases. For
cases where noisy data and outliers may degrade FCM results,
we recommend the use of the object data model with , op-
timized using the fuzzy c-medians algorithm described in [7].
The relational data approach is best saved for cases when ob-
ject data is unavailable or, in special cases for , when the
dimension of the feature space is very high but the number of
data is small. (In this case, it may be computationally cheaper
to form and operate on it rather than on the original feature
data.) We believe the relational approach for exhibits
some robustness properties, but overall we view it as inferior to
the object data approach of Kersten [7]. Our experiments always
used , but we believe the importance of and holds
for any choice of the fuzzification constant.
Authorized licensed use limited to: Richard Hathaway. Downloaded on February 1, 2009 at 13:19 from IEEE Xplore. Restrictions apply.
HATHAWAY et al.: GENERALIZED FUZZY c-MEANS CLUSTERING STRATEGIES 581
Fig. 5. “Two cluster data” terminal prototypes for the object and relational approaches .
Fig. 6. “No cluster data” terminal prototypes for the relational-data approach .
Choices for other than or lead to models which
can provide good clustering results and possibly classifier de-
signs, but the models are more difficult to optimize in the ob-
ject data case. For values near two, the results obtained using
the object and relational approaches are quite similar. However,
it now appears that the existing duality theory stated in [14] is
complete; that is, the object and relational approaches have a
strict duality relationship only when .As values in-
crease above one, the attraction of terminal prototype vectors
to outliers increases. The empirical migration of the prototypes
to the approximate center of the data sets as increases is inter-
esting to us, but no illuminating result regarding this has been
obtained. A partial analysis of the relational case indicates that
there is still a strong dependence on the point of initialization,
even as increases without bound. For example, consider clus-
tering into two clusters. It is not hard
to show that as , the relational based approach is es-
sentially equivalent to that done using NERFCM on the
matrix with all zero entries except . Simple
numerical experiments with different crisp initializations show
that different solutions are possible. Because of this, we believe
there may not be a nice theoretical result regarding the limiting
position of the prototypes as . We conclude by giving
one last question. Why is special in the sense demon-
Authorized licensed use limited to: Richard Hathaway. Downloaded on February 1, 2009 at 13:19 from IEEE Xplore. Restrictions apply.
582 IEEE TRANSACTIONS ON FUZZY SYSTEMS, VOL. 8, NO. 5, OCTOBER 2000
strated by Fig. 6 and does this mean that the original FCM is in
some sense optimal as a quantization tool?
ACKNOWLEDGMENT
The authors would like to thank the editor, associate editor,
and all reviewers for their helpful suggestions for improving this
manuscript.
REFERENCES
[1] J. C. Bezdek, Pattern Recognition with Fuzzy Objective Function Algo-
rithms. New York: Plenum, 1981.
[2] J. C. Bezdek, J. M. Keller, R. Krishnapuram, and N. R. Pal, Fuzzy
Models and Algorithms for Pattern Recognition and Image Pro-
cessing. Norwell, MA: Kluwer, 1999.
[3] R. Krishnapuram and J. M. Keller, “A possibilistic approach to clus-
tering,” IEEE Trans. Fuzzy Syst., vol. 1, pp. 98–110, May 1993.
[4] R. N. Dave, “Characterization and detection of noise in clustering,” Pat-
tern Recogn. Lett., vol. 12, pp. 657–664, 1991.
[5] P. R. Kersten, “The fuzzy median and the fuzzy MAD,” in Proc. ISUMA-
NAFIPS, College Park, MD, 1995, pp. 85–88.
[6] , “Fuzzy order statistics and their application to fuzzy clustering,”
IEEE Trans. Fuzzy Syst., vol. 7, pp. 708–712, Dec. 1999.
[7] , “Implementing the fuzzy c-medians clustering algorithm,” in
Proc. IEEE Conf. Fuzzy Syst., Barcelona, Spain, 1997, pp. 957–962.
[8] S. Miyamoto and Y. Agusta, “An efficient algorithm for fuzzy
c-means and its termination,” Contr. Cybern., vol. 25, pp. 421–436,
1995.
[9] L. Bobrowski and J. Bezdek, “C-means clustering with the and
norms,” IEEE Trans. Syst., Man, Cybern., vol. 21, pp. 545–554,
May/June 1991.
[10] K. Jajuga, “ norm-based fuzzy clustering,” Fuzzy Sets Syst., vol. 39,
pp. 43–50, 1991.
[11] S. Miyamoto and Y. Agusta, “Efficient algorithms for fuzzy c-means
and their termination properties,” in Proc. 5th Conf. Int. Federation
Classification Soc., Kobe, Japan, Mar. 1996, pp. 255–258.
[12] , “Algorithms for and fuzzy c-means and their conver-
gence,” in Studies in Classification, Data Analysis, and Knowledge
Organization: Data Science, Classification, and Related Methods,C.
Hayashi, N. Ohsumi, K. Yajima, Y. Tanaka, H. H. Bock, and Y. Baba,
Eds. Tokyo, Japan: Springer-Verlag, 1998, pp. 295–302.
[13] D. D. Overstreet, “Generalized fuzzy c-means clustering,” M.S. thesis
in mathematics, Georgia Southern Univ., Statesboro, GA, June 1998.
[14] R. J. Hathaway and J. C. Bezdek, “NERF c-means: Non-Euclidean re-
lational fuzzy clustering,” Pattern Recogn., vol. 27, pp. 429–437, 1994.
[15] , “Optimization of clustering criteria by reformulation,” IEEE
Trans. Fuzzy Syst., vol. 1, pp. 241–245,May 1995.
Richard J. Hathaway (M’93) (“BIG Rivets”) re-
ceived the B.S. degree in applied math from the
University of Georgia, Athens, in 1979, and the
Ph.D. degree in mathematical sciences from Rice
University, Houston, TX, in 1983.
He is currently a Professor in the Mathematics and
Computer Science Department, Georgia Southern
University, Statesboro. His research interests include
pattern recognition and numerical optimization.
James C. Bezdek (M’90–SM’90–F’92) (affection-
ately known as “Jackhammer” by coworkers in the
reviewing construction trades) received the B.S.C.E.
degree from the University of Nevada, Reno, in 1969,
and the Ph.D. degree in applied math from Cornell
University, Ithaca, NY, in 1973.
He is currently a Professor in the Computer
Science Department, University of West Florida,
Pensacola. His interests include woodworking, fine
cigars, optimization, motorcycles, pattern recog-
nition, gardening, fishing, image processing, snow
skiing, computational neural networks, blues music, woodworking, and com-
putational medicine.
Dr. Bezdek is the founding editor of the National Journal of Approximate
Reasoning and the IEEE TRANSACTIONS ON FUZZY SYSTEMS. He is a fellow of
IFSA.
Yingkang Hu (“Grinder”) received the B.S. degree
in mathematics from Beijing University of Chemical
Engineering, China, in 1982, and the Ph.D. degree in
mathematics from the University of South Carolina,
Columbia, in 1989.
He is a Professor in the Mathematics and
Computer Science Department, Georgia Southern
University, Statesboro. His research interests include
approximation theory, numerical computation, and
pattern recognition.
Authorized licensed use limited to: Richard Hathaway. Downloaded on February 1, 2009 at 13:19 from IEEE Xplore. Restrictions apply.
