[show abstract][hide abstract] ABSTRACT: Linear discriminant analysis(LDA) is a traditional dimension reduction method which finds projective directions to maximize separability between classes. However, when the number of labeled data points is small, the performance of LDA is degraded severely. In this paper, we propose two improved methods for LDA which utilizes abundant unlabeled data. Instead of using all the unlabeled data as in most of semi- supervised dimension reduction methods, we select confident unlabeled data and develop extended LDA algorithms. In the first method, a graph-based LDA method is developed to utilize confidence scores for chosen unlabeled data so that unlabeled data with a low confidence score contributes smaller than unlabeled data with a high confidence score. In the second method, selected unlabeled data points are used to modify the centroids of classes in an objective function of LDA. Extensive experimental results in text classification demonstrates the effectiveness of the proposed methods compared with other semi-supervised dimension reduction methods.
[show abstract][hide abstract] ABSTRACT: Associative classification has been recently proposed which combines association rule mining and classification, and many studies have shown that associative classifiers give high prediction accuracies compared with other traditional classifiers such as a decision tree. However, in order to apply association rule mining to classification problems, data transformation into the form of transaction data should be preceded before applying association rule mining. In this paper, we propose a discretization method based on Support vector machines, which can improve the performance of association classification greatly. The proposed method finds optimal class boundaries by using SVM, and discretization utilizing distances to the boundaries is performed. Experimental results demonstrate that performing SVM-based discretization for continuous attributes makes associative classification more effective in that it reduces the number of classification rules mined and also improves the prediction accuracies at the same time.
[show abstract][hide abstract] ABSTRACT: Dimension reduction is a preprocessing step by which small number of optimal features are extracted. Among several statistical dimension reduction methods, Linear discriminant analysis (LDA) performs dimension reduction to maximize class separability in the reduced dimensional space. However, in multi-labeled problems, data samples belonging to multiple classes cause contradiction between the maximization of the distances between classes and the minimization of the scatter within classes, since they are placed in the overlapping area of multiple classes. In this paper, we show that in multi-labeled text categorization, the outputs from multiple linear methods can be used to compose new features for low dimensional representation. Especially, we apply least squares regression and a linear support vector machine (SVM) for multiple binary-class problems constructed from a multi-labeled problem and obtain optimal features in a low dimensional space which are fed into another classification algorithm. Extensive experimental results in text categorization are presented comparing with other dimension reduction methods and multi-label classification algorithms.
Computer and Information Technology, 2008. CIT 2008. 8th IEEE International Conference on; 08/2008
[show abstract][hide abstract] ABSTRACT: Associative classification has been recently proposed which combines association rule mining and classification, and many studies have shown that associative classifiers have high prediction accuracies. In order to apply an asso- ciation rule mining to classification problem, data transfor- mation into the form of transaction data should be preceded before applying association rule mining. In this paper, we propose a discretization method based on Support vector machines, which is very effective for association classifica- tion. The proposed method finds optimal class boundaries by using SVM, and discretization utilizing distances to the boundaries is performed. Experimental results demonstrate that performing SVM-based discretization for continuous attributes makes associative classification more effective in that it reduces the number of association rules mined and also improves the prediction accuracies at the same time.
Computational Intelligence and Security, 2007 International Conference on; 01/2008
[show abstract][hide abstract] ABSTRACT: Linear discriminant analysis (LDA) is one of the most popular dimension reduction methods, but it is originally focused on a single-labeled problem. In this paper, we derive the formulation for applying LDA for a multi-labeled problem. We also propose a generalized LDA algorithm which is effective in a high dimensional multi-labeled problem. Experimental results demonstrate that by considering multi-labeled structure, LDA can achieve computational efficiency and also improve classification performances.
[show abstract][hide abstract] ABSTRACT: While much progress has been made in face recognition over the last decades, changes in illumination directions still remain as a difficult problem. In this paper, we propose an efficient image normalization method which can overcome illumination effects effectively. The proposed method is based on intensity distribution transformation. However, instead of applying it globally, transformation in intensity distribution is performed for each column independently using one frontal mean face as a reference image. Since it does not require image processing such as image segmentation, the computational complexity is very low and it can circumvent boundary discontinuity caused by region segmentation. Extensive experimental results using Feret database and extended Yale B database demonstrate the competence of the proposed method.
Proceedings of the 1st ACM SIGMM International Conference on Multimedia Information Retrieval, MIR 2008, Vancouver, British Columbia, Canada, October 30-31, 2008; 01/2008
[show abstract][hide abstract] ABSTRACT: Linear discriminant analysis (LDA) is a dimension reduction method which finds an optimal linear transformation that maximizes the class separability. However, in undersampled problems where the number of data samples is smaller than the dimension of data space, it is difficult to apply LDA due to the singularity of scatter matrices caused by high dimensionality. In order to make LDA applicable, several generalizations of LDA have been proposed recently. In this paper, we present theoretical and algorithmic relationships among several generalized LDA algorithms and compare their computational complexities and performances in text classification and face recognition. Towards a practical dimension reduction method for high dimensional data, an efficient algorithm is proposed, which reduces the computational complexity greatly while achieving competitive prediction accuracies. We also present nonlinear extensions of these LDA algorithms based on kernel methods. It is shown that a generalized eigenvalue problem can be formulated in the kernel-based feature space, and generalized LDA algorithms are applied to solve the generalized eigenvalue problem, resulting in nonlinear discriminant analysis. Performances of these linear and nonlinear discriminant analysis algorithms are compared extensively.
[show abstract][hide abstract] ABSTRACT: Most of classifiers implicitly assume that data samples belong to at least one class among predefined classes. How- ever, all data patterns may not be known at the time of data collection or a new pattern can be emerging over time. Hence ideal classifiers need to be able to recognize an emerging pattern. In this paper, we explore the per- formances and limitations of the existing classification sys- tems in detecting a new class. Also a new method is pro- posed that can monitor the change in class distribution and detect an emerging class. It works under the supervised learning model where along with classification an emerg- ing class with new characteristic is detected so that classi- fication model can be adapted systematically. For detection of an emerging class, we design statistical significance test- ing for signaling change of class distribution. When the alarm for new class generation is set on, candidates for new class members are retrieved for close examination by experts. Our experimental results demonstrate competent performances of the proposed method.
Granular Computing, 2007. GRC 2007. IEEE International Conference on; 12/2007
[show abstract][hide abstract] ABSTRACT: Traditional classification problem assumes that a data sample belongs to one class among the predefined classes. On the other
hand, in a multi-labeled problem such as text categorization, data samples can belong to multiple classes and the task is
to output a set of class labels associated with new unseen data sample. As common in text categorization problem, learning
a classifier in a high dimensional space can be difficult, known as the curse of dimensionality. It has been shown that performing
dimension reduction as a preprocessing step can improve classification performances greatly. Especially, Linear discriminant
analysis (LDA) is one of the most popular dimension reduction methods, which is optimized for classification tasks. However,
in applying LDA for a multi-labeled problem some ambiguities and difficulties can arise. In this paper, we study on applying
LDA for a multi-labeled problem and analyze how an objective function of LDA can be interpreted in multi-labeled setting.
We also propose a LDA algorithm which is effective in a multi-labeled problem. Experimental results demonstrate that by considering
multi-labeled structures LDA can achieve computational efficiency and also improve classification performances greatly.
Machine Learning and Data Mining in Pattern Recognition, 5th International Conference, MLDM 2007, Leipzig, Germany, July 18-20, 2007, Proceedings; 01/2007
[show abstract][hide abstract] ABSTRACT: Feature extraction is an important preprocessing step which is encountered in many areas such as data mining, pattern recognition
and scientific visualization. In this paper, a new method for sparse feature extraction using local manifold learning is proposed.
Similarities in a neighborhood are first computed to explore local geometric structures, producing sparse feature representation.
Based on the constructed similarity matrix, linear dimension reduction is applied to enhance similarities among the elements
in the same class and extract optimal features for classification performances. Since it only computes similarities in a neighborhood,
sparsity in the similarity matrix can give computational efficiency and memory savings. Experimental results demonstrate superior
performances of the proposed method.
Advances in Knowledge Discovery and Data Mining, 10th Pacific-Asia Conference, PAKDD 2006, Singapore, April 9-12, 2006, Proceedings; 01/2006
[show abstract][hide abstract] ABSTRACT: Linear Discriminant Analysis (LDA) has been widely used for linear dimension reduction. However, LDA has some limitations that one of the scatter matrices is required to be nonsingular and the nonlinearly clustered structure is not easily captured. In order to overcome the problems caused by the singularity of the scatter matrices, a generalization of LDA based on the generalized singular value decomposition (GSVD) has been developed recently. In this paper, we propose a nonlinear discriminant analysis based on the kernel method and the generalized singular value decomposition. The GSVD is applied to solve the generalized eigenvalue problem which is formulated in the feature space defined by a nonlinear mapping through kernel functions. Our GSVD-based kernel discriminant anal- ysis is theoretically compared with other kernel-based nonlinear discriminant analysis algorithms. The experimental results show that our method is an effective nonlinear dimension reduction method.
SIAM J. Matrix Analysis Applications. 01/2005; 27:87-102.
[show abstract][hide abstract] ABSTRACT: In this paper, we present a new approach for fingerprint classification based on discrete Fourier transform (DFT) and nonlinear discriminant analysis. Utilizing the DFT and directional filters, a reliable and efficient directional image is constructed from each fingerprint image, and then nonlinear discriminant analysis is applied to the constructed directional images, reducing the dimension dramatically and extracting the discriminant features. The proposed method explores the capability of DFT and directional filtering in dealing with low-quality images and the effectiveness of nonlinear feature extraction method in fingerprint classification. Experimental results demonstrates competitive performance compared with other published results.
[show abstract][hide abstract] ABSTRACT: This paper presents theoretical relationships among several generalized LDA algorithms and proposes computationally efficient approaches for them utilizing the relationships. Generalized LDA algorithms are extended nonlinearly by kernel methods resulting in nonlinear discriminant analysis. Performances and computational complexities of these linear and nonlinear discriminant analysis algorithms are compared.
Data Mining, 2004. ICDM '04. Fourth IEEE International Conference on; 12/2004
[show abstract][hide abstract] ABSTRACT: An optimization criterion is presented for discriminant analysis. The criterion extends the optimization criteria of the classical Linear Discriminant Analysis (LDA) through the use of the pseudoinverse when the scatter matrices are singular. It is applicable regardless of the relative sizes of the data dimension and sample size, overcoming a limitation of classical LDA. The optimization problem can be solved analytically by applying the Generalized Singular Value Decomposition (GSVD) technique. The pseudoinverse has been suggested and used for undersampled problems in the past, where the data dimension exceeds the number of data points. The criterion proposed in this paper provides a theoretical justification for this procedure. An approximation algorithm for the GSVD-based approach is also presented. It reduces the computational complexity by finding subclusters of each cluster and uses their centroids to capture the structure of each cluster. This reduced problem yields much smaller matrices to which the GSVD can be applied efficiently. Experiments on text data, with up to 7,000 dimensions, show that the approximation algorithm produces results that are close to those produced by the exact algorithm.
IEEE Transactions on Pattern Analysis and Machine Intelligence 09/2004; 26(8):982-94. · 4.80 Impact Factor
[show abstract][hide abstract] ABSTRACT: A nonlinear feature extraction method is presented which can reduce the data dimension down to the number of clusters, providing dramatic savings in computational costs. The dimension reducing nonlin- ear transformation is obtained by implicitly mapping the input data into a feature space using a kernel function, and then finding a linear mapping based on an orthonormal basis of centroids in the feature space that maximally separates the between-cluster relationship. The experimental results demonstrate that our method is capable of extracting nonlinear features effectively so that competitive performance of classification can be obtained with linear classifiers in the dimension reduced space.