Karim Abou-Moustafa

Intel

There is an accumulating evidence in the literature that stability of learning algorithms is a key characteristic that permits a learning algorithm to generalize. Despite various insightful results in this direction, there seems to be an overlooked dichotomy in the type of stability-based generalization bounds we have in the literature. On one hand...

There is accumulating evidence in the literature that stability of learning algorithms is a key characteristic that permits a learning algorithm to generalize. Despite various insightful results in this direction, there seems to be an overlooked dichotomy in the type of stability-based generalization bounds we have in the literature. On one hand, t...

There is accumulating evidence in the literature that stability of learning algorithms is a key characteristic that permits a learning algorithm to generalize. Despite various insightful results in this direction, there seems to be an overlooked dichotomy in the type of stability-based generalization bounds we have in the literature. On one hand, t...

We consider a priori generalization bounds developed in terms of cross-validation estimates and the stability of learners. In particular, we first derive an exponential Efron-Stein type tail inequality for the concentration of a general function of n independent random variables. Next, under some reasonable notion of stability, we use this exponent...

Finding the set of nearest neighbors for a query point of interest appears in a variety of algorithms for machine learning and pattern recognition. Examples include k nearest neighbor classification, information retrieval, case-based reasoning, manifold learning, and nonlinear dimensionality reduction. In this work, we propose a new approach for de...

Digitally recorded data have become another critical natural resource in our current research environment. This reality is one of the tremendous victories for decades of research in computer engineering, computer science, electronics, and communications. While this scenario will continue to be the case in the future, our current era has also marked...

We are interested in the following questions. Given a finite data set \(\mathcal {S}\), with neither labels nor side information, and an unsupervised learning algorithm \(\mathsf {A}\), can the generalization of \(\mathsf {A}\) be assessed on \(\mathcal {S}\)? Similarly, given two unsupervised learning algorithms, \(\mathsf {A}_1\) and \(\mathsf {A...

We address the class masking problem in multiclass linear discriminant analysis (LDA). In the multiclass setting, LDA does not maximize each pairwise distance between classes, but rather maximizes the sum of all pairwise distances. This results in serious overlaps between classes that are close to each other in the input space, and degrades classif...

Entropy measures of probability distributions are widely used measures in ecology, biology, genetics, and in other fields, to quantify species diversity of a community. Unfortunately, entropy-based diversity indices, or diversity indices for short, suffer from three problems. First, when computing the diversity for samples withdrawn from communitie...

Manifold learning algorithms rely on a neighbourhood graph to provide an estimate of the data's local topology. Unfortunately, current methods for estimating local topology assume local Euclidean geometry and locally uniform data density, which often leads to poor embeddings of the data. We address these shortcomings by proposing a framework that c...

Manifold learning algorithms rely on a neighbourhood graph to provide an estimate of the data's local topology. Unfortunately, current methods for estimating local topology assume local Euclidean geometry and locally uniform data density, which often leads to poor data embeddings. We address these shortcomings by proposing a framework that combines...

Multivariate Gaussian densities are pervasive in pattern recognition and machine learning. A central operation that appears in most of these areas is to measure the difference between two multivariate Gaussians. Unfortunately, traditional measures based on the Kullback-Leibler (KL) divergence and the Bhattacharyya distance do not satisfy all metric...

Multivariate Gaussian densities are pervasive in pattern recognition and machine learning. A central operation that appears in most of these areas is to measure the difference between two multivariate Gaussians. Unfortunately, traditional measures based on the Kullback- Leibler (KL) divergence and the Bhattacharyya distance do not satisfy all metri...

Many unsupervised learning algorithms make use of kernels that rely on the Euclidean distance between two samples. However, the Euclidean distance is optimal for Gaussian distributed data. In this paper, we relax the global Gaussian assumption made by the Euclidean distance, and propose a locale Gaussian modelling for the immediate neighbourhood of...

Measuring the difference between two multivariate Gaussians is central to statistics and machine learning. Traditional measures based on the Bhattacharyya coefficient or the symmetric Kullback-Leibler divergence do not satisfy metric properties necessary for many algorithms. This paper proposes a metric for Gaussian densities. Similar to the Bhatta...

Linear Discriminant Analysis (LDA) is a popular tool for multiclass discriminative dimensionality reduction. How- ever, LDA suffers from two major problems: (1) It only op- timizes the Bayes error for the case of unimodal Gaussian classes with equal covariances (assuming full rank matri- ces) and, (2) The multiclass extension maximizes the sum of p...

To learn a metric for query-based operations, we combine the concept underlying manifold learning algorithms and the minimum volume ellipsoid metric in a unified algorithm to find the nearest neighbouring points on the manifold on which the query point is lying. Extensive experiments on standard benchmark data sets in the context of classification...

We first investigate the combined effect of data complexity, curse of dimensionality and the definition of the Euclidean distance on the distance measure between points. Then, based on the concepts un- derlying manifold learning algorithms and the minimum volume ellipsoid metric, we design an algorithm that learns a local metric on the lower dimens...

We are interested in learning an adaptive local metric on a lower dimensional manifold for query-based operations. We combine the concept underlying manifold learning algo- rithms and the minimum volume ellipsoid metric to find the nearest neighbouring points to a query point on the man- ifold on which the query point is lying. Extensive experi- me...

We propose an unsupervised "local learning" algorithm for learning a metric in the input space. Geometrically, for a given query point, the algorithm finds the minimum volume ellipsoid (MVE) cover-ing its neighborhood which characterizes the correlations and variances of its neighborhood variables. Algebraically, the algorithm maximizes the determi...

Classification of Time-Series data using discriminative models such as SVMs is very hard due to the variable length of this type of data. On the other hand generative models such as HMMs have become the standard tool for modeling Time-Series data due to their efficiency. This paper proposes a general generative/discriminative hybrid that uses HMMs...

Classification of sequential data using discriminative models such as support vector machines is very hard due to the variable length of this type of data. On the other hand, generative models such as HMMs have become the standard tool for representing sequential data due to their efficiency. This paper proposes a general generative-discriminative...

This paper investigates the effect of HMM structure on the performance of HMM-based classifiers. The investigation is based on the framework of graphical models, the diffusion of credits of HMMs and empirical experiments. Although some researchers have focused on determining the number of states, this study shows that the topology has a stronger in...

Classification of Sequential data using discriminative mod - els such as SVMs is very hard due to the variable length of this type of data. On the other hand, generative models such as HMMs have become the standard tool for represent- ing sequential data due to their efficiency. This paper pro- poses a general generative-discriminative framework th...

Thesis (M.Comp.Sc.)--Concordia University, 2004. Includes bibliographical references.

This thesis targets the problem of poor performance of HMM-based classifiers. First, we study the effect of the structure on the performance of HMMs and see how the number of states and the topology can contribute to the classification performance. As a result, our investigation showed the topology has a stronger contribution to the classification...