Alain Berlinet

• Professor (Now honorary) at Université de Montpellier

The paper aims at answering the following question, in the scalar as well in the vector case: What do the famous Aitken’s $\Delta ^{2}$ and Wynn’s $\varepsilon $ -algorithm exactly do with the terms of the input sequence? Inspecting the rules of these algorithms from a geometric point of view leads to change the question into another one: By wh...

Among recent methods designed for accelerating the EM algorithm without any modification in the structure of EM or in the statistical model, the parabolic acceleration (P-EM) has proved its efficiency. It does not involve any computation of gradient or hessian matrix and can be used as an additional software component of any fixed point algorithm m...

The index of regularity of a measure was introduced by Beirlant, Berlinet and Biau [1] to solve practical problems in nearest neighbour density estimation such as removing bias or selecting the number of neighbours. These authors proved the weak consistency of an estimator based on the nearest neighbour density estimator. In this paper, we study an...

This paper shows that a class of methods for solving linear equations, including the Cauchy–Barzilai–Borwein method, can be interpreted by means of a simple geometric object, the Bézier parabola. This curve is built from the current iterate using a transformation characterizing the system to be solved. The localization of the next iterates in the p...

This paper deals with a special adaptive estimation problem, namely how can one select for each set of i.i.d. data X 1, …, X n the better of two given estimates of the data-generating probability density. Such a problem was studied by Devroye and Lugosi [Combinatorial Methods in Density Estimation, Springer, Berlin, 2001] who proposed a feasible su...

Many convergence results in density estimation can be stated without any restriction on the function to be estimated. Unlike these universal properties, the asymptotic normality of estimators often requires hypotheses on the derivatives of the underlying density and additional conditions on the smoothing parameter. Yet, despite the possible bad loc...

We outline a general method that estimates smooth functionals of a probability distribution from a sample of observations, restricting the framework to local polynomial fitting. The construction of the estimators is based on a weighted least squares criterion and reproducing kernel Hilbert spaces theory. We briefly discuss their asymptotic properti...

L'estimateur des plus proches voisins de la densité est un estimateur simple et facile à mettre en oeuvre. Sa normalité asymptotique a été établie par Moore et Yackel~(1977) sous des hypothèses faisant intervenir les dérivées de la densité. Sans faire d'hypothèse de continuité sur la densité, nous donnons une condition nécessaire et suffisante de c...

Since the first works laying its foundations as a subfield of complex analysis, the theory of reproducing kernels has proved to be a powerful tool in many fields of pure and applied mathematics. The aim of this paper is to give some idea of how and why this theory interacts with probability and statistics.

A new acceleration scheme for optimization procedures is defined through geometric considerations and applied to the EM algorithm. In many cases it is able to circumvent the problem of stagnation. No modification of the original algorithm is required. It is simply used as a software component. Thus the new scheme can be easily implemented to accele...

Dans ce travail, on s'intéresse à la régression non paramétrique locale pour des variables explicatives fonctionnelles. On propose tout d'abord un estimateur de l'opérateur de régression. La construction de cet estimateur est liée à la résolution d'un problème inverse linéaire. On établit des bornes de l'erreur quadratique moyenne en utilisant une...

La dérivée symétrique d'une mesure de probabilité en un point de Lebesgue peut souvent être approximée à l'aide d'un développement faisant intervenir un indice de régularité. La connaissance de cet indice est d'un intérêt pratique. En effet, il permet par exemple de déterminer le comportement local de la mesure étudiée. Il intervient aussi dans l'é...

Nous donnons des théorèmes généraux de normalité asymptotique des estimateurs convergents du mode conditionnel, indépendamment de la structure de dépendance des données et les appliquons au cas d'un processus stationnaire α-mélangeant.We state sufficient conditions for asymptotic normality of convergent estimates of the conditional mode, irrespecti...

The symmetric derivative of a probability measure at a Lebesgue point can often be specified by an exact relation involving a regularity index. Knowledge of this index is of practical interest, for example to specify the local behavior of the measure under study and to evaluate bandwidths or number of neighbors to take into account in smoothing tec...

Let X be a random variable taking values in a Hilbert space and let Y be a random label with values in {0, 1}. Given a collection of classification rules and a learning sample of independent copies of the pair (X, Y ), it is shown how to select optimally and consistently a classifier. As a general strategy, the learning sample observations are firs...

We study a non linear regression model with functional data as inputs and scalar response. We propose a pointwise estimate of the regression function that maps a Hilbert space onto the real line by a local linear method. We provide the asymptotic mean square error. Computations involve a linear inverse problem as well as a representation of the sma...

New acceleration schemes and restarting procedures are defined and studied in view of application to the EM algorithm. In most cases the introduced algorithms circumvent the problems of stagnation and degeneracy. Their behavior is analyzed on real data sets.

It is known that quantizations of primary sources of information reduce the information available for statistical inference. We are interested in the quantizations for which the loss of statistical information can be controlled by the number of cells in the observation space used to quantize observations. If the losses for increasing numbers of cel...

Density estimation raises delicate problems in higher dimensions especially when strong convergence is required and data marginals can be highly correlated. Modified histograms have been introduced to circumvent the problem of low bin counts when convergence is considered in the sense of information divergence. These estimates are defined from some...

It is well-established that one can improve performance of kernel density estimates by varying the bandwidth with the location and/or the sample data at hand. Our interest in this paper is in the data-based selection of a variable bandwidth within an appropriate parameterized class of functions. We present an automatic selection procedure inspired...

Basic definitions and theory Stochastic processes Loève and Karhunen theorems Nonparametric functional estimation Higher order kernels Embedding method for measures The example of moments Learning and decision theory Support vector machines Law of iterated logarithm Strassen theorem Kuelbs theorem

A multivariate modified histogram density estimate depending on a reference density g and a partition P has been proved to have good consistency properties according to several information theoretic criteria. Given an i.i.d. sample, we show how to select automatically both g and P so that the expected L1 error of the corresponding selected estimate...

A modified histogram estimate depending on a reference density has recently been proved to have good consistency properties according to several information theoretic criteria. In the present paper, this modified histogram is studied as a dynamical system in a functional space with the reference density as initial state. Properties of stationary de...

The convergence of measure estimates in the sense of Kullback–Leibler divergence is required in many applications in decision and information theory. Recently, modified histograms have been shown to have good properties with respect to information divergences. For these estimates deterministic optimal bandwidths have been given, but no automatic sm...

1 Theory.- 2 RKHS AND STOCHASTIC PROCESSES.- 3 Nonparametric Curve Estimation.- 4 Measures And Random Measures.- 5 Miscellaneous Applications.- 6 Computational Aspects.- 7 A Collection of Examples.- to Sobolev spaces.- A.l Schwartz-distributions or generalized functions.- A.1.1 Spaces and their topology.- A.1.2 Weak-derivative or derivative in the...

Since its foundation by Borel and Lebesgue around the year 1900 the modern theory of measure, generalizing the basic notions of length, area and volume, has become one of the major fields in Pure and Applied Mathematics. In all human activities one collects measurements subject to variability and leading to the classical concepts of Probability and...

In Chapter 1, we have studied the relationships between reproducing kernels and positive definite functions. In this chapter, the central result due to Loève is that the class of covariance functions of second order-stochastic processes coincide with the class of positive definite functions. This link has been used to translate some problems relate...

Reproducing kernels are often found in nonpararnetric curve estimation in connection with the use of spline functions, which were popularized by Wahba in the statistics literature in the 1970s. A brief introduction to the theory of splines is presented in Section 2. Sections 4 and 5 are devoted to the use of splines in nonparametric estimation of d...

New reproducing kernels with interesting applications continually appear in the literature. In Section 4 of the present chapter we list major examples for which the kernel and the associated norm and space are explicitly described. They can be used to illustrate aspects of the theory or to practically implement some of the tools presented in the bo...

A Reproducing Kernel Hilbert Space (RKHS) is first of all a Hilbert space, that is, the most natural extension of the mathematical model for the actual space where everyday life takes place (the Euclidean space ℝ3). When studying elements of some abstract set S it is convenient to consider them as elements of some other set S′ on which is already d...

The theory of reproducing kernel Hilbert spaces interacts with so many subjects in Probability and Mathematical Statistics that it is impossible to deal with all of them in this book. Besides topics that we were willing to develop and to which a chapter is devoted we have selected a few themes gathered in the present chapter.

In many applications, the choice of Hilbert space and norm is governed by context related modeling reasons and one has to face the problem of computing the corresponding reproducing kernel. Symmetrically, it is of interest to characterize the Hilbert space H K associated with a given kernel K by the Moore-Aronszajn theorem and in particular to give...

This paper presents a general framework for estimating smooth functionals of the probability distribution functions, such as the density, the hazard rate function, the mean residual time, the Lorenz curve, the spectral density, the tail index, the quantile function and many others. This framework is based on maximizing a local asymptotic pseudolike...

We extend the concept of piecewise linear histogram introduced recently by Beirlant, Berlinet and Györfi. The disadvantage of that histogram is that in many models it takes on negative values with probability close to 1. We show that for a wide set of models, the extended class of estimates contains a bona fide density with probability tending to 1...

Barron-type estimators are histogram-based distribution estimators that have been proved to have good consistency properties according to several information theoretic criteria. However they are not continuous. In this paper, we examine a new class of continuous distribution estimators obtained as a combination of Barron-type estimators with the fr...

In a recent paper [Stat. Neerl. 56, No. 3, 301–313 (2002; Zbl 1076.62035)] we have extended the concept of piecewise linear histogram (PLH) introduced by J. Beirlant, A. Berlinet and L. Györfi [Stat. Neerl. 53, No. 3, 287–308 (1999; Zbl 0954.62038)]. The disadvantage of the PLH is that, in many models with probability close to 1, it takes on negati...

We study piecewise linear density estimators from the L1 point of view: the frequency polygons investigated by Scott (1985) and Jones et al. (1997), and a new piecewise linear histogram. In contrast to the earlier proposals, a unique multivariate generalization of the new piecewise linear histogram is available. All these estimators are shown to be...

Let Q be a conditional distribution with L1-median μ and let {Qn} be a sequence of conditional distributions converging in some sense to Q. Then the L1-medians {μn} of distributions {Qn} are natural estimates of μ. In the case where {Qn} is a sequence of kernel estimates we give conditions ensuring that {μn} is a well-defined sequence of continuous...

Barron estimators are modified histograms depending on some reference probability density. They have been proved to have good consistency properties according to several information theoretical criteria. In this Note, Barron estimators are viewed as dynamical systems in a functional space with the reference density as initial state. Properties of s...

We consider the problem of estimating a multidimensional discrete deterministic dynamical system from the first n + 1 observations. We exhibit the optimal rate function r n and show that the nearest neighbor estimator achieves this optimal rate, extending a recent study carried out by Guerre and Mas [18] in the univariate case. Previous results str...

We introduce a modified version ƒnof the piecewiss linear hisiugrimi uf Beirlant et al. (1998) which is a true probability density, i.e., ƒn[d] 0 and [d]ƒn=1. We prove that ƒnestimates the underlying densitv ƒ strongly consistently in the L1mmn, derive large deviation inequalities for the t\ error \ƒn- f\ and prove that £||/"-/|| tends to zero with...

We state sufficient conditions for asymptotic normality of convergent estimates of conditional quantiles, irrespective of data dependence and consider the particular case of α-mixing stationary processes under optimal condition of convergence. We apply this result to confidence intervals building for time series predictors based on nonparametric es...

Let Q be a conditional distribution with L1-median [mu] and let {Qn} be a sequence of conditional distributions converging in some sense to Q. Then the L1-medians {[mu]n} of distributions {Qn} are natural estimates of [mu]. In the case where {Qn} is a sequence of kernel estimates we give conditions ensuring that {[mu]n} is a well-defined sequence o...

We consider Barron density estimates defined as modified histograms. We give convergence results for the Kullback–Leibler divergence and its expectation. We propose an automatic procedure for selecting the asymptotically optimal bandwidth.

We consider Barron density estimates defined as modified histograms. We give convergence results for the Kullback-Leibler divergence and its expectation. We propose an automatic procedure for selecting the asymptotically optimal bandwidth. © 2000 Académie des sciences/Éditions scientifiques et médicales Elsevier SAS.

From consistency results on estimators asymptotically minimizing a general criterion, we derive necessary and sufficient conditions for the convergence of approximate M-estimators in nonlinear regression models with nonstationary and/or dependent errors. Special attention is paid to models with errors satisfying the law of large numbers. The condit...

Statistics based on the sample autocovariances are widely used in time-series analysis. Estimators of the asymptotic covariance between the sample autocovariances are commonly derived from the so-called Bartlett's formula. However, this formula essentially holds for linear processes. This entails that for a wide range of nonlinear time series the a...

This article was originally published in Encyclopedia of Statistical Sciences (Update Volume 3, April 15, 1999) ISBN-10, ‎047123883X ; ISBN-13, ‎978-0471238836

The asymptotic behavior of global errors of functional estimates plays a key role in hypothesis testing and confidence interval building. Whereas for pointwise errors asymptotic normality often easily follows from standard Central Limit Theorems, global errors asymptotics involve some additional techniques such as strong approximation, martingale t...

By extending the information-theoretic arguments of previous papers dealing with the Barron-type density estimates, and their consistency in information divergence and chi-square divergence, the problem of consistency in Csiszar's φ-divergence is motivated for general convex functions φ. The problem of consistency in φ-divergence is solved for all...

Multivariate kernel density estimators are known to systematically deviate from the true value near critical points of the density surface. To overcome this difficulty a method based on Rao-Blackwell's theorem is proposed. Local corrections of kernel density estimators are achieved by conditioning these estimators with respect to locally sufficient...

We state sufficient conditions for asymptotic normality of convergent estimates of the conditional quantiles, irrespective of data dependence, and give an application to α-mixing stationary processes, under optimal conditions. As an application, we use asymptotic normality to construct confidence bands for predictors based on nonparametric estimate...

We state sufficient conditions for asymptotic normality of convergent estimates of the conditional quantiles, irrespective of data dependence, and give an application to α-mixing stationary processes, under optimal conditions. As an application, we use asymptotic normality to construct confidence bands for predictors based on nonparametric estimate...

By studying the geometry of relevant Hilbert spaces, we give a characterization of the identifiable standard representations of multivariate ARMA models in terms of the autocovariance function.

Bartlett’s formula is widely used in time series analysis to provide estimates of the asymptotic covariance between sample autocovariances. However, it is derived under precise assumptions (namely linearity of the underlying process and vanishing of its fourth-order cumulants) and effectiv e computations show that the value given by this formula ca...

We show that the centered and normalized relative entropy error of a consistent histogram based density estimate is asymptotically normal with asymptotic variance less than or equal to 1 for all multivariate densities f which have a finite relative entropy with respect to a given reference density g and which satisfy a mild condition on the boundar...

A transformation [phi] defined on a subset E of the real line and taking real values reduces the variance if and only if [phi] is Lipschitz continuous on E with constant equal to one. This result provides a general method to compare the variance of two random variables.

Let f(n) be a histogram estimate constructed from a sample of i.i.d. real-valued random variables with common continuously differentiable density S. In this paper we prove a central limit theorem for the L(t) error parallel to f(n) - f parallel to. We determine a positive constant 0 < sigma(2) less than or equal to 1 - 2/pi in order that, under the...

The first part of this paper is a review of central limit theorems for L p errors of nonparametric density estimates. For each of them the main probabilistic tools are presented and the assumptions are discussed. Besides the seminal works of P. J. Bickel and M. Rosenblatt [Ann. Stat. 1, 1071-1095 (1973; Zbl 0275.62033); Correction ibid. 3, 1370 (19...

In the double kernel density estimate, the smoothing parameter h is chosen so as to minimize the L 1 distance between two kernel density estimates having identical smoothing factors but different kernels. This method is known to be consistent for any density and to be asymptotically optimal for a certain smooth class of densities. We propose a plug...

Recent literature on functional estimation has shown the importance of kernels with vanishing moments although no general framework was given to build kernels of increasing order apart from some specific methods based on moment relationships. The purpose of the present paper is to develop such a framework and to show how to build higher order kerne...

Approximation of functionals on RKHS RKHS embedding of measures Optimum kernels in probability density estimation Examples of hierarchy of kernels

Recent literature on density and regression function estimation has shown the interest of kernels of order s, i.e. kernels with (s-1) vanishing moments. We define here a multi-stage procedure to build estimates based on increasing order kernels and leading to a data-driven choice of both the order and the smoothing parameter. Some asymptotic as wel...

Revue : Statistique et Analyse des Données

A minimization theorem in Riesz spaces is given and applied to functional estimation. The applications concern density (optimal kernels), spectrum and bispectrum (optimal windows) estimation. Published in "Cahiers du Centre d'Etudes de Recherche Opérationnelle"

We define a wide class of multivariate density estimates for which we give sufficient conditions for strong uniform consistency. This class contains all usual kernel estimators and allows non affine dependence with respect to the observations or the estimation point.

A number of sequence transformations are actually used in statistics to solve different kinds of problems. In the two first parts of this paper we set the statistical problem of estimating the unknown orders of an ARMA process and we give its equivalent formulation in terms of invariance properties of sequence transformations: the most used among t...

RÉSUMÉ Ce travail comprend quatre parties. La première est consacrée à l'étude de certaines propriétés des paramètres relatives à l'espace des observations, aux statistiques adaptées à ces propriétés et aux estimateurs que l'on peut en déduire. Lorsque l'on évoque des propriétés locales d'un paramètre défini sur un ensemble de probabilités P et à...

Compstat Lectures 3, 1984

Alternative Approaches to Time Series Analysis. Publications des Facultés universitaires Saint-Louis. Bruxelles

Publications de l'Institut de Statistique de l'Université de Paris

Random variables taking their values in RKHS. Embedding of classes of measures in RKHS. Special case of the empirical measure: strong approximation. Spline methods in functional estimation. Special case of the probability density.

Berlinet, Alain: Variables aléatoires dans les espaces autoreproduisants et mesure empirique. C. R. Acad. Sci., Paris, Sér. A 290, 973-980 (1980). RKHS embedding of measures. Many books and papers dealing with random variables taking their values in Hilbert spaces have been published. Now, spaces used in applications are often Reproducing Kernel Hi...

Embedding of measures in RKHS. Special case of the empirical measure. Random variables with values in RKHS. Proofs can be found in my PhD thesis (1980) and in my book with Christine Thomas-Agnan (2004): Reproducing Kernel Hilbert Spaces in Probability and Statistics.

At the basis of combinatorial methods in density estimation introduced by De- vroye and Lugosi is the so-called Sche¤é selection rule. We show by an examples that this rule based on L1 errors may not bring the selection closer to optimality than tossing of a coin. As in any estimation problem, the choice of a criterion is at the heart of the matter...