Optimal Estimation and Prediction for Dense Signals in High-Dimensional Linear Models

Article · March 2012with9 Reads
Source: arXiv
Estimation and prediction problems for dense signals are often framed in terms of minimax problems over highly symmetric parameter spaces. In this paper, we study minimax problems over l2-balls for high-dimensional linear models with Gaussian predictors. We obtain sharp asymptotics for the minimax risk that are applicable in any asymptotic setting where the number of predictors diverges and prove that ridge regression is asymptotically minimax. Adaptive asymptotic minimax ridge estimators are also identified. Orthogonal invariance is heavily exploited throughout the paper and, beyond serving as a technical tool, provides additional insight into the problems considered here. Most of our results follow from an apparently novel analysis of an equivalent non-Gaussian sequence model with orthogonally invariant errors. As with many dense estimation and prediction problems, the minimax risk studied here has rate d/n, where d is the number of predictors and n is the number of observations; however, when d is roughly proportional to n the minimax risk is influenced by the spectral distribution of the predictors and is notably different from the linear minimax risk for the Gaussian sequence model (Pinsker, 1980) that often appears in other dense estimation and prediction problems.
    • "Sparse Gaussian predictive density estimation has the attributes of a sparse prediction problem adapted to the peculiarities of the entropy loss function. Point prediction analyses of dense signals in M.1 (Dicker, 2012, Huber and Leeb, 2012, Leeb, 2009) relate the worst-case performance with the spectral distribution of the predictors. Here, we concentrate on the orthogonal model. "
    [Show abstract] [Hide abstract] ABSTRACT: We consider estimating the predictive density under Kullback-Leibler loss in an $\ell_0$ sparse Gaussian sequence model. Explicit expressions of the first order minimax risk along with its exact constant, asymptotically least favorable priors and optimal predictive density estimates are derived. Compared to the sparse recovery results involving point estimation of the normal mean, new decision theoretic phenomena are seen here. Sub-optimal performance of the class of plug-in density estimates reflects the predictive nature of the problem and optimal strategies need diversification of the future risk. We find that minimax optimal strategies lie outside the Gaussian family but can be constructed with threshold predictive density estimates. Novel minimax techniques involving simultaneous calibration of the sparsity adjustment and the risk diversification mechanisms are used to design optimal predictive density estimates.
    Full-text · Article · Nov 2012