Abstract:We study the classical problem of predicting an outcome variable, $Y$, using a linear combination of a $d$-dimensional covariate vector, $\mathbf{X}$. We are interested in linear predictors whose coefficients solve: % \begin{align*} \inf_{\boldsymbol{\beta} \in \mathbb{R}^d} \left( \mathbb{E}_{\mathbb{P}_n} \left[ \left(Y-\mathbf{X}^{\top}\beta \right)^r \right] \right)^{1/r} +\delta \, \rho\left(\boldsymbol{\beta}\right), \end{align*} where $\delta>0$ is a regularization parameter, $\rho:\mathbb{R}^d\to \mathbb{R}_+$ is a convex penalty function, $\mathbb{P}_n$ is the empirical distribution of the data, and $r\geq 1$. We present three sets of new results. First, we provide conditions under which linear predictors based on these estimators % solve a \emph{distributionally robust optimization} problem: they minimize the worst-case prediction error over distributions that are close to each other in a type of \emph{max-sliced Wasserstein metric}. Second, we provide a detailed finite-sample and asymptotic analysis of the statistical properties of the balls of distributions over which the worst-case prediction error is analyzed. Third, we use the distributionally robust optimality and our statistical analysis to present i) an oracle recommendation for the choice of regularization parameter, $\delta$, that guarantees good out-of-sample prediction error; and ii) a test-statistic to rank the out-of-sample performance of two different linear estimators. None of our results rely on sparsity assumptions about the true data generating process; thus, they broaden the scope of use of the square-root lasso and related estimators in prediction problems.

Finite Sample Analysis of Distribution-Free Confidence Ellipsoids for Linear Regression

Finite-Sample Identification of Linear Regression Models with Residual-Permuted Sums

Sample Complexity of the Sign-Perturbed Sums Method

On Adaptive confidence Ellipsoids for sparse high dimensional linear models

Asymptotic Theory of Least Distances Estimate in Multivariate Linear Models

Optimal Simultaneous Confidence Bands in Multiple Linear Regression with Predictor Variables Constrained in an Ellipsoidal Region

Optimal High-Dimensional Shrinkage Covariance Estimation for Elliptical Distributions

Optimal shrinkage covariance matrix estimation under random sampling from elliptical distributions

Some facts about the optimality of the LSE in the Gaussian sequence model with convex constraint

Valid Credible Ellipsoids for Linear Functionals by a Renormalized Bernstein-von Mises Theorem

Finite Sample Confidence Regions for Linear Regression Parameters Using Arbitrary Predictors

Distribution-Free Robust Linear Regression

Linear shrinkage of sample covariance matrix or matrices under elliptical distributions: a review

A Geometric Approach to Confidence Regions and Bands for Functional Parameters

Learning minimal volume uncertainty ellipsoids

Confidence Analysis of Standard Deviational Ellipse and Its Extension into Higher Dimensional Euclidean Space

A simple approach to construct confidence bands for a regression function with incomplete data

Noisy recovery from random linear observations: Sharp minimax rates under elliptical constraints

Finite sample posterior concentration in high-dimensional regression

The out-of-sample prediction error of the square-root-LASSO and related estimators