Abstract:We study high-dimensional least-squares regression within a subgaussian statistical learning framework with heterogeneous noise. It includes $s$-sparse and $r$-low-rank least-squares regression when a fraction $\epsilon$ of the labels are adversarially contaminated. We also present a novel theory of trace-regression with matrix decomposition based on a new application of the product process. For these problems, we show novel near-optimal "subgaussian" estimation rates of the form $r(n,d_{e})+\sqrt{\log(1/\delta)/n}+\epsilon\log(1/\epsilon)$, valid with probability at least $1-\delta$. Here, $r(n,d_{e})$ is the optimal uncontaminated rate as a function of the effective dimension $d_{e}$ but independent of the failure probability $\delta$. These rates are valid uniformly on $\delta$, i.e., the estimators' tuning do not depend on $\delta$. Lastly, we consider noisy robust matrix completion with non-uniform sampling. If only the low-rank matrix is of interest, we present a novel near-optimal rate that is independent of the corruption level $a$. Our estimators are tractable and based on a new "sorted" Huber-type loss. No information on $(s,r,\epsilon,a)$ are needed to tune these estimators. Our analysis makes use of novel $\delta$-optimal concentration inequalities for the multiplier and product processes which could be useful elsewhere. For instance, they imply novel sharp oracle inequalities for Lasso and Slope with optimal dependence on $\delta$. Numerical simulations confirm our theoretical predictions. In particular, "sorted" Huber regression can outperform classical Huber regression.

Multivariate reduced rank regression by signal subspace matching

Multivariate Spatiotemporal Models with Low Rank Coefficient Matrix

Robust reduced-rank regression

Reduced Rank Multivariate Kernel Ridge Regression

Adaptive estimation of the rank of the coefficient matrix in high dimensional multivariate response regression models

Bayesian sparse multiple regression for simultaneous rank reduction and variable selection

Reduced-Rank Estimation for Ill-Conditioned Stochastic Linear Model with High Signal-to-Noise Ratio

Uncertainty Quantification in Bayesian Reduced-Rank Sparse Regressions

Minimax rank estimation for subspace tracking

Rank-Constrained Least-Squares: Prediction and Inference

Unified Low-Rank Matrix Estimate Via Penalized Matrix Least Squares Approximation

Estimation of high-dimensional low-rank matrices

Spectral Algorithm for Low-rank Multitask Regression

A framework of regularized low-rank matrix models for regression and classification

Rank-Constrained Matrix Matching and Least Square Problems

Sparse Reduced Rank Huber Regression in High Dimensions

The statistical rate for support matrix machines under low rankness and row (column) sparsity

Robust Regularized Low-Rank Matrix Models for Regression and Classification

A Random Matrix Approach to Low-Multilinear-Rank Tensor Approximation

Low Rank Spectral Regression Via Matrix Factorization for Efficient Subspace Learning

Outlier-robust sparse/low-rank least-squares regression and robust matrix completion