Minimax rates of l p-losses for high-dimensional linear regression models with additive measurement errors over l q-balls

Xin Li,Dongya Wu

2019-01-01

Abstract:We study minimax rates for high-dimensional linear regression with additive errors under the lp (1 ≤ p < ∞)-losses, where the regression parameter is of weak sparsity. Our lower and upper bounds agree up to constant factors, implying that the proposed estimator is minimax optimal.

What problem does this paper attempt to address?

Sparse Estimation Via ℓ_q Optimization Method in High-Dimensional Linear Regression

Xin Li,Yaohua Hu,Chong Li,Xiaoqi Yang,Tianzi Jiang

2019-01-01

Abstract:In this paper, we discuss the statistical properties of the ℓ_qoptimization methods (0<q≤ 1), including the ℓ_q minimization methodand the ℓ_q regularization method, for estimating a sparse parameter fromnoisy observations in high-dimensional linear regression with either adeterministic or random design. For this purpose, we introduce a generalq-restricted eigenvalue condition (REC) and provide its sufficient conditionsin terms of several widely-used regularity conditions such as sparse eigenvaluecondition, restricted isometry property, and mutual incoherence property. Byvirtue of the q-REC, we exhibit the stable recovery property of the ℓ_qoptimization methods for either deterministic or random designs by showing thatthe ℓ_2 recovery bound O(ϵ^2) for the ℓ_q minimizationmethod and the oracle inequality and ℓ_2 recovery boundO(λ^2/2-qs) for the ℓ_q regularization method holdrespectively with high probability. The results in this paper are nonasymptoticand only assume the weak q-REC. The preliminary numerical results verify theestablished statistical property and demonstrate the advantages of the ℓ_qregularization method over some existing sparse optimization methods.
Optimal RIP Bounds for Sparse Signals Recovery Via ℓ Minimization

Rui Zhang,Song Li

DOI: https://doi.org/10.1016/j.acha.2017.10.004

IF: 2.974

2019-01-01

Applied and Computational Harmonic Analysis

Abstract:In this paper, we present a unified analysis of RIP bounds for sparse signals recovery by using ℓp minimization with 00, δ2k<δ(p)+ϵ is not sufficient to guarantee the exact recovery of all k-sparse signals. We also apply the results to the cases of low rank matrix recovery and the reconstruction of sparse vectors in terms of redundant dictionary. In particular, when p=1, the corresponding constant δ(1)=22, this sharp bound was shown by T. Cai and A. Zhang in their work. Thus, we give a complete characterization for optimal RIP bounds δ2k via ℓp minimization for k-sparse signals recovery with 0
Minimax Rates for High-dimensional Double Sparse Structure over $\ell_q$-balls

Zhifan Li,Yanhang Zhang,Jianxin Yin

DOI: https://doi.org/10.48550/arxiv.2207.11888

2022-01-01

Abstract: In this paper, we focus on the high-dimensional double sparse structure, where the parameter of interest simultaneously encourages group-wise sparsity and element-wise sparsity in each group. Combining Gilbert-Varshamov bound and its variants, we develop a novel lower bound technique for the metric entropy of the parameter space, which is well suited for the double sparse structure over $\ell_q$-balls for $q \in [0,1]$. We prove the lower bounds on estimation error in an information theoretical manner, which is based on our proposed lower bound technique and Fano's inequality. The matching upper bounds are also established, whose proof follows from a direct analysis of the constrained least-squares estimators and results on empirical processes. Moreover, we extend the results over $\ell_q$-balls into the double sparse regression model and establish its minimax rate on the estimation error. Finally, we develop the DSIHT (Double Sparse Iterative Hard Thresholding) algorithm and show its optimality in the minimax sense for solving the double sparse linear regression.
Accuracy Assessment for High-dimensional Linear Regression

T. Tony Cai,Zijian Guo

DOI: https://doi.org/10.1214/17-AOS1604

2018-01-01

Abstract:This paper considers point and interval estimation of the l(q) loss of an estimator in high-dimensional linear regression with random design. We establish the minimax rate for estimating the l(q) loss and the minimax expected length of confidence intervals for the l(q) loss of rate-optimal estimators of the regression vector, including commonly used estimators such as Lasso, scaled Lasso, square-root Lasso and Dantzig Selector. Adaptivity of confidence intervals for the l(q) loss is also studied. Both the setting of the known identity design covariance matrix and known noise level and the setting of unknown design covariance matrix and unknown noise level are studied. The results reveal interesting and significant differences between estimating the l(2) loss and l(q) loss with 1 <= q < 2 as well as between the two settings. New technical tools are developed to establish rate sharp lower bounds for the minimax estimation error and the expected length of minimax and adaptive confidence intervals for the l(q) loss. A significant difference between loss estimation and the traditional parameter estimation is that for loss estimation the constraint is on the performance of the estimator of the regression vector, but the lower bounds are on the difficulty of estimating its l(q) loss. The technical tools developed in this paper can also be of independent interest.
Sparse Estimation Via Lower-Order Penalty Optimization Methods in High-Dimensional Linear Regression.

Li Xin,Hu Yaohua,Li Chong,Yang Xiaoqi,Jiang Tianzi

DOI: https://doi.org/10.1007/s10898-022-01220-5

2022-01-01

Journal of Global Optimization

Abstract:The lower-order penalty optimization methods, including the ℓ _q minimization method and the ℓ _q regularization method (0<q≤ 1) , have been widely applied to find sparse solutions of linear regression problems and gained successful applications in various mathematics and applied science fields. In this paper, we aim to investigate statistical properties of the ℓ _q penalty optimization methods with randomly noisy observations and a deterministic/random design. For this purpose, we introduce a general q -Restricted Eigenvalue Condition (REC) and provide its sufficient conditions in terms of several widely-used regularity conditions such as sparse eigenvalue condition, restricted isometry property, and mutual incoherence property. By virtue of the q -REC, we exhibit the ℓ _2 recovery bounds of order O(ϵ ^2) and O(λ ^2/2-qs) for the ℓ _q minimization method and the ℓ _q regularization method, respectively, with high probability for either deterministic or random designs. The results in this paper are nonasymptotic and only assume the weak q -REC. The preliminary numerical results verify the established statistical properties and demonstrate the advantages of the ℓ _q penalty optimization methods over existing sparse optimization methods.
Weighted Lasso Estimates for Sparse Logistic Regression: Non-Asymptotic Properties with Measurement Errors

Huang Huamei,Gao Yujing,Zhang Huiming,Li Bo

DOI: https://doi.org/10.1007/s10473-021-0112-6

2020-01-01

Acta Mathematica Scientia

Abstract:For high-dimensional models with a focus on classification performance, the ℓ1-penalized logistic regression is becoming important and popular. However, the Lasso estimates could be problematic when penalties of different coefficients are all the same and not related to the data. We propose two types of weighted Lasso estimates, depending upon covariates determined by the McDiarmid inequality. Given sample size n and a dimension of covariates p, the finite sample behavior of our proposed method with a diverging number of predictors is illustrated by non-asymptotic oracle inequalities such as the ℓ1-estimation error and the squared prediction error of the unknown parameters. We compare the performance of our method with that of former weighted estimates on simulated data, then apply it to do real data analysis.
Restricted -Isometry Properties of Nonconvex Matrix Recovery

Min Zhang,Zheng-Hai Huang,Ying Zhang

DOI: https://doi.org/10.1109/TIT.2013.2250577

IF: 2.5

2013-01-01

IEEE Transactions on Information Theory

Abstract:Recently, a nonconvex relaxation of low-rank matrix recovery (LMR), called the Schatten- $p$ quasi-norm minimization ($0< p< 1$), was introduced instead of the previous nuclear norm minimization in order to approximate the problem of LMR closer. In this paper, we introduce a notion of the restricted $p$-isometry constants ( $0< p\\leq 1$) and derive a $p$ -RIP condition for exact reconstruction of LMR via Schatten-$p$ quasi-norm minimization. In particular, we determine how many random, Gaussian measurements are needed for the $p$-RIP condition to hold with high probability, which gives a theoretical result that it needs fewer measurements with small $p$ for exact recovery via Schatten- $p$ quasi-norm minimization than when $p=1$.
Minimax estimation of linear functionals under squared error loss

Meng Zhao,K. B. Kulasekera

DOI: https://doi.org/10.1016/j.jspi.2009.02.015

IF: 1.095

2009-01-01

Journal of Statistical Planning and Inference

Abstract:Under a very general setting, we consider the problem of estimating a linear functional of an unknown vector in a Hilbert space from indirect data contaminated by noise. We then discuss two situations in detail: estimating the signal function in the fractional Brownian motion model and the regression model with correlated errors. In the fractional Brownian motion model, we observe the process which is the sum of a fractional Brownian motion with Hurst index between (12,1) and a drift function that is determined by the signal function. In the regression model with correlated errors, we assume that the errors have long memory. For both estimation problems, we obtain the asymptotic rate for the minimax affine risks over certain types of parameter spaces. In each case, we also show that the minimax affine risk is bounded by 1.25 times the minimax risk.
Minimax Linear Regression under the Quantile Risk

Ayoub El Hanchi,Chris J. Maddison,Murat A. Erdogdu

2024-06-18

Abstract:We study the problem of designing minimax procedures in linear regression under the quantile risk. We start by considering the realizable setting with independent Gaussian noise, where for any given noise level and distribution of inputs, we obtain the exact minimax quantile risk for a rich family of error functions and establish the minimaxity of OLS. This improves on the known lower bounds for the special case of square error, and provides us with a lower bound on the minimax quantile risk over larger sets of distributions. Under the square error and a fourth moment assumption on the distribution of inputs, we show that this lower bound is tight over a larger class of problems. Specifically, we prove a matching upper bound on the worst-case quantile risk of a variant of the recently proposed min-max regression procedure, thereby establishing its minimaxity, up to absolute constants. We illustrate the usefulness of our approach by extending this result to all $p$-th power error functions for $p \in (2, \infty)$. Along the way, we develop a generic analogue to the classical Bayesian method for lower bounding the minimax risk when working with the quantile risk, as well as a tight characterization of the quantiles of the smallest eigenvalue of the sample covariance matrix.

Statistics Theory,Machine Learning
Sparse Recovery Via Nonconvex Regularized M-estimators over ℓ_q-Balls

Xin Li,Dongya Wu,Chong Li,Jinhua Wang,Jen-Chih Yao

DOI: https://doi.org/10.1016/j.csda.2020.107047

2020-01-01

Abstract:The recovery properties of nonconvex regularized M-estimators are analysed, under the general sparsity assumption on the true parameter. In the statistical aspect, the recovery bound for any stationary point of the nonconvex regularized M-estimator is established under some regularity conditions. In the computational aspect, the proximal gradient method is used to solve the nonconvex optimization problem and is proved to achieve a linear convergence rate, by virtue of a slight decomposition of the objective function. In particular, for commonly-used regularizers such as SCAD and MCP, a simpler decomposition is applicable thanks to the assumption on the regularizer, which helps to construct the estimator with better recovery performance. In the aspect of application, theoretical consequences are obtained on the corrupted error-in-variables linear regression model by verifying the required conditions. Finally, statistical and computational results as well as advantages of the assumptions are demonstrated by several numerical experiments. Simulation results show remarkable consistency with the theory under high-dimensional scaling.
High-dimensional Measurement Error Models for Lipschitz Loss

Xin Ma,Suprateek Kundu

DOI: https://doi.org/10.48550/arXiv.2210.15008

2022-10-27

Abstract:Recently emerging large-scale biomedical data pose exciting opportunities for scientific discoveries. However, the ultrahigh dimensionality and non-negligible measurement errors in the data may create difficulties in estimation. There are limited methods for high-dimensional covariates with measurement error, that usually require knowledge of the noise distribution and focus on linear or generalized linear models. In this work, we develop high-dimensional measurement error models for a class of Lipschitz loss functions that encompasses logistic regression, hinge loss and quantile regression, among others. Our estimator is designed to minimize the $L_1$ norm among all estimators belonging to suitable feasible sets, without requiring any knowledge of the noise distribution. Subsequently, we generalize these estimators to a Lasso analog version that is computationally scalable to higher dimensions. We derive theoretical guarantees in terms of finite sample statistical error bounds and sign consistency, even when the dimensionality increases exponentially with the sample size. Extensive simulation studies demonstrate superior performance compared to existing methods in classification and quantile regression problems. An application to a gender classification task based on brain functional connectivity in the Human Connectome Project data illustrates improved accuracy under our approach, and the ability to reliably identify significant brain connections that drive gender differences.

Methodology,Statistics Theory,Machine Learning
Asymptotic Theory of Least Distances Estimate in Multivariate Linear Models

Z. D. Bai,X.R. Chen,B.Q. Miao,C. Radhakrishna Rao

DOI: https://doi.org/10.1080/02331889008802260

IF: 2.346

1990-01-01

Statistics

Abstract:We consider the multivariate linear model where Yi is a p-vector random variable, Xi is a q X p matrix, βA0 is an unknown q-vector parameter and {si} is a sequence of iid p-vector random variable with median vector zero. The estimate βAn of βA0 such that is called the least distances (LD) estimator. It may be recalled that the least squares (LS) estimator is obtained by minimizing the sum of norm squares. In this paper, it is shown that the LD estimator is unique, consistent and has an asymptotic q-variate normal distribution with mean β0 and co variance matrix V which depends on the distribution of the error vectors {ei}. A consistent estimator of V is proposed which together with βAn enables an asymptotic inference on β0. In particular, tests of linear hypotheses on β0 analogous to those of analysis of variance in the GATJSS-MARKOFF linear model are developed. Explicit expressions are obtained in some cases for the asymptotic relative efficiency of the LD compared to the LS estimator.
Minimax rates of convergence for nonparametric location-scale models

Bingxin Zhao,Yuhong Yang

2023-07-04

Abstract:This paper studies minimax rates of convergence for nonparametric location-scale models, which include mean, quantile and expectile regression settings. Under Hellinger differentiability on the error distribution and other mild conditions, we show that the minimax rate of convergence for estimating the regression function under the squared $L_2$ loss is determined by the metric entropy of the nonparametric function class. Different error distributions, including asymmetric Laplace distribution, asymmetric connected double truncated gamma distribution, connected normal-Laplace distribution, Cauchy distribution and asymmetric normal distribution are studied as examples. Applications on low order interaction models and multiple index models are also given.

Statistics Theory
Wasserstein-based Minimax Estimation of Dependence in Multivariate Regularly Varying Extremes

Xuhui Zhang,Jose Blanchet,Youssef Marzouk,Viet Anh Nguyen,Sven Wang

2023-12-15

Abstract:We study minimax risk bounds for estimators of the spectral measure in multivariate linear factor models, where observations are linear combinations of regularly varying latent factors. Non-asymptotic convergence rates are derived for the multivariate Peak-over-Threshold estimator in terms of the $p$-th order Wasserstein distance, and information-theoretic lower bounds for the minimax risks are established. The convergence rate of the estimator is shown to be minimax optimal under a class of Pareto-type models analogous to the standard class used in the setting of one-dimensional observations known as the Hall-Welsh class. When the estimator is minimax inefficient, a novel two-step estimator is introduced and demonstrated to attain the minimax lower bound. Our analysis bridges the gaps in understanding trade-offs between estimation bias and variance in multivariate extreme value theory.

Statistics Theory,Methodology
Characterizing the minimax rate of nonparametric regression under bounded convex constraints

Akshay Prasadan,Matey Neykov

2024-03-10

Abstract:We quantify the minimax rate for a nonparametric regression model over a convex function class $\mathcal{F}$ with bounded diameter. We obtain a minimax rate of ${\varepsilon^{\ast}}^2\wedge\mathrm{diam}(\mathcal{F})^2$ where \[\varepsilon^{\ast} =\sup\{\varepsilon>0:n\varepsilon^2 \le \log M_{\mathcal{F}}^{\operatorname{loc}}(\varepsilon,c)\},\] where $M_{\mathcal{F}}^{\operatorname{loc}}(\cdot, c)$ is the local metric entropy of $\mathcal{F}$ and our loss function is the squared population $L_2$ distance over our input space $\mathcal{X}$. In contrast to classical works on the topic [cf. Yang and Barron, 1999], our results do not require functions in $\mathcal{F}$ to be uniformly bounded in sup-norm. In addition, we prove that our estimator is adaptive to the true point, and to the best of our knowledge this is the first such estimator in this general setting. This work builds on the Gaussian sequence framework of Neykov [2022] using a similar algorithmic scheme to achieve the minimax rate. Our algorithmic rate also applies with sub-Gaussian noise. We illustrate the utility of this theory with examples including multivariate monotone functions, linear functionals over ellipsoids, and Lipschitz classes.

Statistics Theory
Sparse quantile regression

Le-Yu Chen,Sokbae Lee

DOI: https://doi.org/10.1016/j.jeconom.2023.02.014

IF: 3.363

2023-05-10

Journal of Econometrics

Abstract:We consider both l0 -penalized and l0 -constrained quantile regression estimators. For the l0 -penalized estimator, we derive an exponential inequality on the tail probability of excess quantile prediction risk and apply it to obtain non-asymptotic upper bounds on the mean-square parameter and regression function estimation errors. We also derive analogous results for the l0 -constrained estimator. The resulting rates of convergence are nearly minimax-optimal and the same as those for l1 -penalized and non-convex penalized estimators. Further, we characterize expected Hamming loss for the l0 -penalized estimator. We implement the proposed procedure via mixed integer linear programming and also a more scalable first-order approximation algorithm. We illustrate the finite-sample performance of our approach in Monte Carlo experiments and its usefulness in a real data application concerning conformal prediction of infant birth weights (with n≈103 and up to p>103 ). In sum, our l0 -based method produces a much sparser estimator than the l1 -penalized and non-convex penalized approaches without compromising precision.

economics,social sciences, mathematical methods,mathematics, interdisciplinary applications
A note on the minimax risk of sparse linear regression

Yilin Guo,Shubhangi Ghosh,Haolei Weng,Arian Maleki

2024-05-09

Abstract:Sparse linear regression is one of the classical and extensively studied problems in high-dimensional statistics and compressed sensing. Despite the substantial body of literature dedicated to this problem, the precise determination of its minimax risk remains elusive. This paper aims to fill this gap by deriving asymptotically constant-sharp characterization for the minimax risk of sparse linear regression. More specifically, the paper focuses on scenarios where the sparsity level, denoted as k, satisfies the condition $(k \log p)/n {\to} 0$, with p and n representing the number of features and observations respectively. We establish that the minimax risk under isotropic Gaussian random design is asymptotically equal to $2{\sigma}^2k/n log(p/k)$, where ${\sigma}$ denotes the standard deviation of the noise. In addition to this result, we will summarize the existing results in the literature, and mention some of the fundamental problems that have still remained open.

Statistics Theory
Minimax rate of testing in sparse linear regression

Alexandra Carpentier,Olivier Collier,Laëtitia Comminges,Alexandre B. Tsybakov,Yuhao Wang

DOI: https://doi.org/10.48550/arXiv.1804.06494

2018-10-10

Abstract:We consider the problem of testing the hypothesis that the parameter of linear regression model is 0 against an s-sparse alternative separated from 0 in the l2-distance. We show that, in Gaussian linear regression model with p < n, where p is the dimension of the parameter and n is the sample size, the non-asymptotic minimax rate of testing has the form sqrt((s/n) log(1 + sqrt(p)/s )). We also show that this is the minimax rate of estimation of the l2-norm of the regression parameter.

Statistics Theory
Quasi-minimax Estimation in the General Linear Regression Model

Hu Yang,Litong Wang,Lijuan Song

DOI: https://doi.org/10.1016/j.jspi.2008.09.016

IF: 1.095

2008-01-01

Journal of Statistical Planning and Inference

Abstract:This paper is mainly concerned with minimax estimation in the general linear regression model y=Xβ+ε under ellipsoidal restrictions on the parameter space and quadratic loss function. We confine ourselves to estimators that are linear in the response vector y. The minimax estimators of the regression coefficient β are derived under homogeneous condition and heterogeneous condition, respectively. Furthermore, these obtained estimators are the ridge-type estimators and mean dispersion error (MDE) superior to the best linear unbiased estimator b=(X′W-1X)-1X′W-1y under some conditions.
A minimax optimal approach to high-dimensional double sparse linear regression

Yanhang Zhang,Zhifan Li,Jianxin Yin

2023-05-07

Abstract:In this paper, we focus our attention on the high-dimensional double sparse linear regression, that is, a combination of element-wise and group-wise <a class="link-external link-http" href="http://sparsity.To" rel="external noopener nofollow">this http URL</a> address this problem, we propose an IHT-style (iterative hard thresholding) procedure that dynamically updates the threshold at each step. We establish the matching upper and lower bounds for parameter estimation, showing the optimality of our proposal in the minimax sense. Coupled with a novel sparse group information criterion, we develop a fully adaptive procedure to handle unknown group sparsity and noise levels.We show that our adaptive procedure achieves optimal statistical accuracy with fast convergence. Finally, we demonstrate the superiority of our method by comparing it with several state-of-the-art algorithms on both synthetic and real-world datasets.

Statistics Theory

Minimax rates of l p-losses for high-dimensional linear regression models with additive measurement errors over l q-balls

Sparse Estimation Via ℓ_q Optimization Method in High-Dimensional Linear Regression

Optimal RIP Bounds for Sparse Signals Recovery Via ℓ Minimization

Minimax Rates for High-dimensional Double Sparse Structure over $\ell_q$-balls

Accuracy Assessment for High-dimensional Linear Regression

Sparse Estimation Via Lower-Order Penalty Optimization Methods in High-Dimensional Linear Regression.

Weighted Lasso Estimates for Sparse Logistic Regression: Non-Asymptotic Properties with Measurement Errors

Restricted -Isometry Properties of Nonconvex Matrix Recovery

Minimax estimation of linear functionals under squared error loss

Minimax Linear Regression under the Quantile Risk

Sparse Recovery Via Nonconvex Regularized M-estimators over ℓ_q-Balls

High-dimensional Measurement Error Models for Lipschitz Loss

Asymptotic Theory of Least Distances Estimate in Multivariate Linear Models

Minimax rates of convergence for nonparametric location-scale models

Wasserstein-based Minimax Estimation of Dependence in Multivariate Regularly Varying Extremes

Characterizing the minimax rate of nonparametric regression under bounded convex constraints

Sparse quantile regression

A note on the minimax risk of sparse linear regression

Minimax rate of testing in sparse linear regression

Quasi-minimax Estimation in the General Linear Regression Model

A minimax optimal approach to high-dimensional double sparse linear regression