Abstract:The Lasso is biased. Concave penalized least squares estimation (PLSE) takes advantage of signal strength to reduce this bias, leading to sharper error bounds in prediction, coefficient estimation and variable selection. For prediction and estimation, the bias of the Lasso can be also reduced by taking a smaller penalty level than what selection consistency requires, but such smaller penalty level depends on the sparsity of the true coefficient vector. The sorted L1 penalized estimation (Slope) was proposed for adaptation to such smaller penalty levels. However, the advantages of concave PLSE and Slope do not subsume each other. We propose sorted concave penalized estimation to combine the advantages of concave and sorted penalizations. We prove that sorted concave penalties adaptively choose the smaller penalty level and at the same time benefits from signal strength, especially when a significant proportion of signals are stronger than the corresponding adaptively selected penalty levels. A local convex approximation, which extends the local linear and quadratic approximations to sorted concave penalties, is developed to facilitate the computation of sorted concave PLSE and proven to possess desired prediction and estimation error bounds. We carry out a unified treatment of penalty functions in a general optimization setting, including the penalty levels and concavity of the above mentioned sorted penalties and mixed penalties motivated by Bayesian considerations. Our analysis of prediction and estimation errors requires the restricted eigenvalue condition on the design, not beyond, and provides selection consistency under a required minimum signal strength condition in addition. Thus, our results also sharpens existing results on concave PLSE by removing the upper sparse eigenvalue component of the sparse Riesz condition.

Learning with coefficient-based regularization and ℓ 1 −penalty

Concentration Estimates for Learning with Unbounded Sampling

Learning Theory Estimates for Coefficient-Based Regularized Regression

Sparse Estimation Via Lower-Order Penalty Optimization Methods in High-Dimensional Linear Regression.

Kernel Regression with Coefficient-based l(q)-regularization

Concentration Estimates for Learning with ℓ1-Regularizer and Data Dependent Hypothesis Spaces

Learning Rates of L^q Coefficient Regularization Learning with Gaussian Kernel

Optimal Rates for Coefficient-Based Regularized Regression

Sparse Kernel Regression with Coefficient-based $\Ell_q-$Regularization.

Learning Rates of Least-Square Regularized Regression with Polynomial Kernels

Tikhonov Regularization with L^0-Term Complementing a Convex Penalty: L^1 Convergence under Sparsity Constraints

Sparse Estimation Via ℓ_q Optimization Method in High-Dimensional Linear Regression

Online Efficient Learning with Quantized Klms and L-1 Regularization

Compound Poisson Processes, Latent Shrinkage Priors and Bayesian Nonconvex Penalization

On optimal solutions of the constrained $\ell_0$ regularization and its penalty problem

Quantile Regression With L(1)-Regularization And Gaussian Kernels

Least Squares Approximation with a Diverging Number of Parameters

Online Efficient Learning with Quantized KLMS and L<inf>1</inf> Regularization

Sorted Concave Penalized Regression

Coefficient-based regularized distribution regression

Learning rates for regularized least squares ranking algorithm