Abstract:Variable selection is fundamental to high-dimensional statistical modeling, including nonparametric regression. Many approaches in use are stepwise selection procedures, which can be computationally expensive and ignore stochastic errors in the variable selection process. In this article, penalized likelihood approaches are proposed to handle these kinds of problems. The proposed methods select variables and estimate coefficients simultaneously. Hence they enable us to construct confidence intervals for estimated parameters. The proposed approaches are distinguished from others in that the penalty functions are symmetric, nonconcave on (0, ∞), and have singularities at the origin to produce sparse solutions. Furthermore, the penalty functions should be bounded by a constant to reduce bias and satisfy certain conditions to yield continuous solutions. A new algorithm is proposed for optimizing penalized likelihood functions. The proposed ideas are widely applicable. They are readily applied to a variety of parametric models such as generalized linear models and robust regression models. They can also be applied easily to nonparametric modeling by using wavelets and splines. Rates of convergence of the proposed penalized likelihood estimators are established. Furthermore, with proper choice of regularization parameters, we show that the proposed estimators perform as well as the oracle procedure in variable selection; namely, they work as well as if the correct submodel were known. Our simulation shows that the newly proposed methods compare favorably with other variable selection techniques. Furthermore, the standard error formulas are tested to be accurate enough for practical applications.

A few theoretical results for Laplace and arctan penalized ordinary least squares linear regression estimators

Adaptive Lasso in high-dimensional settings

Nonconvex Penalization Using Laplace Exponents and Concave Conjugates.

A new penalized least absolute deviation model for high dimensional sparse linear regression and an efficient sequential linear programming algorithm

Sparse Estimation Via Lower-Order Penalty Optimization Methods in High-Dimensional Linear Regression.

Weighted Lasso Estimates for Sparse Logistic Regression: Non-Asymptotic Properties with Measurement Errors

Calibrating nonconvex penalized regression in ultra-high dimension

High-dimensional generalized linear models and the lasso

A Note on Adaptive Lp Regularization

Two Stage Non-penalized Corrected Least Squares for High Dimensional Linear Models with Measurement error or Missing Covariates

The Influence Function of Penalized Regression Estimators

The Trimmed Lasso: Sparsity and Robustness

Mean and variance estimation in high-dimensional heteroscedastic models with non-convex penalties

Non-Concave Penalization in Linear Mixed-Effects Models and Regularized Selection of Fixed Effects

Variable Selection via Nonconcave Penalized Likelihood and its Oracle Properties

Theoretical Analysis of Leave-one-out Cross Validation for Non-differentiable Penalties under High-dimensional Settings

Penalty parameter selection and asymmetry corrections to Laplace approximations in Bayesian P-splines models

A unifying theory of exactness of linear penalty functions II: parametric penalty functions

Proximal Iteration for Nonlinear Adaptive Lasso

P-splines with an L1 Penalty for Repeated Measures

An RKHS-based approach to double-penalized regression in high-dimensional partially linear models