Abstract:We propose MC+, a fast, continuous, nearly unbiased and accurate method of penalized variable selection in high-dimensional linear regression. The LASSO is fast and continuous, but biased. The bias of the LASSO may prevent consistent variable selection. Subset selection is unbiased but computationally costly. The MC+ has two elements: a minimax concave penalty (MCP) and a penalized linear unbiased selection (PLUS) algorithm. The MCP provides the convexity of the penalized loss in sparse regions to the greatest extent given certain thresholds for variable selection and unbiasedness. The PLUS computes multiple exact local minimizers of a possibly nonconvex penalized loss function in a certain main branch of the graph of critical points of the penalized loss. Its output is a continuous piecewise linear path encompassing from the origin for infinite penalty to a least squares solution for zero penalty. We prove that at a universal penalty level, the MC+ has high probability of matching the signs of the unknowns, and thus correct selection, without assuming the strong irrepresentable condition required by the LASSO. This selection consistency applies to the case of $p\gg n$, and is proved to hold for exactly the MC+ solution among possibly many local minimizers. We prove that the MC+ attains certain minimax convergence rates in probability for the estimation of regression coefficients in $\ell_r$ balls. We use the SURE method to derive degrees of freedom and $C_p$-type risk estimates for general penalized LSE, including the LASSO and MC+ estimators, and prove their unbiasedness. Based on the estimated degrees of freedom, we propose an estimator of the noise level for proper choice of the penalty level.

A globally convergent algorithm for lasso-penalized mixture of linear regression models

On Mixture Regression Shrinkage and Selection via the MR-LASSO

Nearly unbiased variable selection under minimax concave penalty

Model Selection for Exponential Power Mixture Regression Models

Simultaneous Estimation and Variable Selection in Median Regression Using Lasso-type Penalty

Statistical analysis for a penalized EM algorithm in high-dimensional mixture linear regression model

Convergence of Online Learning Algorithm for a Mixture of Multiple Linear Regressions

Proximal Iteration for Nonlinear Adaptive Lasso

New Robust Variable Selection Methods for Linear Regression Models

Outlier detection and robust variable selection via the penalized weighted LAD-LASSO method

Variable Selection for Linear Mixed Effects Model Via Penalization Approaches

Non-Concave Penalization in Linear Mixed-Effects Models and Regularized Selection of Fixed Effects

Variable Selection via Nonconcave Penalized Likelihood and its Oracle Properties

NONCONCAVE PENALIZED M-ESTIMATION WITH A DIVERGING NUMBER OF PARAMETERS

A Fast Solution to the Lasso Problem with Equality Constraints

The linearized alternating direction method of multipliers for low-rank and fused LASSO matrix regression model

Least Squares Approximation with a Diverging Number of Parameters

Variable Selection Using Penalized Empirical Likelihood

Fast selection of nonlinear mixed effect models using penalized likelihood

Stability of a Generalized Debiased Lasso with Applications to Resampling-Based Variable Selection

Combined-Penalized Likelihood Estimations with A Diverging Number of Parameters