Abstract:A central issue of many statistical learning problems is to select an appropriate model from a set of candidate models. Large models tend to inflate the variance (or overfitting), while small models tend to cause biases (or underfitting) for a given fixed dataset. In this work, we address the critical challenge of model selection to strike a balance between model fitting and model complexity, thus gaining reliable predictive power. We consider the task of approaching the theoretical limit of statistical learning, meaning that the selected model has the predictive performance that is as good as the best possible model given a class of potentially misspecified candidate models. We propose a generalized notion of Takeuchi’s information criterion and prove that the proposed method can asymptotically achieve the optimal out-sample prediction loss under reasonable assumptions. It is the first proof of the asymptotic property of Takeuchi’s information criterion to our best knowledge. Our proof applies to a wide variety of nonlinear models, loss functions, and high dimensionality (in the sense that the models’ complexity can grow with sample size). The proposed method can be used as a computationally efficient surrogate for leave-one-out cross-validation. Moreover, for modeling streaming data, we propose an online algorithm that sequentially expands the model complexity to enhance selection stability and reduce computation cost. Experimental studies show that the proposed method has desirable predictive power and significantly less computational cost than some popular methods.

On Model Selection Consistency of the Elastic Net when P >> N

On Model Selection Consistency of Lasso for High-Dimensional Ising Models

Ising Model Selection Using ℓ1ℓ1\ell_{1}-Regularized Linear Regression: A Statistical Mechanics Analysis

Ising Model Selection Using ℓ_1-Regularized Linear Regression: A Statistical Mechanics Analysis

Ising Model Selection Using 𝓁-Regularized Linear Regression.

On Estimation Error Bounds of the Elastic Net when P ≫ N

Nonnegative Elastic Net and application in index tracking

Model Order Selection In Errors-In-Variables System Identification

Asymptotic Behavior for S-estimators in Random Design Linear Model with Long-Range-dependent Errors

Adaptive Lasso in high-dimensional settings

Elastic Net Procedure for Partially Linear Models

The Lasso under Poisson-Like Heteroscedasticity

Variable selection and model prediction based on Lasso, adaptive lasso and elastic net

Asymptotics of the Adaptive Elastic Net Estimation for Conditional Heteroscedastic Time Series Models

Iterative Reweighted Framework Based Algorithms for Sparse Linear Regression with Generalized Elastic Net Penalty

Robust Elastic Net Regression

On Statistical Efficiency in Learning

Model selection via standard error adjusted adaptive lasso

The Lasso under Heteroscedasticity

Non-asymptotic model selection for models of network data with parameter vectors of increasing dimension

Variable selection for uncertain regression models based on elastic net method