Abstract:We propose to address the common problem of linear estimation in linear statistical models by using a model selection approach via penalization. Depending then on the framework in which the linear statistical model is considered namely the regression framework or the inverse problem framework, a data-driven model selection criterion is obtained either under general assumptions, or under the mild assumption of model identifiability respectively. The proposed approach was stimulated by the important recent non-asymptotic model selection results due to Birgé and Massart mainly (Birge and Massart 2007), and our results in this paper, like theirs, are non-asymptotic and turn to be sharp. Our main contribution in this paper resides in the fact that these linear estimators are not necessarily least-squares estimators but can be any linear estimators. The proposed approach finds therefore potential applications in countless fields of engineering and applied science (image science, signal processing,applied statistics, coding, to name a few) in which one is interested in recovering some unknown vector quantity of interest as the one, for example, which achieves the best trade-off between a term of fidelity to data, and a term of regularity or/and parsimony of the solution. The proposed approach provides then such applications with an interesting model selection framework that allows them to achieve such a goal.

Invariance of reparametrization in model selection of neural networks

Information Geometry Approach to the Model Selection of Neural Networks

The Relation Between Intrinsic Complexity and Generalization of a Model and the Geometric Curvature

A New Geometric Approach to the Complexity of Model Selection

Ising Model Selection Using ℓ1ℓ1\ell_{1}-Regularized Linear Regression: A Statistical Mechanics Analysis

Ising Model Selection Using ℓ_1-Regularized Linear Regression: A Statistical Mechanics Analysis

A New Model Selection Criterion Based on Kullback-Leibler Information Divergence and Geometric Curvature

Regularization and variable selection for infinite variance autoregressive models

A New Model Selection Criterion Based on Information Geometry

Reparameterization invariance in approximate Bayesian inference

Rethinking Bias-Variance Trade-off for Generalization of Neural Networks

Generalization and Estimation Error Bounds for Model-based Neural Networks

Geometric Inductive Biases of Deep Networks: The Role of Data and Architecture

Non-asymptotic model selection for linear non least-squares estimation in regression models and inverse problems

Curvature-based Comparison of Two Neural Networks

Model Selection Criterion Applied To Visual Perception

Consistent Variable Selection for a Nonparametric Nonlinear System by Inverse and Contour Regressions.

On the Benefits of Invariance in Neural Networks

On the role of parametrization in models with a misspecified nuisance component

A General Framework of the Consistency for Large Neural Networks

'Deep' Dive into $b \to c$ Anomalies: Standardized and Future-proof Model Selection Using Self-normalizing Neural Networks