Abstract:Nonlinear regression problem is one of the most popular and important statistical tasks. The first methods like least squares estimation go back to Gauss and Legendre. Recent models and developments in statistics and machine learning like Deep Neuronal Networks or Bayesian methods for nonlinear PDE stimulate new research in this direction which has to address the important issues and challenges of modern statistical inference such as huge complexity and parameter dimension of the model, limited samples size, lack of convexity and identifiability among many others. Classical results of nonparametric statistics in terms of rate of convergence fail to explain the mentioned issues because of the curse of dimensionality problem. This note offers a general approach to studying a nonlinear regression problem which enables one to derive finite sample expansions for the loss of the penalized maximum likelihood estimation (pMLE) with explicit error guarantees and obtain sharp loss and risk bounds. An important step of the study called calming allows to make the objective function stochastically linear by extending the parameter space and to reduce the original problem to semiparametric estimation with a special stochastically linear structure. Such models are studied in this paper in the full generality, the results provide finite sample expansions and risk bounds for the full and target parameters. In all results, the remainder is given explicitly and can be evaluated in terms of the effective sample size and effective parameter dimension which allows us to identify the so-called critical parameter dimension. The results are also dimension and coordinate-free. Despite generality, all the presented bounds are nearly sharp and the classical asymptotic results can be obtained as simple corollaries. The obtained general results are specified to nonlinear smooth regression.

Sparse nonlinear regression: Parameter estimation under nonconvexity

Sparse Nonlinear Regression: Parameter Estimation and Asymptotic Inference

Sparse deep neural networks for nonparametric estimation in high-dimensional sparse regression

Linear Programming-Based Sparse Kernel Regression with L1-Norm Minimization for Nonlinear System Modeling

Optimal computational and statistical rates of convergence for sparse nonconvex learning problems

Estimation of Sparse Covariance Matrix Via Non-Convex Regularization

Sparse Estimation Via ℓ_q Optimization Method in High-Dimensional Linear Regression

Optimal Linear Estimation Under Unknown Nonlinear Transform

Nonconvex Sparse Logistic Regression with Weakly Convex Regularization

Finite sample inference in nonlinear regression estimation

Robust Sparse Recovery Via Non-Convex Optimization

Sparse Linear Regression with Beta Process Priors

Estimation of Linear Functionals in High-Dimensional Linear Models: From Sparsity to Nonsparsity

Parameter Estimation of Linear Stochastic Differential Equations with Sparse Observations

Nonconvex Sparse Regularization for Deep Neural Networks and Its Optimality

High-dimensional Inference Via Lipschitz Sparsity-Yielding Regularizers.

Functional Nonlinear Sparse Models

On Sparse Estimation for Semiparametric Linear Transformation Models

Sparse estimation of linear model via Bayesian method

Sparse Recovery Via Nonconvex Regularized M-estimators over ℓ_q-Balls