Abstract:Nonlinear regression problem is one of the most popular and important statistical tasks. The first methods like least squares estimation go back to Gauss and Legendre. Recent models and developments in statistics and machine learning like Deep Neuronal Networks or Bayesian methods for nonlinear PDE stimulate new research in this direction which has to address the important issues and challenges of modern statistical inference such as huge complexity and parameter dimension of the model, limited samples size, lack of convexity and identifiability among many others. Classical results of nonparametric statistics in terms of rate of convergence fail to explain the mentioned issues because of the curse of dimensionality problem. This note offers a general approach to studying a nonlinear regression problem which enables one to derive finite sample expansions for the loss of the penalized maximum likelihood estimation (pMLE) with explicit error guarantees and obtain sharp loss and risk bounds. An important step of the study called calming allows to make the objective function stochastically linear by extending the parameter space and to reduce the original problem to semiparametric estimation with a special stochastically linear structure. Such models are studied in this paper in the full generality, the results provide finite sample expansions and risk bounds for the full and target parameters. In all results, the remainder is given explicitly and can be evaluated in terms of the effective sample size and effective parameter dimension which allows us to identify the so-called critical parameter dimension. The results are also dimension and coordinate-free. Despite generality, all the presented bounds are nearly sharp and the classical asymptotic results can be obtained as simple corollaries. The obtained general results are specified to nonlinear smooth regression.

Least Squares Approximations in Linear Statistical Inverse Learning Problems

Statistical inverse learning problems with random observations

Linear Convergence of Inexact Descent Method and Inexact Proximal Gradient Algorithms for Lower-Order Regularization Problems

Learning linear operators: Infinite-dimensional regression as a well-behaved non-compact inverse problem

Least Squares Rational Approximation

Optimal sampling for least squares approximation with general dictionaries

Randomized Matrix Approximation to Enhance Regularized Projection Schemes in Inverse Problems

Least squares formulation for ill-posed inverse problems and applications

Non-intrusive Least-Squares Functional A Posteriori Error Estimator: Linear and Nonlinear Problems with Plain Convergence

Non-asymptotic error estimates for the Laplace approximation in Bayesian inverse problems

Analysis of Least Squares Regularized Regression in Reproducing Kernel Krein Spaces

Finite sample inference in nonlinear regression estimation

Optimal low-rank approximations of posteriors for linear Gaussian inverse problems on Hilbert spaces

Learning Theory Viewpoint of Approximation by Positive Linear Operators

Learning-Based Low-Rank Approximations

Randomized least-squares with minimal oversampling and interpolation in general spaces

Fast Sparse Least-Squares Regression with Non-Asymptotic Guarantees

Implicit Regularization in Nonconvex Statistical Estimation: Gradient Descent Converges Linearly for Phase Retrieval, Matrix Completion, and Blind Deconvolution

Learned reconstruction methods for inverse problems: sample error estimates

Linearly convergent adjoint free solution of least squares problems by random descent

Local convergence rates of the nonparametric least squares estimator with applications to transfer learning