Abstract:Nonlinear regression problem is one of the most popular and important statistical tasks. The first methods like least squares estimation go back to Gauss and Legendre. Recent models and developments in statistics and machine learning like Deep Neuronal Networks or Bayesian methods for nonlinear PDE stimulate new research in this direction which has to address the important issues and challenges of modern statistical inference such as huge complexity and parameter dimension of the model, limited samples size, lack of convexity and identifiability among many others. Classical results of nonparametric statistics in terms of rate of convergence fail to explain the mentioned issues because of the curse of dimensionality problem. This note offers a general approach to studying a nonlinear regression problem which enables one to derive finite sample expansions for the loss of the penalized maximum likelihood estimation (pMLE) with explicit error guarantees and obtain sharp loss and risk bounds. An important step of the study called calming allows to make the objective function stochastically linear by extending the parameter space and to reduce the original problem to semiparametric estimation with a special stochastically linear structure. Such models are studied in this paper in the full generality, the results provide finite sample expansions and risk bounds for the full and target parameters. In all results, the remainder is given explicitly and can be evaluated in terms of the effective sample size and effective parameter dimension which allows us to identify the so-called critical parameter dimension. The results are also dimension and coordinate-free. Despite generality, all the presented bounds are nearly sharp and the classical asymptotic results can be obtained as simple corollaries. The obtained general results are specified to nonlinear smooth regression.

Inverse Sampling of Degenerate Datasets from a Linear Regression Line

Linear regression with unmatched data: a deconvolution perspective

Optimal Subsampling Approaches for Large Sample Linear Regression

Removing the influence of group variables in high-dimensional predictive modelling

Accelerating Stochastic Gradient Descent Using Antithetic Sampling.

Information-Based Optimal Subdata Selection for Big Data Linear Regression

Models as Approximations I: Consequences Illustrated with Linear Regression

Regression from Dependent Observations

An algebraic-geometric approach for linear regression without correspondences

Subsampling for Big Data Linear Models with Measurement Errors

Optimal Subsampling Algorithms for Big Data Regressions

Inverse Modeling: A Strategy to Cope with Non-linearity

An iterative approach for deriving and solving an accurate regression equation

Finite sample inference in nonlinear regression estimation

Orthogonal Subsampling for Big Data Linear Regression

Stable Neural Stochastic Differential Equations in Analyzing Irregular Time Series Data

Multiple multi-sample testing under arbitrary covariance dependency

Adaptive Sampling Strategies to Construct Equitable Training Datasets

One Step to Efficient Synthetic Data

Adaptive Sampling for Deep Learning via Efficient Nonparametric Proxies

Out-of-Sample Extrapolation with Neuron Editing