Abstract:Nonlinear regression problem is one of the most popular and important statistical tasks. The first methods like least squares estimation go back to Gauss and Legendre. Recent models and developments in statistics and machine learning like Deep Neuronal Networks or Bayesian methods for nonlinear PDE stimulate new research in this direction which has to address the important issues and challenges of modern statistical inference such as huge complexity and parameter dimension of the model, limited samples size, lack of convexity and identifiability among many others. Classical results of nonparametric statistics in terms of rate of convergence fail to explain the mentioned issues because of the curse of dimensionality problem. This note offers a general approach to studying a nonlinear regression problem which enables one to derive finite sample expansions for the loss of the penalized maximum likelihood estimation (pMLE) with explicit error guarantees and obtain sharp loss and risk bounds. An important step of the study called calming allows to make the objective function stochastically linear by extending the parameter space and to reduce the original problem to semiparametric estimation with a special stochastically linear structure. Such models are studied in this paper in the full generality, the results provide finite sample expansions and risk bounds for the full and target parameters. In all results, the remainder is given explicitly and can be evaluated in terms of the effective sample size and effective parameter dimension which allows us to identify the so-called critical parameter dimension. The results are also dimension and coordinate-free. Despite generality, all the presented bounds are nearly sharp and the classical asymptotic results can be obtained as simple corollaries. The obtained general results are specified to nonlinear smooth regression.

Estimation Sample Complexity of a Class of Nonlinear Continuous-time Systems

Adaptive Estimation Of Time-Varying Parameters In Linear Systems

Adaptive Estimation of Time-Varying Parameters in Linearly Parametrized Systems

Parameter estimation for nonlinear time-delay systems with noisy output measurements

State Estimation For Nonlinear Systems By Using Dynamic Optimization

State-Delay Estimation for Nonlinear Systems Using Inexact Output Data

Adaptive Modular Control for a Class of Nonlinear Systems with Unknown Time-Varying Parameters

Identifiability Implies Robust, Globally Exponentially Convergent On-line Parameter Estimation: Application to Model Reference Adaptive Control

Sample Complexity Lower Bounds for Linear System Identification

On Parameter Convergence of Nonlinearly Parameterized Adaptive Systems: Analysis Via Contraction and First Lyapunov's Methods

H-Infinity State Estimation For Discrete-Time Chaotic Systems Based On A Unified Model

State Estimation for Continuous-Discrete-Time Nonlinear Stochastic Systems

On-Line Parameter Estimation For A Class Of Time-Varying Continuous Systems With Bounded Disturbances

Non-linear estimation is easy

Stochastic Approximation for Nonlinear Discrete Stochastic Control: Finite-Sample Bounds

Finite sample inference in nonlinear regression estimation

State Estimation for a Class of Fractional-Order Linear Systems by the Generalized Modulating Functions Method.

Dynamic Learning from Neural Network‐based Control for Sampled‐data Strict‐feedback Nonlinear Systems

Adaptive Parameter Estimation under Finite Excitation

Fast and robust state estimator for a kind of nonlinear systems: one method from two perspectives

Numerical Method for Parameter Inference of Systems of Nonlinear Ordinary Differential Equations with Partial Observations.