Abstract:Nonlinear regression problem is one of the most popular and important statistical tasks. The first methods like least squares estimation go back to Gauss and Legendre. Recent models and developments in statistics and machine learning like Deep Neuronal Networks or Bayesian methods for nonlinear PDE stimulate new research in this direction which has to address the important issues and challenges of modern statistical inference such as huge complexity and parameter dimension of the model, limited samples size, lack of convexity and identifiability among many others. Classical results of nonparametric statistics in terms of rate of convergence fail to explain the mentioned issues because of the curse of dimensionality problem. This note offers a general approach to studying a nonlinear regression problem which enables one to derive finite sample expansions for the loss of the penalized maximum likelihood estimation (pMLE) with explicit error guarantees and obtain sharp loss and risk bounds. An important step of the study called calming allows to make the objective function stochastically linear by extending the parameter space and to reduce the original problem to semiparametric estimation with a special stochastically linear structure. Such models are studied in this paper in the full generality, the results provide finite sample expansions and risk bounds for the full and target parameters. In all results, the remainder is given explicitly and can be evaluated in terms of the effective sample size and effective parameter dimension which allows us to identify the so-called critical parameter dimension. The results are also dimension and coordinate-free. Despite generality, all the presented bounds are nearly sharp and the classical asymptotic results can be obtained as simple corollaries. The obtained general results are specified to nonlinear smooth regression.

Proving Nonsingularity of Coefficient Matrix in Least Squares Normal Equation of Non-Linear Semiparametric Model

The Quadratic Inference Functions in Measurement Error Model for Longitudinal Data

Varying Coefficient Partially Nonlinear Models with Nonstationary Regressors

Nonlinear Robust Modeling Base on Least Trimmed Squares Regression

Generalized nonlinear models and variance function estimation

Robust estimation in a nonlinear cointegration model

The Convergence of Least-Squares Progressive Iterative Approximation for Singular Least-Squares Fitting System.

Asymptotic Theory of Least Distances Estimate in Multivariate Linear Models

On Nonsingularity Verification of Uncertain Matrices over a Quadratically Constrained Set

The Normality Testing of Quasi-Linear Regression

A Fast Scale-Invariant Algorithm for Non-negative Least Squares with Non-negative Data

Insights Into Algorithms for Separable Nonlinear Least Squares Problems

Low-rank matrix estimation via nonconvex spectral regularized methods in errors-in-variables matrix regression

Finite sample inference in nonlinear regression estimation

Strong convergence rates of several estimators in semiparametric varying-coefficient partially linear models

Nonlinear Regression Analysis

Efficient Algorithms for Positive Semi-Definite Total Least Squares Problems, Minimum Rank Problem and Correlation Matrix Computation

A Semiparametric Approach for Modeling Partially Linear Autoregressive Model with Skew Normal Innovations

Efficient Parameters Estimation Method for the Separable Nonlinear Least Squares Problem

Estimating linear covariance models with numerical nonlinear algebra

No penalty no tears: Least squares in high-dimensional linear models