Abstract:Several statistical models for regression of a function $F$ on $\mathbb{R}^d$ without the statistical and computational curse of dimensionality exist, for example by imposing and exploiting geometric assumptions on the distribution of the data (e.g. that its support is low-dimensional), or strong smoothness assumptions on $F$, or a special structure $F$. Among the latter, compositional models assume $F=f\circ g$ with $g$ mapping to $\mathbb{R}^r$ with $r\ll d$, have been studied, and include classical single- and multi-index models and recent works on neural networks. While the case where $g$ is linear is rather well-understood, much less is known when $g$ is nonlinear, and in particular for which $g$'s the curse of dimensionality in estimating $F$, or both $f$ and $g$, may be circumvented. In this paper, we consider a model $F(X):=f(\Pi_\gamma X) $ where $\Pi_\gamma:\mathbb{R}^d\to[0,\rm{len}_\gamma]$ is the closest-point projection onto the parameter of a regular curve $\gamma: [0,\rm{len}_\gamma]\to\mathbb{R}^d$ and $f:[0,\rm{len}_\gamma]\to\mathbb{R}^1$. The input data $X$ is not low-dimensional, far from $\gamma$, conditioned on $\Pi_\gamma(X)$ being well-defined. The distribution of the data, $\gamma$ and $f$ are unknown. This model is a natural nonlinear generalization of the single-index model, which corresponds to $\gamma$ being a line. We propose a nonparametric estimator, based on conditional regression, and show that under suitable assumptions, the strongest of which being that $f$ is coarsely monotone, it can achieve the $one$-$dimensional$ optimal min-max rate for non-parametric regression, up to the level of noise in the observations, and be constructed in time $\mathcal{O}(d^2n\log n)$. All the constants in the learning bounds, in the minimal number of samples required for our bounds to hold, and in the computational complexity are at most low-order polynomials in $d$.

Effective Minkowski Dimension of Deep Nonparametric Regression: Function Approximation and Statistical Theories

Deep Nonparametric Regression on Approximate Manifolds: Non-Asymptotic Error Bounds with Polynomial Prefactors

Adaptive Bayesian Regression on Data with Low Intrinsic Dimensionality

High-Dimensional Analysis for Generalized Nonlinear Regression: from Asymptotics to Algorithm

Deep Neural Networks for Nonparametric Interaction Models with Diverging Dimension

Deep Nonlinear Sufficient Dimension Reduction

A Statistical Analysis for Supervised Deep Learning with Exponential Families for Intrinsically Low-dimensional Data

Deep Fréchet Regression

Deep Learning meets Nonparametric Regression: Are Weight-Decayed DNNs Locally Adaptive?

Sparse deep neural networks for nonparametric estimation in high-dimensional sparse regression

Robust Nonparametric Regression with Deep Neural Networks

Sub-optimality of the Naive Mean Field approximation for proportional high-dimensional Linear Regression

Low dimensional approximation and generalization of multivariate functions on smooth manifolds using deep ReLU neural networks

Moderate-Dimensional Inferences on Quadratic Functionals in Ordinary Least Squares

High-Dimensional Linear Regression via Implicit Regularization

Multiscale regression on unknown manifolds

Nonparametric regression using deep neural networks with ReLU activation function

Nonparametric Estimation via Partial Derivatives

Conditional regression for the Nonlinear Single-Variable Model

High-dimensional analysis of double descent for linear regression with random projections