Abstract: In this paper, we study the properties of robust nonparametric estimation using deep neural networks for regression models with heavy tailed error distributions. We establish the non-asymptotic error bounds for a class of robust nonparametric regression estimators using deep neural networks with ReLU activation under suitable smoothness conditions on the regression function and mild conditions on the error term. In particular, we only assume that the error distribution has a finite p-th moment with p greater than one. We also show that the deep robust regression estimators are able to circumvent the curse of dimensionality when the distribution of the predictor is supported on an approximate lower-dimensional set. An important feature of our error bound is that, for ReLU neural networks with network width and network size (number of parameters) no more than the order of the square of the dimensionality d of the predictor, our excess risk bounds depend sub-linearly on d. Our assumption relaxes the exact manifold support assumption, which could be restrictive and unrealistic in practice. We also relax several crucial assumptions on the data distribution, the target regression function and the neural networks required in the recent literature. Our simulation studies demonstrate that the robust methods can significantly outperform the least squares method when the errors have heavy-tailed distributions and illustrate that the choice of loss function is important in the context of deep nonparametric regression.

Statistical learning by sparse deep neural networks

Sparse Bayesian Approach to Fast Learning Network for Multiclassification.

Sparse-penalized deep neural networks estimator under weak dependence

Nonconvex Sparse Regularization for Deep Neural Networks and Its Optimality

Deep learning from strongly mixing observations: Sparse-penalized regularization and minimax optimality

Penalized deep neural networks estimator with general loss functions under weak dependence

Sparse-Input Neural Networks for High-dimensional Nonparametric Regression and Classification

Minimax optimality of deep neural networks on dependent data via PAC-Bayes bounds

Deep belief networks with self-adaptive sparsity

Beyond Least-Squares: Fast Rates for Regularized Empirical Risk Minimization through Self-Concordance

A Two-Stage Approach for Learning A Sparse Model with Sharp Excess Risk Analysis

Reliable and Efficient Inference of Bayesian Networks from Sparse Data by Statistical Learning Theory

Statistical learning with sparsity

High-dimensional classification by sparse logistic regression

Sparse Nonlinear Regression: Parameter Estimation and Asymptotic Inference

Deep Nonparametric Regression on Approximate Manifolds: Non-Asymptotic Error Bounds with Polynomial Prefactors

A Simple Neural Network for Sparse Optimization with $l_1$ Regularization

A note on the minimax risk of sparse linear regression

Sparse Learning for Large-Scale and High-Dimensional Data: A Randomized Convex-Concave Optimization Approach

Robust Nonparametric Regression with Deep Neural Networks

Excess risk bound for deep learning under weak dependence