Abstract:The foundations of deep learning are supported by the seemingly opposing perspectives of approximation or learning theory. The former advocates for large/expressive models that need not generalize, while the latter considers classes that generalize but may be too small/constrained to be universal approximators. Motivated by real-world deep learning implementations that are both expressive and statistically reliable, we ask: "Is there a class of neural networks that is both large enough to be universal but structured enough to generalize?" This paper constructively provides a positive answer to this question by identifying a highly structured class of ReLU multilayer perceptions (MLPs), which are optimal function approximators and are statistically well-behaved. We show that any $L$-Lipschitz function from $[0,1]^d$ to $[-n,n]$ can be approximated to a uniform $Ld/(2n)$ error on $[0,1]^d$ with a sparsely connected $L$-Lipschitz ReLU MLP of width $\mathcal{O}(dn^d)$, depth $\mathcal{O}(\log(d))$, with $\mathcal{O}(dn^d)$ nonzero parameters, and whose weights and biases take values in $\{0,\pm 1/2\}$ except in the first and last layers which instead have magnitude at-most $n$. Unlike previously known "large" classes of universal ReLU MLPs, the empirical Rademacher complexity of our class remains bounded even when its depth and width become arbitrarily large. Further, our class of MLPs achieves a near-optimal sample complexity of $\mathcal{O}(\log(N)/\sqrt{N})$ when given $N$ i.i.d. normalized sub-Gaussian training samples. We achieve this by avoiding the standard approach to constructing optimal ReLU approximators, which sacrifices regularity by relying on small spikes. Instead, we introduce a new construction that perfectly fits together linear pieces using Kuhn triangulations and avoids these small spikes.

Approximation Bounds for Norm Constrained Neural Networks with Applications to Regression and GANs

Optimal Rates of Approximation by Shallow ReLU Neural Networks and Applications to Nonparametric Regression

Error bounds for approximations with deep ReLU neural networks in $W^{s,p}$ norms

Optimal rates of approximation by shallow ReLU$^k$ neural networks and applications to nonparametric regression

Approximation Error and Complexity Bounds for ReLU Networks on Low-Regular Function Spaces

A general approximation lower bound in $L^p$ norm, with applications to feed-forward neural networks

Deep Nonparametric Regression on Approximate Manifolds: Non-Asymptotic Error Bounds with Polynomial Prefactors

Approximation in Shift-Invariant Spaces with Deep ReLU Neural Networks

Normal approximation of Random Gaussian Neural Networks

Optimal function approximation with ReLU neural networks

On the growth of the parameters of approximating ReLU neural networks

Nonparametric regression using over-parameterized shallow ReLU neural networks

Covering Numbers for Deep ReLU Networks with Applications to Function Approximation and Nonparametric Regression

New advances in universal approximation with neural networks of minimal width

On bounds for norms of reparameterized ReLU artificial neural network parameters: sums of fractional powers of the Lipschitz norm control the network parameter vector

Theoretical Investigation of Generalization Bound for Residual Networks.

Approximation with Random Shallow ReLU Networks with Applications to Model Reference Adaptive Control

Optimal Approximation Rates for Deep ReLU Neural Networks on Sobolev and Besov Spaces

A priori generalization error for two-layer ReLU neural network through minimum norm solution

Low dimensional approximation and generalization of multivariate functions on smooth manifolds using deep ReLU neural networks

Bridging the Gap Between Approximation and Learning via Optimal Approximation by ReLU MLPs of Maximal Regularity