Abstract:The foundations of deep learning are supported by the seemingly opposing perspectives of approximation or learning theory. The former advocates for large/expressive models that need not generalize, while the latter considers classes that generalize but may be too small/constrained to be universal approximators. Motivated by real-world deep learning implementations that are both expressive and statistically reliable, we ask: "Is there a class of neural networks that is both large enough to be universal but structured enough to generalize?" This paper constructively provides a positive answer to this question by identifying a highly structured class of ReLU multilayer perceptions (MLPs), which are optimal function approximators and are statistically well-behaved. We show that any $L$-Lipschitz function from $[0,1]^d$ to $[-n,n]$ can be approximated to a uniform $Ld/(2n)$ error on $[0,1]^d$ with a sparsely connected $L$-Lipschitz ReLU MLP of width $\mathcal{O}(dn^d)$, depth $\mathcal{O}(\log(d))$, with $\mathcal{O}(dn^d)$ nonzero parameters, and whose weights and biases take values in $\{0,\pm 1/2\}$ except in the first and last layers which instead have magnitude at-most $n$. Unlike previously known "large" classes of universal ReLU MLPs, the empirical Rademacher complexity of our class remains bounded even when its depth and width become arbitrarily large. Further, our class of MLPs achieves a near-optimal sample complexity of $\mathcal{O}(\log(N)/\sqrt{N})$ when given $N$ i.i.d. normalized sub-Gaussian training samples. We achieve this by avoiding the standard approach to constructing optimal ReLU approximators, which sacrifices regularity by relying on small spikes. Instead, we introduce a new construction that perfectly fits together linear pieces using Kuhn triangulations and avoids these small spikes.

ResNet with one-neuron hidden layers is a Universal Approximator

New advances in universal approximation with neural networks of minimal width

Deep Network Approximation: Achieving Arbitrary Accuracy with Fixed Number of Neurons

Deep Neural Networks: Multi-Classification and Universal Approximation

Universal Function Approximation by Deep Neural Nets with Bounded Width and ReLU Activations

Minimum Width of Leaky-ReLU Neural Networks for Uniform Universal Approximation

On the Expressive Power of Neural Networks

The Expressive Power of Neural Networks: A View from the Width

Universal approximation with complex-valued deep narrow neural networks

Neural Networks Trained by Weight Permutation are Universal Approximators

Implicit Hypersurface Approximation Capacity in Deep ReLU Networks

Rates of Approximation by ReLU Shallow Neural Networks

Universality of deep convolutional neural networks

Universal approximation property of ODENet and ResNet with a single activation function

An elementary proof of a universal approximation theorem

Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators

Low dimensional approximation and generalization of multivariate functions on smooth manifolds using deep ReLU neural networks

Lp Simultaneous Approximation by Neural Networks with One Hidden Layer

On the Principles of ReLU Networks with One Hidden Layer

Universal Approximation Power of Deep Residual Neural Networks via Nonlinear Control Theory

Bridging the Gap Between Approximation and Learning via Optimal Approximation by ReLU MLPs of Maximal Regularity