Abstract:The foundations of deep learning are supported by the seemingly opposing perspectives of approximation or learning theory. The former advocates for large/expressive models that need not generalize, while the latter considers classes that generalize but may be too small/constrained to be universal approximators. Motivated by real-world deep learning implementations that are both expressive and statistically reliable, we ask: "Is there a class of neural networks that is both large enough to be universal but structured enough to generalize?" This paper constructively provides a positive answer to this question by identifying a highly structured class of ReLU multilayer perceptions (MLPs), which are optimal function approximators and are statistically well-behaved. We show that any $L$-Lipschitz function from $[0,1]^d$ to $[-n,n]$ can be approximated to a uniform $Ld/(2n)$ error on $[0,1]^d$ with a sparsely connected $L$-Lipschitz ReLU MLP of width $\mathcal{O}(dn^d)$, depth $\mathcal{O}(\log(d))$, with $\mathcal{O}(dn^d)$ nonzero parameters, and whose weights and biases take values in $\{0,\pm 1/2\}$ except in the first and last layers which instead have magnitude at-most $n$. Unlike previously known "large" classes of universal ReLU MLPs, the empirical Rademacher complexity of our class remains bounded even when its depth and width become arbitrarily large. Further, our class of MLPs achieves a near-optimal sample complexity of $\mathcal{O}(\log(N)/\sqrt{N})$ when given $N$ i.i.d. normalized sub-Gaussian training samples. We achieve this by avoiding the standard approach to constructing optimal ReLU approximators, which sacrifices regularity by relying on small spikes. Instead, we introduce a new construction that perfectly fits together linear pieces using Kuhn triangulations and avoids these small spikes.

Optimal Deep Neural Networks by Maximization of the Approximation Power

Optimal function approximation with ReLU neural networks

Near-Minimax Optimal Estimation With Shallow ReLU Neural Networks

Optimal rates of approximation by shallow ReLU$^k$ neural networks and applications to nonparametric regression

Neural networks with ReLU powers need less depth

Optimal Rates of Approximation by Shallow ReLU Neural Networks and Applications to Nonparametric Regression

Bridging the Gap Between Approximation and Learning via Optimal Approximation by ReLU MLPs of Maximal Regularity

Neural Network Architecture Optimization through Submodularity and Supermodularity.

Optimal Neural Network Approximation for High-Dimensional Continuous Functions

Deep vs. shallow networks : An approximation theory perspective

Searching for Minimal Optimal Neural Networks

The limitation of neural nets for approximation and optimization

On the Expressive Power of Neural Networks

Nonparametric regression using deep neural networks with ReLU activation function

Deep Neural Networks with ReLU-Sine-Exponential Activations Break Curse of Dimensionality in Approximation on Hölder Class.

Optimization Algorithm Inspired Deep Neural Network Structure Design

Why Deep Neural Networks for Function Approximation?

Adaptive two-layer ReLU neural network: I. Best least-squares approximation

Smooth Function Approximation by Deep Neural Networks with General Activation Functions

Optimal Approximation and Learning Rates for Deep Convolutional Neural Networks

Theoretical Issues in Deep Networks: Approximation, Optimization and Generalization