Abstract:Covering numbers of families of (deep) ReLU networks have been used to characterize their approximation-theoretic performance, upper-bound the prediction error they incur in nonparametric regression, and quantify their classification capacity. These results are based on covering number upper bounds obtained through the explicit construction of coverings. Lower bounds on covering numbers do not seem to be available in the literature. The present paper fills this gap by deriving tight (up to a multiplicative constant) lower and upper bounds on the covering numbers of fully-connected networks with bounded weights, sparse networks with bounded weights, and fully-connected networks with quantized weights. Thanks to the tightness of the bounds, a fundamental understanding of the impact of sparsity, quantization, bounded vs. unbounded weights, and network output truncation can be developed. Furthermore, the bounds allow to characterize the fundamental limits of neural network transformation, including network compression, and lead to sharp upper bounds on the prediction error in nonparametric regression through deep networks. Specifically, we can remove a $\log^6(n)$-factor in the best-known sample complexity rate in the estimation of Lipschitz functions through deep networks thereby establishing optimality. Finally, we identify a systematic relation between optimal nonparametric regression and optimal approximation through deep networks, unifying numerous results in the literature and uncovering general underlying principles.

Quantified advantage of discontinuous weight selection in approximations with deep neural networks

Three Quantization Regimes for ReLU Networks

On the optimal approximation of Sobolev and Besov functions using deep ReLU neural networks

Why Deep Neural Networks for Function Approximation?

ReLU neural network approximation to piecewise constant functions

A note about why deep learning is deep: A discontinuous approximation perspective

Optimal Approximation Rates for Deep ReLU Neural Networks on Sobolev and Besov Spaces

Rates of Approximation by ReLU Shallow Neural Networks

Covering Numbers for Deep ReLU Networks with Applications to Function Approximation and Nonparametric Regression

Low dimensional approximation and generalization of multivariate functions on smooth manifolds using deep ReLU neural networks

New advances in universal approximation with neural networks of minimal width

Approximation with Random Shallow ReLU Networks with Applications to Model Reference Adaptive Control

Uniform Approximation with Quadratic Neural Networks

Approximation in Shift-Invariant Spaces with Deep ReLU Neural Networks

Deep neural network approximation of composite functions without the curse of dimensionality

Efficient uniform approximation using Random Vector Functional Link networks

Error bounds for approximations with deep ReLU neural networks in $W^{s,p}$ norms

Approximation Rates for Shallow ReLU$^k$ Neural Networks on Sobolev Spaces via the Radon Transform

Deep Neural Networks with ReLU-Sine-Exponential Activations Break Curse of Dimensionality in Approximation on Hölder Class.

Optimal Neural Network Approximation for High-Dimensional Continuous Functions

Neural networks with ReLU powers need less depth