Abstract:Covering numbers of families of (deep) ReLU networks have been used to characterize their approximation-theoretic performance, upper-bound the prediction error they incur in nonparametric regression, and quantify their classification capacity. These results are based on covering number upper bounds obtained through the explicit construction of coverings. Lower bounds on covering numbers do not seem to be available in the literature. The present paper fills this gap by deriving tight (up to a multiplicative constant) lower and upper bounds on the covering numbers of fully-connected networks with bounded weights, sparse networks with bounded weights, and fully-connected networks with quantized weights. Thanks to the tightness of the bounds, a fundamental understanding of the impact of sparsity, quantization, bounded vs. unbounded weights, and network output truncation can be developed. Furthermore, the bounds allow to characterize the fundamental limits of neural network transformation, including network compression, and lead to sharp upper bounds on the prediction error in nonparametric regression through deep networks. Specifically, we can remove a $\log^6(n)$-factor in the best-known sample complexity rate in the estimation of Lipschitz functions through deep networks thereby establishing optimality. Finally, we identify a systematic relation between optimal nonparametric regression and optimal approximation through deep networks, unifying numerous results in the literature and uncovering general underlying principles.

Error bounds for approximations with deep ReLU neural networks in $W^{s,p}$ norms

Optimal Approximation Rates for Deep ReLU Neural Networks on Sobolev and Besov Spaces

On the optimal approximation of Sobolev and Besov functions using deep ReLU neural networks

Approximation Error and Complexity Bounds for ReLU Networks on Low-Regular Function Spaces

Approximation Rates for Shallow ReLU$^k$ Neural Networks on Sobolev Spaces via the Radon Transform

Approximation in $L^p(μ)$ with deep ReLU neural networks

Approximation Bounds for Norm Constrained Neural Networks with Applications to Regression and GANs

Nearly Optimal Approximation Rates for Deep Super ReLU Networks on Sobolev Spaces

Approximation in Shift-Invariant Spaces with Deep ReLU Neural Networks

Rates of Approximation by ReLU Shallow Neural Networks

On the Expressive Power of Neural Networks

Neural networks with ReLU powers need less depth

Three Quantization Regimes for ReLU Networks

Optimal rates of approximation by shallow ReLU$^k$ neural networks and applications to nonparametric regression

Optimal Rates of Approximation by Shallow ReLU Neural Networks and Applications to Nonparametric Regression

ReLU neural network approximation to piecewise constant functions

Covering Numbers for Deep ReLU Networks with Applications to Function Approximation and Nonparametric Regression

Solving the Poisson Equation with Dirichlet data by shallow ReLU$^α$-networks: A regularity and approximation perspective

Upper and lower bounds for the Lipschitz constant of random neural networks

Deep ReLU networks and high-order finite element methods II: Chebyshev emulation

On the growth of the parameters of approximating ReLU neural networks