Abstract:The Universal Approximation Theorem posits that neural networks can theoretically possess unlimited approximation capacity with a suitable activation function and a freely chosen or trained set of parameters. However, a more practical scenario arises when these neural parameters, especially the nonlinear weights and biases, are bounded. This leads us to question: \textbf{Does the approximation capacity of a neural network remain universal, or does it have a limit when the parameters are practically bounded? And if it has a limit, how can it be measured?} Our theoretical study indicates that while universal approximation is theoretically feasible, in practical numerical scenarios, Deep Neural Networks (DNNs) with any analytic activation functions (such as Tanh and Sigmoid) can only be approximated by a finite-dimensional vector space under a bounded nonlinear parameter space (NP space), whether in a continuous or discrete sense. Based on this study, we introduce the concepts of \textit{$\epsilon$ outer measure} and \textit{Numerical Span Dimension (NSdim)} to quantify the approximation capacity limit of a family of networks both theoretically and practically. Furthermore, drawing on our new theoretical study and adopting a fresh perspective, we strive to understand the relationship between back-propagation neural networks and random parameter networks (such as the Extreme Learning Machine (ELM)) with both finite and infinite width. We also aim to provide fresh insights into regularization, the trade-off between width and depth, parameter space, width redundancy, condensation, and other related important issues.

Spectral Barron space and deep neural network approximation

Embedding Inequalities for Barron-type Spaces

Weighted Sobolev Approximation Rates for Neural Networks on Unbounded Domains

The Barron Space and the Flow-Induced Function Spaces for Neural Network Models

Optimal Approximation Rates for Deep ReLU Neural Networks on Sobolev and Besov Spaces

Space-Time Approximation with Shallow Neural Networks in Fourier Lebesgue spaces

Deep Nonparametric Regression on Approximate Manifolds: Non-Asymptotic Error Bounds with Polynomial Prefactors

Lp Simultaneous Approximation by Neural Networks with One Hidden Layer

On the optimal approximation of Sobolev and Besov functions using deep ReLU neural networks

Spectral Neural Operators

Approximation in Shift-Invariant Spaces with Deep ReLU Neural Networks

Universal Approximation Theorem for Neural Networks

Basis Operator Network: A Neural Network-Based Model for Learning Nonlinear Operators Via Neural Basis

Mixed Laplace approximation for marginal posterior and Bayesian inference in error-in-operator model

Numerical Approximation Capacity of Neural Networks with Bounded Parameters: Do Limits Exist, and How Can They Be Measured?

Sharp Bounds on the Approximation Rates, Metric Entropy, and $n$-widths of Shallow Neural Networks

Approximation smooth and sparse functions by deep neural networks without saturation

Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionality

Approximation Bounds by Neural Networks in L Ω P [-4Pt]

Some observations on high-dimensional partial differential equations with Barron data

High-dimensional classification problems with Barron regular boundaries under margin conditions