Abstract:The Universal Approximation Theorem posits that neural networks can theoretically possess unlimited approximation capacity with a suitable activation function and a freely chosen or trained set of parameters. However, a more practical scenario arises when these neural parameters, especially the nonlinear weights and biases, are bounded. This leads us to question: \textbf{Does the approximation capacity of a neural network remain universal, or does it have a limit when the parameters are practically bounded? And if it has a limit, how can it be measured?} Our theoretical study indicates that while universal approximation is theoretically feasible, in practical numerical scenarios, Deep Neural Networks (DNNs) with any analytic activation functions (such as Tanh and Sigmoid) can only be approximated by a finite-dimensional vector space under a bounded nonlinear parameter space (NP space), whether in a continuous or discrete sense. Based on this study, we introduce the concepts of \textit{$\epsilon$ outer measure} and \textit{Numerical Span Dimension (NSdim)} to quantify the approximation capacity limit of a family of networks both theoretically and practically. Furthermore, drawing on our new theoretical study and adopting a fresh perspective, we strive to understand the relationship between back-propagation neural networks and random parameter networks (such as the Extreme Learning Machine (ELM)) with both finite and infinite width. We also aim to provide fresh insights into regularization, the trade-off between width and depth, parameter space, width redundancy, condensation, and other related important issues.

The Limit of Best Generalized Peak Norm Approximations

On approximation using a peak norm

Almost Sure Max-Limits for Nonstationary Gaussian Sequence

On Best Simultaneous Approximation

On best restricted range approximation in continuouscomplex-valued function spaces

Generalized weight approximation by maximal families with applications

On Nonlinear Simultaneous Approximation Problems

On Well Posed Generalized Best Approximation Problems

Limit Theory of Restricted Range Approximations of Complex-Valued Continuous Functions

On Best Uniform Restricted Range Approximation in Complex-Valued Continuous Function Spaces

Nonlinearly Constrained Best Approximation in Hilbert Spaces: the Strong CHIP and the Basic Constraint Qualification.

On Best Approximation by Nonconvex Sets and Perturbation of Nonconvex Inequality Systems in Hilbert Spaces.

The Limit Points of the Strong Law of Large Numbers under the Sub-Linear Expectations

General Characterization of Best Approximation and Its Application to Uniform Approximation with Restricted Ranges

Best Simultaneous Approximation of Functions and a Generalized Minimax Theorem

A Characterization Of Best Approximations With Restricted Ranges

Numerical Approximation Capacity of Neural Networks with Bounded Parameters: Do Limits Exist, and How Can They Be Measured?

On the Generalization Properties of Minimum-norm Solutions for Over-parameterized Neural Network Models

The Generalization Error of the Minimum-norm Solutions for Over-parameterized Neural Networks

Norm inequalities in and a geometric constant

Low-Rank Approximation via Generalized Reweighted Iterative Nuclear and Frobenius Norms