Abstract:The Universal Approximation Theorem posits that neural networks can theoretically possess unlimited approximation capacity with a suitable activation function and a freely chosen or trained set of parameters. However, a more practical scenario arises when these neural parameters, especially the nonlinear weights and biases, are bounded. This leads us to question: \textbf{Does the approximation capacity of a neural network remain universal, or does it have a limit when the parameters are practically bounded? And if it has a limit, how can it be measured?} Our theoretical study indicates that while universal approximation is theoretically feasible, in practical numerical scenarios, Deep Neural Networks (DNNs) with any analytic activation functions (such as Tanh and Sigmoid) can only be approximated by a finite-dimensional vector space under a bounded nonlinear parameter space (NP space), whether in a continuous or discrete sense. Based on this study, we introduce the concepts of \textit{$\epsilon$ outer measure} and \textit{Numerical Span Dimension (NSdim)} to quantify the approximation capacity limit of a family of networks both theoretically and practically. Furthermore, drawing on our new theoretical study and adopting a fresh perspective, we strive to understand the relationship between back-propagation neural networks and random parameter networks (such as the Extreme Learning Machine (ELM)) with both finite and infinite width. We also aim to provide fresh insights into regularization, the trade-off between width and depth, parameter space, width redundancy, condensation, and other related important issues.

Lower Estimation of Approximation Rate for Neural Networks

Characterization of Degree of Approximation for Neural Networks with One Hidden Layer

Pointwise Approximation for Neural Networks

The Estimate for Approximation Error of Neural Networks: A Constructive Approach

Essential Rate For Approximation By Spherical Neural Networks

Lp Simultaneous Approximation by Neural Networks with One Hidden Layer

Approximation Bounds by Neural Networks in L Ω P [-4Pt]

Convergence Rates for A Class of Neural Networks with Logarithmic Function

The essential approximation order for neural networks with trigonometric hidden layer units

Numerical Approximation Capacity of Neural Networks with Bounded Parameters: Do Limits Exist, and How Can They Be Measured?

Approximation Capability of a Novel Neural Network Model for Dynamic Systems

A general approximation lower bound in $L^p$ norm, with applications to feed-forward neural networks

Why Deep Neural Networks for Function Approximation?

Deep Network Approximation: Achieving Arbitrary Accuracy with Fixed Number of Neurons

Exact and Approximate Interpolation for Neural Networks with Single Hidden Layer

Approximation Capability of Interpolation Neural Networks

Optimal Rates of Approximation by Shallow ReLU Neural Networks and Applications to Nonparametric Regression

The errors of approximation for feedforward neural networks in the Lp metric

Some queries on (quote)Comments on 'Approximation capability in C(R~/sup n/) by multilayer feedforward networks and related problems'(quote)

Approximation of functions from Korobov spaces by shallow neural networks

Partial-neurons-based state estimation for artificial neural networks under constrained bit rate: The finite-time case