Abstract:The Universal Approximation Theorem posits that neural networks can theoretically possess unlimited approximation capacity with a suitable activation function and a freely chosen or trained set of parameters. However, a more practical scenario arises when these neural parameters, especially the nonlinear weights and biases, are bounded. This leads us to question: \textbf{Does the approximation capacity of a neural network remain universal, or does it have a limit when the parameters are practically bounded? And if it has a limit, how can it be measured?} Our theoretical study indicates that while universal approximation is theoretically feasible, in practical numerical scenarios, Deep Neural Networks (DNNs) with any analytic activation functions (such as Tanh and Sigmoid) can only be approximated by a finite-dimensional vector space under a bounded nonlinear parameter space (NP space), whether in a continuous or discrete sense. Based on this study, we introduce the concepts of \textit{$\epsilon$ outer measure} and \textit{Numerical Span Dimension (NSdim)} to quantify the approximation capacity limit of a family of networks both theoretically and practically. Furthermore, drawing on our new theoretical study and adopting a fresh perspective, we strive to understand the relationship between back-propagation neural networks and random parameter networks (such as the Extreme Learning Machine (ELM)) with both finite and infinite width. We also aim to provide fresh insights into regularization, the trade-off between width and depth, parameter space, width redundancy, condensation, and other related important issues.

Approximation Capability of a Novel Neural Network Model for Dynamic Systems

A New Neural Network Model And Its Application

Approximation of dynamic system by recurrent neural network

Approximation Of Interval Models By Neural Networks

Approximation Property of the Modified Elman Network

Neural Network Approximations of Compositional Functions With Applications to Dynamical Systems

Approximations of Continuous Functionals by Neural Networks with Application to Dynamic Systems

Theoretical Research on Neural Networks in Application to Dynamic System Modeling

Standard neural network model and its application

The Estimate for Approximation Error of Neural Networks: A Constructive Approach

On the Approximation and Complexity of Deep Neural Networks to Invariant Functions

Asynchronous Lebesgue Approximation Model for Distributed Continuous-Time Nonlinear Systems

A Method to Model Nonlinear Systems by Neural Networks

An approximation by neural networkswith a fixed weight

A Study of Online Function Approximation System Based on BP Neural Network

Approximation to Nonlinear Discrete-Time Systems by Recurrent Neural Networks.

Approximation Capability of Elman Neural Network

Approximation Capability of Interpolation Neural Networks

Numerical Approximation Capacity of Neural Networks with Bounded Parameters: Do Limits Exist, and How Can They Be Measured?

Approximation Performance Analysis of Recurrent Neural Networks.

Approximate non-linear model predictive control with safety-augmented neural networks