Abstract:Abstract One of the reasons why many neural networks are capable of replicating complicated tasks or functions is their universal approximation property. Though the past few decades have seen tremendous advances in theories of neural networks, a single constructive and elementary framework for neural network universality remains unavailable. This paper is an effort to provide a unified and constructive framework for the universality of a large class of activation functions including most of the existing ones. At the heart of the framework is the concept of neural network approximate identity (nAI). The main result is: any nAI activation function is universal in the space of continuous functions on compacta. It turns out that most of the existing activation functions are nAI, and thus universal. The framework induces several advantages over the contemporary counterparts. First, it is constructive with elementary means from functional analysis, probability theory, and numerical analysis. Second, it is one of the first unified and constructive attempts that is valid for most of the existing activation functions. Third, it provides new proofs for most activation functions. Fourth, for a given activation and error tolerance, the framework provides precisely the architecture of the corresponding one-hidden neural network with a predetermined number of neurons and the values of weights/biases. Fifth, the framework allows us to abstractly present the first universal approximation with a favorable non-asymptotic rate. Sixth, our framework also provides insights into the developments, and hence providing constructive derivations, of some of the existing approaches.

Universal Approximation to Nonlinear Operators by Neural Networks with Arbitrary Activation Functions and Its Application to Dynamical Systems

Universal Approximation of Multiple Nonlinear Operators by Neural Networks

Approximations of Continuous Functionals by Neural Networks with Application to Dynamic Systems

Approximation Capability to Functions of Several Variables, Nonlinear Functionals and Operators by Radial Basis Function Neural Networks

Universal approximation results for neural networks with non-polynomial activation function over non-compact domains

Universal Approximation Capability of Ebf Neural Networks with Arbitrary Activation Functions

Approximation Capability of a Novel Neural Network Model for Dynamic Systems

Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators

A unified and constructive framework for the universality of neural networks

Universal Approximation Power of Deep Residual Neural Networks via Nonlinear Control Theory

Theoretical Research on Neural Networks in Application to Dynamic System Modeling

Capability of Neural Networks in Computing the Outputs of Dynamic Systems with Inputs Defined on the Whole Space

Universal Approximation Theorem for Neural Networks

Neural Network Approximations of Compositional Functions With Applications to Dynamical Systems

A Minimal Control Family of Dynamical Syetem for Universal Approximation

Approximation of neural network operators based on the Joukowski transformation

Universal approximation with complex-valued deep narrow neural networks

Quantitative Approximation for Neural Operators in Nonlinear Parabolic Equations

Numerical Approximation Capacity of Neural Networks with Bounded Parameters: Do Limits Exist, and How Can They Be Measured?

Neural Operator: Learning Maps Between Function Spaces