Abstract:Abstract One of the reasons why many neural networks are capable of replicating complicated tasks or functions is their universal approximation property. Though the past few decades have seen tremendous advances in theories of neural networks, a single constructive and elementary framework for neural network universality remains unavailable. This paper is an effort to provide a unified and constructive framework for the universality of a large class of activation functions including most of the existing ones. At the heart of the framework is the concept of neural network approximate identity (nAI). The main result is: any nAI activation function is universal in the space of continuous functions on compacta. It turns out that most of the existing activation functions are nAI, and thus universal. The framework induces several advantages over the contemporary counterparts. First, it is constructive with elementary means from functional analysis, probability theory, and numerical analysis. Second, it is one of the first unified and constructive attempts that is valid for most of the existing activation functions. Third, it provides new proofs for most activation functions. Fourth, for a given activation and error tolerance, the framework provides precisely the architecture of the corresponding one-hidden neural network with a predetermined number of neurons and the values of weights/biases. Fifth, the framework allows us to abstractly present the first universal approximation with a favorable non-asymptotic rate. Sixth, our framework also provides insights into the developments, and hence providing constructive derivations, of some of the existing approaches.

Universal Approximation Capability of Ebf Neural Networks with Arbitrary Activation Functions

Universal Approximation to Nonlinear Operators by Neural Networks with Arbitrary Activation Functions and Its Application to Dynamical Systems

Approximation Capability to Functions of Several Variables, Nonlinear Functionals and Operators by Radial Basis Function Neural Networks

Universal Approximation Using Radial-Basis-Function Networks

Universal approximation results for neural networks with non-polynomial activation function over non-compact domains

L P Approximation Capability of RBF Neural Networks

Numerical Approximation Capacity of Neural Networks with Bounded Parameters: Do Limits Exist, and How Can They Be Measured?

Function Approximation with Spiked Random Networks

Function Approximation by Random Neural Networks with a Bounded Number of Layers

On the universal approximation property of radial basis function neural networks

Investigation of Neural Networks for Function Approximation

Fuzzy RBF Neural Network Based Function Approximation

Universal approximation with complex-valued deep narrow neural networks

Deep Network Approximation: Achieving Arbitrary Accuracy with Fixed Number of Neurons

Some queries on (quote)Comments on 'Approximation capability in C(R~/sup n/) by multilayer feedforward networks and related problems'(quote)

Approximation Rates for Neural Networks with General Activation Functions

A unified and constructive framework for the universality of neural networks

Function Approximation with Randomly Initialized Neural Networks for Approximate Model Reference Adaptive Control

Universal Approximation Theorem for Neural Networks

On the universal approximation theorem of fuzzy neural networks with random membership function parameters