Abstract:This paper develops simple feed-forward neural networks that achieve the universal approximation property for all continuous functions with a fixed finite number of neurons. These neural networks are simple because they are designed with a simple, computable, and continuous activation function $\sigma$ leveraging a triangular-wave function and the softsign function. We first prove that $\sigma$-activated networks with width $36d(2d+1)$ and depth $11$ can approximate any continuous function on a $d$-dimensional hypercube within an arbitrarily small error. Hence, for supervised learning and its related regression problems, the hypothesis space generated by these networks with a size not smaller than $36d(2d+1)\times 11$ is dense in the continuous function space $C([a,b]^d)$ and therefore dense in the Lebesgue spaces $L^p([a,b]^d)$ for $p\in [1,\infty)$. Furthermore, we show that classification functions arising from image and signal classification are in the hypothesis space generated by $\sigma$-activated networks with width $36d(2d+1)$ and depth $12$ when there exist pairwise disjoint bounded closed subsets of $\mathbb{R}^d$ such that the samples of the same class are located in the same subset. Finally, we use numerical experimentation to show that replacing the rectified linear unit (ReLU) activation function by ours would improve the experiment results.

Approximations of Continuous Functionals by Neural Networks with Application to Dynamic Systems

Universal Approximation to Nonlinear Operators by Neural Networks with Arbitrary Activation Functions and Its Application to Dynamical Systems

Approximation Capability of a Novel Neural Network Model for Dynamic Systems

Neural Network Approximations of Compositional Functions With Applications to Dynamical Systems

Theoretical Research on Neural Networks in Application to Dynamic System Modeling

An approximation by neural networkswith a fixed weight

Approximation Capability to Functions of Several Variables, Nonlinear Functionals and Operators by Radial Basis Function Neural Networks

Function Approximation with Randomly Initialized Neural Networks for Approximate Model Reference Adaptive Control

On the Approximation and Complexity of Deep Neural Networks to Invariant Functions

Optimal Neural Network Approximation for High-Dimensional Continuous Functions

Universal approximation results for neural networks with non-polynomial activation function over non-compact domains

Capability of Neural Networks in Computing the Outputs of Dynamic Systems with Inputs Defined on the Whole Space

Why Deep Neural Networks for Function Approximation?

Deep Neural Network Approximation of Composition Functions: with Application to PINNs

Deep Network Approximation: Achieving Arbitrary Accuracy with Fixed Number of Neurons

Neural Networks are Integrable

Smooth Function Approximation by Deep Neural Networks with General Activation Functions

Deep Learning via Dynamical Systems: An Approximation Perspective

Approximation of RKHS Functionals by Neural Networks

A Minimal Control Family of Dynamical Syetem for Universal Approximation