Abstract:This paper develops simple feed-forward neural networks that achieve the universal approximation property for all continuous functions with a fixed finite number of neurons. These neural networks are simple because they are designed with a simple, computable, and continuous activation function $\sigma$ leveraging a triangular-wave function and the softsign function. We first prove that $\sigma$-activated networks with width $36d(2d+1)$ and depth $11$ can approximate any continuous function on a $d$-dimensional hypercube within an arbitrarily small error. Hence, for supervised learning and its related regression problems, the hypothesis space generated by these networks with a size not smaller than $36d(2d+1)\times 11$ is dense in the continuous function space $C([a,b]^d)$ and therefore dense in the Lebesgue spaces $L^p([a,b]^d)$ for $p\in [1,\infty)$. Furthermore, we show that classification functions arising from image and signal classification are in the hypothesis space generated by $\sigma$-activated networks with width $36d(2d+1)$ and depth $12$ when there exist pairwise disjoint bounded closed subsets of $\mathbb{R}^d$ such that the samples of the same class are located in the same subset. Finally, we use numerical experimentation to show that replacing the rectified linear unit (ReLU) activation function by ours would improve the experiment results.

Approximation error of single hidden layer neural networks with fixed weights

On the approximation by single hidden layer feedforward neural networks with fixed weights

Measure theoretic results for approximation by neural networks with limited weights

Approximation capability of two hidden layer feedforward neural networks with fixed weights

An approximation by neural networkswith a fixed weight

Universal Approximation Theorem for Neural Networks

Deep Network Approximation: Achieving Arbitrary Accuracy with Fixed Number of Neurons

Universal approximation results for neural networks with non-polynomial activation function over non-compact domains

Universal Approximation Theorem for Vector- and Hypercomplex-Valued Neural Networks

Universal Approximation Theorem and error bounds for quantum neural networks and quantum reservoirs

Multi-layer random features and the approximation power of neural networks

Why Deep Neural Networks for Function Approximation?

Function Approximation with Randomly Initialized Neural Networks for Approximate Model Reference Adaptive Control

Approximation and interpolation of deep neural networks

Neural Networks: Further Insights into Error Function, Generalized Weights and Others

An elementary proof of a universal approximation theorem

Approximation by random networks with bounded number of layers

Neural networks: deep, shallow, or in between?

On the Approximation and Complexity of Deep Neural Networks to Invariant Functions

Optimal Neural Network Approximation for High-Dimensional Continuous Functions