Abstract:This paper develops simple feed-forward neural networks that achieve the universal approximation property for all continuous functions with a fixed finite number of neurons. These neural networks are simple because they are designed with a simple, computable, and continuous activation function $\sigma$ leveraging a triangular-wave function and the softsign function. We first prove that $\sigma$-activated networks with width $36d(2d+1)$ and depth $11$ can approximate any continuous function on a $d$-dimensional hypercube within an arbitrarily small error. Hence, for supervised learning and its related regression problems, the hypothesis space generated by these networks with a size not smaller than $36d(2d+1)\times 11$ is dense in the continuous function space $C([a,b]^d)$ and therefore dense in the Lebesgue spaces $L^p([a,b]^d)$ for $p\in [1,\infty)$. Furthermore, we show that classification functions arising from image and signal classification are in the hypothesis space generated by $\sigma$-activated networks with width $36d(2d+1)$ and depth $12$ when there exist pairwise disjoint bounded closed subsets of $\mathbb{R}^d$ such that the samples of the same class are located in the same subset. Finally, we use numerical experimentation to show that replacing the rectified linear unit (ReLU) activation function by ours would improve the experiment results.

Universal Approximation by Phase Series and Fixed-Weight Networks

Universal Approximation Theorem for Neural Networks

Universal Approximation Theorem and error bounds for quantum neural networks and quantum reservoirs

Universal Approximation of Multiple Nonlinear Operators by Neural Networks

Universal approximation results for neural networks with non-polynomial activation function over non-compact domains

Fourier Neural Networks for Function Approximation

On the approximation by single hidden layer feedforward neural networks with fixed weights

Universal Approximation Theorem for Vector- and Hypercomplex-Valued Neural Networks

Global universal approximation of functional input maps on weighted spaces

Noncompact uniform universal approximation

A Survey on Universal Approximation Theorems

An approximation by neural networkswith a fixed weight

Weighted Sobolev Approximation Rates for Neural Networks on Unbounded Domains

Universal approximation with complex-valued deep narrow neural networks

Approximation error of single hidden layer neural networks with fixed weights

Neural Networks Trained by Weight Permutation are Universal Approximators

An elementary proof of a universal approximation theorem

On Expressive Power of Quantized Neural Networks under Fixed-Point Arithmetic

Deep Network Approximation: Achieving Arbitrary Accuracy with Fixed Number of Neurons