Abstract:The universal approximation theorem is generalised to uniform convergence on the (noncompact) input space $\mathbb{R}^n$. All continuous functions that vanish at infinity can be uniformly approximated by neural networks with one hidden layer, for all activation functions $\varphi$ that are continuous, nonpolynomial, and asymptotically polynomial at $\pm\infty$. When $\varphi$ is moreover bounded, we exactly determine which functions can be uniformly approximated by neural networks, with the following unexpected results. Let $\overline{\mathcal{N}_\varphi^l(\mathbb{R}^n)}$ denote the vector space of functions that are uniformly approximable by neural networks with $l$ hidden layers and $n$ inputs. For all $n$ and all $l\geq2$, $\overline{\mathcal{N}_\varphi^l(\mathbb{R}^n)}$ turns out to be an algebra under the pointwise product. If the left limit of $\varphi$ differs from its right limit (for instance, when $\varphi$ is sigmoidal) the algebra $\overline{\mathcal{N}_\varphi^l(\mathbb{R}^n)}$ ($l\geq2$) is independent of $\varphi$ and $l$, and equals the closed span of products of sigmoids composed with one-dimensional projections. If the left limit of $\varphi$ equals its right limit, $\overline{\mathcal{N}_\varphi^l(\mathbb{R}^n)}$ ($l\geq1$) equals the (real part of the) commutative resolvent algebra, a C*-algebra which is used in mathematical approaches to quantum theory. In the latter case, the algebra is independent of $l\geq1$, whereas in the former case $\overline{\mathcal{N}_\varphi^2(\mathbb{R}^n)}$ is strictly bigger than $\overline{\mathcal{N}_\varphi^1(\mathbb{R}^n)}$.

Deep, narrow sigmoid belief networks are universal approximators

Deep Narrow Boltzmann Machines are Universal Approximators

Deep Network Approximation: Achieving Arbitrary Accuracy with Fixed Number of Neurons

Neural Variational Inference and Learning in Belief Networks

New advances in universal approximation with neural networks of minimal width

Constructive Universal High-Dimensional Distribution Generation through Deep ReLU Networks

On the capacity of deep generative networks for approximating distributions

A Fast Learning Algorithm for Deep Belief Nets

Universality of deep convolutional neural networks

Universal approximation with complex-valued deep narrow neural networks

Deep Neural Networks: Multi-Classification and Universal Approximation

ResNet with one-neuron hidden layers is a Universal Approximator

On the approximation by single hidden layer feedforward neural networks with fixed weights

Universal Approximation of Markov Kernels by Shallow Stochastic Feedforward Networks

Approximating smooth functions by deep neural networks with sigmoid activation function

Noncompact uniform universal approximation

Universal Approximation Theorem for Vector- and Hypercomplex-Valued Neural Networks

Learning Functions: When Is Deep Better Than Shallow

An elementary proof of a universal approximation theorem

Universal Consistency of Wide and Deep ReLU Neural Networks and Minimax Optimal Convergence Rates for Kolmogorov-Donoho Optimal Function Classes

Wide Deep Neural Networks with Gaussian Weights are Very Close to Gaussian Processes