Abstract:The universal approximation theorem is generalised to uniform convergence on the (noncompact) input space $\mathbb{R}^n$. All continuous functions that vanish at infinity can be uniformly approximated by neural networks with one hidden layer, for all activation functions $\varphi$ that are continuous, nonpolynomial, and asymptotically polynomial at $\pm\infty$. When $\varphi$ is moreover bounded, we exactly determine which functions can be uniformly approximated by neural networks, with the following unexpected results. Let $\overline{\mathcal{N}_\varphi^l(\mathbb{R}^n)}$ denote the vector space of functions that are uniformly approximable by neural networks with $l$ hidden layers and $n$ inputs. For all $n$ and all $l\geq2$, $\overline{\mathcal{N}_\varphi^l(\mathbb{R}^n)}$ turns out to be an algebra under the pointwise product. If the left limit of $\varphi$ differs from its right limit (for instance, when $\varphi$ is sigmoidal) the algebra $\overline{\mathcal{N}_\varphi^l(\mathbb{R}^n)}$ ($l\geq2$) is independent of $\varphi$ and $l$, and equals the closed span of products of sigmoids composed with one-dimensional projections. If the left limit of $\varphi$ equals its right limit, $\overline{\mathcal{N}_\varphi^l(\mathbb{R}^n)}$ ($l\geq1$) equals the (real part of the) commutative resolvent algebra, a C*-algebra which is used in mathematical approaches to quantum theory. In the latter case, the algebra is independent of $l\geq1$, whereas in the former case $\overline{\mathcal{N}_\varphi^2(\mathbb{R}^n)}$ is strictly bigger than $\overline{\mathcal{N}_\varphi^1(\mathbb{R}^n)}$.

Universal Approximation of Markov Kernels by Shallow Stochastic Feedforward Networks

Deep Narrow Boltzmann Machines are Universal Approximators

Universal approximation with complex-valued deep narrow neural networks

An elementary proof of a universal approximation theorem

Universal Approximation Theorem for Neural Networks

Universality and Optimality of Structured Deep Kernel Networks

On the approximation by single hidden layer feedforward neural networks with fixed weights

Deep Network Approximation: Achieving Arbitrary Accuracy with Fixed Number of Neurons

Optimal approximating Markov chains for Bayesian inference

Optimal Neural Network Approximation for High-Dimensional Continuous Functions

Universal Approximation Using Radial-Basis-Function Networks

Universal Approximation of Multiple Nonlinear Operators by Neural Networks

Function Approximation with Spiked Random Networks

Recurrent Neural Networks Are Universal Approximators with Stochastic Inputs

Neural networks: deep, shallow, or in between?

Approximation of functions from Korobov spaces by shallow neural networks

Approximation and interpolation of deep neural networks

Universal Approximation Theorem for Vector- and Hypercomplex-Valued Neural Networks

Noncompact uniform universal approximation

Universal approximation results for neural networks with non-polynomial activation function over non-compact domains