Abstract:The learning speed of feed-forward neural networks is notoriously slow and has presented a bottleneck in deep learning applications for several decades. For instance, gradient-based learning algorithms, which are used extensively to train neural networks, tend to work slowly when all of the network parameters must be iteratively tuned. To counter this, both researchers and practitioners have tried introducing randomness to reduce the learning requirement. Based on the original construction of Igelnik and Pao, single layer neural-networks with random input-to-hidden layer weights and biases have seen success in practice, but the necessary theoretical justification is lacking. In this paper, we begin to fill this theoretical gap. We provide a (corrected) rigorous proof that the Igelnik and Pao construction is a universal approximator for continuous functions on compact domains, with approximation error decaying asymptotically like $O(1/\sqrt{n})$ for the number $n$ of network nodes. We then extend this result to the non-asymptotic setting, proving that one can achieve any desired approximation error with high probability provided $n$ is sufficiently large. We further adapt this randomized neural network architecture to approximate functions on smooth, compact submanifolds of Euclidean space, providing theoretical guarantees in both the asymptotic and non-asymptotic forms. Finally, we illustrate our results on manifolds with numerical experiments.

Approximation to a Compact Set of Functions by Feedforward Neural Networks.

Approximations of Continuous Functionals by Neural Networks with Application to Dynamic Systems

Deep Network Approximation: Achieving Arbitrary Accuracy with Fixed Number of Neurons

Approximation error of single hidden layer neural networks with fixed weights

An approximation by neural networkswith a fixed weight

Approximation Capability of a Novel Neural Network Model for Dynamic Systems

On the approximation by single hidden layer feedforward neural networks with fixed weights

A general approximation lower bound in $L^p$ norm, with applications to feed-forward neural networks

Universal Approximation Theorem for Neural Networks

Why Deep Neural Networks for Function Approximation?

Fuzzy RBF Neural Network Based Function Approximation

Function Approximation with Spiked Random Networks

On the Approximation and Complexity of Deep Neural Networks to Invariant Functions

Function Approximation by Random Neural Networks with a Bounded Number of Layers

Function Approximation with Randomly Initialized Neural Networks for Approximate Model Reference Adaptive Control

Some queries on (quote)Comments on 'Approximation capability in C(R~/sup n/) by multilayer feedforward networks and related problems'(quote)

Approximation of RKHS Functionals by Neural Networks

Geometric Interpretation of Nonlinear Approximation Capability for Feedforward Neural Networks.

Approximation by random networks with bounded number of layers

Optimal Neural Network Approximation for High-Dimensional Continuous Functions

Random Vector Functional Link Networks for Function Approximation on Manifolds