Abstract:Recurrent neural networks (RNNs) are known to be universal approximators of dynamic systems under fairly mild and general assumptions. However, RNNs usually suffer from the issues of vanishing and exploding gradients in standard RNN training. Reservoir computing (RC), a special RNN where the recurrent weights are randomized and left untrained, has been introduced to overcome these issues and has demonstrated superior empirical performance especially in scenarios where training samples are extremely limited. On the other hand, the theoretical grounding to support this observed performance has yet been fully developed. In this work, we show that RC can universally approximate a general linear time-invariant (LTI) system. Specifically, we present a clear signal processing interpretation of RC and utilize this understanding in the problem of approximating a generic LTI system. Under this setup, we analytically characterize the optimum probability density function for configuring (instead of training and/or randomly generating) the recurrent weights of the underlying RNN of the RC. Extensive numerical evaluations are provided to validate the optimality of the derived distribution for configuring the recurrent weights of the RC to approximate a general LTI system. Our work results in clear signal processing-based model interpretability of RC and provides theoretical explanation/justification for the power of randomness in randomly generating instead of training RC's recurrent weights. Furthermore, it provides a complete optimum analytical characterization for configuring the untrained recurrent weights, marking an important step towards explainable machine learning (XML) to incorporate domain knowledge for efficient learning.

A universal approximation theorem for nonlinear resistive networks

A Fast Algorithm to Simulate Nonlinear Resistive Networks

Universal Approximation Power of Deep Residual Neural Networks via Nonlinear Control Theory

Universal Approximation Theorem for Vector- and Hypercomplex-Valued Neural Networks

Deep Network Approximation: Achieving Arbitrary Accuracy with Fixed Number of Neurons

Function Approximation with Randomly Initialized Neural Networks for Approximate Model Reference Adaptive Control

Universal Approximation to Nonlinear Operators by Neural Networks with Arbitrary Activation Functions and Its Application to Dynamical Systems

Noncompact uniform universal approximation

Efficient uniform approximation using Random Vector Functional Link networks

Universal Approximation of Linear Time-Invariant (LTI) Systems through RNNs: Power of Randomness in Reservoir Computing

On the Expressive Power of Neural Networks

Approximate Nonlinear Output Regulation Based on the Universal Approximation Theorem

The complex dynamics of memristive circuits: analytical results and universal slow relaxation

An elementary proof of a universal approximation theorem

Learning to Approximate Functions Using Nb-doped SrTiO$_3$ Memristors

Universal Approximation of Multiple Nonlinear Operators by Neural Networks

Universal Approximation Theorem and error bounds for quantum neural networks and quantum reservoirs

Efficiency and controllability of stochastic boolean function generation by a random network of non-linear nanoparticle junctions

On the approximation by single hidden layer feedforward neural networks with fixed weights

Approximation Error and Complexity Bounds for ReLU Networks on Low-Regular Function Spaces