Abstract:The echo state network (ESN) is a recurrent neural network that has yielded state-of-the-art results in many areas owing to its rapid learning ability and the fact that the weights of input neurons and hidden neurons are fixed throughout the learning process. However, the setting procedure for initializing the ESN's recurrent structure may lead to difficulties in designing a sound reservoir that matches a specific task. This paper proposes an improved pre-training method to adjust the model's parameters and topology to obtain an adaptive reservoir for a given application. Two strategies, namely global random selection and ensemble training, are introduced to pre-train the randomly initialized ESN model. Specifically, particle swarm optimization is applied to optimize chosen fixed and global weight values within the network, and the reliability and stability of the pre-trained model are enhanced by employing the ensemble training strategy. In addition, we test the feasibility of the model for time series prediction on six benchmarks and two real-life datasets. The experimental results show a clear enhancement in the ESN learning results. Furthermore, the proposed global random selection and ensemble training strategies are also applied to pre-train the extreme learning machine (ELM), which has a similar training process to the ESN model. Numerical experiments are subsequently carried out on the above-mentioned eight datasets. The experimental findings consistently show that the performance of the proposed pre-trained ELM model is also improved significantly. The suggested two strategies can thus enhance the ESN and ELM models' prediction accuracy and adaptability.

Universality and Approximation Bounds for Echo State Networks with Random Weights

Echo State Networks with Self-Normalizing Activations on the Hyper-Sphere

Learning the Deterministically Constructed Echo State Networks.

Approximation with Random Shallow ReLU Networks with Applications to Model Reference Adaptive Control

Echo State Networks for Mobile Robot Modeling and Control

On the Expressive Power of Neural Networks

Approximation in Shift-Invariant Spaces with Deep ReLU Neural Networks

Approximation Error and Complexity Bounds for ReLU Networks on Low-Regular Function Spaces

Ensemble and Pre-Training Approach for Echo State Network and Extreme Learning Machine Models

Deep Neural Networks with ReLU-Sine-Exponential Activations Break Curse of Dimensionality in Approximation on Hölder Class.

An echo state network with interacting reservoirs for modeling and analysis of nonlinear systems

Long-Short Term Echo State Network for Time Series Prediction

Feature-Based Echo-State Networks: A Step Towards Interpretability and Minimalism in Reservoir Computer

Variability of echo state network prediction horizon for partially observed dynamical systems

Hybrid Regularized Echo State Network for Multivariate Chaotic Time Series Prediction

On the Approximation Properties of Random ReLU Features

Universal Approximation Capability of Ebf Neural Networks with Arbitrary Activation Functions

Deep Network Approximation: Achieving Arbitrary Accuracy with Fixed Number of Neurons

Numerical Approximation Capacity of Neural Networks with Bounded Parameters: Do Limits Exist, and How Can They Be Measured?

fastESN: Fast Echo State Network

Automatic-differentiated Physics-Informed Echo State Network (API-ESN)