Abstract:The echo state network (ESN) is a recurrent neural network that has yielded state-of-the-art results in many areas owing to its rapid learning ability and the fact that the weights of input neurons and hidden neurons are fixed throughout the learning process. However, the setting procedure for initializing the ESN's recurrent structure may lead to difficulties in designing a sound reservoir that matches a specific task. This paper proposes an improved pre-training method to adjust the model's parameters and topology to obtain an adaptive reservoir for a given application. Two strategies, namely global random selection and ensemble training, are introduced to pre-train the randomly initialized ESN model. Specifically, particle swarm optimization is applied to optimize chosen fixed and global weight values within the network, and the reliability and stability of the pre-trained model are enhanced by employing the ensemble training strategy. In addition, we test the feasibility of the model for time series prediction on six benchmarks and two real-life datasets. The experimental results show a clear enhancement in the ESN learning results. Furthermore, the proposed global random selection and ensemble training strategies are also applied to pre-train the extreme learning machine (ELM), which has a similar training process to the ESN model. Numerical experiments are subsequently carried out on the above-mentioned eight datasets. The experimental findings consistently show that the performance of the proposed pre-trained ELM model is also improved significantly. The suggested two strategies can thus enhance the ESN and ELM models' prediction accuracy and adaptability.

Design of Echo State Network with Multi-Objective Optimization Algorithm and L1 Regularization

Multi-objective Sparse Echo State Network

Design of Incremental Echo State Network Using Leave-One-Out Cross-Validation.

An Improved Echo State Network via L 1-Norm Regularization

A Novel Echo State Network Design Method Based on Differential Evolution Algorithm

Long-Short Term Echo State Network for Time Series Prediction

Self-Adaptive Particle Swarm Optimization-Based Echo State Network for Time Series Prediction

Optimizing the echo state network with a binary particle swarm optimization algorithm

Automatic topology optimization of echo state network based on particle swarm optimization

Chain-Structure Echo State Network With Stochastic Optimization: Methodology and Application

Tailoring Echo State Networks for Optimal Learning

L1/2 Norm Regularized Echo State Network for Chaotic Time Series Prediction.

Dynamical Regularized Echo State Network for Time Series Prediction

Design of Sparse Bayesian Echo State Network for Time Series Prediction

An Echo State Network Based on Levenberg-Marquardt Algorithm

Multi-steps prediction of chaotic time series based on echo state network

Online Sequential Echo State Network with Sparse RLS Algorithm for Time Series Prediction.

Error-Driven Chained Multiple-Subnetwork Echo State Network for Time-Series Prediction

Growing Echo-State Network With Multiple Subreservoirs.

Ensemble and Pre-Training Approach for Echo State Network and Extreme Learning Machine Models

An Effective Online Sparse Learning Algorithm for Echo State Networks