Abstract:Echo state networks (ESNs) are a powerful form of reservoir computing that only require training of linear output weights whilst the internal reservoir is formed of fixed randomly connected neurons. With a correctly scaled connectivity matrix, the neurons' activity exhibits the echo-state property and responds to the input dynamics with certain timescales. Tuning the timescales of the network can be necessary for treating certain tasks, and some environments require multiple timescales for an efficient representation. Here we explore the timescales in hierarchical ESNs, where the reservoir is partitioned into two smaller linked reservoirs with distinct properties. Over three different tasks (NARMA10, a reconstruction task in a volatile environment, and psMNIST), we show that by selecting the hyper-parameters of each partition such that they focus on different timescales, we achieve a significant performance improvement over a single ESN. Through a linear analysis, and under the assumption that the timescales of the first partition are much shorter than the second's (typically corresponding to optimal operating conditions), we interpret the feedforward coupling of the partitions in terms of an effective representation of the input signal, provided by the first partition to the second, whereby the instantaneous input signal is expanded into a weighted combination of its time derivatives. Furthermore, we propose a data-driven approach to optimise the hyper-parameters through a gradient descent optimisation method that is an online approximation of backpropagation through time. We demonstrate the application of the online learning rule across all the tasks considered.

Multi-Reservoir Echo State Network with Five-Elements Cycle

Learning the Deterministically Constructed Echo State Networks.

An echo state network with interacting reservoirs for modeling and analysis of nonlinear systems

Feature-Based Echo-State Networks: A Step Towards Interpretability and Minimalism in Reservoir Computer

Long-Short Term Echo State Network for Time Series Prediction

Ensemble and Pre-Training Approach for Echo State Network and Extreme Learning Machine Models

MP-CRJ: Multi-parallel cycle reservoirs with jumps for industrial multivariate time series predictions

fastESN: Fast Echo State Network

Exploring deep echo state networks for image classification: a multi-reservoir approach

Impact of time-history terms on reservoir dynamics and prediction accuracy in echo state networks

Deep Echo State Network With Multiple Adaptive Reservoirs for Time Series Prediction

Multi-Reservoir Fully Connected Echo State Network for Wind Power Prediction

Analysis of Wide and Deep Echo State Networks for Multiscale Spatiotemporal Time Series Forecasting

Exploiting Multiple Timescales in Hierarchical Echo State Networks

Memory augmented echo state network for time series prediction

Review on Echo State Networks

Fractional-integer-order echo state network for time series prediction

Attention Based Echo State Network

Multi-reservoir echo state computing for solar irradiance prediction: A fast yet efficient deep learning approach

Generating Oscillation Activity with Echo State Network to Mimic the Behavior of a Simple Central Pattern Generator

Hybrid Backpropagation Parallel Reservoir Networks