Abstract:Recent work has shown that machine learning (ML) models can skillfully forecast the dynamics of unknown chaotic systems. Short-term predictions of the state evolution and long-term predictions of the statistical patterns of the dynamics ("climate") can be produced by employing a feedback loop, whereby the model is trained to predict forward only one time step, then the model output is used as input for multiple time steps. In the absence of mitigating techniques, however, this feedback can result in artificially rapid error growth ("instability"). One established mitigating technique is to add noise to the ML model training input. Based on this technique, we formulate a new penalty term in the loss function for ML models with memory of past inputs that deterministically approximates the effect of many small, independent noise realizations added to the model input during training. We refer to this penalty and the resulting regularization as Linearized Multi-Noise Training (LMNT). We systematically examine the effect of LMNT, input noise, and other established regularization techniques in a case study using reservoir computing, a machine learning method using recurrent neural networks, to predict the spatiotemporal chaotic Kuramoto-Sivashinsky equation. We find that reservoir computers trained with noise or with LMNT produce climate predictions that appear to be indefinitely stable and have a climate very similar to the true system, while the short-term forecasts are substantially more accurate than those trained with other regularization techniques. Finally, we show the deterministic aspect of our LMNT regularization facilitates fast reservoir computer regularization hyperparameter tuning.

Good and bad predictions: Assessing and improving the replication of chaotic attractors by means of reservoir computing

Application of next-generation reservoir computing for predicting chaotic systems from partial observations

Model-Free Prediction of Chaotic Systems Using High Efficient Next-generation Reservoir Computing

Chaotic attractor reconstruction using small reservoirs -- the influence of topology

Nonlinear Neural Dynamics and Classification Accuracy in Reservoir Computing

Long-term Prediction of Chaotic Systems with Machine Learning

Large Sampling Intervals for Learning and Predicting Chaotic Systems with Reservoir Computing

Predicting Chaotic System Behavior using Machine Learning Techniques

Backpropagation algorithms and Reservoir Computing in Recurrent Neural Networks for the forecasting of complex spatiotemporal dynamics

Reservoir Computing with Error Correction: Long-term Behaviors of Stochastic Dynamical Systems

Catch-22s of reservoir computing

Stabilizing machine learning prediction of dynamics: Novel noise-inspired regularization tested with reservoir computing

A systematic exploration of reservoir computing for forecasting complex spatiotemporal dynamics

Reservoir computing with logistic map

Model scale versus domain knowledge in statistical forecasting of chaotic systems

Controlling dynamical systems to complex target states using machine learning: next-generation vs. classical reservoir computing

Long-term prediction of chaotic systems with recurrent neural networks

Complex Network Perspective on Modelling Chaotic Systems Via Machine Learning*

Mapping topological characteristics of dynamical systems into neural networks: A reservoir computing approach

Learning unseen coexisting attractors

Learning spatiotemporal chaos using next-generation reservoir computing