Abstract:The development of data-informed predictive models for dynamical systems is of widespread interest in many disciplines. We present a unifying framework for blending mechanistic and machine-learning approaches to identify dynamical systems from noisily and partially observed data. We compare pure data-driven learning with hybrid models which incorporate imperfect domain knowledge, referring to the discrepancy between an assumed truth model and the imperfect mechanistic model as model error. Our formulation is agnostic to the chosen machine learning model, is presented in both continuous- and discrete-time settings, and is compatible both with model errors that exhibit substantial memory and errors that are memoryless. First, we study memoryless linear (w.r.t. parametric-dependence) model error from a learning theory perspective, defining excess risk and generalization error. For ergodic continuous-time systems, we prove that both excess risk and generalization error are bounded above by terms that diminish with the square-root of T T , the time-interval over which training data is specified. Secondly, we study scenarios that benefit from modeling with memory, proving universal approximation theorems for two classes of continuous-time recurrent neural networks (RNNs): both can learn memory-dependent model error, assuming that it is governed by a finite-dimensional hidden variable and that, together, the observed and hidden variables form a continuous-time Markovian system. In addition, we connect one class of RNNs to reservoir computing, thereby relating learning of memory-dependent error to recent work on supervised learning between Banach spaces using random features. Numerical results are presented (Lorenz ’63, Lorenz ’96 Multiscale systems) to compare purely data-driven and hybrid approaches, finding hybrid methods less datahungry and more parametrically efficient. We also find that, while a continuous-time framing allows for robustness to irregular sampling and desirable domain- interpretability, a discrete-time framing can provide similar or better predictive performance, especially when data are undersampled and the vector field defining the true dynamics cannot be identified. Finally, we demonstrate numerically how data assimilation can be leveraged to learn hidden dynamics from noisy, partially-observed data, and illustrate challenges in representing memory by this approach, and in the training of such models.

Modeling Error and Nonuniqueness of the Continuous-Time Models Learned via Runge–Kutta Methods

An Error Analysis Framework for Neural Network Modeling of Dynamical Systems

A framework for machine learning of model error in dynamical systems

Error modeling for surrogates of dynamical systems using machine learning

Nonlinear Recursive Minimum Model Error Estimation

Estimation of ordinary differential equation models with discretization error quantification

Lessons Learned from Quantitative Dynamical Modeling in Systems Biology

Modeling of Nonlinear Continous Time Dynamic System with Fuzzy Neural Networks

Discrepancy Modeling Framework: Learning Missing Physics, Modeling Systematic Residuals, and Disambiguating between Deterministic and Random Effects

On the Effects of Modeling Errors on Distributed Continuous-time Filtering

Learning About Structural Errors in Models of Complex Dynamical Systems

Probabilistic Error Estimate for Numerical Discretization of High-Index Saddle Dynamics with Inaccurate Models

Modeling Latent Non-Linear Dynamical System over Time Series

A Control-Theoretic Study on Runge-Kutta Methods with Application to Real-Time Fault-Tolerant Control of Nonlinear Systems

Learning Dynamics from Noisy Measurements using Deep Learning with a Runge-Kutta Constraint

Differential Equations for Modeling Asynchronous Algorithms

Modeling Unknown Stochastic Dynamical System Subject to External Excitation

Efficient Exploration in Continuous-time Model-based Reinforcement Learning

Modeling still matters: a surprising instance of catastrophic floating point errors in mathematical biology and numerical methods for ODEs

Explicit and implicit error inhibiting schemes with post-processing

Time-series machine-learning error models for approximate solutions to parameterized dynamical systems