Abstract:Recently, Neural Ordinary Differential Equations has emerged as a powerful framework for modeling physical simulations without explicitly defining the ODEs governing the system, but instead learning them via machine learning. However, the question: "Can Bayesian learning frameworks be integrated with Neural ODE's to robustly quantify the uncertainty in the weights of a Neural ODE?" remains unanswered. In an effort to address this question, we primarily evaluate the following categories of inference methods: (a) The No-U-Turn MCMC sampler (NUTS), (b) Stochastic Gradient Hamiltonian Monte Carlo (SGHMC) and (c) Stochastic Langevin Gradient Descent (SGLD). We demonstrate the successful integration of Neural ODEs with the above Bayesian inference frameworks on classical physical systems, as well as on standard machine learning datasets like MNIST, using GPU acceleration. On the MNIST dataset, we achieve a posterior sample accuracy of 98.5% on the test ensemble of 10,000 images. Subsequently, for the first time, we demonstrate the successful integration of variational inference with normalizing flows and Neural ODEs, leading to a powerful Bayesian Neural ODE object. Finally, considering a predator-prey model and an epidemiological system, we demonstrate the probabilistic identification of model specification in partially-described dynamical systems using universal ordinary differential equations. Together, this gives a scientific machine learning tool for probabilistic estimation of epistemic uncertainties.

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is to combine the Bayesian learning framework with Neural Ordinary Differential Equations (Neural ODEs) in order to robustly quantify the uncertainty in the weights of Neural ODEs. Specifically, the paper explores the following issues: 1. **Combination of Bayesian Framework and Neural ODEs**: How to combine Bayesian learning methods (such as No - U - Turn Sampler (NUTS), Stochastic Gradient Hamiltonian Monte Carlo (SGHMC) and Stochastic Langevin Gradient Descent (SGLD)) with Neural ODEs to quantify the uncertainty of model parameters. 2. **Application on Classical Physical Systems and Standard Machine Learning Datasets**: The paper verifies the effectiveness of these methods on classical physical systems (such as the Spiral ODE and Lotka - Volterra model) and standard machine learning datasets (such as MNIST), and shows their performance in image classification tasks. 3. **Combination of Variational Inference and Normalizing Flows**: The paper for the first time shows the method of combining variational inference with normalizing flows and Neural ODEs, which improves the predictive ability of the model. 4. **Model Identification of Partially Described Dynamical Systems**: The paper uses Universal Ordinary Differential Equations (UDEs) to identify the missing terms in partially described dynamical systems. For example, in a predator - prey model and an epidemiological model, the missing model terms are successfully recovered. ### Main Contributions - **Implementation of Bayesian Neural ODEs**: The paper successfully combines Bayesian inference methods with Neural ODEs, achieving robust quantification of the uncertainty of model parameters. - **High - Performance Bayesian Inference Methods**: By using methods such as NUTS, SGHMC and SGLD, the paper shows efficient Bayesian inference performance on multiple classical physical systems and standard datasets. - **Combination of Variational Inference and Normalizing Flows**: By introducing normalizing flows, the performance of variational inference in Neural ODEs is improved. - **Model Identification of Partially Described Dynamical Systems**: The paper proposes a method that can identify the missing model terms in partially described dynamical systems, thus providing a new scientific machine - learning tool for estimating epistemic uncertainty. ### Experimental Results - **Classical Physical Systems**: On the Spiral ODE and Lotka - Volterra model, the prediction results of Bayesian Neural ODEs are consistent with the real data, and the posterior samples are well - mixed. - **MNIST Dataset**: On the MNIST dataset, Bayesian Neural ODEs using the SGHMC method achieve a 98.5% test - set accuracy and can quantify the confidence of predictions. - **Variational Inference and Normalizing Flows**: By introducing normalizing flows, the prediction performance of variational inference is improved. - **Partially Described Dynamical Systems**: In the predator - prey model and the epidemiological model, the missing model terms are successfully recovered, showing the potential of this method in identifying complex systems. In conclusion, by combining the Bayesian learning framework with Neural ODEs, this paper not only improves the prediction performance of the model, but also can robustly quantify the uncertainty of model parameters, providing new tools and methods for scientific computing and machine learning.

Bayesian Neural Ordinary Differential Equations

Uncertainty and Structure in Neural Ordinary Differential Equations

Dealing with Stochasticity in Biological ODE Models

Beyond Predictions in Neural ODEs: Identification and Interventions.

Bayesian polynomial neural networks and polynomial neural ordinary differential equations

Probabilistic solution of chaotic dynamical system inverse problems using Bayesian Artificial Neural Networks

Infinitely Deep Bayesian Neural Networks with Stochastic Differential Equations

Neural ordinary differential equations with irregular and noisy data

Inferring the Langevin Equation with Uncertainty via Bayesian Neural Networks

Efficient Amortised Bayesian Inference for Hierarchical and Nonlinear Dynamical Systems

Bayesian Numerical Methods for Nonlinear Partial Differential Equations

Neural ODEs with Irregular and Noisy Data

Bayesian Deep Learning Framework for Uncertainty Quantification in Stochastic Partial Differential Equations

Evaluating Uncertainty Quantification approaches for Neural PDEs in scientific applications

Bayesian Physics-Informed Neural Networks for real-world nonlinear dynamical systems

Constrained Neural Ordinary Differential Equations with Stability Guarantees

Learning Neural Event Functions for Ordinary Differential Equations

Distributional Gradient Matching for Learning Uncertain Neural Dynamics Models

Stiff neural ordinary differential equations

Collocation based training of neural ordinary differential equations