Abstract:Unrolling training trajectories over time strongly influences the inference accuracy of neural network-augmented physics simulators. We analyze this in three variants of training neural time-steppers. In addition to one-step setups and fully differentiable unrolling, we include a third, less widely used variant: unrolling without temporal gradients. Comparing networks trained with these three modalities disentangles the two dominant effects of unrolling, training distribution shift and long-term gradients. We present detailed study across physical systems, network sizes and architectures, training setups, and test scenarios. It also encompasses two simulation modes: In prediction setups, we rely solely on neural networks to compute a trajectory. In contrast, correction setups include a numerical solver that is supported by a neural network. Spanning these variations, our study provides the empirical basis for our main findings: Non-differentiable but unrolled training with a numerical solver in a correction setup can yield substantial improvements over a fully differentiable prediction setup not utilizing this solver. The accuracy of models trained in a fully differentiable setup differs compared to their non-differentiable counterparts. Differentiable ones perform best in a comparison among correction networks as well as among prediction setups. For both, the accuracy of non-differentiable unrolling comes close. Furthermore, we show that these behaviors are invariant to the physical system, the network architecture and size, and the numerical scheme. These results motivate integrating non-differentiable numerical simulators into training setups even if full differentiability is unavailable. We show the convergence rate of common architectures to be low compared to numerical algorithms. This motivates correction setups combining neural and numerical parts which utilize benefits of both.

Correcting auto-differentiation in neural-ODE training

Efficient, Accurate and Stable Gradients for Neural ODEs

Adaptive Feedforward Gradient Estimation in Neural ODEs

ODEN: A Framework to Solve Ordinary Differential Equations using Artificial Neural Networks

Opening the Blackbox: Accelerating Neural Differential Equations by Regularizing Internal Solver Heuristics

Training Stiff Neural Ordinary Differential Equations with Explicit Exponential Integration Methods

Implicit regularization of deep residual networks towards neural ODEs

Understanding Latent Timescales in Neural Ordinary Differential Equation Models for Advection-Dominated Dynamical Systems

On Tuning Neural ODE for Stability, Consistency and Faster Convergence

Feedback Favors the Generalization of Neural ODEs

Symplectic Adjoint Method for Exact Gradient of Neural ODE with Minimal Memory

Faster Training of Neural ODEs Using Gauß-Legendre Quadrature

Near-optimal control of dynamical systems with neural ordinary differential equations

Automatic differentiation and the optimization of differential equation models in biology

Differentiability in Unrolled Training of Neural Physics Simulators on Transient Dynamics

Understanding Automatic Differentiation Pitfalls

Do Residual Neural Networks discretize Neural Ordinary Differential Equations?

Locally Regularized Neural Differential Equations: Some Black Boxes Were Meant to Remain Closed!

Accuracy and Architecture Studies of Residual Neural Network solving Ordinary Differential Equations

The Neural Network Approach to Inverse Problems in Differential Equations

On the Correctness of Automatic Differentiation for Neural Networks with Machine-Representable Parameters