Abstract:Unrolling training trajectories over time strongly influences the inference accuracy of neural network-augmented physics simulators. We analyze this in three variants of training neural time-steppers. In addition to one-step setups and fully differentiable unrolling, we include a third, less widely used variant: unrolling without temporal gradients. Comparing networks trained with these three modalities disentangles the two dominant effects of unrolling, training distribution shift and long-term gradients. We present detailed study across physical systems, network sizes and architectures, training setups, and test scenarios. It also encompasses two simulation modes: In prediction setups, we rely solely on neural networks to compute a trajectory. In contrast, correction setups include a numerical solver that is supported by a neural network. Spanning these variations, our study provides the empirical basis for our main findings: Non-differentiable but unrolled training with a numerical solver in a correction setup can yield substantial improvements over a fully differentiable prediction setup not utilizing this solver. The accuracy of models trained in a fully differentiable setup differs compared to their non-differentiable counterparts. Differentiable ones perform best in a comparison among correction networks as well as among prediction setups. For both, the accuracy of non-differentiable unrolling comes close. Furthermore, we show that these behaviors are invariant to the physical system, the network architecture and size, and the numerical scheme. These results motivate integrating non-differentiable numerical simulators into training setups even if full differentiability is unavailable. We show the convergence rate of common architectures to be low compared to numerical algorithms. This motivates correction setups combining neural and numerical parts which utilize benefits of both.

On instabilities in neural network-based physics simulators

On the Stability and Convergence of Physics Informed Neural Networks

Some Results on Neural Network Stability, Consistency, and Convergence: Insights into Non-IID Data, High-Dimensional Settings, and Physics-Informed Neural Networks

An Analysis of Physics-Informed Neural Networks

Transient Stability Analysis with Physics-Informed Neural Networks

Towards Stability of Autoregressive Neural Operators

Dynamical stability and chaos in artificial neural network trajectories along training

Characterizing possible failure modes in physics-informed neural networks

Physics guided neural networks for modelling of non-linear dynamics

Inverse-Dirichlet Weighting Enables Reliable Training of Physics Informed Neural Networks

Physically Consistent Neural ODEs for Learning Multi-Physics Systems

On the weight dynamics of learning networks

Evaluating Error Bound for Physics-Informed Neural Networks on Linear Dynamical Systems

OnsagerNet: Learning Stable and Interpretable Dynamics Using a Generalized Onsager Principle

Bayesian Physics-Informed Neural Networks for real-world nonlinear dynamical systems

Equivariant Neural Simulators for Stochastic Spatiotemporal Dynamics

Differentiability in Unrolled Training of Neural Physics Simulators on Transient Dynamics

Adversarial uncertainty quantification in physics-informed neural networks

Physics-informed Neural Networks for Encoding Dynamics in Real Physical Systems

The Challenges of the Nonlinear Regime for Physics-Informed Neural Networks