Abstract:Democratization of machine learning requires architectures that automatically adapt to new problems. Neural Differential Equations (NDEs) have emerged as a popular modeling framework by removing the need for ML practitioners to choose the number of layers in a recurrent model. While we can control the computational cost by choosing the number of layers in standard architectures, in NDEs the number of neural network evaluations for a forward pass can depend on the number of steps of the adaptive ODE solver. But, can we force the NDE to learn the version with the least steps while not increasing the training cost? Current strategies to overcome slow prediction require high order automatic differentiation, leading to significantly higher training time. We describe a novel regularization method that uses the internal cost heuristics of adaptive differential equation solvers combined with discrete adjoint sensitivities to guide the training process towards learning NDEs that are easier to solve. This approach opens up the blackbox numerical analysis behind the differential equation solver's algorithm and directly uses its local error estimates and stiffness heuristics as cheap and accurate cost estimates. We incorporate our method without any change in the underlying NDE framework and show that our method extends beyond Ordinary Differential Equations to accommodate Neural Stochastic Differential Equations. We demonstrate how our approach can halve the prediction time and, unlike other methods which can increase the training time by an order of magnitude, we demonstrate similar reduction in training times. Together this showcases how the knowledge embedded within state-of-the-art equation solvers can be used to enhance machine learning.

Accelerating Neural ODEs Using Model Order Reduction

Neural Ordinary Differential Equations for Model Order Reduction of Stiff Systems

Balanced Neural ODEs: nonlinear model order reduction and Koopman operator approximations

Non-intrusive model reduction of large-scale, nonlinear dynamical systems using deep learning

Opening the Blackbox: Accelerating Neural Differential Equations by Regularizing Internal Solver Heuristics

Understanding Latent Timescales in Neural Ordinary Differential Equation Models for Advection-Dominated Dynamical Systems

Time Dependence in Non-Autonomous Neural ODEs

Efficient, Accurate and Stable Gradients for Neural ODEs

How to train your neural ODE: the world of Jacobian and kinetic regularization

Accelerating Neural-ODE Inference on FPGAs with Two-Stage Structured Pruning and History-based Stepsize Search.

Adaptive Feedforward Gradient Estimation in Neural ODEs

Proximal Implicit ODE Solvers for Accelerating Learning Neural ODEs

On Tuning Neural ODE for Stability, Consistency and Faster Convergence

On the balance between the training time and interpretability of neural ODE for time series modelling

Neural Flows: Efficient Alternative to Neural ODEs

Second-Order Neural ODE Optimizer

Enhancing Representation Power of Deep Neural Networks with Negligible Parameter Growth for Industrial Applications

Implicit regularization of deep residual networks towards neural ODEs

Neural Ordinary Differential Equations for Data-Driven Reduced Order Modeling of Environmental Hydrodynamics

Do Residual Neural Networks discretize Neural Ordinary Differential Equations?

Reduced-order modeling for parameterized PDEs via implicit neural representations