Abstract:Democratization of machine learning requires architectures that automatically adapt to new problems. Neural Differential Equations (NDEs) have emerged as a popular modeling framework by removing the need for ML practitioners to choose the number of layers in a recurrent model. While we can control the computational cost by choosing the number of layers in standard architectures, in NDEs the number of neural network evaluations for a forward pass can depend on the number of steps of the adaptive ODE solver. But, can we force the NDE to learn the version with the least steps while not increasing the training cost? Current strategies to overcome slow prediction require high order automatic differentiation, leading to significantly higher training time. We describe a novel regularization method that uses the internal cost heuristics of adaptive differential equation solvers combined with discrete adjoint sensitivities to guide the training process towards learning NDEs that are easier to solve. This approach opens up the blackbox numerical analysis behind the differential equation solver's algorithm and directly uses its local error estimates and stiffness heuristics as cheap and accurate cost estimates. We incorporate our method without any change in the underlying NDE framework and show that our method extends beyond Ordinary Differential Equations to accommodate Neural Stochastic Differential Equations. We demonstrate how our approach can halve the prediction time and, unlike other methods which can increase the training time by an order of magnitude, we demonstrate similar reduction in training times. Together this showcases how the knowledge embedded within state-of-the-art equation solvers can be used to enhance machine learning.

Adaptive Feedforward Gradient Estimation in Neural ODEs

Efficient, Accurate and Stable Gradients for Neural ODEs

ODEN: A Framework to Solve Ordinary Differential Equations using Artificial Neural Networks

An optimal control framework for adaptive neural ODEs

Neural Jump Ordinary Differential Equations: Consistent Continuous-Time Prediction and Filtering

Faster Training of Neural ODEs Using Gauß-Legendre Quadrature

Beyond Predictions in Neural ODEs: Identification and Interventions.

Distributional Gradient Matching for Learning Uncertain Neural Dynamics Models

Second-Order Neural ODE Optimizer

Neural Flows: Efficient Alternative to Neural ODEs

Feedback Favors the Generalization of Neural ODEs

Accelerating Neural ODEs Using Model Order Reduction

How to train your neural ODE: the world of Jacobian and kinetic regularization

Do Residual Neural Networks discretize Neural Ordinary Differential Equations?

Nonparametric Filtering, Estimation and Classification using Neural Jump ODEs

Symplectic Adjoint Method for Exact Gradient of Neural ODE with Minimal Memory

Opening the Blackbox: Accelerating Neural Differential Equations by Regularizing Internal Solver Heuristics

A Self-Adaptive Penalty Method for Integrating Prior Knowledge Constraints into Neural ODEs

Efficient optimization of ODE neuron models using gradient descent

A feedforward neural network framework for approximating the solutions to nonlinear ordinary differential equations

A memory-efficient neural ODE framework based on high-level adjoint differentiation