Abstract:Derivatives of differential equation solutions are commonly for parameter estimation, fitting neural differential equations, and as model diagnostics. However, with a litany of choices and a Cartesian product of potential methods, it can be difficult for practitioners to understand which method is likely to be the most effective on their particular application. In this manuscript we investigate the performance characteristics of Discrete Local Sensitivity Analysis implemented via Automatic Differentiation (DSAAD) against continuous adjoint sensitivity analysis. Non-stiff and stiff biological and pharmacometric models, including a PDE discretization, are used to quantify the performance of sensitivity analysis methods. Our benchmarks show that on small stiff and non-stiff systems of ODEs (approximately < 100 parameters+ODEs), forward-mode DSAAD is more efficient than both reverse-mode and continuous forward/adjoint sensitivity analysis. The scalability of continuous adjoint methods is shown to be more efficient than discrete adjoints and forward methods after crossing this size range. These comparative studies demonstrate a trade-off between memory usage and performance in the continuous adjoint methods that should be considered when choosing the technique, while numerically unstable backsolve techniques from the machine learning literature are demonstrated as unsuitable for most scientific models. The performance of adjoint methods is shown to be heavily tied to the reverse-mode AD method used for the vector-Jacobian product calculations, with tape-based AD methods shown to be 2 orders of magnitude slower on nonlinear partial differential equations than static AD techniques. In addition, these results demonstrate the out-of-the-box applicability of DSAAD to differential-algebraic equations, delay differential equations, and hybrid differential equation systems where the event timing and effects are dependent on model parameters, showcasing an ease of implementation advantage for DSAAD approaches. Together, these benchmarks provide a guide to help practitioners to quickly identify the best mixture of continuous sensitivities and automatic differentiation for their applications.

A C++ implementation of the discrete adjoint sensitivity analysis method for explicit adaptive Runge-Kutta methods enabled by automatic adjoint differentiation and SIMD vectorization

A Comparison of Automatic Differentiation and Continuous Sensitivity Analysis for Derivatives of Differential Equation Solutions

A Novel Analytical Second‐order Sensitivity Calculation Approach Using the Finite Element Method for Chemical Engineering Problems

A discrete adjoint gradient approach for equality and inequality constraints in dynamics

On improving the efficiency of ADER methods

Level set topology optimization with sparse automatic differentiation

A Comparative Study of Sensitivity Computations in ESDIRK-Based Optimal Control Problems

An Efficient ADER Discontinuous Galerkin Scheme for Directly Solving Hamilton-Jacobi Equation

A novel reduced basis method for adjoint sensitivity analysis of dynamic topology optimization

Analysis for Implicit and Implicit-Explicit ADER and DeC Methods for Ordinary Differential Equations, Advection-Diffusion and Advection-Dispersion Equations

Application of Automatic Differentiation in Power System Trajectory Sensitivity Analysis

Efficient computation of adjoint sensitivities at steady-state in ODE models of biochemical reaction networks

Time-vectorized numerical integration for systems of ODEs

Adjoint-Based Design Optimization of Stability Constrained Systems

Batched Second-Order Adjoint Sensitivity for Reduced Space Methods

Improving the Efficiency of Gradient Descent Algorithms Applied to Optimization Problems with Dynamical Constraints

Sensitivity analysis of differential-algebraic equations and partial differential equations

High order ADER-DG method with local DG predictor for solutions of differential-algebraic systems of equations

Stability-Optimized High Order Methods and Stiffness Detection for Pathwise Stiff Stochastic Differential Equations

PETSc/TS: A Modern Scalable ODE/DAE Solver Library