Abstract:Derivatives of differential equation solutions are commonly for parameter estimation, fitting neural differential equations, and as model diagnostics. However, with a litany of choices and a Cartesian product of potential methods, it can be difficult for practitioners to understand which method is likely to be the most effective on their particular application. In this manuscript we investigate the performance characteristics of Discrete Local Sensitivity Analysis implemented via Automatic Differentiation (DSAAD) against continuous adjoint sensitivity analysis. Non-stiff and stiff biological and pharmacometric models, including a PDE discretization, are used to quantify the performance of sensitivity analysis methods. Our benchmarks show that on small stiff and non-stiff systems of ODEs (approximately < 100 parameters+ODEs), forward-mode DSAAD is more efficient than both reverse-mode and continuous forward/adjoint sensitivity analysis. The scalability of continuous adjoint methods is shown to be more efficient than discrete adjoints and forward methods after crossing this size range. These comparative studies demonstrate a trade-off between memory usage and performance in the continuous adjoint methods that should be considered when choosing the technique, while numerically unstable backsolve techniques from the machine learning literature are demonstrated as unsuitable for most scientific models. The performance of adjoint methods is shown to be heavily tied to the reverse-mode AD method used for the vector-Jacobian product calculations, with tape-based AD methods shown to be 2 orders of magnitude slower on nonlinear partial differential equations than static AD techniques. In addition, these results demonstrate the out-of-the-box applicability of DSAAD to differential-algebraic equations, delay differential equations, and hybrid differential equation systems where the event timing and effects are dependent on model parameters, showcasing an ease of implementation advantage for DSAAD approaches. Together, these benchmarks provide a guide to help practitioners to quickly identify the best mixture of continuous sensitivities and automatic differentiation for their applications.

Application of Automatic Differentiation in Power System Trajectory Sensitivity Analysis

Wind Power System State Estimation with Automatic Differentiation Technique

An Automatic Differentiation Based Interior-point Method for Optimal Power Flow

Efficient method based on adjoint sensitivity analysis for transient stability constrained optimal power flow

Application of Automatic Differentiation Technique to Transient Simulation of Distributed Generation System

An Efficient Implementation of Automatic Differentiation in Interior Point Optimal Power Flow

A Fast Dynamic Optimization Approach for Transient Stability Emergency Control

Calculating sensitivity or gradient for geophysical inverse problems using automatic and implicit differentiation

Computing the High Order Derivatives with Automatic Differentiation and Its Application in Chebyshev's Method

Fast Emergency Control Strategy Calculation Based on Dynamic Equivalence and Integral Sensitivity

Analytic Deep Learning and Stepwise Integrated Gradients-based Power System Transient Stability Preventive Control

Complexity Analysis for Hybrid Differentiation in Process System Optimization

Fast Automatic Differentiation and Its Application to Flight Vehicle Parameter Optimization

Analysis and Application of Power Control Sensitivity Based on Trajectory Eigenvalues

An Efficient Parallel Sequential Approach for Transient Stability Emergency Control of Large-Scale Power System

A Transient Stability Numerical Integration Algorithm for Variable Step Sizes Based on Virtual Input

A Comparison of Automatic Differentiation and Continuous Sensitivity Analysis for Derivatives of Differential Equation Solutions

Adaptive DAE solving algorithm for power system transient simulation via matrix exponential operator

Study of the Automatic Differentiation Method for Calculating Sensitivities in the MDO of Flying Vehicles

Multi-dimensional Derivative Order Controlled Multi-step Taylor Series Method for Transient Stability Calculation

Constant Jacobian matrix and its application to fast trajectory simulation of power systems