Abstract:Derivatives of differential equation solutions are commonly for parameter estimation, fitting neural differential equations, and as model diagnostics. However, with a litany of choices and a Cartesian product of potential methods, it can be difficult for practitioners to understand which method is likely to be the most effective on their particular application. In this manuscript we investigate the performance characteristics of Discrete Local Sensitivity Analysis implemented via Automatic Differentiation (DSAAD) against continuous adjoint sensitivity analysis. Non-stiff and stiff biological and pharmacometric models, including a PDE discretization, are used to quantify the performance of sensitivity analysis methods. Our benchmarks show that on small stiff and non-stiff systems of ODEs (approximately < 100 parameters+ODEs), forward-mode DSAAD is more efficient than both reverse-mode and continuous forward/adjoint sensitivity analysis. The scalability of continuous adjoint methods is shown to be more efficient than discrete adjoints and forward methods after crossing this size range. These comparative studies demonstrate a trade-off between memory usage and performance in the continuous adjoint methods that should be considered when choosing the technique, while numerically unstable backsolve techniques from the machine learning literature are demonstrated as unsuitable for most scientific models. The performance of adjoint methods is shown to be heavily tied to the reverse-mode AD method used for the vector-Jacobian product calculations, with tape-based AD methods shown to be 2 orders of magnitude slower on nonlinear partial differential equations than static AD techniques. In addition, these results demonstrate the out-of-the-box applicability of DSAAD to differential-algebraic equations, delay differential equations, and hybrid differential equation systems where the event timing and effects are dependent on model parameters, showcasing an ease of implementation advantage for DSAAD approaches. Together, these benchmarks provide a guide to help practitioners to quickly identify the best mixture of continuous sensitivities and automatic differentiation for their applications.

Numerical Differentiation in the Measurement Model

Numerical Differentiation and Its Applications

Equivalent linear analysis and optimization of tracking differentiator

Automatic numerical differentiation by maximum likelihood estimation of state-space model

Numerical Differentiation from A Viewpoint of Regularization Theory

Measurement System Based on Nonrecursive Filters with the Optimal Correction of the Dynamic Measurement Error

Differentiation of Discrete Data with Unequal Measurement Intervals and Quantification of Uncertainty in Differentiation Using Bayesian Compressive Sampling

A Comparison of Automatic Differentiation and Continuous Sensitivity Analysis for Derivatives of Differential Equation Solutions

Evaluation of the Degree of Rate Control via Automatic Differentiation

Numerical Resolution of Differential Equations Using the Finite Difference Method in the Real and Complex Domain

Toward state estimation by high gain differentiators with automatic differentiation

Three Approximations of Numerical Solution's by Finite Element Method for Resolving Space-Time Partial Differential Equations Involving Fractional Derivative's Order

Numerical simulation of fractional-order reaction–diffusion equations with the Riesz and Caputo derivatives

Formulas for Numerical Differentiation on a Uniform Mesh in the Presence of a Boundary Layer

Numerical Recovery of the Diffusion Coefficient in Diffusion Equations from Terminal Measurement

First and Second Order Numerical Differentiation with Tikhonov Regularization

Real-time numerical differentiation of sampled data using adaptive input and state estimation

Construction Of Difference Schemes For Nonlinear Singular Perturbed Equations By Approximation Of Coefficients

Numerical Differentiation for the Second Order Derivatives of Functions of Two Variables

A NEW NUMERICAL DIFFERENTIATION ALGORITHM WITH REGULARIZATION