Abstract:Derivatives of differential equation solutions are commonly for parameter estimation, fitting neural differential equations, and as model diagnostics. However, with a litany of choices and a Cartesian product of potential methods, it can be difficult for practitioners to understand which method is likely to be the most effective on their particular application. In this manuscript we investigate the performance characteristics of Discrete Local Sensitivity Analysis implemented via Automatic Differentiation (DSAAD) against continuous adjoint sensitivity analysis. Non-stiff and stiff biological and pharmacometric models, including a PDE discretization, are used to quantify the performance of sensitivity analysis methods. Our benchmarks show that on small stiff and non-stiff systems of ODEs (approximately < 100 parameters+ODEs), forward-mode DSAAD is more efficient than both reverse-mode and continuous forward/adjoint sensitivity analysis. The scalability of continuous adjoint methods is shown to be more efficient than discrete adjoints and forward methods after crossing this size range. These comparative studies demonstrate a trade-off between memory usage and performance in the continuous adjoint methods that should be considered when choosing the technique, while numerically unstable backsolve techniques from the machine learning literature are demonstrated as unsuitable for most scientific models. The performance of adjoint methods is shown to be heavily tied to the reverse-mode AD method used for the vector-Jacobian product calculations, with tape-based AD methods shown to be 2 orders of magnitude slower on nonlinear partial differential equations than static AD techniques. In addition, these results demonstrate the out-of-the-box applicability of DSAAD to differential-algebraic equations, delay differential equations, and hybrid differential equation systems where the event timing and effects are dependent on model parameters, showcasing an ease of implementation advantage for DSAAD approaches. Together, these benchmarks provide a guide to help practitioners to quickly identify the best mixture of continuous sensitivities and automatic differentiation for their applications.

A Derivative-Free Variant Called DFSA of Dekkers and Aarts' Continuous Simulated Annealing Algorithm.

Derivative-Free Optimization via Finite Difference Approximation: An Experimental Study

Adaptive State-Dependent Diffusion for Derivative-Free Optimization

A Differential Evolution Algorithm Based on Local Search and Boundary Reflection for Global Optimization.

Combined Strategy of Improved Simulated Annealing and Genetic Algorithm for Inverse Problem

An adaptive sampling algorithm for simulation-based optimization with descriptive complexity constraints

A Simultaneous Strategy For Dynamic Optimization Based On Symbolic Derivation

A Surrogate-Assisted Differential Evolution with fitness-independent parameter adaptation for high-dimensional expensive optimization

An Improved Simulated Annealing Algorithm for the Optimization of Mathematical Functions

Swarm-based gradient descent meets simulated annealing

An Adaptive Sampling Algorithm for Simulation-Based Optimization With Descriptive Complexity Preference

Numerical simulation of differential-algebraic equations with embedded global optimization criteria

A Comparison of Automatic Differentiation and Continuous Sensitivity Analysis for Derivatives of Differential Equation Solutions

A Hybrid Cuckoo Search and Simulated Annealing Algorithm

DSA: Decentralized Double Stochastic Averaging Gradient Algorithm

A sinusoidal differential evolution algorithm for numerical optimisation

Differential simulated annealing: a robust and efficient global optimization algorithm for parameter estimation of biological networks.

An open-source solver for finding global solutions to constrained derivative-free optimization problems

A Two-Stage Ensemble of Differential Evolution Variants for Numerical Optimization

A differential evolution algorithm with a superior-inferior mutation scheme

SFSADE: an improved self-adaptive differential evolution algorithm with a shuffled frog-leaping strategy