Abstract:Automatic differentiation (AD) is a technique for augmenting computer programs to compute derivatives. The essence of AD in its forward accumulation mode is to attach perturbations to each number, and propagate these through the computation by overloading the arithmetic operators. When derivatives are nested, the distinct derivative calculations, and their associated perturbations, must be distinguished. This is typically accomplished by creating a unique tag for each derivative calculation and tagging the perturbations. We exhibit a subtle bug, present in fielded implementations which support derivatives of higher-order functions, in which perturbations are confused despite the tagging machinery, leading to incorrect results. The essence of the bug is as follows: a unique tag is needed for each derivative calculation, but in existing implementations unique tags are created when taking the derivative of a function at a point. When taking derivatives of higher-order functions, these need not correspond! We exhibit a simple example: a higher-order function f whose derivative at a point x , namely f ′( x ), is itself a function which calculates a derivative. This situation arises naturally when taking derivatives of curried functions. Two potential solutions are presented, and their deficiencies discussed. One uses eta expansion to delay the creation of fresh tags in order to put them into one-to-one correspondence with derivative calculations. The other wraps outputs of derivative operators with tag substitution machinery. Both solutions seem very difficult to implement without violating the desirable complexity guarantees of forward AD.

Automatic Differentiation via Effects and Handlers: An Implementation in Frank

AbstractDifferentiation.jl: Backend-Agnostic Differentiable Programming in Julia

Verifying an Effect-Handler-Based Define-By-Run Reverse-Mode AD Library

Do be do be do

Automatic Differentiation in Prolog

Forward- or Reverse-Mode Automatic Differentiation: What's the Difference?

Composing Automatic Differentiation with Custom Derivatives of Higher-Order Functions

Using Rewrite Strategies for Efficient Functional Automatic Differentiation

Understanding Automatic Differentiation Pitfalls

Forward-Mode Automatic Differentiation of Compiled Programs

Automatic Differentiation for Solid Mechanics

FastAD: Expression Template-Based C++ Library for Fast and Memory-Efficient Automatic Differentiation

Automatic differentiation in machine learning: a survey

Correctness of Automatic Differentiation via Diffeologies and Categorical Gluing

Mini-symposium on automatic differentiation and its applications in the financial industry

Automatic Differentiation for ML-family languages: correctness via logical relations

Perturbation confusion in forward automatic differentiation of higher-order functions

Accelerated Inference for Partially Observed Markov Processes using Automatic Differentiation

An Automatic Differentiation Based Interior-point Method for Optimal Power Flow

AD-HOC: A C++ Expression Template package for high-order derivatives backpropagation

Applying Extended Automatic Differentiation Technique to Process System Optimization Problems