Automatic Differentiation: Theory and Practice

Mario Lezcano-Casado

DOI: https://doi.org/10.48550/arXiv.2207.06114

IF: 5.414

2022-07-13

Machine Learning

Abstract:We present the classical coordinate-free formalism for forward and backward mode ad in the real and complex setting. We show how to formally derive the forward and backward formulae for a number of matrix functions starting from basic principles.

What problem does this paper attempt to address?

Forward- or Reverse-Mode Automatic Differentiation: What's the Difference?

Birthe van den Berg,Tom Schrijvers,James McKinna,Alexander Vandenbroucke

2023-08-09

Abstract:Automatic differentiation (AD) has been a topic of interest for researchers in many disciplines, with increased popularity since its application to machine learning and neural networks. Although many researchers appreciate and know how to apply AD, it remains a challenge to truly understand the underlying processes. From an algebraic point of view, however, AD appears surprisingly natural: it originates from the differentiation laws. In this work we use Algebra of Programming techniques to reason about different AD variants, leveraging Haskell to illustrate our observations. Our findings stem from three fundamental algebraic abstractions: (1) the notion of module over a semiring, (2) Nagata's construction of the 'idealization of a module', and (3) Kronecker's delta function, that together allow us to write a single-line abstract definition of AD. From this single-line definition, and by instantiating our algebraic structures in various ways, we derive different AD variants, that have the same extensional behaviour, but different intensional properties, mainly in terms of (asymptotic) computational complexity. We show the different variants equivalent by means of Kronecker isomorphisms, a further elaboration of our Haskell infrastructure which guarantees correctness by construction. With this framework in place, this paper seeks to make AD variants more comprehensible, taking an algebraic perspective on the matter.

Programming Languages
An Automatic Adjoint Theorem and Its Applications

JD Wu,SJ Lu

DOI: https://doi.org/10.1090/s0002-9939-01-06285-2

IF: 0.9709

2001-01-01

Proceedings of the American Mathematical Society

Abstract:In this paper, we prove the following automatic adjoint theorem: For any sequence spaces E ( X ) E(X) and F ( Y ) F(Y) , if E ( X ) E(X) has the signed-weak gliding hump property and A A is an infinite matrix which transforms E ( X ) E(X) into F ( Y ) F(Y) , then the transpose matrix A ′ A’ of A A transforms F ( Y ) β F(Y)^\beta into E ( X ) β E(X)^\beta , and for any x ∈ E ( X ) x\in E(X) and T ∈ F ( Y ) β T\in F(Y)^\beta , [ A x , T ] = [ x , A ′ T ] [Ax,T]=[x,A’T] . That is, the adjoint operator of A A automatically exists and is just the transpose matrix A ′ A’ of A A . From the theorem we obtain a class of infinite matrix topological algebras ( λ , μ ) (\lambda ,\mu ) , and prove also a λ \lambda -multiplier convergence theorem of Orlicz-Pettis type. The theorem improves substantially the famous Stiles’ Orlicz-Pettis theorem.
Composing Automatic Differentiation with Custom Derivatives of Higher-Order Functions

Sam Estep

2024-08-15

Abstract:Recent theoretical work on automatic differentiation (autodiff) has focused on characteristics such as correctness and efficiency while assuming that all derivatives are automatically generated by autodiff using program transformation, with the exception of a fixed set of derivatives for primitive operations. However, in practice this assumption is insufficient: the programmer often needs to provide custom derivatives for composite functions to achieve efficiency and numerical stability. In this work, we start from the untyped lambda calculus with a reverse-mode autodiff operator, extend it with an operator to attach manual derivatives, and demonstrate its utility via several examples.

Programming Languages
Automatic Differentiation in Prolog

Tom Schrijvers,Birthe van den Berg,Fabrizio Riguzzi

DOI: https://doi.org/10.48550/arXiv.2305.07878

2023-05-13

Abstract:Automatic differentiation (AD) is a range of algorithms to compute the numeric value of a function's (partial) derivative, where the function is typically given as a computer program or abstract syntax tree. AD has become immensely popular as part of many learning algorithms, notably for neural networks. This paper uses Prolog to systematically derive gradient-based forward- and reverse-mode AD variants from a simple executable specification: evaluation of the symbolic derivative. Along the way we demonstrate that several Prolog features (DCGs, co-routines) contribute to the succinct formulation of the algorithm. We also discuss two applications in probabilistic programming that are enabled by our Prolog algorithms. The first is parameter learning for the Sum-Product Loop Language and the second consists of both parameter learning and variational inference for probabilistic logic programming.

Programming Languages
Correctness of Automatic Differentiation via Diffeologies and Categorical Gluing

Mathieu Huot,Sam Staton,Matthijs Vákár

DOI: https://doi.org/10.48550/arXiv.2001.02209

2020-04-01

Abstract:We present semantic correctness proofs of Automatic Differentiation (AD). We consider a forward-mode AD method on a higher order language with algebraic data types, and we characterise it as the unique structure preserving macro given a choice of derivatives for basic operations. We describe a rich semantics for differentiable programming, based on diffeological spaces. We show that it interprets our language, and we phrase what it means for the AD method to be correct with respect to this semantics. We show that our characterisation of AD gives rise to an elegant semantic proof of its correctness based on a gluing construction on diffeological spaces. We explain how this is, in essence, a logical relations argument. Finally, we sketch how the analysis extends to other AD methods by considering a continuation-based method.

Programming Languages,Logic in Computer Science
A tutorial on automatic differentiation with complex numbers

Nicholas Krämer

2024-10-12

Abstract:Automatic differentiation is everywhere, but there exists only minimal documentation of how it works in complex arithmetic beyond stating "derivatives in $\mathbb{C}^d$" $\cong$ "derivatives in $\mathbb{R}^{2d}$" and, at best, shallow references to Wirtinger calculus. Unfortunately, the equivalence $\mathbb{C}^d \cong \mathbb{R}^{2d}$ becomes insufficient as soon as we need to derive custom gradient rules, e.g., to avoid differentiating "through" expensive linear algebra functions or differential equation simulators. To combat such a lack of documentation, this article surveys forward- and reverse-mode automatic differentiation with complex numbers, covering topics such as Wirtinger derivatives, a modified chain rule, and different gradient conventions while explicitly avoiding holomorphicity and the Cauchy--Riemann equations (which would be far too restrictive). To be precise, we will derive, explain, and implement a complex version of Jacobian-vector and vector-Jacobian products almost entirely with linear algebra without relying on complex analysis or differential geometry. This tutorial is a call to action, for users and developers alike, to take complex values seriously when implementing custom gradient propagation rules -- the manuscript explains how.

Mathematical Software,Machine Learning,Numerical Analysis
Automatic Differentiation for Solid Mechanics

Andrea Vigliotti,Ferdinando Auricchio

DOI: https://doi.org/10.1007/s11831-019-09396-y

IF: 9.7

2020-01-20

Archives of Computational Methods in Engineering

Abstract:Automatic differentiation (AD) is an ensemble of techniques that allows to evaluate accurate numerical derivatives of a mathematical function expressed in a computer programming language. In this paper we use AD for stating and solving solid mechanics problems. Given a finite element discretization of the domain, we evaluate the free energy of the solid as the integral of its strain energy density, and we make use of AD for directly obtaining the residual force vector and the tangent stiffness matrix of the problem, as the gradient and the Hessian of the free energy respectively. The result is a remarkable simplification in the statement and the solution of complex problems involving non trivial constraints systems and both geometrical and material non linearities. Together with the continuum mechanics theoretical basis, and with a description of the specific AD technique adopted, the paper illustrates the solution of a number of solid mechanics problems, with the aim of presenting a convenient numerical implementation approach, made easily available by recent programming languages, to the solid mechanics community.

computer science, interdisciplinary applications,engineering, multidisciplinary,mathematics
Automatic Differentiation for ML-family languages: correctness via logical relations

Fernando Lucatelli Nunes,Matthijs Vákár

2024-06-14

Abstract:We give a simple, direct and reusable logical relations technique for languages with term and type recursion and partially defined differentiable functions. We demonstrate it by working out the case of Automatic Differentiation (AD) correctness: namely, we present a correctness proof of a dual numbers style AD code transformation for realistic functional languages in the ML-family. We also show how this code transformation provides us with correct forward- and reverse-mode AD. The starting point is to interpret a functional programming language as a suitable freely generated categorical structure. In this setting, by the universal property of the syntactic categorical structure, the dual numbers AD code transformation and the basic $\omega$-cpo semantics arise as structure preserving functors. The proof follows, then, by a novel logical relations argument. The key to much of our contribution is a powerful monadic logical relations technique for term recursion and recursive types. It provides us with a semantic correctness proof based on a simple approach for denotational semantics, making use only of the very basic concrete model of $\omega$-cpos.

Programming Languages
ON AUTOMATIC DIFFERENTIATION AND ALGORITHMIC LINEARIZATION

Andreas Griewank

DOI: https://doi.org/10.1590/0101-7438.2014.034.03.0621

2014-12-01

Pesquisa Operacional

Abstract:We review the methods and applications of automatic differentiation, a research and development activity, which has evolved in various computational fields since the mid 1950's. Starting from very simple basic principles that are familiar from school, one arrives at various theoretical and practical challenges. The resulting activity encompasses mathematical research and software development; it is now oftenreferredtoas algorithmic differentiation. From a geometrical and algebraic point of view, differentiation amounts to linearization, a concept that naturally extends to infinite dimensional spaces. In contract to other surveys, we will emphasize this interpretation as it has become more important recently and also facilitates the treatment of nonsmooth problems by piecewise linearization.
Automatic differentiation in machine learning: a survey

Atilim Gunes Baydin,Barak A. Pearlmutter,Alexey Andreyevich Radul,Jeffrey Mark Siskind

DOI: https://doi.org/10.48550/arXiv.1502.05767

2018-02-05

Abstract:Derivatives, mostly in the form of gradients and Hessians, are ubiquitous in machine learning. Automatic differentiation (AD), also called algorithmic differentiation or simply "autodiff", is a family of techniques similar to but more general than backpropagation for efficiently and accurately evaluating derivatives of numeric functions expressed as computer programs. AD is a small but established field with applications in areas including computational fluid dynamics, atmospheric sciences, and engineering design optimization. Until very recently, the fields of machine learning and AD have largely been unaware of each other and, in some cases, have independently discovered each other's results. Despite its relevance, general-purpose AD has been missing from the machine learning toolbox, a situation slowly changing with its ongoing adoption under the names "dynamic computational graphs" and "differentiable programming". We survey the intersection of AD and machine learning, cover applications where AD has direct relevance, and address the main implementation techniques. By precisely defining the main differentiation techniques and their interrelationships, we aim to bring clarity to the usage of the terms "autodiff", "automatic differentiation", and "symbolic differentiation" as these are encountered more and more in machine learning settings.

Symbolic Computation,Machine Learning
Automatic differentiation of Sylvester, Lyapunov, and algebraic Riccati equations

Ta-Chu Kao,Guillaume Hennequin

DOI: https://doi.org/10.48550/arXiv.2011.11430

2020-11-23

Optimization and Control

Abstract:Sylvester, Lyapunov, and algebraic Riccati equations are the bread and butter of control theorists. They are used to compute infinite-horizon Gramians, solve optimal control problems in continuous or discrete time, and design observers. While popular numerical computing frameworks (e.g., scipy) provide efficient solvers for these equations, these solvers are still largely missing from most automatic differentiation libraries. Here, we derive the forward and reverse-mode derivatives of the solutions to all three types of equations, and showcase their application on an inverse control problem.
Automatic Differentiation: Inverse Accumulation Mode

Barak A. Pearlmutter,Jeffrey Mark Siskind

2024-11-28

Abstract:We show that, under certain circumstances, it is possible to automatically compute Jacobian-inverse-vector and Jacobian-inverse-transpose-vector products about as efficiently as Jacobian-vector and Jacobian-transpose-vector products. The key insight is to notice that the Jacobian corresponding to the use of one basis function is of a form whose sparsity is invariant to inversion. The main restriction of the method is a constraint on the number of active variables, which suggests a variety of techniques or generalization to allow the constraint to be enforced or relaxed. This technique has the potential to allow the efficient direct calculation of Newton steps as well as other numeric calculations of interest.

Numerical Analysis
Decomposing reverse-mode automatic differentiation

Roy Frostig,Matthew J. Johnson,Dougal Maclaurin,Adam Paszke,Alexey Radul

DOI: https://doi.org/10.48550/arXiv.2105.09469

2021-05-20

Abstract:We decompose reverse-mode automatic differentiation into (forward-mode) linearization followed by transposition. Doing so isolates the essential difference between forward- and reverse-mode AD, and simplifies their joint implementation. In particular, once forward-mode AD rules are defined for every primitive operation in a source language, only linear primitives require an additional transposition rule in order to arrive at a complete reverse-mode AD implementation. This is how reverse-mode AD is written in JAX and Dex.

Programming Languages,Machine Learning
Automating Variational Differentiation

Kangbo Li,Anil Damle

2024-06-24

Abstract:Many problems in Physics and Chemistry are formulated as the minimization of a functional. Therefore, methods for solving these problems typically require differentiating maps whose input and/or output are functions -- commonly referred to as variational differentiation. Such maps are not addressed at the mathematical level by the chain rule, which underlies modern symbolic and algorithmic differentiation (AD) systems. Although there are algorithmic solutions such as tracing and reverse accumulation, they do not provide human readability and introduce strict programming constraints that bottleneck performance, especially in high-performance computing (HPC) environments. In this manuscript, we propose a new computer theoretic model of differentiation by combining the pullback of the $\mathbf{B}$ and $\mathbf{C}$ combinators from the combinatory logic. Unlike frameworks based on the chain rule, this model differentiates a minimal complete basis for the space of computable functions. Consequently, the model is capable of analytic backpropagation and variational differentiation while supporting complex numbers. To demonstrate the generality of this approach we build a system named CombDiff, which can differentiate nontrivial variational problems such as Hartree-Fock (HF) theory and multilayer perceptrons.

Mathematical Software,Programming Languages,Numerical Analysis
ALGORITHMS OF AUTOMATIC DIFFERENTIATION

Cheng Qiang,Zhang Haibin,Wang Bin

DOI: https://doi.org/10.3321/j.issn:0254-7791.2009.01.002

2009-01-01

Abstract:We study fundamentals and techniques of automatic differentiation by discussing the way for calculating different kinds of first-and second-order derivatives.First we present the first-and second-order differentiation models in a simple way,then discuss the costs in terms of operations count and memory for calculating different derivatives with different differentiation models,finally discuss some typical algorithms in solving the nonlinear problems via automatic differentiation with respect to the computational costs,both in operations count and in memory.Several fundamental techniques including checkpointing,forward accumulating and backward accumulating are discussed.
Automatic Differentiable Monte Carlo: Theory and Application

Shi-Xin Zhang,Zhou-Quan Wan,Hong Yao

DOI: https://doi.org/10.1103/physrevresearch.5.033041

2023-01-01

Physical Review Research

Abstract:Differentiable programming has emerged as a key programming paradigmempowering rapid developments of deep learning while its applications toimportant computational methods such as Monte Carlo remain largely unexplored.Here we present the general theory enabling infinite-order automaticdifferentiation on expectations computed by Monte Carlo with unnormalizedprobability distributions, which we call "automatic differentiable Monte Carlo"(ADMC). By implementing ADMC algorithms on computational graphs, one can alsoleverage state-of-the-art machine learning frameworks and techniques totraditional Monte Carlo applications in statistics and physics. We illustratethe versatility of ADMC by showing some applications: fast search of phasetransitions and accurately finding ground states of interacting many-bodymodels in two dimensions. ADMC paves a promising way to innovate Monte Carlo invarious aspects to achieve higher accuracy and efficiency, e.g. easing orsolving the sign problem of quantum many-body models through ADMC.
Understanding Automatic Differentiation Pitfalls

Jan Hückelheim,Harshitha Menon,William Moses,Bruce Christianson,Paul Hovland,Laurent Hascoët

2023-05-12

Abstract:Automatic differentiation, also known as backpropagation, AD, autodiff, or algorithmic differentiation, is a popular technique for computing derivatives of computer programs accurately and efficiently. Sometimes, however, the derivatives computed by AD could be interpreted as incorrect. These pitfalls occur systematically across tools and approaches. In this paper we broadly categorize problematic usages of AD and illustrate each category with examples such as chaos, time-averaged oscillations, discretizations, fixed-point loops, lookup tables, and linear solvers. We also review debugging techniques and their effectiveness in these situations. With this article we hope to help readers avoid unexpected behavior, detect problems more easily when they occur, and have more realistic expectations from AD tools.

Numerical Analysis,Artificial Intelligence,Computational Engineering, Finance, and Science
Forward-Mode Automatic Differentiation of Compiled Programs

Max Aehle,Johannes Blühdorn,Max Sagebaum,Nicolas R. Gauger

2023-07-07

Abstract:Algorithmic differentiation (AD) is a set of techniques that provide partial derivatives of computer-implemented functions. Such a function can be supplied to state-of-the-art AD tools via its source code, or via an intermediate representation produced while compiling its source code. We present the novel AD tool Derivgrind, which augments the machine code of compiled programs with forward-mode AD logic. Derivgrind leverages the Valgrind instrumentation framework for a structured access to the machine code, and a shadow memory tool to store dot values. Access to the source code is required at most for the files in which input and output variables are defined. Derivgrind's versatility comes at the price of scaling the run-time by a factor between 30 and 75, measured on a benchmark based on a numerical solver for a partial differential equation. Results of our extensive regression test suite indicate that Derivgrind produces correct results on GCC- and Clang-compiled programs, including a Python interpreter, with a small number of exceptions. While we provide a list of scenarios that Derivgrind does not handle correctly, nearly all of them are academic counterexamples or originate from highly optimized math libraries. As long as differentiating those is avoided, Derivgrind can be applied to an unprecedentedly wide range of cross-language or partially closed-source software with little integration efforts.

Mathematical Software
Differential Calculus, Tensor Products and the Importance of Notation

Jonathan H. Manton

DOI: https://doi.org/10.48550/arXiv.1208.0197

2013-10-02

Abstract:An efficient coordinate-free notation is elucidated for differentiating matrix expressions and other functions between higher-dimensional vector spaces. This method of differentiation is known, but not explained well, in the literature. Teaching it early in the curriculum would avoid the tedium of element-wise differentiation and provide a better footing for understanding more advanced applications of calculus. Additionally, it is shown to lead naturally to tensor products, a topic previously considered too difficult to motivate quickly in elementary ways.

History and Overview

Automatic Differentiation: Theory and Practice

Forward- or Reverse-Mode Automatic Differentiation: What's the Difference?

An Automatic Adjoint Theorem and Its Applications

Composing Automatic Differentiation with Custom Derivatives of Higher-Order Functions

Automatic Differentiation in Prolog

Correctness of Automatic Differentiation via Diffeologies and Categorical Gluing

A tutorial on automatic differentiation with complex numbers

Automatic Differentiation for Solid Mechanics

Automatic Differentiation for ML-family languages: correctness via logical relations

ON AUTOMATIC DIFFERENTIATION AND ALGORITHMIC LINEARIZATION

Automatic differentiation in machine learning: a survey

Automatic differentiation of Sylvester, Lyapunov, and algebraic Riccati equations

Automatic Differentiation: Inverse Accumulation Mode

Decomposing reverse-mode automatic differentiation

Automating Variational Differentiation

ALGORITHMS OF AUTOMATIC DIFFERENTIATION

Automatic Differentiable Monte Carlo: Theory and Application

Understanding Automatic Differentiation Pitfalls

Forward-Mode Automatic Differentiation of Compiled Programs

Differential Calculus, Tensor Products and the Importance of Notation