Abstract:Automatic differentiation (autodiff) has revolutionized machine learning. It allows to express complex computations by composing elementary ones in creative ways and removes the burden of computing their derivatives by hand. More recently, differentiation of optimization problem solutions has attracted widespread attention with applications such as optimization layers, and in bi-level problems such as hyper-parameter optimization and meta-learning. However, so far, implicit differentiation remained difficult to use for practitioners, as it often required case-by-case tedious mathematical derivations and implementations. In this paper, we propose automatic implicit differentiation, an efficient and modular approach for implicit differentiation of optimization problems. In our approach, the user defines directly in Python a function $F$ capturing the optimality conditions of the problem to be differentiated. Once this is done, we leverage autodiff of $F$ and the implicit function theorem to automatically differentiate the optimization problem. Our approach thus combines the benefits of implicit differentiation and autodiff. It is efficient as it can be added on top of any state-of-the-art solver and modular as the optimality condition specification is decoupled from the implicit differentiation mechanism. We show that seemingly simple principles allow to recover many existing implicit differentiation methods and create new ones easily. We demonstrate the ease of formulating and solving bi-level optimization problems using our framework. We also showcase an application to the sensitivity analysis of molecular dynamics.

ON AUTOMATIC DIFFERENTIATION AND ALGORITHMIC LINEARIZATION

Automatic differentiation in machine learning: a survey

ALGORITHMS OF AUTOMATIC DIFFERENTIATION

A Step-by-step Introduction to the Implementation of Automatic Differentiation

Automatic differentiation and the optimization of differential equation models in biology

Understanding Automatic Differentiation Pitfalls

Automatic Differentiation of Optimization Algorithms with Time-Varying Updates

Differentiating Through Linear Solvers

Applications of Automatic Differentiation in Image Registration

Automating Variational Differentiation

Automatic Differentiation for Solid Mechanics

Distortion-free tight confinement and step-like decay of fs pulses in free space

Composing Automatic Differentiation with Custom Derivatives of Higher-Order Functions

A tutorial on automatic differentiation with complex numbers

Efficient and Modular Implicit Differentiation

Differentiable Programming for Differential Equations: A Review

Mini-symposium on automatic differentiation and its applications in the financial industry

Automatic Differentiation: Theory and Practice

Complexity Analysis for Hybrid Differentiation in Process System Optimization

Automatic differentiation of Sylvester, Lyapunov, and algebraic Riccati equations