Abstract:Automatic differentiation (autodiff) has revolutionized machine learning. It allows to express complex computations by composing elementary ones in creative ways and removes the burden of computing their derivatives by hand. More recently, differentiation of optimization problem solutions has attracted widespread attention with applications such as optimization layers, and in bi-level problems such as hyper-parameter optimization and meta-learning. However, so far, implicit differentiation remained difficult to use for practitioners, as it often required case-by-case tedious mathematical derivations and implementations. In this paper, we propose automatic implicit differentiation, an efficient and modular approach for implicit differentiation of optimization problems. In our approach, the user defines directly in Python a function $F$ capturing the optimality conditions of the problem to be differentiated. Once this is done, we leverage autodiff of $F$ and the implicit function theorem to automatically differentiate the optimization problem. Our approach thus combines the benefits of implicit differentiation and autodiff. It is efficient as it can be added on top of any state-of-the-art solver and modular as the optimality condition specification is decoupled from the implicit differentiation mechanism. We show that seemingly simple principles allow to recover many existing implicit differentiation methods and create new ones easily. We demonstrate the ease of formulating and solving bi-level optimization problems using our framework. We also showcase an application to the sensitivity analysis of molecular dynamics.

Implicit discrete-time implementation of robust exact differentiators – a toolbox

Proper Implicit Discretization of Arbitrary-Order Robust Exact Differentiators

Optimal robust exact first-order differentiators with Lipschitz continuous output

Equivalent linear analysis and optimization of tracking differentiator

AlgDiff: An open source toolbox for the design, analysis and discretisation of algebraic differentiators

Robust Commensurate Fractional Differentiators for a Class of Fractional Order Linear Systems

Design and Implementation of Fractional Differentiators, Part II: Signal Based Methods

A note about high-order semi-implicit differentiation: application to a numerical integration scheme with Taylor-based compensated error

Efficient and Modular Implicit Differentiation

Identifiability Implies Robust, Globally Exponentially Convergent On-line Parameter Estimation: Application to Model Reference Adaptive Control

Design and Implementation of Fractional Differentiators, Part I: System Based Methods

Designing predefined-time differentiators with bounded time-varying gains

Adaptive Real-Time Numerical Differentiation with Variable-Rate Forgetting and Exponential Resetting

A Discrete-Time Equivalent-Input-Disturbance Approach Based on Joint Minimum-Variance Unbiased Disturbance and State Estimation

Modulating functions based model-free fractional order differentiators using a sliding integration window

An Efficient IMEX Compact Scheme for the Coupled Time Fractional Integro-Differential Equations Arising from Option Pricing with Jumps

Robust implementable regulator design of linear systems with non-vanishing measurements

An arbitrary-order predefined-time exact differentiator for signals with exponential growth bound

Rapid-convergent nonlinear differentiator

A new method for converting impulsive Riemann–Liouville fractional order system into the integral equation

Robust Implicit Adaptive Low Rank Time-Stepping Methods for Matrix Differential Equations