Differentiable Programming for Differential Equations: A Review

Facundo Sapienza,Jordi Bolibar,Frank Schäfer,Brian Groenke,Avik Pal,Victor Boussange,Patrick Heimbach,Giles Hooker,Fernando Pérez,Per-Olof Persson,Christopher Rackauckas

2024-06-14

Abstract:The differentiable programming paradigm is a cornerstone of modern scientific computing. It refers to numerical methods for computing the gradient of a numerical model's output. Many scientific models are based on differential equations, where differentiable programming plays a crucial role in calculating model sensitivities, inverting model parameters, and training hybrid models that combine differential equations with data-driven approaches. Furthermore, recognizing the strong synergies between inverse methods and machine learning offers the opportunity to establish a coherent framework applicable to both fields. Differentiating functions based on the numerical solution of differential equations is non-trivial. Numerous methods based on a wide variety of paradigms have been proposed in the literature, each with pros and cons specific to the type of problem investigated. Here, we provide a comprehensive review of existing techniques to compute derivatives of numerical solutions of differential equations. We first discuss the importance of gradients of solutions of differential equations in a variety of scientific domains. Second, we lay out the mathematical foundations of the various approaches and compare them with each other. Third, we cover the computational considerations and explore the solutions available in modern scientific software. Last but not least, we provide best-practices and recommendations for practitioners. We hope that this work accelerates the fusion of scientific models and data, and fosters a modern approach to scientific modelling.

Numerical Analysis,Dynamical Systems,Computational Physics,Machine Learning

What problem does this paper attempt to address?

This paper reviews different differentiable programming methods for computing gradients of numerical solutions to differential equations. The focus of the research is on efficiently calculating the gradients of these models, which is crucial for sensitivity analysis, parameter inversion, mixed model training (combining data-driven methods), and integrating scientific models with data. The paper covers multiple scientific disciplines, including machine learning, computational physics and optimization design, earth sciences, biology, and ecology, and discusses the mathematical foundations of different methods, such as finite differences, automatic differentiation (forward and reverse mode), complex-step differentiation, symbolic differentiation, forward sensitivity equations, and adjoint methods (discrete and continuous form). In addition, the implementation of these methods is explored from a computer science perspective, including direct methods and solver-based methods, as well as their advantages and disadvantages. The paper also discusses how these techniques can be extended to higher-order differential equations, partial differential equations, and chaotic systems, and provides recommendations and conclusions aimed at accelerating the modernization of scientific modeling.