Abstract:Optimization is at the heart of machine learning, statistics, and many applied scientific disciplines. It also has a long history in physics, ranging from the minimal action principle to finding ground states of disordered systems such as spin glasses. Proximal algorithms form a class of methods that are broadly applicable and are particularly well-suited to nonsmooth, constrained, large-scale, and distributed optimization problems. There are essentially five proximal algorithms currently known, each proposed in seminal work: Forward-backward splitting, Tseng splitting, Douglas-Rachford, alternating direction method of multipliers, and the more recent Davis-Yin. These methods sit on a higher level of abstraction compared to gradient-based ones, with deep roots in nonlinear functional analysis. In this paper we show that all of these methods are actually different discretizations of a single differential equation, namely, the simple gradient flow which dates back to Cauchy (1847). An important aspect behind many of the success stories in machine learning relies on "accelerating" the convergence of first-order methods. However, accelerated methods are notoriously difficult to analyze, counterintuitive, and without an underlying guiding principle. We show that similar discretization schemes applied to Newton's equation with an additional dissipative force, which we refer to as accelerated gradient flow, allow us to obtain accelerated variants of all these proximal algorithms-the majority of which are new although some recover known cases in the literature. Furthermore, we extend these methods to stochastic settings, allowing us to make connections with Langevin and Fokker-Planck equations. Similar ideas apply to gradient descent, heavy ball, and Nesterov's method which are simpler. Our results therefore provide a unified framework from which several important optimization methods are nothing but simulations of classical dissipative systems.

Gradient-enhanced stochastic optimization of high-fidelity simulations

A Gradient-Based Method Assisted by Surrogate Model for Robust Optimization of Turbomachinery Blades

Secondary Flow Reduction By Blade Redesign And Endwall Contouring Using An Adjoint Optimization Method

Gradient-Based Response Surface Approximations for Design Optimization

Aerodynamic Shape Optimization of a Transonic Fan by an Adjoint-Response Surface Method

Particle Swarm Optimization of Aerodynamic Shapes with Nonuniform Shape Parameter–Based Radial Basis Function

Flow Reconstructions and Aerodynamic Shape Optimization of Turbomachinery Blades by POD-based Hybrid Models

Adjoint-Response Surface Method in Aerodynamic Shape Optimization of Turbomachinery Blades

Rapid Aerodynamic Shape Optimization Under Parametric and Turbulence Model Uncertainty: A Stochastic Gradient Approach

Shape Gradient Methods for Shape Optimization of an Unsteady Multiscale Fluid–Structure Interaction Model

A General Multi-objective Bayesian Optimization Framework for the Design of Hybrid Schemes towards Adaptive Complex Flow Simulations

Progressive augmentation of Reynolds stress tensor models for secondary flow prediction by computational fluid dynamics driven surrogate optimisation

Dynamic Machine Learning Global Optimization Algorithm and Its Application to Aerodynamics

Deterministic Langevin Unconstrained Optimization with Normalizing Flows

Customized data-driven RANS closures for bi-fidelity LES–RANS optimization

Gradient flows and proximal splitting methods: A unified view on accelerated and stochastic optimization

On the Application of Gradient Based Reconstruction for Flow Simulations on Generalized Curvilinear and Dynamic Mesh Domains

A turbulent eddy-viscosity surrogate modeling framework for Reynolds-averaged Navier-Stokes simulations

Predictions of steady and unsteady flows using machine-learned surrogate models

Multidisciplinary optimization of the SRV2-O radial compressor using an adjoint-based approach

An Adjoint Method for a High-Order Discretization of Deforming Domain Conservation Laws for Optimization of Flow Problems