Abstract:Optimization is at the heart of machine learning, statistics, and many applied scientific disciplines. It also has a long history in physics, ranging from the minimal action principle to finding ground states of disordered systems such as spin glasses. Proximal algorithms form a class of methods that are broadly applicable and are particularly well-suited to nonsmooth, constrained, large-scale, and distributed optimization problems. There are essentially five proximal algorithms currently known, each proposed in seminal work: Forward-backward splitting, Tseng splitting, Douglas-Rachford, alternating direction method of multipliers, and the more recent Davis-Yin. These methods sit on a higher level of abstraction compared to gradient-based ones, with deep roots in nonlinear functional analysis. In this paper we show that all of these methods are actually different discretizations of a single differential equation, namely, the simple gradient flow which dates back to Cauchy (1847). An important aspect behind many of the success stories in machine learning relies on "accelerating" the convergence of first-order methods. However, accelerated methods are notoriously difficult to analyze, counterintuitive, and without an underlying guiding principle. We show that similar discretization schemes applied to Newton's equation with an additional dissipative force, which we refer to as accelerated gradient flow, allow us to obtain accelerated variants of all these proximal algorithms-the majority of which are new although some recover known cases in the literature. Furthermore, we extend these methods to stochastic settings, allowing us to make connections with Langevin and Fokker-Planck equations. Similar ideas apply to gradient descent, heavy ball, and Nesterov's method which are simpler. Our results therefore provide a unified framework from which several important optimization methods are nothing but simulations of classical dissipative systems.

Existence of standing pulse solutions to a skew-gradient system

Boundary Stabilization of a Class of Reaction–advection–diffusion Systems Via a Gradient-Based Optimization Approach

Structure preserving integration and model order reduction of skew-gradient reaction-diffusion systems

Index theory for traveling waves in reaction diffusion systems with skew gradient structure

Small-Disturbance Input-to-State Stability of Perturbed Gradient Flows: Applications to LQR Problem

Pulse solutions for an extended Klausmeier model with spatially varying coefficients

Adaptive Non-reversible Stochastic Gradient Langevin Dynamics

Emergence of non-trivial solutions from trivial solutions in reaction–diffusion equations for pattern formation

Ground states of a class of gradient systems of Choquard type with general nonlinearity

Existence of Solutions for Nonlinear Impulsive Qk -Difference Equations with First-Order Qk -Derivatives

Non-equilibrium steady states as saddle points and EDP-convergence for slow-fast gradient systems

Existence and Stability of Standing Pulses in Neural Networks : I Existence

Dynamical behavior of a class of nonsmooth gradient-like systems

Stable asymmetric spike equilibria for the Gierer–Meinhardt model with a precursor field

Slow localized patterns in singularly perturbed 2-component reaction-diffusion equations

Gradient flows and proximal splitting methods: A unified view on accelerated and stochastic optimization

Accelerated Gradient and Skew-Symmetric Splitting Methods for a Class of Monotone Operator Equations

Pattern Formations in Nonlinear Reaction-Diffusion Systems with Strong Localized Impurities

Multi-spike solutions of a hybrid reaction-transport model

Unfolding a Hopf bifurcation in a linear reaction-diffusion equation with strongly localized impurity existence of breathing pulses