Abstract:This paper settles an open and challenging question pertaining to the design of simple and optimal high-order methods for solving smooth and monotone variational inequalities (VIs). A VI involves finding $x^\star \in \mathcal{X}$ such that $\langle F(x), x - x^\star\rangle \geq 0$ for all $x \in \mathcal{X}$. We consider the setting in which $F$ is smooth with up to $(p-1)^{th}$-order derivatives. For $p = 2$, the cubic regularized Newton method was extended to VIs with a global rate of $O(\epsilon^{-1})$. An improved rate of $O(\epsilon^{-2/3}\log\log(1/\epsilon))$ can be obtained via an alternative second-order method, but this method requires a nontrivial line-search procedure as an inner loop. Similarly, high-order methods based on line-search procedures have been shown to achieve a rate of $O(\epsilon^{-2/(p+1)}\log\log(1/\epsilon))$. As emphasized by Nesterov, however, such procedures do not necessarily imply practical applicability in large-scale applications, and it would be desirable to complement these results with a simple high-order VI method that retains the optimality of the more complex methods. We propose a $p^{th}$-order method that does \textit{not} require any line search procedure and provably converges to a weak solution at a rate of $O(\epsilon^{-2/(p+1)})$. We prove that our $p^{th}$-order method is optimal in the monotone setting by establishing a matching lower bound under a generalized linear span assumption. Our method with restarting attains a linear rate for smooth and uniformly monotone VIs and a local superlinear rate for smooth and strongly monotone VIs. Our method also achieves a global rate of $O(\epsilon^{-2/p})$ for solving smooth and nonmonotone VIs satisfying the Minty condition and when augmented with restarting it attains a global linear and local superlinear rate for smooth and nonmonotone VIs satisfying the uniform/strong Minty condition.

A Variational Perspective on High-Resolution ODEs

Accelerated optimization algorithms and ordinary differential equations: the convex non Euclidean case

Variational Analysis of a Nonconvex and Nonsmooth Optimization Problem: An Introduction

An SDE Perspective on Stochastic Inertial Gradient Dynamics with Time-Dependent Viscosity and Geometric Damping

Comparative analysis of accelerated gradient algorithms for convex optimization: high and super resolution ODE approach

Discrete approximation of the minimizing movement scheme for evolution equations of Wasserstein gradient flow type with nonlinear mobility

Monotone discretizations of levelset convex geometric PDEs

Zeroth-order Gradient and Quasi-Newton Methods for Nonsmooth Nonconvex Stochastic Optimization

Acceleration via Symplectic Discretization of High-Resolution Differential Equations

Non-smooth variational problems and applications

Optimizing Variational Physics-Informed Neural Networks Using Least Squares

Direct Runge-Kutta Discretization Achieves Acceleration.

Variational formulation based on duality to solve partial differential equations: Use of B-splines and machine learning approximants

The stochastic Ravine accelerated gradient method with general extrapolation coefficients

High-Probability Bounds for Stochastic Optimization and Variational Inequalities: the Case of Unbounded Variance

Perseus: A Simple and Optimal High-Order Method for Variational Inequalities

Optimistic Dual Extrapolation for Coherent Non-monotone Variational Inequalities

Variational Extrapolation of Implicit Schemes for General Gradient Flows

Robust SDE-Based Variational Formulations for Solving Linear PDEs via Deep Learning

Smooth over-parameterized solvers for non-smooth structured optimization

Stochastic Methods in Variational Inequalities: Ergodicity, Bias and Refinements