Abstract:Primal-dual algorithms for the resolution of convex-concave saddle point problems usually come with one or several step size parameters. Within the range where convergence is guaranteed, choosing well the step size can make the difference between a slow or a fast algorithm. A usual way to adaptively set step sizes is to ensure that there is a fair balance between primal and dual variable's amount of change. In this work, we show how to find even better step sizes for the primal-dual hybrid gradient. Getting inspiration from quadratic problems, we base our method on a spectral radius estimation procedure and try to minimize this spectral radius, which is directly related to the rate of convergence. Building on power iterations, we could produce spectral radius estimates that are always smaller than 1 and work also in the case of conjugate principal eigenvalues. For strongly convex quadratics, we show that our step size rule yields an algorithm as fast as inertial gradient descent. Moreover, since our spectral radius estimates only rely on residual norms, our method can be readily adapted to more general convex-concave saddle point problems. In a second part, we extend these results to a randomized version of PDHG called PURE-CD. We design a statistical test to compare observed convergence rates and decide whether a step size is better than another. Numerical experiments on least squares, sparse SVM, TV-L1 denoising and TV-L2 denoising problems support our findings.

Analysis of Gradient Descent with Varying Step Sizes using Integral Quadratic Constraints

Gradient-based Cooperative Control of quasi-Linear Parameter Varying Vehicles with Noisy Gradients

From Stability to Chaos: Analyzing Gradient Descent Dynamics in Quadratic Regression

Gradient Methods with Adaptive Step-Sizes

Convergence Analysis of Gradient Algorithms on Riemannian Manifolds Without Curvature Constraints and Application to Riemannian Mass

A Two Stepsize SQP Method for Nonlinear Equality Constrained Stochastic Optimization

Time-Varying Feedback Optimization for Quadratic Programs with Heterogeneous Gradient Step Sizes

Two efficient gradient methods with approximately optimal stepsizes based on regularization models for unconstrained optimization

Convergence Analysis of Non-Strongly-Monotone Stochastic Quasi-Variational Inequalities

Exact worst-case convergence rates of gradient descent: a complete analysis for all constant stepsizes over nonconvex and convex functions

A Tutorial on a Lyapunov-Based Approach to the Analysis of Iterative Optimization Algorithms

Analytic analysis of the worst-case complexity of the gradient method with exact line search and the Polyak stepsize

Accelerated Objective Gap and Gradient Norm Convergence for Gradient Descent via Long Steps

Global Convergence of Non-Convex Gradient Descent for Computing Matrix Squareroot

Variable Metric Proximal Gradient Method with Diagonal Barzilai-Borwein Stepsize

On a Family of Relaxed Gradient Descent Methods for Quadratic Minimization

The q-gradient method for global optimization

Eigen-componentwise convergence of SGD on quadratic programming

Monitoring the Convergence Speed of PDHG to Find Better Primal and Dual Step Sizes

Quadratic Gradient: Combining Gradient Algorithms and Newton's Method as One

Concavifiability and convergence: necessary and sufficient conditions for gradient descent analysis