Abstract:Line search procedures are often employed in primal-dual methods for bilinear saddle point problems, especially when the norm of the linear operator is large or difficult to compute. In this paper, we demonstrate that line search is unnecessary by introducing a novel primal-dual method, the auto-conditioned primal-dual hybrid gradient (AC-PDHG) method, which achieves optimal complexity for solving bilinear saddle point problems. AC-PDHG is fully adaptive to the linear operator, using only past iterates to estimate its norm. We further tailor AC-PDHG to solve linearly constrained problems, providing convergence guarantees for both the optimality gap and constraint violation. Moreover, we explore an important class of linearly constrained problems where both the objective and constraints decompose into two parts. By incorporating the design principles of AC-PDHG into the preconditioned alternating direction method of multipliers (ADMM), we propose the auto-conditioned alternating direction method of multipliers (AC-ADMM), which guarantees convergence based solely on one part of the constraint matrix and fully adapts to it, eliminating the need for line search. Finally, we extend both AC-PDHG and AC-ADMM to solve bilinear problems with an additional smooth term. By integrating these methods with a novel acceleration scheme, we attain optimal iteration complexities under the single-oracle setting.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to design an adaptive method without line search in order to achieve the optimal iteration complexity when solving bilinear saddle - point problems and linearly - constrained optimization problems. Specifically, the paper proposes two new algorithms: 1. **Auto - Conditioned Primal - Dual Hybrid Gradient (AC - PDHG)**: - **Problem Background**: When solving bilinear saddle - point problems, traditional primal - dual methods usually need to estimate the norm $\|A\|$ of the linear operator $A$, which is often very time - consuming and difficult to calculate accurately in large - scale problems. In addition, relying on the global constant $\|A\|$ may lead to conservative step - size selection, thus affecting the convergence speed of the algorithm. - **Main Contribution**: AC - PDHG realizes the optimal convergence rate without line search by introducing an additional sequence of search points $\{\bar{x}_t\}$ as "proximal centers" and combining Nesterov's smoothing technique. This method can automatically adapt to $\|A\|$ only by inputting the diameter $D_Y$ of the dual feasible region and the required precision $\epsilon$. 2. **Auto - Conditioned Alternating Direction Method of Multipliers (AC - ADMM)**: - **Problem Background**: For linearly - constrained optimization problems, the traditional ADMM method needs to solve complex sub - problems in each step of the update, and its convergence speed depends on the norms of the constraint matrices $B$ and $K$. This may also lead to conservative step - size selection. - **Main Contribution**: AC - ADMM realizes the optimal convergence rate without line search by introducing a locally estimated $\|K\|$ to determine the step - size strategy through a design idea similar to that of AC - PDHG. This method only depends on the matrix $K$ and not on $B$. ### Formula Summary - **Bilinear Saddle - Point Problem**: \[ \min_{x\in X}\max_{y\in Y}f(x)+\langle Ax,y\rangle - g(y) \] where $X\subseteq\mathbb{R}^n$ and $Y\subseteq\mathbb{R}^m$ are closed convex sets, $A\in\mathbb{R}^{m\times n}$ is a linear mapping, and $f:X\rightarrow\mathbb{R}$ and $g:Y\rightarrow\mathbb{R}$ are appropriate lower - semicontinuous convex functions. - **Linearly - Constrained Optimization Problem**: \[ \min_{x\in X}\{f(x):Ax = b\} \] which can be transformed into a bilinear saddle - point problem through Lagrangian duality. - **Update Rule of AC - PDHG**: \[ y_0=\arg\min_{y\in Y}\left\{\langle - Ax_0,y\rangle+g(y)+\frac{\mu_d}{2}\|\tilde{y}_0 - y\|^2\right\} \] \[ x_t=\arg\min_{x\in X}\left\{\eta_t[\langle A^{\top}y_{t - 1},x\rangle+f(x)]+\frac{1}{2}\|\bar{x}_{t - 1}-x\|^2\right\} \] \[ \bar{x}_t=(1-\beta_t)\bar{x}

Auto-conditioned primal-dual hybrid gradient method and alternating direction method of multipliers

An Efficient Interior-Point Algorithm with New Non-Monotone Line Search Filter Method for Nonlinear Constrained Programming

Acceleration of Primal–Dual Methods by Preconditioning and Simple Subproblem Procedures

Understanding the Convergence of the Preconditioned PDHG Method: A View of Indefinite Proximal ADMM

Dual Descent Augmented Lagrangian Method and Alternating Direction Method of Multipliers

Accelerating preconditioned ADMM via degenerate proximal point mappings

A Proximal Point Analysis of the Preconditioned Alternating Direction Method of Multipliers.

Accelerated nonlinear primal-dual hybrid gradient methods with applications to supervised machine learning

Accelerated Primal-Dual Proximal Gradient Splitting Methods for Convex-Concave Saddle-Point Problems

A Generalized Alternating Direction Method of Multipliers with Semi-Proximal Terms for Convex Composite Conic Programming

A Golden Ratio Proximal Alternating Direction Method of Multipliers for Separable Convex Optimization

A Hybrid Bregman Alternating Direction Method of Multipliers for the Linearly Constrained Difference-of-convex Problems

A convex combination based primal-dual algorithm with linesearch for general convex-concave saddle point problems

Accelerated primal-dual methods with enlarged step sizes and operator learning for nonsmooth optimal control problems

Primal Dual Alternating Proximal Gradient Algorithms for Nonsmooth Nonconvex Minimax Problems with Coupled Linear Constraints

An Inertial Proximal Partially Symmetric ADMM-based Algorithm for Linearly Constrained Multi-Block Nonconvex Optimization Problems with Applications.

A Bregman-style Partially Symmetric Alternating Direction Method of Multipliers for Nonconvex Multi-block Optimization

A proximal point algorithm revisit on the alternating direction method of multipliers

Several Variants of the Primal-Dual Hybrid Gradient Algorithm with Applications

Parallel Direction Method of Multipliers

An adaptive linearized alternating direction multiplier method with a relaxation step for convex programming