Abstract:Augmented Lagrangian method (ALM) has been popularly used for solving constrained optimization problems. Practically, subproblems for updating primal variables in the framework of ALM usually can only be solved inexactly. The convergence and local convergence speed of ALM have been extensively studied. However, the global convergence rate of the inexact ALM is still open for problems with nonlinear inequality constraints. In this paper, we work on general convex programs with both equality and inequality constraints. For these problems, we establish the global convergence rate of the inexact ALM and estimate its iteration complexity in terms of the number of gradient evaluations to produce a primal and/or primal-dual solution with a specified accuracy. We first establish an ergodic convergence rate result of the inexact ALM that uses constant penalty parameters or geometrically increasing penalty parameters. Based on the convergence rate result, we then apply Nesterov's optimal first-order method on each primal subproblem and estimate the iteration complexity of the inexact ALM. We show that if the objective is convex, then \(O(\varepsilon ^{-1})\) gradient evaluations are sufficient to guarantee a primal \(\varepsilon \)-solution in terms of both primal objective and feasibility violation. If the objective is strongly convex, the result can be improved to \(O(\varepsilon ^{-\frac{1}{2}}|\log \varepsilon |)\). To produce a primal-dual \(\varepsilon \)-solution, more gradient evaluations are needed for convex case, and the number is \(O(\varepsilon ^{-\frac{4}{3}})\), while for strongly convex case, the number is still \(O(\varepsilon ^{-\frac{1}{2}}|\log \varepsilon |)\). Finally, we establish a nonergodic convergence rate result of the inexact ALM that uses geometrically increasing penalty parameters. This result is established only for the primal problem. We show that the nonergodic iteration complexity result is in the same order as that for the ergodic result. Numerical experiments on quadratically constrained quadratic programming are conducted to compare the performance of the inexact ALM with different settings.

On Iteration Complexity of a First-Order Primal-Dual Method for Nonlinear Convex Cone Programming

First-Order Primal-Dual Augmented Lagrangian Method for Nonlinear Cone Constrained Composite Convex Optimization

A First-Order Primal-Dual Method for Nonconvex Constrained Optimization Based on the Augmented Lagrangian

Auxiliary Problem Principle of Augmented Lagrangian with Varying Core Functions for Large-Scale Structured Convex Problems

Stochastic Primal-Dual Coordinate Method for Nonlinear Convex Cone Programs

Convergence analysis of approximate primal solutions in dual first-order methods

On the Complexity Analysis of the Primal Solutions for the Accelerated Randomized Dual Coordinate Ascent

A nonsmooth primal-dual method with interwoven PDE constraint solver

Iteration-complexity of first-order augmented Lagrangian methods for convex programming

A New Primal-Dual Algorithm for a Class of Nonlinear Compositional Convex Optimization Problems

On the Linear Convergence of the General First Order Primal-Dual Algorithm

Iteration complexity of inexact augmented Lagrangian methods for constrained convex programming

A Fast Single-Loop Primal-Dual Algorithm for Non-Convex Functional Constrained Optimization

Linear Convergence of Randomized Primal-Dual Coordinate Method for Large-scale Linear Constrained Convex Programming

Convergence rate analysis of fast primal-dual methods with scalings for linearly constrained convex optimization problems

A Smooth Primal-Dual Optimization Framework for Nonsmooth Composite Convex Minimization

Fast Primal–dual Algorithm Via Dynamical System for a Linearly Constrained Convex Optimization Problem

Nearly Optimal Linear Convergence of Stochastic Primal-Dual Methods for Linear Programming

Faster Accelerated First-order Methods for Convex Optimization with Strongly Convex Function Constraints