Abstract:Augmented Lagrangian method (ALM) has been popularly used for solving constrained optimization problems. Practically, subproblems for updating primal variables in the framework of ALM usually can only be solved inexactly. The convergence and local convergence speed of ALM have been extensively studied. However, the global convergence rate of the inexact ALM is still open for problems with nonlinear inequality constraints. In this paper, we work on general convex programs with both equality and inequality constraints. For these problems, we establish the global convergence rate of the inexact ALM and estimate its iteration complexity in terms of the number of gradient evaluations to produce a primal and/or primal-dual solution with a specified accuracy. We first establish an ergodic convergence rate result of the inexact ALM that uses constant penalty parameters or geometrically increasing penalty parameters. Based on the convergence rate result, we then apply Nesterov's optimal first-order method on each primal subproblem and estimate the iteration complexity of the inexact ALM. We show that if the objective is convex, then \(O(\varepsilon ^{-1})\) gradient evaluations are sufficient to guarantee a primal \(\varepsilon \)-solution in terms of both primal objective and feasibility violation. If the objective is strongly convex, the result can be improved to \(O(\varepsilon ^{-\frac{1}{2}}|\log \varepsilon |)\). To produce a primal-dual \(\varepsilon \)-solution, more gradient evaluations are needed for convex case, and the number is \(O(\varepsilon ^{-\frac{4}{3}})\), while for strongly convex case, the number is still \(O(\varepsilon ^{-\frac{1}{2}}|\log \varepsilon |)\). Finally, we establish a nonergodic convergence rate result of the inexact ALM that uses geometrically increasing penalty parameters. This result is established only for the primal problem. We show that the nonergodic iteration complexity result is in the same order as that for the ergodic result. Numerical experiments on quadratically constrained quadratic programming are conducted to compare the performance of the inexact ALM with different settings.

Morozov's principle for the augmented Lagrangian method applied to linear inverse problems

Linear Convergence of Inexact Descent Method and Inexact Proximal Gradient Algorithms for Lower-Order Regularization Problems

On convergence and convergence rates for Ivanov and Morozov regularization and application to some parameter identification problems in elliptic PDEs

An augmented Lagrangian algorithm for total bounded variation regularization based image deblurring

A structured L-BFGS method and its application to inverse problems

Inexact Augmented Lagrangian Methods for Conic Programs: Quadratic Growth and Linear Convergence

The inexact power augmented Lagrangian method for constrained nonconvex optimization

A New Insight on Augmented Lagrangian Method with Applications in Machine Learning

Convergence and adaptive discretization of the IRGNM Tikhonov and the IRGNM Ivanov method under a tangential cone condition in Banach space

Iteration complexity of inexact augmented Lagrangian methods for constrained convex programming

Stochastic variance reduced gradient method for linear ill-posed inverse problems

An Inexact Augmented Lagrangian Framework for Nonconvex Optimization with Nonlinear Constraints

A modified iteratively regularized Landweber iteration method: Hölder stability and convergence rates

Weakly Convex Regularisers for Inverse Problems: Convergence of Critical Points and Primal-Dual Optimisation

A convergence analysis of the iteratively regularized Gauss-Newton method under Lipschitz condition

Global Convergence Analysis of the Power Proximal Point and Augmented Lagrangian Method

Iterative regularization for low complexity regularizers

Generalized Convergence Rates Results for Linear Inverse Problems in Hilbert Spaces

Superiorized Regularization of Inverse Problems

Posterior-Variance–Based Error Quantification for Inverse Problems in Imaging