Abstract:Augmented Lagrangian method (ALM) has been popularly used for solving constrained optimization problems. Practically, subproblems for updating primal variables in the framework of ALM usually can only be solved inexactly. The convergence and local convergence speed of ALM have been extensively studied. However, the global convergence rate of the inexact ALM is still open for problems with nonlinear inequality constraints. In this paper, we work on general convex programs with both equality and inequality constraints. For these problems, we establish the global convergence rate of the inexact ALM and estimate its iteration complexity in terms of the number of gradient evaluations to produce a primal and/or primal-dual solution with a specified accuracy. We first establish an ergodic convergence rate result of the inexact ALM that uses constant penalty parameters or geometrically increasing penalty parameters. Based on the convergence rate result, we then apply Nesterov's optimal first-order method on each primal subproblem and estimate the iteration complexity of the inexact ALM. We show that if the objective is convex, then \(O(\varepsilon ^{-1})\) gradient evaluations are sufficient to guarantee a primal \(\varepsilon \)-solution in terms of both primal objective and feasibility violation. If the objective is strongly convex, the result can be improved to \(O(\varepsilon ^{-\frac{1}{2}}|\log \varepsilon |)\). To produce a primal-dual \(\varepsilon \)-solution, more gradient evaluations are needed for convex case, and the number is \(O(\varepsilon ^{-\frac{4}{3}})\), while for strongly convex case, the number is still \(O(\varepsilon ^{-\frac{1}{2}}|\log \varepsilon |)\). Finally, we establish a nonergodic convergence rate result of the inexact ALM that uses geometrically increasing penalty parameters. This result is established only for the primal problem. We show that the nonergodic iteration complexity result is in the same order as that for the ergodic result. Numerical experiments on quadratically constrained quadratic programming are conducted to compare the performance of the inexact ALM with different settings.

Complexity Guarantees for Nonconvex Newton-MR Under Inexact Hessian Information

A Newton-MR algorithm with complexity guarantees for nonconvex smooth unconstrained optimization

Stochastic Optimization for Non-convex Problem with Inexact Hessian Matrix, Gradient, and Function

Inexact Proximal Cubic Regularized Newton Methods for Convex Optimization

Stochastic Optimization for Nonconvex Problem with Inexact Hessian Matrix, Gradient, and Function

Convergence of Inexact Steepest Descent Algorithm for Multiobjective Optimizations on Riemannian Manifolds Without Curvature Constraints

Improved Complexity for Smooth Nonconvex Optimization: A Two-Level Online Learning Approach with Quasi-Newton Methods

Extended Newton Methods for Multiobjective Optimization: Majorizing Function Technique and Convergence Analysis

Convex and Non-convex Optimization Under Generalized Smoothness

Yet another fast variant of Newton’s method for nonconvex optimization

First and zeroth-order implementations of the regularized Newton method with lazy approximated Hessians

Gauss-Newton Methods for A Class of Nonsmooth Optimization Problems

An inexact regularized proximal Newton method for nonconvex and nonsmooth optimization

Worst-case evaluation complexity and optimality of second-order methods for nonconvex smooth optimization

On Some Versions of Subspace Optimization Methods with Inexact Gradient Information

Stochastic Second-order Methods for Non-convex Optimization with Inexact Hessian and Gradient

Iteration complexity of inexact augmented Lagrangian methods for constrained convex programming

A Multilevel Low-Rank Newton Method with Super-linear Convergence Rate and its Application to Non-convex Problems

Riemannian Adaptive Regularized Newton Methods with Hölder Continuous Hessians

Convergence and Complexity Guarantee for Inexact First-order Riemannian Optimization Algorithms