Abstract:In this paper, we study a variant of the quadratic penalty method for linearly constrained convex problems, which has already been widely used but actually lacks theoretical justification. Namely, the penalty parameter steadily increases and the penalized objective function is minimized inexactly rather than exactly, e.g., with only one step of the proximal gradient descent. For such a variant of the quadratic penalty method, we give counterexamples to show that it may not give a solution to the original constrained problem. By choosing special penalty parameters, we ensure the convergence and further establish the convergence rates of $O\left(\frac{1}{\sqrt{K}}\right)$ for the generally convex problems and $O\left(\frac{1}{K}\right)$ for strongly convex ones, where $K$ is the number of iterations. Furthermore, by adopting Nesterov's extrapolation we show that the convergence rates can be improved to $O\left(\frac{1}{K}\right)$ for the generally convex problems and $O\left(\frac{1}{K^2}\right)$ for strongly convex ones. When applied to the decentralized distributed optimization, the penalty methods studied in this paper become the widely used distributed gradient method and the fast distributed gradient method. However, due to the totally different analysis framework, we can improve their $O\left(\frac{\log K}{\sqrt{K}}\right)$ and $O\left(\frac{\log K}{K}\right)$ convergence rates to $O\left(\frac{1}{\sqrt{K}}\right)$ and $O\left(\frac{1}{K}\right)$ with fewer assumptions on the network topology for general convex problems. Using our analysis framework, we also extend the fast distributed gradient method to a communication efficient version, i.e., finding an $\varepsilon$ solution in $O\left(\frac{1}{\varepsilon}\right)$ communications and $O\left(\frac{1}{\varepsilon^{2+\delta}}\right)$ computations for the non-smooth problems, where $\delta$ is a small constant.

Decentralized non-convex optimization via bi-level SQP and ADMM

An essentially decentralized interior point method for control

A Decentralized Primal-Dual Method with Quasi-Newton Tracking

Monotone Splitting SQP Algorithms for Two-block Nonconvex Optimization Problems with General Linear Constraints and Applications

A Globally Convergent Stabilized SQP Method

An ADMM-based SQP method for separably smooth nonconvex optimization

A sequential quadratic programming method for nonsmooth stochastic optimization with upper-C^2 objective

On Distributed Nonconvex Optimisation Via Modified ADMM

Proximal ADMM for Nonconvex and Nonsmooth Optimization

Global and local convergence of a nonmonotone SQP method for constrained nonlinear optimization

Can Decentralized Stochastic Minimax Optimization Algorithms Converge Linearly for Finite-Sum Nonconvex-Nonconcave Problems?

Linearized ADMM for Nonconvex Nonsmooth Optimization with Convergence Analysis

A stabilized SQP method: superlinear convergence

On the Performance of SQP Methods for Nonlinear Optimization

A comparative study of SQP-type algorithms for nonlinear and nonconvex mixed-integer optimization

A Unified Inexact Stochastic ADMM for Composite Nonconvex and Nonsmooth Optimization

Convergence Rates Analysis of The Quadratic Penalty Method and Its Applications to Decentralized Distributed Optimization

A QCQP-based splitting SQP algorithm for two-block nonconvex constrained optimization problems with application

An adaptive stochastic sequential quadratic programming with differentiable exact augmented lagrangians

On the Divergence of Decentralized Non-Convex Optimization

Decentralized Stochastic Subgradient Methods for Nonsmooth Nonconvex Optimization