Abstract:In this paper, we study a variant of the quadratic penalty method for linearly constrained convex problems, which has already been widely used but actually lacks theoretical justification. Namely, the penalty parameter steadily increases and the penalized objective function is minimized inexactly rather than exactly, e.g., with only one step of the proximal gradient descent. For such a variant of the quadratic penalty method, we give counterexamples to show that it may not give a solution to the original constrained problem. By choosing special penalty parameters, we ensure the convergence and further establish the convergence rates of $O\left(\frac{1}{\sqrt{K}}\right)$ for the generally convex problems and $O\left(\frac{1}{K}\right)$ for strongly convex ones, where $K$ is the number of iterations. Furthermore, by adopting Nesterov's extrapolation we show that the convergence rates can be improved to $O\left(\frac{1}{K}\right)$ for the generally convex problems and $O\left(\frac{1}{K^2}\right)$ for strongly convex ones. When applied to the decentralized distributed optimization, the penalty methods studied in this paper become the widely used distributed gradient method and the fast distributed gradient method. However, due to the totally different analysis framework, we can improve their $O\left(\frac{\log K}{\sqrt{K}}\right)$ and $O\left(\frac{\log K}{K}\right)$ convergence rates to $O\left(\frac{1}{\sqrt{K}}\right)$ and $O\left(\frac{1}{K}\right)$ with fewer assumptions on the network topology for general convex problems. Using our analysis framework, we also extend the fast distributed gradient method to a communication efficient version, i.e., finding an $\varepsilon$ solution in $O\left(\frac{1}{\varepsilon}\right)$ communications and $O\left(\frac{1}{\varepsilon^{2+\delta}}\right)$ computations for the non-smooth problems, where $\delta$ is a small constant.

A Decentralized Proximal Point-type Method for Non-convex Non-concave Saddle Point Problems

A Decentralized Proximal Point-type Method for Saddle Point Problems

Decentralized Distributed Optimization for Saddle Point Problems

Decentralized saddle point problems via non-Euclidean mirror prox

A Unified Distributed Method for Constrained Networked Optimization via Saddle-Point Dynamics

The Mirror-Prox Sliding Method for Non-smooth decentralized saddle-point problems

A Stochastic Proximal Point Algorithm for Saddle-Point Problems

A Communication-Efficient Decentralized Newton's Method with Provably Faster Convergence

Distributed Multiproximal Algorithm for Nonsmooth Convex Optimization With Coupled Inequality Constraints

Distributed Coordination for Nonsmooth Convex Optimization via Saddle-Point Dynamics

Decentralized Optimization with Coupled Constraints

Globally-Constrained Decentralized Optimization with Variable Coupling

A globally convergent proximal Newton-type method in nonsmooth convex optimization

On the Divergence of Decentralized Non-Convex Optimization

A One-Sample Decentralized Proximal Algorithm for Non-Convex Stochastic Composite Optimization

A neural network for constrained saddle point problems: an approximation approach

Decentralized Proximal Optimization Method with Consensus Procedure

A Feasible Level Proximal Point Method for Nonconvex Sparse Constrained Optimization

Convergence Rates Analysis of The Quadratic Penalty Method and Its Applications to Decentralized Distributed Optimization

Switch and Conquer: Efficient Algorithms By Switching Stochastic Gradient Oracles For Decentralized Saddle Point Problems

Decentralized Non-Convex Learning With Linearly Coupled Constraints: Algorithm Designs and Application to Vertical Learning Problem