Abstract:In this paper, we study a variant of the quadratic penalty method for linearly constrained convex problems, which has already been widely used but actually lacks theoretical justification. Namely, the penalty parameter steadily increases and the penalized objective function is minimized inexactly rather than exactly, e.g., with only one step of the proximal gradient descent. For such a variant of the quadratic penalty method, we give counterexamples to show that it may not give a solution to the original constrained problem. By choosing special penalty parameters, we ensure the convergence and further establish the convergence rates of $O\left(\frac{1}{\sqrt{K}}\right)$ for the generally convex problems and $O\left(\frac{1}{K}\right)$ for strongly convex ones, where $K$ is the number of iterations. Furthermore, by adopting Nesterov's extrapolation we show that the convergence rates can be improved to $O\left(\frac{1}{K}\right)$ for the generally convex problems and $O\left(\frac{1}{K^2}\right)$ for strongly convex ones. When applied to the decentralized distributed optimization, the penalty methods studied in this paper become the widely used distributed gradient method and the fast distributed gradient method. However, due to the totally different analysis framework, we can improve their $O\left(\frac{\log K}{\sqrt{K}}\right)$ and $O\left(\frac{\log K}{K}\right)$ convergence rates to $O\left(\frac{1}{\sqrt{K}}\right)$ and $O\left(\frac{1}{K}\right)$ with fewer assumptions on the network topology for general convex problems. Using our analysis framework, we also extend the fast distributed gradient method to a communication efficient version, i.e., finding an $\varepsilon$ solution in $O\left(\frac{1}{\varepsilon}\right)$ communications and $O\left(\frac{1}{\varepsilon^{2+\delta}}\right)$ computations for the non-smooth problems, where $\delta$ is a small constant.

Computational Convergence Analysis of Distributed Gradient Tracking for Smooth Convex Optimization Using Dissipativity Theory

Convergence in High Probability of Distributed Stochastic Gradient Descent Algorithms

Accelerated Primal-Dual Algorithms for Distributed Smooth Convex Optimization over Networks

A Unified Contraction Analysis of a Class of Distributed Algorithms for Composite Optimization

Distributed Algorithms for Composite Optimization: Unified Framework and Convergence Analysis

Distributed Optimization via Energy Conservation Laws in Dilated Coordinates

Convergence Analysis of Gradient Algorithms on Riemannian Manifolds Without Curvature Constraints and Application to Riemannian Mass

Convex and Non-convex Optimization Under Generalized Smoothness

Global Convergence of Langevin Dynamics Based Algorithms for Nonconvex Optimization

Distributed Adaptive Gradient Algorithm with Gradient Tracking for Stochastic Non-Convex Optimization

Convergence Rates Analysis of The Quadratic Penalty Method and Its Applications to Decentralized Distributed Optimization

Decentralized projected Riemannian gradient method for smooth optimization on compact submanifolds

On Decomposition and Convergence of Distributed Optimization Algorithms

Compressed Gradient Tracking Algorithms for Distributed Nonconvex Optimization

Distributed Stochastic Consensus Optimization With Momentum for Nonconvex Nonsmooth Problems

On the Computation-Communication Trade-Off with A Flexible Gradient Tracking Approach

Quantized Distributed Gradient Tracking Algorithm with Linear Convergence in Directed Networks

Gradient-tracking based Distributed Optimization with Guaranteed Optimality under Noisy Information Sharing

Data Dependent Convergence for Distributed Stochastic Optimization