Abstract:In this paper, we study a variant of the quadratic penalty method for linearly constrained convex problems, which has already been widely used but actually lacks theoretical justification. Namely, the penalty parameter steadily increases and the penalized objective function is minimized inexactly rather than exactly, e.g., with only one step of the proximal gradient descent. For such a variant of the quadratic penalty method, we give counterexamples to show that it may not give a solution to the original constrained problem. By choosing special penalty parameters, we ensure the convergence and further establish the convergence rates of $O\left(\frac{1}{\sqrt{K}}\right)$ for the generally convex problems and $O\left(\frac{1}{K}\right)$ for strongly convex ones, where $K$ is the number of iterations. Furthermore, by adopting Nesterov's extrapolation we show that the convergence rates can be improved to $O\left(\frac{1}{K}\right)$ for the generally convex problems and $O\left(\frac{1}{K^2}\right)$ for strongly convex ones. When applied to the decentralized distributed optimization, the penalty methods studied in this paper become the widely used distributed gradient method and the fast distributed gradient method. However, due to the totally different analysis framework, we can improve their $O\left(\frac{\log K}{\sqrt{K}}\right)$ and $O\left(\frac{\log K}{K}\right)$ convergence rates to $O\left(\frac{1}{\sqrt{K}}\right)$ and $O\left(\frac{1}{K}\right)$ with fewer assumptions on the network topology for general convex problems. Using our analysis framework, we also extend the fast distributed gradient method to a communication efficient version, i.e., finding an $\varepsilon$ solution in $O\left(\frac{1}{\varepsilon}\right)$ communications and $O\left(\frac{1}{\varepsilon^{2+\delta}}\right)$ computations for the non-smooth problems, where $\delta$ is a small constant.

On the convergence analysis of the decentralized projected gradient descent method

Decentralized projected Riemannian gradient method for smooth optimization on compact submanifolds

On the Divergence of Decentralized Non-Convex Optimization

On the convergence of the distributed proximal point algorithm

Decentralized Stochastic Subgradient Methods for Nonsmooth Nonconvex Optimization

Convergence Results of a Nested Decentralized Gradient Method for Non-strongly Convex Problems

Convergence Rates Analysis of The Quadratic Penalty Method and Its Applications to Decentralized Distributed Optimization

Double Averaging and Gradient Projection: Convergence Guarantees for Decentralized Constrained Optimization

Convergence Analysis of Gradient Algorithms on Riemannian Manifolds Without Curvature Constraints and Application to Riemannian Mass

A Distributed Gradient Algorithm Based on Randomized Block-Coordinate and Projection-Free over Networks

Convergence in High Probability of Distributed Stochastic Gradient Descent Algorithms

A Proximal Gradient Algorithm for Decentralized Nondifferentiable Optimization.

On Nonconvex Decentralized Gradient Descent

Decentralized projected Riemannian stochastic recursive momentum method for smooth optimization on compact submanifolds

Tighter Analysis for Decentralized Stochastic Gradient Method: Impact of Data Homogeneity

On the Convergence of Decentralized Adaptive Gradient Methods

Decentralized Conjugate Gradient and Memoryless BFGS Methods

Decentralized Riemannian Conjugate Gradient Method on the Stiefel Manifold

Linear convergence of decentralized estimation for statistical estimation using gradient method

Can Decentralized Stochastic Minimax Optimization Algorithms Converge Linearly for Finite-Sum Nonconvex-Nonconcave Problems?

Improving the Sample and Communication Complexity for Decentralized Non-Convex Optimization: A Joint Gradient Estimation and Tracking Approach