Abstract:In this paper, we study a variant of the quadratic penalty method for linearly constrained convex problems, which has already been widely used but actually lacks theoretical justification. Namely, the penalty parameter steadily increases and the penalized objective function is minimized inexactly rather than exactly, e.g., with only one step of the proximal gradient descent. For such a variant of the quadratic penalty method, we give counterexamples to show that it may not give a solution to the original constrained problem. By choosing special penalty parameters, we ensure the convergence and further establish the convergence rates of $O\left(\frac{1}{\sqrt{K}}\right)$ for the generally convex problems and $O\left(\frac{1}{K}\right)$ for strongly convex ones, where $K$ is the number of iterations. Furthermore, by adopting Nesterov's extrapolation we show that the convergence rates can be improved to $O\left(\frac{1}{K}\right)$ for the generally convex problems and $O\left(\frac{1}{K^2}\right)$ for strongly convex ones. When applied to the decentralized distributed optimization, the penalty methods studied in this paper become the widely used distributed gradient method and the fast distributed gradient method. However, due to the totally different analysis framework, we can improve their $O\left(\frac{\log K}{\sqrt{K}}\right)$ and $O\left(\frac{\log K}{K}\right)$ convergence rates to $O\left(\frac{1}{\sqrt{K}}\right)$ and $O\left(\frac{1}{K}\right)$ with fewer assumptions on the network topology for general convex problems. Using our analysis framework, we also extend the fast distributed gradient method to a communication efficient version, i.e., finding an $\varepsilon$ solution in $O\left(\frac{1}{\varepsilon}\right)$ communications and $O\left(\frac{1}{\varepsilon^{2+\delta}}\right)$ computations for the non-smooth problems, where $\delta$ is a small constant.

Penalty Method for Constrained Distributed Quaternion-Variable Optimization

A Neurodynamic Approach to Nonsmooth Quaternion Distributed Convex Optimization With Inequality and Affine Equality Constraints

Distributed Continuous-Time Algorithm for Constrained Optimization of Networked Euler–Lagrange Systems

A general framework for constrained convex quaternion optimization

A Hessian-based zeroing neurodynamic approach for quaternion-variable time-varying constrained optimization problems

Network Optimization via Smooth Exact Penalty Functions Enabled by Distributed Gradient Computation

Quantization Avoids Saddle Points in Distributed Optimization

Distributed Continuous-Time Algorithms for Time-Varying Constrained Convex Optimization

Exponentially Convergent Algorithm Design for Constrained Distributed Optimization via Non-smooth Approach

Distributed Unbalanced Optimization Design over Nonidentical Constraints

A Unified Distributed Method for Constrained Networked Optimization via Saddle-Point Dynamics

Matrix-valued distributed stochastic optimization with constraints

Convergence Rates Analysis of The Quadratic Penalty Method and Its Applications to Decentralized Distributed Optimization

Implicit Tracking-Based Distributed Constraint-Coupled Optimization

Distributed Quasi-Newton Method for Multi-Agent Optimization

Globally-Constrained Decentralized Optimization with Variable Coupling

Distributed Optimization with Quantized Gradient Descent

Neurodynamic approaches for multi-agent distributed optimization

Distributed Constrained Optimization With Delayed Subgradient Information Over Time-Varying Network Under Adaptive Quantization

A Method for Transforming Non-Convex Optimization Problem to Distributed Form

Distributed Nonconvex Optimization: Gradient-free Iterations and $ε$-Globally Optimal Solution