Abstract:Consider the consensus problem of minimizing f(x) = Sigma(n)(i=1) fi(x), where x is an element of R-p and each f(i) is only known to the individual agent i in a connected network of n agents. To solve this problem and obtain the solution, all the agents collaborate with their neighbors through information exchange. This type of decentralized computation does not need a fusion center, offers better network load balance, and improves data privacy. This paper studies the decentralized gradient descent method [A. Nedic and A. Ozdaglar, IEEE Trans. Automat. Control, 54 (2009), pp. 48-61], in which each agent i updates its local variable x((i)) is an element of R-n by combining the average of its neighbors' with a local negative-gradient step -alpha del f(i)(x((i))). The method is described by the iteration x((i)) (k + 1) <- Sigma(n)(j=1) w(ij)x((j))(k)-alpha del f(i)(x((i))(k)), for each agent i, where w(ij) is nonzero only if i and j are neighbors or i = j and the matrix W = [w(ij)] is an element of R-n (x) (n) is symmetric and doubly stochastic. This paper analyzes the convergence of this iteration and derives its rate of convergence under the assumption that each f(i) is proper closed convex and lower bounded, del f(i) is Lipschitz continuous with constant L-fi > 0, and the stepsize a is fixed. Provided that alpha <= min{(1 + lambda(n)) (W))/L-h, 1/L-(f) over bar}, where L-h = max(i){L-fi} and L-(f) over bar = (1)(n) Sigma(n)(i=1) L-fi, the objective errors of all the local solutions and the networkwide mean solution reduce at rates of O(1/k) until they reach a level of O(alpha). If fi are strongly convex with modulus mu(fi), and alpha <= min{(1+lambda(n)(W))/L-h, 1/(L-f+mu(f))}, where mu(f) - 1/n Sigma(n)(i=1) mu(fi), then all the local solutions and the mean solution converge to the global minimizer x* at a linear rate until reaching an O(alpha)-neighborhood of x*. We also develop an iteration for decentralized basis pursuit and establish its linear convergence to an O(alpha)-neighborhood of the true sparse signal. This analysis reveals how the convergence of x((i))(k + 1) <- Sigma(n)(j=1) wijx(j)(k) - alpha del f(i)(x((i))(k)), for each agent i, depends on the stepsize, function convexity, and network spectrum.

On Linear Convergence of Adaptive Sign-based Gradient Descent

On Faster Convergence of Scaled Sign Gradient Descent

Sign Stochastic Gradient Descents Without Bounded Gradient Assumption for the Finite Sum Minimization

Linear Convergence of Adaptive Stochastic Gradient Descent

Characterization of Linear Convergence of Gradient Descent

Linear Convergence Rate in Convex Setup is Possible! Gradient Descent Method Variants under $(L_0,L_1)$-Smoothness

A gradient descent akin method for constrained optimization: algorithms and applications

Linear convergence of decentralized estimation for statistical estimation using gradient method

On the Convergence of Decentralized Gradient Descent.

On the Convergence of Adaptive Gradient Methods for Nonconvex Optimization

Universal Gradient Descent Ascent Method for Nonconvex-Nonconcave Minimax Optimization

Gradient descent with adaptive stepsize converges (nearly) linearly under fourth-order growth

Efficient Sign-Based Optimization: Accelerating Convergence via Variance Reduction

Convergence Analysis of Adaptive Gradient Methods under Refined Smoothness and Noise Assumptions

On the Unstable Convergence Regime of Gradient Descent

A Methodology Establishing Linear Convergence of Adaptive Gradient Methods under PL Inequality

Directional Smoothness and Gradient Methods: Convergence and Adaptivity

Adaptive Multi-Hierarchical signSGD for Communication-Efficient Distributed Optimization

Convergence in High Probability of Distributed Stochastic Gradient Descent Algorithms

Linear convergence of forward-backward accelerated algorithms without knowledge of the modulus of strong convexity