Abstract:The computational bottleneck in distributed optimization methods, which is based on projected gradient descent, is due to the computation of a full gradient vector and projection step. This is a particular problem for large datasets. To reduce the computational complexity of existing methods, we combine the randomized block-coordinate descent and the Frank–Wolfe techniques, and then propose a distributed randomized block-coordinate projection-free algorithm over networks, where each agent randomly chooses a subset of the coordinates of its gradient vector and the projection step is eschewed in favor of a much simpler linear optimization step. Moreover, the convergence performance of the proposed algorithm is also theoretically analyzed. Specifically, we rigorously prove that the proposed algorithm can converge to optimal point at rate of 𝒪(1/t) under convexity and 𝒪(1/t^2) under strong convexity, respectively. Here, t is the number of iterations. Furthermore, the proposed algorithm can converge to a stationary point, where the “Frank-Wolfe” gap is equal to zero, at a rate 𝒪(1/√(t)) under non-convexity. To evaluate the computational benefit of the proposed algorithm, we use the proposed algorithm to solve the multiclass classification problems by simulation experiments on two datasets, i.e., aloi and news20. The results shows that the proposed algorithm is faster than the existing distributed optimization algorithms due to its lower computation per iteration. Furthermore, the results also show that well-connected graphs or smaller graphs leads to faster convergence rate, which can confirm the theoretical results.

Decentralized stochastic subgradient projection optimization algorithms over random networks

Distributed Stochastic Optimization With Unbounded Subgradients Over Randomly Time-Varying Networks

Distributed Constrained Stochastic Subgradient Algorithms Based on Random Projection and Asynchronous Broadcast over Networks

Distributed Stochastic Subgradient Projection Algorithms Based on Weight-Balancing over Time-Varying Directed Graphs.

Distributed Heterogeneous Multi-Agent Optimization with Stochastic Sub-Gradient

Distributed Stochastic Subgradient Projection Algorithms for Convex Optimization

Distributed Stochastic Algorithm for Global Optimization in Networked System

A privacy-preserving decentralized randomized block-coordinate subgradient algorithm over time-varying networks

Distributed Subgradient-Free Stochastic Optimization Algorithm for Nonsmooth Convex Functions over Time-Varying Networks

Distributed Projection Subgradient Algorithm over Time-Varying General Unbalanced Directed Graphs

A Decentralized Nesterov Gradient Method for Stochastic Optimization over Unbalanced Directed Networks

An Accelerated Decentralized Stochastic Optimization Algorithm with Inexact Model

An Event-Triggering Algorithm for Decentralized Stochastic Optimization over Networks

Convergence of Distributed Gradient-Tracking-Based Optimization Algorithms with Random Graphs

Distributed Proximal Gradient Algorithm for Nonconvex Optimization Over Time-Varying Networks

Stochastic Block-Coordinate Gradient Projection Algorithms for Submodular Maximization.

A Distributed Gradient Algorithm Based on Randomized Block-Coordinate and Projection-Free over Networks

On the Convergence Properties of a Distributed Projected Subgradient Algorithm

A Distributed Random Projection Gradient Method for Stochastic Optimization Problems in Multi-Agent Systems

Convergence in High Probability of Distributed Stochastic Gradient Descent Algorithms

Random Sleep Scheme-Based Distributed Optimization Algorithm over Unbalanced Time-Varying Networks