Abstract:Distributed stochastic non-convex optimization problems have recently received attention due to the growing interest of signal processing, computer vision, and natural language processing communities in applications deployed over distributed learning systems (e.g., federated learning). We study the setting where the data is distributed across the nodes of a time-varying directed network, a topology suitable for modeling dynamic networks experiencing communication delays and straggler effects. The network nodes, which can access only their local objectives and query a stochastic first-order oracle to obtain gradient estimates, collaborate to minimize a global objective function by exchanging messages with their neighbors. We propose an algorithm, novel to this setting, that leverages stochastic gradient descent with momentum and gradient tracking to solve distributed non-convex optimization problems over time-varying networks. To analyze the algorithm, we tackle the challenges that arise when analyzing dynamic network systems which communicate gradient acceleration components. We prove that the algorithm's oracle complexity is $\mathcal{O}(1/\epsilon^{1.5})$, and that under Polyak-$Ł$ojasiewicz condition the algorithm converges linearly to a steady error state. The proposed scheme is tested on several learning tasks: a non-convex logistic regression experiment on the MNIST dataset, an image classification task on the CIFAR-10 dataset, and an NLP classification test on the IMDB dataset. We further present numerical simulations with an objective that satisfies the PL condition. The results demonstrate superior performance of the proposed framework compared to the existing related methods.

Distributed Time-Varying Convex Optimization with Dynamic Quantization

An Initialization-Free Continuous-Time Distributed Algorithm for Convex Optimization Problem

Distributed Constrained Optimization With Delayed Subgradient Information Over Time-Varying Network Under Adaptive Quantization

Distributed constrained optimization with periodic dynamic quantization

Distributed Continuous-Time Algorithms for Time-Varying Constrained Convex Optimization

Distributed Online Optimization Under Dynamic Adaptive Quantization

Distributed Time-Varying Convex Optimization for a Class of Nonlinear Multiagent Systems

Quantized Distributed Nonconvex Optimization Algorithms with Linear Convergence under the Polyak--$Ł$ojasiewicz Condition

Distributed Optimization via Gradient Descent with Event-Triggered Zooming over Quantized Communication

DQC-ADMM: Decentralized Dynamic ADMM with Quantized and Censored Communications

Distributed Algorithms for Composite Optimization: Unified Framework and Convergence Analysis

Quantization Avoids Saddle Points in Distributed Optimization

Logarithmically Quantized Distributed Optimization over Dynamic Multi-Agent Networks

Distributed online constrained convex optimization with event-triggered communication

Distributed Continuous‐time Constrained Convex Optimization with General Time‐varying Cost Functions

Distributed Adaptive Time-Varying Optimization with Global Asymptotic Convergence

Distributed Convex Optimization "Over-the-Air" in Dynamic Environments

Distributed Optimization with Quantized Gradient Descent

Accelerated Distributed Stochastic Non-Convex Optimization over Time-Varying Directed Networks

Distributed Economic Dispatch Algorithm with Quantized Communication Mechanism

Neurodynamic Algorithms for Constrained Distributed Convex Optimization over Fixed or Switching Topology with Time-Varying Communication Delay