Abstract:Distributed stochastic non-convex optimization problems have recently received attention due to the growing interest of signal processing, computer vision, and natural language processing communities in applications deployed over distributed learning systems (e.g., federated learning). We study the setting where the data is distributed across the nodes of a time-varying directed network, a topology suitable for modeling dynamic networks experiencing communication delays and straggler effects. The network nodes, which can access only their local objectives and query a stochastic first-order oracle to obtain gradient estimates, collaborate to minimize a global objective function by exchanging messages with their neighbors. We propose an algorithm, novel to this setting, that leverages stochastic gradient descent with momentum and gradient tracking to solve distributed non-convex optimization problems over time-varying networks. To analyze the algorithm, we tackle the challenges that arise when analyzing dynamic network systems which communicate gradient acceleration components. We prove that the algorithm's oracle complexity is $\mathcal{O}(1/\epsilon^{1.5})$, and that under Polyak-$Ł$ojasiewicz condition the algorithm converges linearly to a steady error state. The proposed scheme is tested on several learning tasks: a non-convex logistic regression experiment on the MNIST dataset, an image classification task on the CIFAR-10 dataset, and an NLP classification test on the IMDB dataset. We further present numerical simulations with an objective that satisfies the PL condition. The results demonstrate superior performance of the proposed framework compared to the existing related methods.

Distributed Weakly Convex Optimization under Random Time-Delay Interference

Distributed Stochastic Optimization for Weakly Coupled Systems with Applications to Wireless Communications

A Distributed Convex Optimization Algorithm over Directed Networks with Time Delay

Delayed Stochastic Algorithms for Distributed Weakly Convex Optimization

Delayed Algorithms for Distributed Stochastic Weakly Convex Optimization

Neurodynamic Algorithms for Constrained Distributed Convex Optimization over Fixed or Switching Topology with Time-Varying Communication Delay

Accelerated Convergence Algorithm for Distributed Constrained Optimization under Time-Varying General Directed Graphs.

Distributed Online Constrained Optimization With Feedback Delays

Delay effects on distributed constrained optimization over double‐integrator multi‐agent systems

Distributed Optimization Based on a Multiagent System in the Presence of Communication Delays

Distributed online constrained convex optimization with event-triggered communication

Distributed Stochastic Constrained Composite Optimization Over Time-Varying Network With a Class of Communication Noise

Computation-Efficient Distributed Algorithm for Convex Optimization over Time-Varying Networks with Limited Bandwidth Communication

A Generic Algorithm Framework for Distributed Optimization over the Time-Varying Network with Communication Delays

Distributed fixed-point algorithms for dynamic convex optimization over decentralized and unbalanced wireless networks

Distributed Time-Varying Convex Optimization with Dynamic Quantization

Decentralized Multi-Task Online Convex Optimization Under Random Link Failures

Accelerated Distributed Stochastic Non-Convex Optimization over Time-Varying Directed Networks

Distributed Online Optimization Subject to Long-Term Constraints and Time-Varying Topology: an Event-Triggered and Bandit Feedback Approach

Distributed Multi-Agent Optimization Subject to Nonidentical Constraints and Communication Delays.

Decentralized Multitask Online Convex Optimization Under Random Link Failures