Abstract:Distributed stochastic non-convex optimization problems have recently received attention due to the growing interest of signal processing, computer vision, and natural language processing communities in applications deployed over distributed learning systems (e.g., federated learning). We study the setting where the data is distributed across the nodes of a time-varying directed network, a topology suitable for modeling dynamic networks experiencing communication delays and straggler effects. The network nodes, which can access only their local objectives and query a stochastic first-order oracle to obtain gradient estimates, collaborate to minimize a global objective function by exchanging messages with their neighbors. We propose an algorithm, novel to this setting, that leverages stochastic gradient descent with momentum and gradient tracking to solve distributed non-convex optimization problems over time-varying networks. To analyze the algorithm, we tackle the challenges that arise when analyzing dynamic network systems which communicate gradient acceleration components. We prove that the algorithm's oracle complexity is $\mathcal{O}(1/\epsilon^{1.5})$, and that under Polyak-$Ł$ojasiewicz condition the algorithm converges linearly to a steady error state. The proposed scheme is tested on several learning tasks: a non-convex logistic regression experiment on the MNIST dataset, an image classification task on the CIFAR-10 dataset, and an NLP classification test on the IMDB dataset. We further present numerical simulations with an objective that satisfies the PL condition. The results demonstrate superior performance of the proposed framework compared to the existing related methods.

Decentralized Convex Optimization on Time-Varying Networks with Application to Wasserstein Barycenters

Accelerated Distributed Stochastic Non-Convex Optimization over Time-Varying Directed Networks

Lower Bounds and Optimal Algorithms for Non-Smooth Convex Decentralized Optimization over Time-Varying Networks

Non-Smooth Setting of Stochastic Decentralized Convex Optimization Problem Over Time-Varying Graphs

Distributed Algorithms for Composite Optimization: Unified Framework and Convergence Analysis

Decentralized Finite-Sum Optimization over Time-Varying Networks

Nonlinear Perturbation-based Non-Convex Optimization over Time-Varying Networks

Distributed Convex Optimization "Over-the-Air" in Dynamic Environments

Distributed Subgradient-Free Stochastic Optimization Algorithm for Nonsmooth Convex Functions over Time-Varying Networks

A Unified Algorithmic Framework for Distributed Composite Optimization.

Distributed fixed-point algorithms for dynamic convex optimization over decentralized and unbalanced wireless networks

Constrained Distributed Nonconvex Optimization over Time-varying Directed Graphs

Distributed Time-Varying Convex Optimization with Dynamic Quantization

A Unified Contraction Analysis of a Class of Distributed Algorithms for Composite Optimization

Decentralized Optimization in Time-Varying Networks with Arbitrary Delays

Distributed Algorithm for Time-Varying Convex Optimization with Fixed-Time Convergence

A Distributed Convex Optimization Algorithm over Directed Networks with Time Delay

Decentralized optimization with affine constraints over time-varying networks

Accelerated Primal-Dual Algorithms for Distributed Smooth Convex Optimization over Networks

Distributed Stochastic Mirror Descent Algorithm over Time-varying Network

Distributed Randomized Gradient-free Convex Optimization with Set Constraints over Time-varying Weight-unbalanced Digraphs