Abstract:Distributed stochastic non-convex optimization problems have recently received attention due to the growing interest of signal processing, computer vision, and natural language processing communities in applications deployed over distributed learning systems (e.g., federated learning). We study the setting where the data is distributed across the nodes of a time-varying directed network, a topology suitable for modeling dynamic networks experiencing communication delays and straggler effects. The network nodes, which can access only their local objectives and query a stochastic first-order oracle to obtain gradient estimates, collaborate to minimize a global objective function by exchanging messages with their neighbors. We propose an algorithm, novel to this setting, that leverages stochastic gradient descent with momentum and gradient tracking to solve distributed non-convex optimization problems over time-varying networks. To analyze the algorithm, we tackle the challenges that arise when analyzing dynamic network systems which communicate gradient acceleration components. We prove that the algorithm's oracle complexity is $\mathcal{O}(1/\epsilon^{1.5})$, and that under Polyak-$Ł$ojasiewicz condition the algorithm converges linearly to a steady error state. The proposed scheme is tested on several learning tasks: a non-convex logistic regression experiment on the MNIST dataset, an image classification task on the CIFAR-10 dataset, and an NLP classification test on the IMDB dataset. We further present numerical simulations with an objective that satisfies the PL condition. The results demonstrate superior performance of the proposed framework compared to the existing related methods.

Distributed Online Optimization in Time-Varying Unbalanced Networks Without Explicit Subgradients

Privacy-Preserving Distributed Online Optimization Over Unbalanced Digraphs via Subgradient Rescaling

Distributed Randomized Gradient-free Convex Optimization with Set Constraints over Time-varying Weight-unbalanced Digraphs

Distributed Constrained Optimization over Unbalanced Time-Varying Digraphs: A Randomized Constraint Solving Algorithm

Distributed Subgradient-Free Stochastic Optimization Algorithm for Nonsmooth Convex Functions over Time-Varying Networks

Online Distributed Constrained Optimization over General Unbalanced Digraphs

Distributed Stochastic Optimization With Unbounded Subgradients Over Randomly Time-Varying Networks

Online Distributed Optimization With Nonconvex Objective Functions Under Unbalanced Digraphs: Dynamic Regret Analysis

Online Distributed Optimization with Nonconvex Objective Functions Via Dynamic Regrets

Accelerated Distributed Stochastic Non-Convex Optimization over Time-Varying Directed Networks

Random Sleep Scheme-Based Distributed Optimization Algorithm over Unbalanced Time-Varying Networks

Distributed Projection Subgradient Algorithm over Time-Varying General Unbalanced Directed Graphs

Distributed Proximal Gradient Algorithm for Nonconvex Optimization Over Time-Varying Networks

Decentralized Online Learning Methods Based on Weight-Balancing over Time-Varying Digraphs

Distributed Continuous-Time Algorithm for Nonsmooth Aggregative Optimization over Weight-Unbalanced Digraphs

Distributed Stochastic Subgradient Projection Algorithms Based on Weight-Balancing over Time-Varying Directed Graphs.

Distributed Online Optimization with Coupled Inequality Constraints over Unbalanced Directed Networks

Multi-Step Subgradient Methods for Distributed Optimization over Unbalanced Digraphs with Local Constraint Sets

Distributed Convex Optimization with Inequality Constraints over Time-Varying Unbalanced Digraphs.

Online distributed optimization with stochastic gradients: high probability bound of regrets

Regularized dual gradient distributed method for constrained convex optimization over unbalanced directed graphs