Abstract:Distributed stochastic non-convex optimization problems have recently received attention due to the growing interest of signal processing, computer vision, and natural language processing communities in applications deployed over distributed learning systems (e.g., federated learning). We study the setting where the data is distributed across the nodes of a time-varying directed network, a topology suitable for modeling dynamic networks experiencing communication delays and straggler effects. The network nodes, which can access only their local objectives and query a stochastic first-order oracle to obtain gradient estimates, collaborate to minimize a global objective function by exchanging messages with their neighbors. We propose an algorithm, novel to this setting, that leverages stochastic gradient descent with momentum and gradient tracking to solve distributed non-convex optimization problems over time-varying networks. To analyze the algorithm, we tackle the challenges that arise when analyzing dynamic network systems which communicate gradient acceleration components. We prove that the algorithm's oracle complexity is $\mathcal{O}(1/\epsilon^{1.5})$, and that under Polyak-$Ł$ojasiewicz condition the algorithm converges linearly to a steady error state. The proposed scheme is tested on several learning tasks: a non-convex logistic regression experiment on the MNIST dataset, an image classification task on the CIFAR-10 dataset, and an NLP classification test on the IMDB dataset. We further present numerical simulations with an objective that satisfies the PL condition. The results demonstrate superior performance of the proposed framework compared to the existing related methods.

A Distributed Algorithm For Dictionary Learning Over Networks

Accelerated Primal-Dual Algorithms for Distributed Smooth Convex Optimization over Networks

Decentralized Non-Convex Learning With Linearly Coupled Constraints: Algorithm Designs and Application to Vertical Learning Problem

Distributed Analysis Dictionary Learning Using a Diffusion Strategy

L0 Norm Based Dictionary Learning by Proximal Methods with Global Convergence

Distributed Online Private Learning of Convex Nondecomposable Objectives

A Push-Pull Gradient Method for Distributed Optimization in Networks.

Distributed Dictionary Learning over Heterogeneous Clients Using Local Adaptive Dictionaries

Accelerated Distributed Stochastic Non-Convex Optimization over Time-Varying Directed Networks

A Decentralized Proximal Point-type Method for Saddle Point Problems

A Forward-Backward Bregman Splitting Scheme for Regularized Distributed Optimization Problems

Distributed Robust Optimization in Networked System.

Distributed Primal-Dual Proximal Algorithms for Convex Optimization Involving Three Composite Functions

A Deep Reinforcement Learning Approach to Efficient Distributed Optimization

Proximal ADMM for Nonconvex and Nonsmooth Optimization

An Efficient Orthonormalization-Free Approach for Sparse Dictionary Learning and Dual Principal Component Pursuit

Decentralized Learning with Lazy and Approximate Dual Gradients

Multi-consensus decentralized primal-dual fixed point algorithm for distributed learning

An Asynchronous Distributed Proximal Gradient Method for Composite Convex Optimization

A New Randomized Block-Coordinate Primal-Dual Proximal Algorithm for Distributed Optimization

A Mixing-Accelerated Primal-Dual Proximal Algorithm for Distributed Nonconvex Optimization