Abstract:Distributed stochastic non-convex optimization problems have recently received attention due to the growing interest of signal processing, computer vision, and natural language processing communities in applications deployed over distributed learning systems (e.g., federated learning). We study the setting where the data is distributed across the nodes of a time-varying directed network, a topology suitable for modeling dynamic networks experiencing communication delays and straggler effects. The network nodes, which can access only their local objectives and query a stochastic first-order oracle to obtain gradient estimates, collaborate to minimize a global objective function by exchanging messages with their neighbors. We propose an algorithm, novel to this setting, that leverages stochastic gradient descent with momentum and gradient tracking to solve distributed non-convex optimization problems over time-varying networks. To analyze the algorithm, we tackle the challenges that arise when analyzing dynamic network systems which communicate gradient acceleration components. We prove that the algorithm's oracle complexity is $\mathcal{O}(1/\epsilon^{1.5})$, and that under Polyak-$Ł$ojasiewicz condition the algorithm converges linearly to a steady error state. The proposed scheme is tested on several learning tasks: a non-convex logistic regression experiment on the MNIST dataset, an image classification task on the CIFAR-10 dataset, and an NLP classification test on the IMDB dataset. We further present numerical simulations with an objective that satisfies the PL condition. The results demonstrate superior performance of the proposed framework compared to the existing related methods.

Dual-based Online Learning of Dynamic Network Topologies

Online Topology Inference from Streaming Stationary Graph Signals with Partial Connectivity Information

Online Learning Of Expanding Graphs

Online Graph Learning under Smoothness Priors

Online Proximal ADMM for Graph Learning from Streaming Smooth Signals

Online Joint Topology Identification and Signal Estimation from Streams with Missing Data

Topology Learning of unknown Networked Linear Dynamical System excited by Cyclostationary inputs

Online Network Inference from Graph-Stationary Signals with Hidden Nodes

Projection-free Distributed Online Learning in Networks

Accelerated Distributed Stochastic Non-Convex Optimization over Time-Varying Directed Networks

Online Deep Learning: Learning Deep Neural Networks on the Fly

Topological exploration of artificial neuronal network dynamics

Discretized Distributed Optimization over Dynamic Digraphs

Network Topology Inference from Smooth Signals Under Partial Observability

Learning to Learn Graph Topologies

Random Feature Approximation for Online Nonlinear Graph Topology Identification

Online Graph Filtering Over Expanding Graphs

Simultaneous Topology Estimation and Synchronization of Dynamical Networks with Time-varying Topology

An Incremental Neural Network for Online Supervised Learning and Topology Learning

Dynamic Topological Data Analysis of Functional Human Brain Networks

Dynamic Neural Dowker Network: Approximating Persistent Homology in Dynamic Directed Graphs