Abstract:Distributed stochastic non-convex optimization problems have recently received attention due to the growing interest of signal processing, computer vision, and natural language processing communities in applications deployed over distributed learning systems (e.g., federated learning). We study the setting where the data is distributed across the nodes of a time-varying directed network, a topology suitable for modeling dynamic networks experiencing communication delays and straggler effects. The network nodes, which can access only their local objectives and query a stochastic first-order oracle to obtain gradient estimates, collaborate to minimize a global objective function by exchanging messages with their neighbors. We propose an algorithm, novel to this setting, that leverages stochastic gradient descent with momentum and gradient tracking to solve distributed non-convex optimization problems over time-varying networks. To analyze the algorithm, we tackle the challenges that arise when analyzing dynamic network systems which communicate gradient acceleration components. We prove that the algorithm's oracle complexity is $\mathcal{O}(1/\epsilon^{1.5})$, and that under Polyak-$Ł$ojasiewicz condition the algorithm converges linearly to a steady error state. The proposed scheme is tested on several learning tasks: a non-convex logistic regression experiment on the MNIST dataset, an image classification task on the CIFAR-10 dataset, and an NLP classification test on the IMDB dataset. We further present numerical simulations with an objective that satisfies the PL condition. The results demonstrate superior performance of the proposed framework compared to the existing related methods.

A Fixed-Time Proximal Gradient Neurodynamic Network With Time-Varying Coefficients for Composite Optimization Problems and Sparse Optimization Problems With Log-Sum Function

Fixed-Time Neurodynamic Optimization Algorithms and Application to Circuits Design

A varying-parameter fixed-time gradient-based dynamic network for convex optimization

Adaptive Fixed-Time Proximal Gradient Method for Non-Smooth Optimization: the Fractional Approach

Linearized Proximal Algorithms with Adaptive Stepsizes for Convex Composite Optimization with Applications

On Convergence Rates of Linearized Proximal Algorithms for Convex Composite Optimization with Applications.

A discrete-time neural network for optimization problems with hybrid constraints.

A Unified Algorithmic Framework for Distributed Composite Optimization.

Distributed Algorithms for Composite Optimization: Unified Framework and Convergence Analysis

Fixed-Time Gradient Flows for Solving Constrained Optimization: A Unified Approach

Asynchronous Proximal Stochastic Gradient Algorithm for Composition Optimization Problems

Design and Analysis of a Novel Distributed Gradient Neural Network for Solving Consensus Problems in a Predefined Time

A Fractional-Order Gradient Neural Solution to Time-Variant Quadratic Programming With Application to Robot Motion Planning

Accelerated Primal-Dual Projection Neurodynamic Approach With Time Scaling for Linear and Set Constrained Convex Optimization Problems

A Novel Neural Network for Nonlinear Convex Programming

Time-varying neurodynamic optimization approaches with fixed-time convergence for sparse signal reconstruction

A mixture varying-gain dynamic learning network for solving nonlinear and nonconvex constrained optimization problems

Nonconvex Stochastic Bregman Proximal Gradient Method with Application to Deep Learning

Accelerated Distributed Stochastic Non-Convex Optimization over Time-Varying Directed Networks

Decentralized Stochastic Proximal Gradient Descent with Variance Reduction over Time-varying Networks

Distributed Proximal Alternating Direction Method of Multipliers for Constrained Composite Optimization Over Directed Networks