Abstract:Decentralized optimization over time-varying graphs has been increasingly common in modern machine learning with massive data stored on millions of mobile devices, such as in federated learning. This paper revisits the widely used accelerated gradient tracking and extends it to time-varying graphs. We prove that the practical single loop accelerated gradient tracking needs $O((\frac{\gamma}{1-\sigma_{\gamma}})^2\sqrt{\frac{L}{\epsilon}})$ and $O((\frac{\gamma}{1-\sigma_{\gamma}})^{1.5}\sqrt{\frac{L}{\mu}}\log\frac{1}{\epsilon})$ iterations to reach an $\epsilon$-optimal solution over time-varying graphs when the problems are nonstrongly convex and strongly convex, respectively, where $\gamma$ and $\sigma_{\gamma}$ are two common constants charactering the network connectivity, $L$ and $\mu$ are the smoothness and strong convexity constants, respectively, and one iteration corresponds to one gradient oracle call and one communication round. Our convergence rates improve significantly over the ones of $O(\frac{1}{\epsilon^{5/7}})$ and $O((\frac{L}{\mu})^{5/7}\frac{1}{(1-\sigma)^{1.5}}\log\frac{1}{\epsilon})$, respectively, which were proved in the original literature of accelerated gradient tracking only for static graphs, where $\frac{\gamma}{1-\sigma_{\gamma}}$ equals $\frac{1}{1-\sigma}$ when the network is time-invariant. When combining with a multiple consensus subroutine, the dependence on the network connectivity constants can be further improved to $O(1)$ and $O(\frac{\gamma}{1-\sigma_{\gamma}})$ for the gradient oracle and communication round complexities, respectively. When the network is static, by employing the Chebyshev acceleration, our complexities exactly match the lower bounds without hiding any poly-logarithmic factor for both nonstrongly convex and strongly convex problems.

A varying-parameter fixed-time gradient-based dynamic network for convex optimization

A gradient-type noise-tolerant finite-time neural network for convex optimization

A discrete-time neural network for optimization problems with hybrid constraints.

Fixed-Time Gradient Flows for Solving Constrained Optimization: A Unified Approach

A Fixed-Time Proximal Gradient Neurodynamic Network With Time-Varying Coefficients for Composite Optimization Problems and Sparse Optimization Problems With Log-Sum Function

Accelerated Distributed Stochastic Non-Convex Optimization over Time-Varying Directed Networks

Convergence in High Probability of Distributed Stochastic Gradient Descent Algorithms

Accelerated Gradient-free Neural Network Training by Multi-convex Alternating Optimization

A Novel Neural Network for Nonlinear Convex Programming

Fast convex optimization via closed-loop time scaling of gradient dynamics

A privacy-preserving decentralized randomized block-coordinate subgradient algorithm over time-varying networks

Breaking the Convergence Barrier: Optimization via Fixed-Time Convergent Flows

A Convergence Theory Towards Practical Over-parameterized Deep Neural Networks

Accelerated Gradient Tracking over Time-varying Graphs for Decentralized Optimization

Global Convergence of Langevin Dynamics Based Algorithms for Nonconvex Optimization

Finite-Time Nonconvex Optimization Using Time-Varying Dynamical Systems

Nonlinear Perturbation-based Non-Convex Optimization over Time-Varying Networks

A mixture varying-gain dynamic learning network for solving nonlinear and nonconvex constrained optimization problems

Dynamic of Stochastic Gradient Descent with State-Dependent Noise

Subgradient-based Feedback Neural Networks for Non-Differentiable Convex Optimization Problems

A Dynamical View on Optimization Algorithms of Overparameterized Neural Networks