Abstract:Decentralized optimization is an emerging paradigm in distributed learning in which agents achieve network-wide solutions by peer-to-peer communication without the central server. Since communication tends to be slower than computation, when each agent communicates with only a few neighboring agents per iteration, they can complete iterations faster than with more agents or a central server. However, the total number of iterations to reach a network-wide solution is affected by the speed at which the agents' information is ``mixed'' by communication. We found that popular communication topologies either have large maximum degrees (such as stars and complete graphs) or are ineffective at mixing information (such as rings and grids). To address this problem, we propose a new family of topologies, EquiTopo, which has an (almost) constant degree and a network-size-independent consensus rate that is used to measure the mixing efficiency. In the proposed family, EquiStatic has a degree of $\Theta(\ln(n))$, where $n$ is the network size, and a series of time-dependent one-peer topologies, EquiDyn, has a constant degree of 1. We generate EquiDyn through a certain random sampling procedure. Both of them achieve an $n$-independent consensus rate. We apply them to decentralized SGD and decentralized gradient tracking and obtain faster communication and better convergence, theoretically and empirically. Our code is implemented through BlueFog and available at \url{https://github.com/kexinjinnn/EquiTopo}

Asynchronous Broadcast-Based Decentralized Learning in Sensor Networks.

Decentralized Learning over Wireless Networks with Broadcast-Based Subgraph Sampling

Faster Convergence with Less Communication: Broadcast-Based Subgraph Sampling for Decentralized Learning over Wireless Networks

Distributed Local Broadcasting Algorithms in the Physical Interference Model.

Decentralized Learning over Wireless Networks: The Effect of Broadcast with Random Access

A Graph Neural Network Based Decentralized Learning Scheme

Decentralized learning for wireless communications and networking

Asynchronous Federated Learning over Wireless Communication Networks.

Scale-Robust Timely Asynchronous Decentralized Learning

Asynchronous Decentralized Learning over Unreliable Wireless Networks

Learn to Sense: a Meta-learning Based Distributed Sensing Algorithm in Wireless Sensor Networks

Distributed Consensus Filtering Over Sensor Networks With Asynchronous Measurements

Distributed Censored Regression over Networks

Chronic right-sided intervertebral disk herniation at T13-L1 and acute left-sided disk herniation at L2-3.

Communication-Efficient Topologies for Decentralized Learning with O(1) Consensus Rate

Decentralized Edge Learning via Unreliable Device-to-Device Communications

An Implementation of Decentralized Consensus Building in Sensor Networks

Learn to Sense: A Meta-Learning-Based Sensing and Fusion Framework for Wireless Sensor Networks

Distributed Kalman Filtering for Sensor Network with Balanced Topology

Energy-efficient Decentralized Learning via Graph Sparsification

Asynchronous Decentralized Online Learning