Abstract:We study federated machine learning at the wireless network edge, where limited power wireless devices, each with its own dataset, build a joint model with the help of a remote parameter server (PS). We consider a bandwidth-limited fading multiple access channel (MAC) from the wireless devices to the PS, and propose various techniques to implement distributed stochastic gradient descent (DSGD). We first propose a digital DSGD (D-DSGD) scheme, in which one device is selected opportunistically for transmission at each iteration based on the channel conditions; the scheduled device quantizes its gradient estimate to a finite number of bits imposed by the channel condition, and transmits these bits to the PS in a reliable manner. Next, motivated by the additive nature of the wireless MAC, we propose a novel analog communication scheme, referred to as the compressed analog DSGD (CA-DSGD), where the devices first sparsify their gradient estimates while accumulating error, and project the resultant sparse vector into a low-dimensional vector for bandwidth reduction. Numerical results show that D-DSGD outperforms other digital approaches in the literature; however, in general the proposed CA-DSGD algorithm converges faster than the D-DSGD scheme and other schemes in the literature, and reaches a higher level of accuracy. We have observed that the gap between the analog and digital schemes increases when the datasets of devices are not independent and identically distributed (i.i.d.). Furthermore, the performance of the CA-DSGD scheme is shown to be robust against imperfect channel state information (CSI) at the devices. Overall these results show clear advantages for the proposed analog over-the-air DSGD scheme, which suggests that learning and communication algorithms should be designed jointly to achieve the best end-to-end performance in machine learning applications at the wireless edge.

A Distributed Learning based on Robust Diffusion SGD over Adaptive Networks with noisy output data

Variance Reduced Diffusion Adaptation for Online Learning over Networks

Convergence behavior of diffusion stochastic gradient descent algorithm

A(DP)$^2$SGD: Asynchronous Decentralized Parallel Stochastic Gradient Descent with Differential Privacy

Asynchronous Byzantine-Robust Stochastic Aggregation with Variance Reduction for Distributed Learning

Stochastic Analysis of Diffusion Dual Sign Algorithm for Cyclostationary White Gaussian Inputs and Nonstationary System

Byzantine-Robust Distributed Learning: Towards Optimal Statistical Rates

Efficient Byzantine-Resilient Stochastic Gradient Desce

Federated Learning over Wireless Fading Channels

Maximum Total Correntropy Diffusion Adaptation over Networks with Noisy Links

A(DP)$^2$2SGD: Asynchronous Decentralized Parallel Stochastic Gradient Descent with Differential Privacy

Decentralized Federated Learning with Unreliable Communications

Self-regularized Nonlinear Diffusion Algorithm Based on Levenberg Gradient Descent

Robust distributed estimation based on a generalized correntropy logarithmic difference algorithm over wireless sensor networks

Diffusion adagrad minimum kernel risk sensitive mean p-power loss algorithm

Holdout SGD: Byzantine Tolerant Federated Learning

Adaptive Stochastic Gradient Descent for Fast and Communication-Efficient Distributed Learning

Robust Fully-Asynchronous Methods for Distributed Training over General Architecture

Diffusion Strategies Outperform Consensus Strategies for Distributed Estimation over Adaptive Networks

Distributed Stochastic Gradient Descent with Staleness: A Stochastic Delay Differential Equation Based Framework

Robust Distributed Learning Against Both Distributional Shifts and Byzantine Attacks