Abstract:We study federated machine learning at the wireless network edge, where limited power wireless devices, each with its own dataset, build a joint model with the help of a remote parameter server (PS). We consider a bandwidth-limited fading multiple access channel (MAC) from the wireless devices to the PS, and propose various techniques to implement distributed stochastic gradient descent (DSGD). We first propose a digital DSGD (D-DSGD) scheme, in which one device is selected opportunistically for transmission at each iteration based on the channel conditions; the scheduled device quantizes its gradient estimate to a finite number of bits imposed by the channel condition, and transmits these bits to the PS in a reliable manner. Next, motivated by the additive nature of the wireless MAC, we propose a novel analog communication scheme, referred to as the compressed analog DSGD (CA-DSGD), where the devices first sparsify their gradient estimates while accumulating error, and project the resultant sparse vector into a low-dimensional vector for bandwidth reduction. Numerical results show that D-DSGD outperforms other digital approaches in the literature; however, in general the proposed CA-DSGD algorithm converges faster than the D-DSGD scheme and other schemes in the literature, and reaches a higher level of accuracy. We have observed that the gap between the analog and digital schemes increases when the datasets of devices are not independent and identically distributed (i.i.d.). Furthermore, the performance of the CA-DSGD scheme is shown to be robust against imperfect channel state information (CSI) at the devices. Overall these results show clear advantages for the proposed analog over-the-air DSGD scheme, which suggests that learning and communication algorithms should be designed jointly to achieve the best end-to-end performance in machine learning applications at the wireless edge.

Device Selection of Distributed Primal-Dual Algorithms over Wireless Networks.

Accelerated Primal-Dual Algorithms for Distributed Smooth Convex Optimization over Networks

Distributed Mimo Network Optimization Based On Duality And Local Message Passing

Primal-Dual Algorithm for Distributed Optimization with Coupled Constraints

Asymptotic Properties of Primal-Dual Algorithm for Distributed Stochastic Optimization Over Random Networks

Distributed and Optimal Reduced Primal-Dual Algorithm for Uplink OFDM Resource Allocation

Adaptive Cluster Head Selection and Spectrum Allocation for D2D-Enabled Collaborative Learning

Distributed Extremum Seeking And Cooperative Control For Mobile Communication

Federated Learning over Wireless Fading Channels

Optimizing Synchronization Delay for Digital Twin over Wireless Networks

Distributed Extremum Seeking and Formation Control for Nonholonomic Mobile Network.

Decentralized Edge Learning via Unreliable Device-to-Device Communications

Distributed joint optimization of relay selection and subchannel pairing in OFDM based relay networks

Accelerating Distributed Optimization: A Primal-Dual Perspective on Local Steps

Deep Reinforcement Learning for Joint Channel Selection and Power Control in D2D Networks

Hybrid Centralized-Distributed Resource Allocation Based on Deep Reinforcement Learning for Cooperative D2D Communications

Distributed Asynchronous Dual Free Stochastic Dual Coordinate Ascent

Distributed fixed-point algorithms for dynamic convex optimization over decentralized and unbalanced wireless networks

Multi-consensus decentralized primal-dual fixed point algorithm for distributed learning

A primal-dual perspective for distributed TD-learning

Online Power Control for Distributed Multitask Learning Over Noisy Fading Wireless Channels