Abstract:To afford the huge computational cost, large-scale deep neural networks (DNN) are usually trained on the distributed system, especially the widely-used parameter server architecture, consisting of a parameter server as well as multiple local workers with powerful GPU cards. During the training, local workers frequently pull the global model and push their computed gradients from/to the parameter server. Due to the limited bandwidth, such frequent communication will cause severe bottleneck for the training acceleration. As recent attempts to address this problem, quantization methods have been proposed to compress the gradients for efficient communication. However, such methods overlook the effects of compression on the model performance such that they either suffer from a low compression ratio or an accuracy drop. In this paper, to better address this problem, we investigate the distributed deep learning as a multi-agent system (MAS) problem. Specifically, 1) local workers and the parameter server are separate agents in the system; 2) the objective of these agents is to maximize the efficacy of the learned model through their cooperative interactions; 3) the strategy of the agents describes how they take actions, i.e. communicate their computed gradients or the global model; 4) rational agents always select the best-response strategy with the optimal utility. Inspired by this, we design a MAS approach for distributed training of DNN. In our method, the agents first estimate the utility (i.e., the benefit to help improve the model) of each action (i.e., transferring a subset of the gradients or the global model), and then take the best-response strategy based on their estimated utilities mixed with e-random exploration. We call our new method Slim-DP as it, being different from the standard data-parallelism, only communicates a subset of the gradient or the global model. Our experimental results demonstrate that our proposed Slim-DP can reduce more communication cost and achieve better speedup without loss of accuracy than the standard data parallelism and its quantization version.

SNAP: A Communication Efficient Distributed Machine Learning Framework for Edge Computing

Efficient Partitioning and Communication Scheme-Based Distributed Edge Computing to Accelerate Deep Neural Network

Condense: A Framework for Device and Frequency Adaptive Neural Network Models on the Edge.

AccEPT: an Acceleration Scheme for Speeding Up Edge Pipeline-parallel Training

Decentralized Edge Learning via Unreliable Device-to-Device Communications

Edge-Based Communication Optimization for Distributed Federated Learning

Adaptive Federated Learning in Resource Constrained Edge Computing Systems

Guest Editorial: Communication Technologies for Efficient Edge Learning

An Efficient Asynchronous Federated Learning Protocol for Edge Devices

An Introduction to Communication Efficient Edge Machine Learning

Learning-Driven Decentralized Machine Learning in Resource-Constrained Wireless Edge Computing.

DynaComm: Accelerating Distributed CNN Training between Edges and Clouds through Dynamic Communication Scheduling

Distributed Machine Learning through Heterogeneous Edge Systems

Communication-Efficient Separable Neural Network for Distributed Inference on Edge Devices

Communication-efficient Decentralized Machine Learning over Heterogeneous Networks

EdgePS: Selective Parameter Aggregation for Distributed Machine Learning in Edge Computing

When Less is More: Achieving Faster Convergence in Distributed Edge Machine Learning

Snowball: Energy Efficient and Accurate Federated Learning with Coarse-to-Fine Compression over Heterogeneous Wireless Edge Devices

Accelerating DNN Training in Wireless Federated Edge Learning Systems

Slim-DP: A Multi-Agent System for Communication-Efficient Distributed Deep Learning

Consensus Based Multi-Layer Perceptrons for Edge Computing