Abstract:To afford the huge computational cost, large-scale deep neural networks (DNN) are usually trained on the distributed system, especially the widely-used parameter server architecture, consisting of a parameter server as well as multiple local workers with powerful GPU cards. During the training, local workers frequently pull the global model and push their computed gradients from/to the parameter server. Due to the limited bandwidth, such frequent communication will cause severe bottleneck for the training acceleration. As recent attempts to address this problem, quantization methods have been proposed to compress the gradients for efficient communication. However, such methods overlook the effects of compression on the model performance such that they either suffer from a low compression ratio or an accuracy drop. In this paper, to better address this problem, we investigate the distributed deep learning as a multi-agent system (MAS) problem. Specifically, 1) local workers and the parameter server are separate agents in the system; 2) the objective of these agents is to maximize the efficacy of the learned model through their cooperative interactions; 3) the strategy of the agents describes how they take actions, i.e. communicate their computed gradients or the global model; 4) rational agents always select the best-response strategy with the optimal utility. Inspired by this, we design a MAS approach for distributed training of DNN. In our method, the agents first estimate the utility (i.e., the benefit to help improve the model) of each action (i.e., transferring a subset of the gradients or the global model), and then take the best-response strategy based on their estimated utilities mixed with e-random exploration. We call our new method Slim-DP as it, being different from the standard data-parallelism, only communicates a subset of the gradient or the global model. Our experimental results demonstrate that our proposed Slim-DP can reduce more communication cost and achieve better speedup without loss of accuracy than the standard data parallelism and its quantization version.

A Hybrid Data and Model Transfer Framework for Distributed Machine Learning

Hierarchical federated learning based on wireless D2D networks

Enhanced Hybrid Hierarchical Federated Edge Learning Over Heterogeneous Networks

Decentralized Federated Learning under Communication Delays.

Decentralized Edge Learning via Unreliable Device-to-Device Communications

Communication-Efficient Learning of Deep Networks from Decentralized Data

Joint Dynamic Data and Model Parallelism for Distributed Training of DNNs over Heterogeneous Infrastructure

Distributed Model Reuse with Multiple Classifiers

Efficient and Less Centralized Federated Learning

Communication Efficient Federated Learning With Heterogeneous Structured Client Models

Slim-DP: A Multi-Agent System for Communication-Efficient Distributed Deep Learning

Distributed Machine Learning in D2D-Enabled Heterogeneous Networks: Architectures, Performance, and Open Challenges

Distributed Machine Learning Based Link Allocation Strategy

Meta-Learning for Speeding Up Large Model Inference in Decentralized Environments

A Hybrid Federated Learning Framework with Dynamic Task Allocation for Multi-Party Distributed Load Prediction

Enhancing Decentralized Federated Learning with Model Pruning and Adaptive Communication

A Hierarchical Communication Algorithm for Distributed Deep Learning Training.

Optimizing Hierarchical Federated Learning: A Reinforcement Learning Approach

Decentralized Federated Learning with Data Feature Transmission and Neighbor Selection.

Decentralized Deep Learning using Momentum-Accelerated Consensus

A Hierarchical Knowledge Transfer Framework for Heterogeneous Federated Learning.