Abstract:To afford the huge computational cost, large-scale deep neural networks (DNN) are usually trained on the distributed system, especially the widely-used parameter server architecture, consisting of a parameter server as well as multiple local workers with powerful GPU cards. During the training, local workers frequently pull the global model and push their computed gradients from/to the parameter server. Due to the limited bandwidth, such frequent communication will cause severe bottleneck for the training acceleration. As recent attempts to address this problem, quantization methods have been proposed to compress the gradients for efficient communication. However, such methods overlook the effects of compression on the model performance such that they either suffer from a low compression ratio or an accuracy drop. In this paper, to better address this problem, we investigate the distributed deep learning as a multi-agent system (MAS) problem. Specifically, 1) local workers and the parameter server are separate agents in the system; 2) the objective of these agents is to maximize the efficacy of the learned model through their cooperative interactions; 3) the strategy of the agents describes how they take actions, i.e. communicate their computed gradients or the global model; 4) rational agents always select the best-response strategy with the optimal utility. Inspired by this, we design a MAS approach for distributed training of DNN. In our method, the agents first estimate the utility (i.e., the benefit to help improve the model) of each action (i.e., transferring a subset of the gradients or the global model), and then take the best-response strategy based on their estimated utilities mixed with e-random exploration. We call our new method Slim-DP as it, being different from the standard data-parallelism, only communicates a subset of the gradient or the global model. Our experimental results demonstrate that our proposed Slim-DP can reduce more communication cost and achieve better speedup without loss of accuracy than the standard data parallelism and its quantization version.

SimpleFSDP: Simpler Fully Sharded Data Parallel with torch.compile

PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel

OF-WFBP: A near-optimal communication mechanism for tensor fusion in distributed deep learning

OSDP: Optimal Sharded Data Parallel for Distributed Deep Learning

Exploiting Simultaneous Communications to Accelerate Data Parallel Distributed Deep Learning

FPDeep: Scalable Acceleration of CNN Training on Deeply-Pipelined FPGA Clusters

Optimizing DNN Compilation for Distributed Training with Joint OP and Tensor Fusion

SDP4Bit: Toward 4-bit Communication Quantization in Sharded Data Parallelism for LLM Training

Accelerating Distributed K-FAC with Smart Parallelism of Computing and Communication Tasks

FSP: Towards Flexible Synchronous Parallel Frameworks for Distributed Machine Learning

FRED: Flexible REduction-Distribution Interconnect and Communication Implementation for Wafer-Scale Distributed Training of DNN Models

ZeroPP: Unleashing Exceptional Parallelism Efficiency through Tensor-Parallelism-Free Methodology

Accelerated FDPS — Algorithms to Use Accelerators with FDPS

Octopus-DF: Unified DataFrame-based cross-platform data analytic system

FP-DNN: an Automated Framework for Mapping Deep Neural Networks Onto FPGAs with RTL-HLS Hybrid Templates

TorchTitan: One-stop PyTorch native solution for production ready LLM pre-training

Optimizing execution for pipelined‐based distributed deep learning in a heterogeneously networked GPU cluster

DP-FFN: Block-Based Dynamic Pooling for Accelerating Feed-Forward Layers in Transformers

ParallelSFL: A Novel Split Federated Learning Framework Tackling Heterogeneity Issues

SDCC: Software-Defined Collective Communication for Distributed Training

Slim-DP: A Multi-Agent System for Communication-Efficient Distributed Deep Learning