Abstract:It is usually infeasible to fit and train an entire large deep neural network (DNN) model using a single edge device due to the limited resources. To facilitate intelligent applications across edge devices, researchers have proposed partitioning a large model into several sub-models, and deploying each of them to a different edge device to collaboratively train a DNN model. However, the communication overhead caused by the large amount of data transmitted from one device to another during training, as well as the sub-optimal partition point due to the inaccurate latency prediction of computation at each edge device can significantly slow down training. In this paper, we propose AccEPT, an acceleration scheme for accelerating the edge collaborative pipeline-parallel training. In particular, we propose a light-weight adaptive latency predictor to accurately estimate the computation latency of each layer at different devices, which also adapts to unseen devices through continuous learning. Therefore, the proposed latency predictor leads to better model partitioning which balances the computation loads across participating devices. Moreover, we propose a bit-level computation-efficient data compression scheme to compress the data to be transmitted between devices during training. Our numerical results demonstrate that our proposed acceleration approach is able to significantly speed up edge pipeline parallel training up to 3 times faster in the considered experimental settings.

PipePar: A Pipelined Hybrid Parallel Approach for Accelerating Distributed DNN Training

PipeDream: Fast and Efficient Pipeline Parallel DNN Training

GraphPipe: Improving Performance and Scalability of DNN Training with Graph Pipeline Parallelism

BaPipe: Exploration of Balanced Pipeline Parallelism for DNN Training

Extendable Multi-Device Collaborative Pipeline Parallel Inference in the Edge-Cloud Scenario

DAPPLE: A Pipelined Data Parallel Approach for Training Large Models

HetPipe: Enabling Large DNN Training on (Whimpy) Heterogeneous GPU Clusters through Integration of Pipelined Model Parallelism and Data Parallelism

Efficient and Robust Parallel DNN Training through Model Parallelism on Multi-GPU Platform

Coded Parallelism for Distributed Deep Learning.

GNNPipe: Scaling Deep GNN Training with Pipelined Model Parallelism

vPipe: A Virtualized Acceleration System for Achieving Efficient and Scalable Pipeline Parallel DNN Training

XPipe: Efficient Pipeline Model Parallelism for Multi-GPU DNN Training

HPH: Hybrid Parallelism on Heterogeneous Clusters for Accelerating Large-scale DNNs Training.

ElasticPipe

PipeMare: Asynchronous Pipeline Parallel DNN Training

Adaptive Partitioning and Efficient Scheduling for Distributed DNN Training in Heterogeneous IoT Environment

Optimizing DNN Training with Pipeline Model Parallelism for Enhanced Performance in Embedded Systems

A bidirectional DNN partition mechanism for efficient pipeline parallel training in cloud

AccEPT: an Acceleration Scheme for Speeding Up Edge Pipeline-parallel Training

Optimizing execution for pipelined‐based distributed deep learning in a heterogeneously networked GPU cluster