Abstract:Many distributed training techniques like Parameter Server and AllReduce have been proposed to take advantage of the increasingly large data and rich features. However, stragglers frequently occur in distributed training due to resource contention and hardware heterogeneity, which significantly hampers the training efficiency. Previous works only address part of the stragglers and could not adaptively solve various stragglers in practice. Additionally, it is challenging to use a systematic framework to address all stragglers because different stragglers require diverse data allocation and fault-tolerance mechanisms. Therefore, this paper proposes a unified distributed training framework called AntDT (Ant Distributed Training Framework) to adaptively solve the straggler problems. Firstly, the framework consists of four components, including the Stateful Dynamic Data Sharding service, Monitor, Controller, and Agent. These components work collaboratively to efficiently distribute workloads and provide a range of pre-defined straggler mitigation methods with fault tolerance, thereby hiding messy details of data allocation and fault handling. Secondly, the framework provides a high degree of flexibility, allowing for the customization of straggler mitigation solutions based on the specific circumstances of the cluster. Leveraging this flexibility, we introduce two straggler mitigation solutions, namely AntDT-ND for non-dedicated clusters and AntDT-DD for dedicated clusters, as practical examples to resolve various types of stragglers at Ant Group. Justified by our comprehensive experiments and industrial deployment statistics, AntDT outperforms other SOTA methods more than 3x in terms of training efficiency. Additionally, in Alipay's homepage recommendation scenario, using AntDT reduces the training duration of the ranking model from 27.8 hours to just 5.4 hours.

EP4DDL: addressing straggler problem in heterogeneous distributed deep learning

WBSP: Addressing Stragglers in Distributed Machine Learning with Worker-Busy Synchronous Parallel

Straggler-Aware Gradient Aggregation for Large-Scale Distributed Deep Learning System

GSSP: Eliminating Stragglers Through Grouping Synchronous for Distributed Deep Learning in Heterogeneous Cluster.

A Comprehensive Inspection of the Straggler Problem

Falcon: Addressing Stragglers in Heterogeneous Parameter Server Via Multiple Parallelism

Falcon: Towards Computation-Parallel Deep Learning in Heterogeneous Parameter Server

LBB: load-balanced batching for efficient distributed learning on heterogeneous GPU cluster

DSANA: A Distributed Machine Learning Acceleration Solution Based on Dynamic Scheduling and Network Acceleration

Joint Dynamic Data and Model Parallelism for Distributed Training of DNNs over Heterogeneous Infrastructure

Joint Dynamic Grouping and Gradient Coding for Time-Critical Distributed Machine Learning in Heterogeneous Edge Networks

DLB: A Dynamic Load Balance Strategy for Distributed Training of Deep Neural Networks

A Load-Balancing Strategy Based on Multi-Task Learning in a Distributed Training Environment

AntDT: A Self-Adaptive Distributed Training Framework for Leader and Straggler Nodes

Design and Optimization of Hierarchical Gradient Coding for Distributed Learning at Edge Devices

Efficient Distributed Machine Learning with Trigger Driven Parallel Training.

Straggler-aware Distributed Learning: Communication Computation Latency Trade-off

Accelerating Distributed Learning in Non-Dedicated Environments

Scaling Deep Learning on GPU and Knights Landing clusters

DPro-SM - A distributed framework for proactive straggler mitigation using LSTM

Heterogeneity-Aware Distributed Parameter Servers