Abstract:Synchronous strategies with data parallelism are widely utilized in distributed training of Deep Neural Networks (DNNs), largely owing to their easy implementation yet promising performance. In these strategies, the workers with different computational capabilities need to wait for each other because of the essential gradient or weight synchronization. This will inevitably cause the high-performance workers to waste time waiting for the weak computational workers, which in turn results in the inefficiency of the cluster. In this paper, we propose a Dynamic Load Balance (DLB) strategy for the distributed training of DNNs to tackle this issue. Specifically, the performance of each worker is evaluated first based on the performance demonstration during the previous training epochs, and then the batch size and dataset partition are adaptively adjusted in consideration of the current performance of the workers. As a result, the waiting cost among the workers will be eliminated, thereby the utilization of the clusters is highly improved. Furthermore, the essential theoretical analysis has also been provided to justify the convergence of the proposed algorithm. Extensive experiments have been conducted on the CIFAR10 and CIFAR100 benchmark datasets with four state-of-the-art DNN models. The experimental results indicate that the proposed algorithm can significantly improve the utilization of the distributed cluster. In addition, the proposed algorithm can also prevent the load imbalance of the distributed DNN training from being affected by the disturbance and can be employed flexibly in conjunction with the other synchronous distributed DNN training methods.

Locality-aware and Fault-tolerant Batching for Machine Learning on Distributed Datasets

Optimize Scheduling of Federated Learning on Battery-powered Mobile Devices

LBB: load-balanced batching for efficient distributed learning on heterogeneous GPU cluster

Semi-Dynamic Load Balancing: Efficient Distributed Learning in Non-Dedicated Environments

EP4DDL: addressing straggler problem in heterogeneous distributed deep learning

Fast Distributed Deep Learning Via Worker-adaptive Batch Sizing

Adaptive Partitioning and Efficient Scheduling for Distributed DNN Training in Heterogeneous IoT Environment

SLoB: Suboptimal Load Balancing Scheduling in Local Heterogeneous GPU Clusters for Large Language Model Inference

Communication-Efficient Adaptive Batch Size Strategies for Distributed Local Gradient Methods

DLB: A Dynamic Load Balance Strategy for Distributed Training of Deep Neural Networks

Joint Dynamic Data and Model Parallelism for Distributed Training of DNNs over Heterogeneous Infrastructure

FedGSync: Jointly Optimized Weak Synchronization and Gradient Transmission for Fast Distributed Machine Learning in Heterogeneous WAN

FLSGD: Free Local SGD with Parallel Synchronization

Robust Fully-Asynchronous Methods for Distributed Training over General Architecture

Online Scheduling Algorithm for Heterogeneous Distributed Machine Learning Jobs

AdaCoOpt: Leverage the Interplay of Batch Size and Aggregation Frequency for Federated Learning.

Optimizing Communication Topology for Collaborative Learning Across Datacenters

Cloudless-Training: A Framework to Improve Efficiency of Geo-Distributed ML Training

A Scalable, High-Performance, and Fault-Tolerant Network Architecture for Distributed Machine Learning

Distributed Non-Convex Optimization with Sublinear Speedup under Intermittent Client Availability

LOAM: Low-latency Communication, Caching, and Computation Placement in Data-Intensive Computing Networks