Abstract:Synchronous strategies with data parallelism, such as the Synchronous StochasticGradient Descent (S-SGD) and the model averaging methods, are widely utilizedin distributed training of Deep Neural Networks (DNNs), largely owing to itseasy implementation yet promising performance. Particularly, each worker ofthe cluster hosts a copy of the DNN and an evenly divided share of the datasetwith the fixed mini-batch size, to keep the training of DNNs convergence. In thestrategies, the workers with different computational capability, need to wait foreach other because of the synchronization and delays in network transmission,which will inevitably result in the high-performance workers wasting <a class="link-external link-http" href="http://computation.Consequently" rel="external noopener nofollow">this http URL</a>, the utilization of the cluster is relatively low. To alleviate thisissue, we propose the Dynamic Batch Size (DBS) strategy for the distributedtraining of DNNs. Specifically, the performance of each worker is evaluatedfirst based on the fact in the previous epoch, and then the batch size and datasetpartition are dynamically adjusted in consideration of the current performanceof the worker, thereby improving the utilization of the cluster. To verify theeffectiveness of the proposed strategy, extensive experiments have been conducted,and the experimental results indicate that the proposed strategy can fully utilizethe performance of the cluster, reduce the training time, and have good robustnesswith disturbance by irrelevant tasks. Furthermore, rigorous theoretical analysis hasalso been provided to prove the convergence of the proposed strategy.

Distributed Deep Learning for Question Answering

Multi-agent Deep Reinforcement Learning Algorithm for Distributed Economic Dispatch in Smart Grid.

Target-Value-Competition-Based Multi-Agent Deep Reinforcement Learning Algorithm for Distributed Nonconvex Economic Dispatch

DaSGD: Squeezing SGD Parallelization Performance in Distributed Training Using Delayed Averaging

A Hierarchical Communication Algorithm for Distributed Deep Learning Training.

Adjacent Leader Decentralized Stochastic Gradient Descent

Distributed Active Learning.

CADA: Communication-Adaptive Distributed Adam

EDiT: A Local-SGD-Based Efficient Distributed Training Method for Large Language Models

Network-Aware Distributed Machine Learning Over Wide Area Network

Distributed Adaptive Subgradient Algorithms for Online Learning over Time-Varying Networks

A DAG Model of Synchronous Stochastic Gradient Descent in Distributed Deep Learning

Asynchronous Stochastic Gradient Descent with Delay Compensation for Distributed Deep Learning.

A Survey From Distributed Machine Learning to Distributed Deep Learning

Communication-Efficient Distributed Deep Learning: Survey, Evaluation, and Challenges.

AutoDDL: Automatic Distributed Deep Learning With Near-Optimal Bandwidth Cost

Adaptive Worker Grouping For Communication-Efficient and Straggler-Tolerant Distributed SGD

EP4DDL: addressing straggler problem in heterogeneous distributed deep learning

Peering Beyond the Gradient Veil with Distributed Auto Differentiation

DISTRIBUTED HIGH-PERFORMANCE COMPUTING METHODS FOR ACCELERATING DEEP LEARNING TRAINING

DBS: Dynamic Batch Size For Distributed Deep Neural Network Training