Abstract:The demand for large-scale deep learning is increasing, and distributed training is the current mainstream solution. Ring AllReduce is widely used as a data parallel decentralized algorithm. However, in a heterogeneous environment, each worker calculates the same amount of data, so that there is a lot of waiting time loss among different workers, which makes the algorithm unable to adapt well to heterogeneous clusters. Resources are not used as they should be. In this paper, we design an implementation of static allocation algorithm. The dataset is artificially allocated to each worker, and samples are drawn proportionally for training, thereby speeding up the training speed of the network in a heterogeneous environment. We verify the convergence and influence on training speed of the network model under this algorithm on one machine with multi-card and multi-machine with multi-card. On this basis of feasibility, we propose a self-adaptive allocation algorithm that allows each machine to find the data it needs to adapt to the current environment. The self-adaptive allocation algorithm can reduce the training time by nearly one-third to half compared to the same proportional <a class="link-external link-http" href="http://allocation.In" rel="external noopener nofollow">this http URL</a> order to better show the applicability of the algorithm in heterogeneous clusters, We replace a poorly performing worker with a good performing worker or add a poorly performing worker to the heterogeneous cluster. Experimental results show that training time will decrease as the overall performance improves. Therefore, it means that resources are fully used. Further, this algorithm is not only suitable for straggler problems, but also for most heterogeneous situations. It can be used as a plug-in for AllReduce and its variant algorithms.

A Fair Scheduling Algorithm for Adaptive Heterogeneous Resources in Data Centers

Multiple Resources Scheduling for Diverse Workloads in Heterogeneous Datacenter

A Novel Job Scheduling Model to Enhance Efficiency and Overall User Fairness of Cloud Computing Environment.

Smart-DRS: A Strategy of Dynamic Resource Scheduling in Cloud Data Center

An Integrated Dynamic Resource Scheduling Framework in On-Demand Clouds

An Integrated Dynamic Resource Scheduling Framework in On-Demand Clouds.

Low Complexity Hierarchical Scheduling for Diverse Datacenter Jobs.

A Resource Co-Allocation Method for Load-Balance Scheduling over Big Data Platforms

Energy Aware Task Scheduling in Data Centers.

Online Task Scheduling for Fog Computing with Multi-Resource Fairness

Multi-resource Fair Allocation with Bounded Number of Tasks in Cloud Computing Systems

DRESS: Dynamic RESource-Reservation Scheme for Congested Data-Intensive Computing Platforms

Energy-Efficient Real-Time Task Allocation in a Data Center.

Research on Heterogeneous Computation Resource Allocation based on Data-driven Method

ReAssigner: A Plug-and-Play Virtual Machine Scheduling Intensifier for Heterogeneous Requests

Task allocation for decentralized training in heterogeneous environment

Design and realization of hybrid resource management system for heterogeneous cluster

Multi-Tiered On-Demand Resource Scheduling for VM-Based Data Center

Optimal Resource Efficiency with Fairness in Heterogeneous GPU Clusters

Cluster Scheduler on Heterogeneous Cloud

RCFS: rate and cost fair CPU scheduling strategy in edge nodes