Abstract:The demand for large-scale deep learning is increasing, and distributed training is the current mainstream solution. Ring AllReduce is widely used as a data parallel decentralized algorithm. However, in a heterogeneous environment, each worker calculates the same amount of data, so that there is a lot of waiting time loss among different workers, which makes the algorithm unable to adapt well to heterogeneous clusters. Resources are not used as they should be. In this paper, we design an implementation of static allocation algorithm. The dataset is artificially allocated to each worker, and samples are drawn proportionally for training, thereby speeding up the training speed of the network in a heterogeneous environment. We verify the convergence and influence on training speed of the network model under this algorithm on one machine with multi-card and multi-machine with multi-card. On this basis of feasibility, we propose a self-adaptive allocation algorithm that allows each machine to find the data it needs to adapt to the current environment. The self-adaptive allocation algorithm can reduce the training time by nearly one-third to half compared to the same proportional <a class="link-external link-http" href="http://allocation.In" rel="external noopener nofollow">this http URL</a> order to better show the applicability of the algorithm in heterogeneous clusters, We replace a poorly performing worker with a good performing worker or add a poorly performing worker to the heterogeneous cluster. Experimental results show that training time will decrease as the overall performance improves. Therefore, it means that resources are fully used. Further, this algorithm is not only suitable for straggler problems, but also for most heterogeneous situations. It can be used as a plug-in for AllReduce and its variant algorithms.

An Accurate and Efficient Large-scale Regression Method through Best Friend Clustering

Distributed Affinity Propagation Clustering Based on MapReduce

A Parallel Varied Density-Based Clustering Algorithm with Optimized Data Partition

A Unified Framework for Representation-Based Subspace Clustering of Out-of-Sample and Large-Scale Data.

A Hybrid Approach to Clustering in Very Large Databases

Scalable Co-Clustering for Large-Scale Data through Dynamic Partitioning and Hierarchical Merging

Online Job Scheduling in Distributed Machine Learning Clusters

Distributed structural clustering on large graph

Boosting cluster tree with reciprocal nearest neighbors scoring

Large-Scale Clustering With Structured Optimal Bipartite Graph

Optimizing Task Placement and Online Scheduling for Distributed GNN Training Acceleration

Parallel Massive Clustering of Discrete Distributions

Distributed Kernel K-Means for Large Scale Clustering

A Communication-Efficient Parallel Method for Group-Lasso.

Large-Scale Clustering on 100 M-Scale Datasets Using a Single T4 GPU via Recall KNN and Subgraph Segmentation

Accurate, Efficient and Scalable Graph Embedding

A New Clustering Method Suitable for Large Scale Data

Clusterwise Functional Linear Regression Models.

HPH: Hybrid Parallelism on Heterogeneous Clusters for Accelerating Large-scale DNNs Training.

Task allocation for decentralized training in heterogeneous environment

A Scalable Approach for General Correlation Clustering