Abstract:Deep learning is a growing technique used to solve complex artificial intelligence (AI) problems. Large-scale deep learning has become a significant issue as a result of the expansion of datasets and the complexity of deep learning models. For training large-scale models, the cloud can be used as a distributed HPC (high-performance computing) tool with benefits in cost and flexibility. However, one of the major performance barriers in distributed deep learning in a distributed HPC environment is the network. The performance is often limited by heavy traffic like many stochastic gradient descent transfers for distributed communication. There are many network studies in distributed deep learning to solve these problems, but most research only focuses on improving communication performance and applying new methods or algorithms like overlapping parameter synchronization to minimize communication delay rather than considering the actual network. In this paper, we are focusing on the actual network, especially in a distributed HPC environment. In such an environment, if cluster nodes are assigned to different zones/regions which means a set of an appropriate number of distributed HPC nodes when performing distributed deep learning tasks, performance degradation due to network delay may occur. The proposed network optimization algorithm ensures that distributed work is placed in the same zone as much as possible to reduce network delay. Furthermore, scoring using network monitoring tools like loss, delay, and throughput is applied to select the optimal node within the zone. Our proposal has been validated on the Kubernetes platform, an open source orchestrator for the automatic management and deployment of micro-services. The performance of distributed deep learning is improved through the proposed scheduler.

Iteration Acceleration for Distributed Learning Systems

A Hybrid Solution to improve Iteration Efficiency in the Distributed Learning.

An Accelerated Iterative Learning Algorithm Based on Feedback Terms for Fault Estimation of Nonlinear Parabolic Distributed Parameter Systems

Joint Dynamic Data and Model Parallelism for Distributed Training of DNNs over Heterogeneous Infrastructure

EP4DDL: addressing straggler problem in heterogeneous distributed deep learning

An Incremental Iterative Acceleration Architecture in Distributed Heterogeneous Environments With GPUs for Deep Learning

DALU: Adaptive Learning Rate Update in Distributed Deep Learning

Online Job Scheduling in Distributed Machine Learning Clusters

A General Distributed Dual Coordinate Optimization Framework for Regularized Loss Minimization

Locality-aware and Fault-tolerant Batching for Machine Learning on Distributed Datasets

A Dynamic Weighting Strategy to Mitigate Worker Node Failure in Distributed Deep Learning

Accelerated Distributed Approximate Newton Method

DISTRIBUTED HIGH-PERFORMANCE COMPUTING METHODS FOR ACCELERATING DEEP LEARNING TRAINING

Hybrid Parameter Update: Alleviating Imbalance Impacts for Distributed Deep Learning.

A Distributed and Scalable Machine Learning Approach for Big Data

Straggler-Resilient Decentralized Learning via Adaptive Asynchronous Updates

Distributed Asynchronous Dual Free Stochastic Dual Coordinate Ascent

MUSIC: Accelerated Convergence for Distributed Optimization With Inexact and Exact Methods

Communication-Efficient Distributed Learning via Sparse and Adaptive Stochastic Gradient

An Optimal Network-Aware Scheduling Technique for Distributed Deep Learning in Distributed HPC Platforms

Leveraging Deep Learning to Improve Performance of Distributed Optimal Frequency Control Under Communication Failures