Abstract:As deep learning has been recently applied to a wide variety of fields, there is a growing demand for models that can handle large input data, including high-resolution images. Therefore, model parallelism was proposed to train a model whose size exceeds the memory capacity of an accelerator, but this method has a very slow training speed due to bubbles. To solve this problem, GPipe reduced the bubbles by proposing a micro-batch concept, according to which a mini-batch is divided into smaller units. However, the improvement in the training speed is still limited to a certain extent of micro-batch size, because the smaller the micro-batches, the less the computation and input/output (I/O) efficiency. To overcome the limitations, we proposed acceleration through prediction and synchronization steps based on process cooperation to train compute-intensive models. In the prediction step, the input data of all processes for the forward pass are calculated concurrently using the weights shared in the synchronization step in advance, and the results are gathered into each corresponding process via an all-to-all collective operation. This can increase computational efficiency and reduce bubbles by minimizing the idle state of the device. Additionally, the proposed method requires minimal memory because it does not have to store the activations in memory. Thus, compared to the GPipe, the proposed method achieved performance improvements of 15.3%, 34.5%, and 25.8% with the VGG16bn, ResNet50, and InceptionV3 models with four devices, respectively, and the memory used for training was reduced by up to 75.0%.

A Prediction Model For Parallel Back Propagation Neural Network On Smp-Cluster

Hybrid Performance Modeling And Analyzing Of Parallel Systems

WBSP: Addressing Stragglers in Distributed Machine Learning with Worker-Busy Synchronous Parallel

Run-time prediction algorithm for parallel jobs in grid

An Implement of Parallel Module Network Learning Algorithm on Distributed Memory Multiprocessors

An Efficient Parallel Computing Model On The Smp Cluster

Parallelization of Bayesian Network Based SNPs Pattern Analysis and Performance Characterization on SMP/HT

Parallel Module Network Learning on Distributed Memory Multiprocessors

Parallelization of Module Network Structure Learning and Performance Tuning on SMP

Parallelizing Back Propagation Neural Network on Speculative Multicores

Efficient and Robust Parallel DNN Training through Model Parallelism on Multi-GPU Platform

Coded Parallelism for Distributed Deep Learning.

Proteus: Simulating the Performance of Distributed DNN Training

A Mechanism of Predicting Network Simulation Performance Based on BP Nets

Parallelizing Backpropagation Neural Network Using MapReduce and Cascading Model.

Hybrid Parallel Programming Model for Hierarchical NoC

HPH: Hybrid Parallelism on Heterogeneous Clusters for Accelerating Large-scale DNNs Training.

Accelerated Synchronous Model Parallelism Using Cooperative Process for Training Compute-Intensive Models

Performance Modeling and Analysis of Distributed Deep Neural Network Training with Parameter Server

Performance and Energy Consumption of Parallel Machine Learning Algorithms