Abstract:Modern deep neural networks cannot be often trained on a single GPU due to large model size and large data size. Model parallelism splits a model for multiple GPUs, but making it scalable and seamless is challenging due to different information sharing among GPUs with communication overhead. Specifically, we identify two key issues to make the parallelism being inefficient and inaccurate; an efficient pipelining technique is crucial to maximize GPU utilization and normalizations in deep neural networks may affect the performance due to different statistics sharing of mini-batch. In this work, we address these issues by investigating efficient pipelining for model parallelism and effective normalizations in model / data parallelisms when training a model with large mini-batch in multiple GPUs so that the model performance in accuracy can not be compromised. Firstly, we propose a novel method to search for an optimal micro-batch size considering the number of GPUs and memory size for model parallelism. For efficient pipelining, mini-batch is usually divided into smaller batches (called micro-batch). To maximize the utilization of GPU computing resources, training should be performed with the optimal micro-batch size. Our proposed micro-batch size search algorithm achieved increased image throughput by up to 12% and improved trainable mini-batch size by 25% as compared to the conventional model parallelism method. Secondly, we investigate normalizations in distributed deep learning training for different parallelisms. Our experiments using different normalization methods suggested that the performance with batch normalization can be improved by sharing the batch information among GPUs when performing data parallelism. It was also confirmed that group normalization helped minimizing accuracy degradation when performing model parallelism with pipelining and yielded consistent accuracies for diverse mini-batch sizes.

GLP4NN: A Convergence-invariant and Network-agnostic Light-weight Parallelization Framework for Deep Neural Networks on Modern GPUs

Brief Announcement: On the Limits of Parallelizing Convolutional Neural Networks on GPUs

A Unified CPU-GPU Protocol for GNN Training

A 4D Hybrid Algorithm to Scale Parallel Training to Thousands of GPUs

DF-GNN: Dynamic Fusion Framework for Attention Graph Neural Networks on GPUs

Using GPI-2 for Distributed Memory Paralleliziation of the Caffe Toolbox to Speed up Deep Neural Network Training

Efficient and Robust Parallel DNN Training through Model Parallelism on Multi-GPU Platform

Performance Modeling and Evaluation of Distributed Deep Learning Frameworks on GPUs

ParallelFusion

Parallel and Distributed Graph Neural Networks: An In-Depth Concurrency Analysis

Faster Multi-GPU Training with PPLL: A Pipeline Parallelism Framework Leveraging Local Learning

FastGL: A GPU-Efficient Framework for Accelerating Sampling-Based GNN Training at Large Scale

MaxK-GNN: Towards Theoretical Speed Limits for Accelerating Graph Neural Networks Training

A Practical Implementation of GPU based Accelerator for Deep Neural Networks

CNNLab: a Novel Parallel Framework for Neural Networks using GPU and FPGA-a Practical Study with Trade-off Analysis

TC-GNN: Bridging Sparse GNN Computation and Dense Tensor Cores on GPUs

Towards accelerating model parallelism in distributed deep learning systems

HiHGNN: Accelerating HGNNs through Parallelism and Data Reusability Exploitation

fuseGNN: Accelerating Graph Convolutional Neural Network Training on GPGPU

GNNPipe: Scaling Deep GNN Training with Pipelined Model Parallelism

A Metaprogramming and Autotuning Framework for Deploying Deep Learning Applications