Abstract:Foundation models are becoming the dominant deep learning technologies. Pretraining a foundation model is always time-consumed due to the large scale of both the model parameter and training dataset. Besides being computing-intensive, the training process is extremely memory-intensive and communication-intensive. These features make it necessary to apply 3D parallelism, which integrates data parallelism, pipeline model parallelism and tensor model parallelism, to achieve high training efficiency. To achieve this goal, some custom software frameworks such as Megatron-LM and DeepSpeed are developed. However, current 3D parallelism frameworks still meet two issues: i) they are not transparent to model developers, which need to manually modify the model to parallelize training. ii) their utilization of computation, GPU memory and network bandwidth are not sufficient. We propose Merak, an automated 3D parallelism deep learning training framework with high resource utilization. Merak automatically deploys with an automatic model partitioner, which uses a graph sharding algorithm on a proxy representation of the model. Merak also presents the non-intrusive API for scaling out foundation model training with minimal code modification. In addition, we design a high-performance 3D parallel runtime engine in Merak. It uses several techniques to exploit available training resources, including shifted critical path pipeline schedule that brings a higher computation utilization, stage-aware recomputation that makes use of idle worker memory, and sub-pipelined tensor model parallelism that overlaps communication and computation. Experiments on 64 GPUs show Merak can speedup the training performance over the state-of-the-art 3D parallelism frameworks of models with 1.5, 2.5, 8.3, and 20 billion parameters by up to 1.42X, 1.39X, 1.43X, and 1.61X, respectively.

Scaling Distributed Deep Learning Workloads beyond the Memory Capacity with KARMA

Woodpecker-DL: Accelerating Deep Neural Networks via Hardware-Aware Multifaceted Optimizations

Hardware-aware training for large-scale and diverse deep learning inference workloads using in-memory computing-based accelerators

Breaking the Memory Wall: A Study of I/O Patterns and GPU Memory Utilization for Hybrid CPU-GPU Offloaded Optimizers

Harnessing Manycore Processors with Distributed Memory for Accelerated Training of Sparse and Recurrent Models

Pipeline Gradient-based Model Training on Analog In-memory Accelerators

Deep Optimizer States: Towards Scalable Training of Transformer Models Using Interleaved Offloading

DeMo: Decoupled Momentum Optimization

Work-in-Progress: Furion: Alleviating Overheads for Deep Learning Framework on Single Machine

Scaling Deep Learning on GPU and Knights Landing clusters

An Efficient 2D Method for Training Super-Large Deep Learning Models

Merak: An Efficient Distributed DNN Training Framework with Automated 3D Parallelism for Giant Foundation Models

Accelerated Synchronous Model Parallelism Using Cooperative Process for Training Compute-Intensive Models

MAD Max Beyond Single-Node: Enabling Large Machine Learning Model Acceleration on Distributed Systems

Optimizing Offload Performance in Heterogeneous MPSoCs

Accelerating Large Language Model Training with 4D Parallelism and Memory Consumption Estimator

Data-parallel distributed training of very large models beyond GPU capacity

FASTERMOE: Modeling and Optimizing Training of Large-Scale Dynamic Pre-Trained Models

Hardware Scaling Trends and Diminishing Returns in Large-Scale Distributed Training

Efficient Arbitrary Precision Acceleration for Large Language Models on GPU Tensor Cores

Can Flexible Multi-Core Scheduling Help to Execute Machine Learning Algorithms Resource-Efficiently?