Abstract:Training of large-scale deep learning models necessitates parallelizing the model and data across numerous devices, and the choice of parallelism strategy substantially depends on the training workloads such as memory consumption, computation cost, and communication cost. Current approaches generally assume uniform training workloads across samples in a given task. Thus, existing systems are designed to adopt a static parallelism strategy throughout one training process. Nevertheless, when training models with sequence inputs, this assumption fails due to the sequence length variation across samples. Consequently, training with a static parallelism strategy would result in sub-optimal performance. In this paper, we first reveal the under-explored fact that the optimal parallelism strategy varies even for the sequences within a single mini-batch. Motivated by this, we present HotSPa, a novel system that adopts multiple parallelism strategies for efficient training with sequence inputs. To be specific, given a mini-batch of training sequences, HotSPa partitions them into multiple groups and applies different parallelism strategies to process each group individually. To enable the hot switching between strategies, HotSPa transfers model parameters and accumulated gradients among the devices on the fly. Significant solutions are proposed with the hope of seamless and rapid parallelism hot switching. Firstly, we design a graph compiler, which generates distributed computation graphs for different parallelism strategies simultaneously, and orchestrates them to share a single model storage backbone. Secondly, we develop a simple yet effective hot switch planner, which heuristically deduces communication plans to accelerate the transition of model partitioning given any pairs of strategies. Extensive experiments on large language model training demonstrate that HotSPa can be up to 2.99× faster than Megatron-LM and DeepSpeed that utilize static parallelism strategies. Source code is available: https://github.com/PKU-DAIR/Hetu.

Parallelizing Big Data Machine Learning Applications With Model Rotation

A Parallel Function Evaluation Approach for Solution to Large-Scale Equation-Oriented Models.

Rethinking Local-to-global Representation Learning for Rotation-Invariant Point Cloud Analysis

Adaptive Partitioning and Efficient Scheduling for Distributed DNN Training in Heterogeneous IoT Environment

High-performance Kernel Machines with Implicit Distributed Optimization and Randomization

Parallelized Computation and Backpropagation Under Angle-Parametrized Orthogonal Matrices

On Optimizing the Communication of Model Parallelism

RotNet: A Rotationally Invariant Graph Neural Network for Quantum Mechanical Calculations

Performance and Energy Consumption of Parallel Machine Learning Algorithms

HPH: Hybrid Parallelism on Heterogeneous Clusters for Accelerating Large-scale DNNs Training.

Efficient and Robust Parallel DNN Training through Model Parallelism on Multi-GPU Platform

Communication Efficient Parallel Algorithms for Optimization on Manifolds

High Performance LDA Through Collective Model Communication Optimization

Integrated Model, Batch and Domain Parallelism in Training Neural Networks

Towards accelerating model parallelism in distributed deep learning systems

Optimal distributed parallel algorithms for deep learning framework Tensorflow

Hemingway: Modeling Distributed Optimization Algorithms

Enabling Parallelism Hot Switching for Efficient Training of Large Language Models

Computron: Serving Distributed Deep Learning Models with Model Parallel Swapping

Optimus-CC: Efficient Large NLP Model Training with 3D Parallelism Aware Communication Compression

Modeling Data Movement Performance on Heterogeneous Architectures