Abstract:Training of large-scale deep learning models necessitates parallelizing the model and data across numerous devices, and the choice of parallelism strategy substantially depends on the training workloads such as memory consumption, computation cost, and communication cost. Current approaches generally assume uniform training workloads across samples in a given task. Thus, existing systems are designed to adopt a static parallelism strategy throughout one training process. Nevertheless, when training models with sequence inputs, this assumption fails due to the sequence length variation across samples. Consequently, training with a static parallelism strategy would result in sub-optimal performance. In this paper, we first reveal the under-explored fact that the optimal parallelism strategy varies even for the sequences within a single mini-batch. Motivated by this, we present HotSPa, a novel system that adopts multiple parallelism strategies for efficient training with sequence inputs. To be specific, given a mini-batch of training sequences, HotSPa partitions them into multiple groups and applies different parallelism strategies to process each group individually. To enable the hot switching between strategies, HotSPa transfers model parameters and accumulated gradients among the devices on the fly. Significant solutions are proposed with the hope of seamless and rapid parallelism hot switching. Firstly, we design a graph compiler, which generates distributed computation graphs for different parallelism strategies simultaneously, and orchestrates them to share a single model storage backbone. Secondly, we develop a simple yet effective hot switch planner, which heuristically deduces communication plans to accelerate the transition of model partitioning given any pairs of strategies. Extensive experiments on large language model training demonstrate that HotSPa can be up to 2.99× faster than Megatron-LM and DeepSpeed that utilize static parallelism strategies. Source code is available: https://github.com/PKU-DAIR/Hetu.

Training Example Selection in Instance-Based Learning Parallelization

Hybrid Simulated Annealing Algorithm Based on the Parallel Strategy

Enabling Parallelism Hot Switching for Efficient Training of Large Language Models

An Online Profile Guided Optimization Approach For Speculative Parallel Threading

Performance and Energy Consumption of Parallel Machine Learning Algorithms

Parallelization of Module Network Structure Learning and Performance Tuning on SMP

Scheduling optimization of parallel linear algebra algorithms using Supervised Learning

Parallel Approaches to Neighborhood Rough Sets: Classification and Feature Selection

Constructing Skeleton for Parallel Applications with Machine Learning Methods

Heuristic Parallel Selective Ensemble Algorithm Based on Clustering and Improved Simulated Annealing

Selective Ensemble Learning with Parallel Optimization and Hierarchical Selection

Single-machine and parallel-machine serial-batching scheduling problems with position-based learning effect and linear setup time

Data-Efficient Performance Modeling for Configurable Big Data Frameworks by Reducing Information Overlap Between Training Examples

A New Method for a Class of Parallel Batch Machine Scheduling Problem

Multi-View Learning for Parallelism Discovery of Sequential Programs

Unrelated Parallel Machines Scheduling With A Truncated Job-Dependent Learning Effect

Data-Centric and Heterogeneity-Adaptive Sequence Parallelism for Efficient LLM Training

A Parallel SVM Training Algorithm on Large-Scale Classification Problems

On the Parallelization Optimization Strategy for High Performance Computing Software

Unrelated Parallel Machine Scheduling with Job Splitting, Setup Time, Learning Effect, Processing Cost and Machine Eligibility.

An Adaptive Placement and Parallelism Framework for Accelerating RLHF Training