Abstract:Deep learning (DL) jobs use multi-dimensional parallelism, i.e. combining data, model, and pipeline parallelism, to use large GPU clusters efficiently. Long-running jobs may experience changes to their GPU allocation: (i) resource elasticity during training adds or removes GPUs; (ii) hardware maintenance may require redeployment on different GPUs; and (iii) GPU failures force jobs to run with fewer devices. Current DL frameworks tie jobs to a set of GPUs and thus lack support for these scenarios. In particular, they cannot change the multi-dimensional parallelism of an already-running job in an efficient and model-independent way. We describe Scalai, a state management library for DL systems that enables jobs to change their parallelism dynamically after the GPU allocation is updated at runtime. Scalai achieves this through a new abstraction, a parallelizable tensor collection (PTC), that externalizes the job state during training. After a GPU change, Scalai uses the PTC to transform the job state: the PTC repartitions the dataset state under data parallelism and exposes it to DL workers through a virtual file system; and the PTC obtains the model state as partitioned checkpoints and transforms them to reflect the new parallelization configuration. For efficiency, Scalai executes PTC transformations in parallel with minimum data movement between workers. Our experiments show that Scalai enables DL jobs to support dynamic parallelization with low overhead.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to dynamically change the multi - dimensional parallel strategy at runtime to adapt to the changes of GPU resources in deep - learning tasks, while maintaining the consistency and accuracy of training results. Specifically, the paper focuses on the following situations that may cause changes in GPU resource allocation when deep - learning tasks are executed on large - scale GPU clusters: 1. **Elastic adjustment**: In order to maintain high cluster utilization, deep - learning tasks may need to dynamically increase or decrease the number of allocated GPUs according to available resources. 2. **Redeployment**: Due to hardware maintenance or other reasons, deep - learning tasks may need to migrate from the currently used GPUs to other GPUs. 3. **Fault recovery**: Running tasks may lose some GPU resources due to hardware failures, network outages or software errors, and need to continue execution after recovery. Existing deep - learning frameworks usually bind tasks to specific sets of GPUs and lack the ability to support the above scenarios, especially when the task is already running, and cannot change the multi - dimensional parallel strategy efficiently and independently of the model. This leads to several challenges, including: - **Impact on convergence**: Changing the number of GPUs may affect the convergence of the task, resulting in different performance of the finally trained model. - **Performance impact**: The optimal multi - dimensional parallel configuration depends on specific GPU resources. When resources change, the originally optimal configuration may no longer be applicable and needs to be re - optimized. To solve these problems, the paper proposes a state - management library named Tenplex, which realizes the ability to dynamically adjust the multi - dimensional parallel strategy when GPU resources change by introducing a new abstraction - Parallelizable Tensor Collection (PTC). Specifically, the main contributions of Tenplex include: 1. **Externalizing DL task states**: Tenplex extracts task states from the deep - learning system and represents these states using PTC, including data - set states and model states. 2. **Transforming DL task states**: When GPU resources change, Tenplex transforms task states through PTC to adapt to the new parallel configuration. 3. **Optimizing DL task state changes**: Tenplex efficiently completes state changes by parallelizing the PTC transformation process and minimizing the amount of data transfer. Through these techniques, Tenplex can support deep - learning tasks to dynamically adjust GPU resources at runtime while maintaining low overhead, thereby improving resource utilization and task flexibility.

Tenplex: Dynamic Parallelism for Deep Learning using Parallelizable Tensor Collections

Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning

Deep Optimizer States: Towards Scalable Training of Transformer Models Using Interleaved Offloading

Enabling Parallelism Hot Switching for Efficient Training of Large Language Models

VELTAIR: towards high-performance multi-tenant deep learning services via adaptive compilation and scheduling

Automatic Task Parallelization of Dataflow Graphs in ML/DL models

HPH: Hybrid Parallelism on Heterogeneous Clusters for Accelerating Large-scale DNNs Training.

Accelerating Heterogeneous Tensor Parallelism via Flexible Workload Control

Collage: Seamless Integration of Deep Learning Backends with Automatic Placement

Slapo: A Schedule Language for Progressive Optimization of Large Deep Learning Model Training

Scheduling Deep Learning Jobs in Multi-Tenant GPU Clusters via Wise Resource Sharing

Horizontally Fused Training Array: An Effective Hardware Utilization Squeezer for Training Novel Deep Learning Models

Computron: Serving Distributed Deep Learning Models with Model Parallel Swapping

Salus: Fine-Grained GPU Sharing Primitives for Deep Learning Applications

PiPar: Pipeline Parallelism for Collaborative Machine Learning

Parallax: Sparsity-aware Data Parallel Training of Deep Neural Networks

TAP: Accelerating Large-Scale DNN Training Through Tensor Automatic Parallelisation

Amazon SageMaker Model Parallelism: A General and Flexible Framework for Large Model Training

Harnessing Deep Learning and HPC Kernels via High-Level Loop and Tensor Abstractions on CPU Architectures

Dynamic Resource Allocation for Deep Learning Clusters with Separated Compute and Storage

A Hybrid Parallelization Approach for Distributed and Scalable Deep Learning