Abstract:Deep learning (DL) workloads include throughput- intensive training tasks and latency-sensitive inference tasks. The dominant practice today is to provision dedicated GPU clusters for training and inference separately. Due to the need to meet strict Service-Level Objectives (SLOs), GPU clusters are often over-provisioned based on the peak load with limited sharing between applications and task types. We present PipeSwitch, a system that enables unused cycles of an inference application to be filled by training or other inference applications. It allows multiple DL applications to time-share the same GPU with the entire GPU memory and millisecond-scale switching overhead. With PipeSwitch, GPU utilization can be significantly improved without sacrificing SLOs. We achieve so by introducing pipelined context switching. The key idea is to leverage the layered structure of neural network models and their layer-by-layer computa- tion pattern to pipeline model transmission over the PCIe and task execution in the GPU with model-aware grouping. We also design unified memory management and active-standby worker switching mechanisms to accompany the pipelining and ensure process-level isolation. We have built a PipeSwitch prototype and integrated it with PyTorch. Experiments on a variety of DL models and GPU cards show that PipeSwitch only incurs a task startup overhead of 3.6-6.6 ms and a total overhead of 5.4-34.6 ms (10-50 x better than NVIDIA MPS), and achieves near 100% GPU utilization.

Versapipe: a versatile programming framework for pipelined computing on GPU.

Scalable-Grain Pipeline Parallelization Method For Multi-Core Systems

An Efficient Task-Parallel Pipeline Programming Framework.

Software Pipelining for Graphic Processing Unit Acceleration: Partition, Scheduling and Granularity

Pipeflow: an Efficient Task-Parallel Pipeline Programming Framework Using Modern C++.

vPipe: A Virtualized Acceleration System for Achieving Efficient and Scalable Pipeline Parallel DNN Training

Exploiting the Task-Pipelined Parallelism of Stream Programs on Many-Core GPUs.

BitPipe: Bidirectional Interleaved Pipeline Parallelism for Accelerating Large Models Training

Enabling Data Movement and Computation Pipelining in Deep Learning Compiler

Efficient Modeling and Real-Time Rendering of Massive Urban Pipelines Based on GPU

IMPLEMENTATION FRAMEWORK FOR BINARY STREAM PATTERN EXTRACTION UNDER CPU/GPU

HG-PIPE: Vision Transformer Acceleration with Hybrid-Grained Pipeline

BaPipe: Exploration of Balanced Pipeline Parallelism for DNN Training

A Translation Framework for Virtual Execution Environment on CPU/GPU Architecture

A Programming Framework Based on Multi-GPU

PipeSwitch: Fast Pipelined Context Switching for Deep Learning Applications

GraphPipe: Improving Performance and Scalability of DNN Training with Graph Pipeline Parallelism

Pipeline Parallelism for Inference on Heterogeneous Edge Computing

A Pipeline-Based ODE Solving Framework

XPipe: Efficient Pipeline Model Parallelism for Multi-GPU DNN Training