Abstract:Many emerging cyber-physical systems, such as autonomous vehicles and robots, rely heavily on artificial intelligence and machine learning algorithms to perform important system operations. Since these highly parallel applications are computationally intensive, they need to be accelerated by graphics processing units (GPUs) to meet stringent timing constraints. However, despite the wide adoption of GPUs, efficiently scheduling multiple GPU applications while providing rigorous real-time guarantees remains challenging. Each GPU application has multiple CPU execution and memory copy segments, with GPU kernels running on different hardware resources. Because of the complicated interactions between heterogeneous segments of parallel tasks, high schedulability is hard to achieve with conventional approaches. This paper proposes RTGPU, which combines fine-grain GPU partitioning on the system-side with a novel scheduling algorithm on the theory-side. We start by building a model for CPU and memory copy segments. Leveraging persistent threads, we then implement fine-grained GPU partitioning with improved performance through interleaved execution. To reap the benefits of fine-grained GPU partitioning and schedule multiple parallel GPU applications, we propose a novel real-time scheduling algorithm based on federated scheduling and grid search with uniprocessor fixed-priority scheduling. Our approach provides real-time guarantees to meet hard deadlines and achieves over 11% improvement in system throughput and up to 57% schedulability improvement compared with previous work. We validate and evaluate RTGPU on NVIDIA GPU systems. Our system-side techniques can be applied on mainstream GPUs, and the proposed scheduling theory can be used in general heterogeneous computing platforms which have a similar task execution pattern.

Towards Fast Setup and High Throughput of GPU Serverless Computing

FaST-GShare: Enabling Efficient Spatio-Temporal GPU Sharing in Serverless Computing for Deep Learning Inference

Kernel-as-a-Service: A Serverless Interface to GPUs

Enabling Efficient Spatio-Temporal GPU Sharing for Network Function Virtualization

A server-based approach for predictable GPU access with improved analysis

GPU-enabled Function-as-a-Service for Machine Learning Inference

DxPU: Large Scale Disaggregated GPU Pools in the Datacenter

Serverless Computing on Heterogeneous Computers.

ESG: Pipeline-Conscious Efficient Scheduling of DNN Workflows on Serverless Platforms with Shareable GPUs

Parallel Processing of Massive Remote Sensing Images in a GPU Architecture.

PARIS and ELSA: An Elastic Scheduling Algorithm for Reconfigurable Multi-GPU Inference Servers

Serverless Platforms on the Edge: A Performance Analysis

FaaSwap: SLO-Aware, GPU-Efficient Serverless Inference via Model Swapping

GPU consolidation for cloud games: Are we there yet?

RTGPU: Real-Time GPU Scheduling of Hard Deadline Parallel Tasks With Fine-Grain Utilization

Towards Seamless Serverless Computing Across an Edge-Cloud Continuum

FaaSGraph: Enabling Scalable, Efficient, and Cost-Effective Graph Processing with Serverless Computing

A holistic approach to build real-time stream processing system with GPU

GPU Scheduling for Short Tasks in Private Cloud

A Load-Aware Pluggable Cloud Framework for Real-Time Video Processing.

FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelines