Abstract:Many emerging cyber-physical systems, such as autonomous vehicles and robots, rely heavily on artificial intelligence and machine learning algorithms to perform important system operations. Since these highly parallel applications are computationally intensive, they need to be accelerated by graphics processing units (GPUs) to meet stringent timing constraints. However, despite the wide adoption of GPUs, efficiently scheduling multiple GPU applications while providing rigorous real-time guarantees remains challenging. Each GPU application has multiple CPU execution and memory copy segments, with GPU kernels running on different hardware resources. Because of the complicated interactions between heterogeneous segments of parallel tasks, high schedulability is hard to achieve with conventional approaches. This paper proposes RTGPU, which combines fine-grain GPU partitioning on the system-side with a novel scheduling algorithm on the theory-side. We start by building a model for CPU and memory copy segments. Leveraging persistent threads, we then implement fine-grained GPU partitioning with improved performance through interleaved execution. To reap the benefits of fine-grained GPU partitioning and schedule multiple parallel GPU applications, we propose a novel real-time scheduling algorithm based on federated scheduling and grid search with uniprocessor fixed-priority scheduling. Our approach provides real-time guarantees to meet hard deadlines and achieves over 11% improvement in system throughput and up to 57% schedulability improvement compared with previous work. We validate and evaluate RTGPU on NVIDIA GPU systems. Our system-side techniques can be applied on mainstream GPUs, and the proposed scheduling theory can be used in general heterogeneous computing platforms which have a similar task execution pattern.

FLARE: Flexibly Sharing Commodity GPUs to Enforce QoS and Improve Utilization

A Virtual Multi-Channel GPU Fair Scheduling Method for Virtual Machines.

Efficient Resource Sharing Through GPU Virtualization on Accelerated High Performance Computing Systems

Efficient GPU Spatial-Temporal Multitasking

FLUX: Fast Software-based Communication Overlap On GPUs Through Kernel Fusion

Enabling Efficient Spatio-Temporal GPU Sharing for Network Function Virtualization

Simultaneous Multikernel GPU: Multi-tasking throughput processors via fine-grained sharing

Exploring the Diversity of Multiple Job Deployments over GPUs for Efficient Resource Sharing

Parallel Transient Stability-Constrained Optimal Power Flow Using GPU as Coprocessor.

Enhanced GPU Resource Utilization through Fairness-aware Task Scheduling

FLARE: A Fast, Secure, and Memory-Efficient Distributed Analytics Framework

MuxFlow: Efficient and Safe GPU Sharing in Large-Scale Production Deep Learning Clusters

Poly: Efficient Heterogeneous System and Application Management for Interactive Applications

GPU Scheduling for Short Tasks in Private Cloud

Improving GPU Performance Through Resource Sharing

RTGPU: Real-Time GPU Scheduling of Hard Deadline Parallel Tasks With Fine-Grain Utilization

Optimizing Hardware Resource Partitioning and Job Allocations on Modern GPUs under Power Caps

Exploiting the Task-Pipelined Parallelism of Stream Programs on Many-Core GPUs.

KubeGPU: efficient sharing and isolation mechanisms for GPU resource management in container cloud

Optimal Workload Placement on Multi-Instance GPUs