Abstract:Many emerging cyber-physical systems, such as autonomous vehicles and robots, rely heavily on artificial intelligence and machine learning algorithms to perform important system operations. Since these highly parallel applications are computationally intensive, they need to be accelerated by graphics processing units (GPUs) to meet stringent timing constraints. However, despite the wide adoption of GPUs, efficiently scheduling multiple GPU applications while providing rigorous real-time guarantees remains challenging. Each GPU application has multiple CPU execution and memory copy segments, with GPU kernels running on different hardware resources. Because of the complicated interactions between heterogeneous segments of parallel tasks, high schedulability is hard to achieve with conventional approaches. This paper proposes RTGPU, which combines fine-grain GPU partitioning on the system-side with a novel scheduling algorithm on the theory-side. We start by building a model for CPU and memory copy segments. Leveraging persistent threads, we then implement fine-grained GPU partitioning with improved performance through interleaved execution. To reap the benefits of fine-grained GPU partitioning and schedule multiple parallel GPU applications, we propose a novel real-time scheduling algorithm based on federated scheduling and grid search with uniprocessor fixed-priority scheduling. Our approach provides real-time guarantees to meet hard deadlines and achieves over 11% improvement in system throughput and up to 57% schedulability improvement compared with previous work. We validate and evaluate RTGPU on NVIDIA GPU systems. Our system-side techniques can be applied on mainstream GPUs, and the proposed scheduling theory can be used in general heterogeneous computing platforms which have a similar task execution pattern.

Exploring the Diversity of Multiple Job Deployments over GPUs for Efficient Resource Sharing

Optimizing Hardware Resource Partitioning and Job Allocations on Modern GPUs under Power Caps

Efficient Resource Sharing Through GPU Virtualization on Accelerated High Performance Computing Systems

Hierarchical Resource Partitioning on Modern GPUs: A Reinforcement Learning Approach

GPU Scheduling for Short Tasks in Private Cloud

A Virtual Multi-Channel GPU Fair Scheduling Method for Virtual Machines.

Simultaneous Multikernel GPU: Multi-tasking throughput processors via fine-grained sharing

Multi-Tenant Virtual GPUs for Optimising Performance of a Financial Risk Application

Improving GPU Performance Through Resource Sharing

RTGPU: Real-Time GPU Scheduling of Hard Deadline Parallel Tasks With Fine-Grain Utilization

PILOT: a Runtime System to Manage Multi-tenant GPU Unified Memory Footprint

Optimal Workload Placement on Multi-Instance GPUs

ParvaGPU: Efficient Spatial GPU Sharing for Large-Scale DNN Inference in Cloud Environments

Process variation-aware workload partitioning algorithms for GPUs supporting spatial-multitasking

Parallel Transient Stability-Constrained Optimal Power Flow Using GPU as Coprocessor.

KubeGPU: efficient sharing and isolation mechanisms for GPU resource management in container cloud

MISO: Exploiting Multi-Instance GPU Capability on Multi-Tenant Systems for Machine Learning

Efficient GPU Spatial-Temporal Multitasking

Unleashing the Power of Preemptive Priority-based Scheduling for Real-Time GPU Tasks

Enhanced GPU Resource Utilization through Fairness-aware Task Scheduling

Orchestrated Co-scheduling, Resource Partitioning, and Power Capping on CPU-GPU Heterogeneous Systems via Machine Learning