Abstract:Many emerging cyber-physical systems, such as autonomous vehicles and robots, rely heavily on artificial intelligence and machine learning algorithms to perform important system operations. Since these highly parallel applications are computationally intensive, they need to be accelerated by graphics processing units (GPUs) to meet stringent timing constraints. However, despite the wide adoption of GPUs, efficiently scheduling multiple GPU applications while providing rigorous real-time guarantees remains challenging. Each GPU application has multiple CPU execution and memory copy segments, with GPU kernels running on different hardware resources. Because of the complicated interactions between heterogeneous segments of parallel tasks, high schedulability is hard to achieve with conventional approaches. This paper proposes RTGPU, which combines fine-grain GPU partitioning on the system-side with a novel scheduling algorithm on the theory-side. We start by building a model for CPU and memory copy segments. Leveraging persistent threads, we then implement fine-grained GPU partitioning with improved performance through interleaved execution. To reap the benefits of fine-grained GPU partitioning and schedule multiple parallel GPU applications, we propose a novel real-time scheduling algorithm based on federated scheduling and grid search with uniprocessor fixed-priority scheduling. Our approach provides real-time guarantees to meet hard deadlines and achieves over 11% improvement in system throughput and up to 57% schedulability improvement compared with previous work. We validate and evaluate RTGPU on NVIDIA GPU systems. Our system-side techniques can be applied on mainstream GPUs, and the proposed scheduling theory can be used in general heterogeneous computing platforms which have a similar task execution pattern.

Towards QoS-Aware and Resource-Efficient GPU Microservices Based on Spatial Multitasking GPUs In Datacenters

Astraea: towards QoS-aware and resource-efficient multi-stage GPU services

A Virtual Multi-Channel GPU Fair Scheduling Method for Virtual Machines.

Efficient GPU Spatial-Temporal Multitasking

Efficient Resource Sharing Through GPU Virtualization on Accelerated High Performance Computing Systems

Enabling Efficient Spatio-Temporal GPU Sharing for Network Function Virtualization

FLARE: Flexibly Sharing Commodity GPUs to Enforce QoS and Improve Utilization

KubeGPU: efficient sharing and isolation mechanisms for GPU resource management in container cloud

Exploiting the Task-Pipelined Parallelism of Stream Programs on Many-Core GPUs.

GPU Scheduling for Short Tasks in Private Cloud

Improving Multi-Application Concurrency Support Within the GPU Memory System

A user mode CPU–GPU scheduling framework for hybrid workloads

Simultaneous Multikernel GPU: Multi-tasking throughput processors via fine-grained sharing

Towards Performance-Aware Allocation for Accelerated Machine Learning on GPU-SSD Systems

RTGPU: Real-Time GPU Scheduling of Hard Deadline Parallel Tasks With Fine-Grain Utilization

Optimizing Hardware Resource Partitioning and Job Allocations on Modern GPUs under Power Caps

PILOT: a Runtime System to Manage Multi-tenant GPU Unified Memory Footprint

Orchestrated Co-scheduling, Resource Partitioning, and Power Capping on CPU-GPU Heterogeneous Systems via Machine Learning

Multi-Service Edge Computing Management With Multi-Stage Coalition Game Task Offloading

QoS-Aware and Resource Efficient Microservice Deployment in Cloud-Edge Continuum

Concurrent analytical query processing with GPUs