Abstract:Many emerging cyber-physical systems, such as autonomous vehicles and robots, rely heavily on artificial intelligence and machine learning algorithms to perform important system operations. Since these highly parallel applications are computationally intensive, they need to be accelerated by graphics processing units (GPUs) to meet stringent timing constraints. However, despite the wide adoption of GPUs, efficiently scheduling multiple GPU applications while providing rigorous real-time guarantees remains challenging. Each GPU application has multiple CPU execution and memory copy segments, with GPU kernels running on different hardware resources. Because of the complicated interactions between heterogeneous segments of parallel tasks, high schedulability is hard to achieve with conventional approaches. This paper proposes RTGPU, which combines fine-grain GPU partitioning on the system-side with a novel scheduling algorithm on the theory-side. We start by building a model for CPU and memory copy segments. Leveraging persistent threads, we then implement fine-grained GPU partitioning with improved performance through interleaved execution. To reap the benefits of fine-grained GPU partitioning and schedule multiple parallel GPU applications, we propose a novel real-time scheduling algorithm based on federated scheduling and grid search with uniprocessor fixed-priority scheduling. Our approach provides real-time guarantees to meet hard deadlines and achieves over 11% improvement in system throughput and up to 57% schedulability improvement compared with previous work. We validate and evaluate RTGPU on NVIDIA GPU systems. Our system-side techniques can be applied on mainstream GPUs, and the proposed scheduling theory can be used in general heterogeneous computing platforms which have a similar task execution pattern.

Process variation-aware workload partitioning algorithms for GPUs supporting spatial-multitasking

Efficient GPU Spatial-Temporal Multitasking

Simultaneous Multikernel GPU: Multi-tasking throughput processors via fine-grained sharing

Improving Multi-Application Concurrency Support Within the GPU Memory System

SPGPU: Spatially Programmed GPU

Exploring the Diversity of Multiple Job Deployments over GPUs for Efficient Resource Sharing

Optimizing Hardware Resource Partitioning and Job Allocations on Modern GPUs under Power Caps

Improving GPU Performance Through Resource Sharing

A Virtual Multi-Channel GPU Fair Scheduling Method for Virtual Machines.

Exploiting the Task-Pipelined Parallelism of Stream Programs on Many-Core GPUs.

Efficient Resource Sharing Through GPU Virtualization on Accelerated High Performance Computing Systems

Enhanced GPU Resource Utilization through Fairness-aware Task Scheduling

Hierarchical Resource Partitioning on Modern GPUs: A Reinforcement Learning Approach

Understanding Data Partition for Applications on CPU-GPU Integrated Processors.

Exploring Cache Bypassing and Partitioning for Multi-Tasking on GPUs

Orchestrated Co-scheduling, Resource Partitioning, and Power Capping on CPU-GPU Heterogeneous Systems via Machine Learning

A CPU-GPGPU Scheduler Based on Data Transmission Bandwidth of Workload

Task Scheduling Greedy Heuristics for GPU Heterogeneous Cluster Involving the Weights of the Processor

RTGPU: Real-Time GPU Scheduling of Hard Deadline Parallel Tasks With Fine-Grain Utilization

A Self-Adaptive Workload Balancing Algorithm on Gpu Clusters

An Optimized GP-GPU Warp Scheduling Algorithm for Sparse Matrix-Vector Multiplication