Abstract:Many emerging cyber-physical systems, such as autonomous vehicles and robots, rely heavily on artificial intelligence and machine learning algorithms to perform important system operations. Since these highly parallel applications are computationally intensive, they need to be accelerated by graphics processing units (GPUs) to meet stringent timing constraints. However, despite the wide adoption of GPUs, efficiently scheduling multiple GPU applications while providing rigorous real-time guarantees remains challenging. Each GPU application has multiple CPU execution and memory copy segments, with GPU kernels running on different hardware resources. Because of the complicated interactions between heterogeneous segments of parallel tasks, high schedulability is hard to achieve with conventional approaches. This paper proposes RTGPU, which combines fine-grain GPU partitioning on the system-side with a novel scheduling algorithm on the theory-side. We start by building a model for CPU and memory copy segments. Leveraging persistent threads, we then implement fine-grained GPU partitioning with improved performance through interleaved execution. To reap the benefits of fine-grained GPU partitioning and schedule multiple parallel GPU applications, we propose a novel real-time scheduling algorithm based on federated scheduling and grid search with uniprocessor fixed-priority scheduling. Our approach provides real-time guarantees to meet hard deadlines and achieves over 11% improvement in system throughput and up to 57% schedulability improvement compared with previous work. We validate and evaluate RTGPU on NVIDIA GPU systems. Our system-side techniques can be applied on mainstream GPUs, and the proposed scheduling theory can be used in general heterogeneous computing platforms which have a similar task execution pattern.

A holistic approach to build real-time stream processing system with GPU

TATA: Throughput-Aware TAsk Placement in Heterogeneous Stream Processing with Deep Reinforcement Learning

Exploiting the Task-Pipelined Parallelism of Stream Programs on Many-Core GPUs.

RTGPU: Real-Time GPU Scheduling of Hard Deadline Parallel Tasks With Fine-Grain Utilization

A Controllable Pipeline Framework of Block Ciphers on GPU for Streaming Data

Video Stream Processing on a High Performance Reconfigurable Architecture

Stream Execution Model of Graphics Processing Unit

A GPU-based Streaming Algorithm for High-Resolution Cloth Simulation.

LMStream: When Distributed Micro-Batch Stream Processing Systems Meet GPU

Throughput Optimization For Streaming Applications On Cpu-Fpga Heterogeneous Systems

A Real-time Virtual Reality Adaptive Streaming System

A Scalable Software Framework for Stateful Stream Data Processing on Multiple GPUs and Applications

Hardware-Conscious Stream Processing

A Load-Aware Pluggable Cloud Framework for Real-Time Video Processing.

Unleashing the Power of Preemptive Priority-based Scheduling for Real-Time GPU Tasks

Accelerating GPU Message Communication for Autonomous Navigation Systems.

Fine-Grained Multi-Query Stream Processing on Integrated Architectures

SPEED: Streaming Partition and Parallel Acceleration for Temporal Interaction Graph Embedding

Distributed and Parallel Processing for Real-Time and Dynamic Spatio-Temporal Graph

A 400Gbit Ethernet core enabling High Data Rate Streaming from FPGAs to Servers and GPUs in Radio Astronomy

An architecture design of GPU-accelerated VoD streaming servers with network coding.