Abstract:GPUs are widely used to accelerate many important classes of workloads today. However, we observe that several important emerging classes of workloads, including simulation engines for deep reinforcement learning and dynamic neural networks, are unable to fully utilize the massive parallelism that GPUs offer. These applications tend to have kernels that are small in size, i.e., have few thread blocks that do not saturate compute resources. Executing independent kernels concurrently is a promising approach to improve parallelism and utilization. However, this inter-kernel concurrency is difficult to leverage in such workloads with existing approaches: First, the inter-kernel dependencies and computational graph are input-dependent and vary each time the application is executed. Second, the computational graphs tend to be irregular, requiring fine-grain scheduling and synchronization; thus incurring significant synchronization overheads if kernel execution is parallelized. In this work, we propose ACS, a framework that enables lightweight detection of inter-kernel dependencies and low overhead kernel scheduling at runtime. The key idea behind ACS is to perform inter-kernel dependency checks for a small window of kernels at runtime, similar to out-of order instruction scheduling. This enables concurrent execution of kernels in applications whose computational graphs are input dependent and require fine-grained scheduling. We propose ACS-SW, a software-only open-source implementation of ACS and ACS-HW, a hardware-software cooperative implementation. ACS-HW further reduces synchronization overheads by reducing communication between the CPU and GPU. We evaluate ACS for deep RL simulation and dynamic DNNs on both real hardware and a GPU simulator. We demonstrate speedups of up to 2.19x (1.56x on average) by improving GPU utilization with concurrent kernel execution.

High-Integrity GPU Designs for Critical Real-Time Automotive Systems

Enabling predictable parallelism in single-GPU systems with persistent CUDA threads

Intelligent Orchestration of ADAS Pipelines on Next Generation Automotive Platforms

RTGPU: Real-Time GPU Scheduling of Hard Deadline Parallel Tasks With Fine-Grain Utilization

CPU Cache GPU Cache Memory Cache Accelerator Isolation & protection Error detection Weak resilient domain Error recovery

Understanding the Effects of Permanent Faults in GPU's Parallelism Management and Control Units

ACS: Concurrent Kernel Execution on Irregular, Input-Dependent Computational Graphs

Towards Reliable AI Applications Via Algorithm-Based Fault Tolerance on NVDLA

Performance Analysis and Optimization Opportunities for NVIDIA Automotive GPUs

Asymmetric Resilience: Exploiting Task-Level Idempotency for Transient Error Recovery in Accelerator-Based Systems

Software-Based Fault Recovery via Adaptive Diversity for COTS Multi-Core Processors

An Efficient Sequential Consistency Implementation with Dynamic Race Detection for GPUs

COMPUTER-AIDED DESIGN OF FAULT-TOLERANT HARDWARE ARCHITECTURES FOR AUTONOMOUS DRIVING SYSTEMS

GPU Acceleration of Real-Time Control Loops

Unleashing the Power of Preemptive Priority-based Scheduling for Real-Time GPU Tasks

An Evaluation Of The Nvidia Tx1 For Supporting Real-Time Computer-Vision Workloads

Fault Tolerant One-Sided Matrix Decompositions on Heterogeneous Systems with GPUs.

Confidential Computing on Heterogeneous CPU-GPU Systems: Survey and Future Directions

3 System-level Dependability for Multicore and Real-time Systems

Optimization of a Line Detection Algorithm for Autonomous Vehicles on a RISC-V with Accelerator

Comparative analysis of soft-error sensitivity in LU decomposition algorithms on diverse GPUs