Abstract:Many intelligent applications like autonomous driving and virtual reality require running both latency-critical and best-effort DNN inference tasks to achieve both real time and work conserving on GPU. However, commodity GPUs lack efficient preemptive scheduling support and state-of-the-art approaches either have to monopolize GPU or let the real-time tasks to wait for best-effort tasks to complete, which causes low utilization or high latency, or both. This paper presents REEF, the first GPU-accelerated DNN inference serving system that enables microsecond-scale kernel preemption and controlled concurrent execution in GPU scheduling. REEF is novel in two ways. First, based on the observation that DNN inference kernels as mostly idempotent, REEF devises a reset-based preemption scheme that launches a real-time kernel on the GPU by proactively killing and restoring best-effort kernels at microsecond-scale. Second, since DNN inference kernels have varied parallelism and predictable latency, REEF proposes a dynamic kernel padding mechanism that dynamically pads the real-time kernel with appropriate best-effort kernels to fully utilize the GPU with negligible overhead. Evaluation using a new DNN inference serving benchmark (DISB) with diverse workloads and a real-world trace on an AMD GPU shows that REEF only incurs less than 2% overhead in the end-to-end latency for real-time tasks but increases the overall throughput by up to 7.7x, compared to dedicating the GPU to real-time tasks. To demonstrate the feasibility of our approaches on closed-source GPUs, we further ported and evaluated a restricted version of REEF on an NVIDIA GPU with a reduction of the preemption latency by up to 12.3x (from 6.3x).

HyperBlocker: Accelerating Rule-based Blocking in Entity Resolution using GPUs

Using GPU to Shorten the Match Time of Rule Reasoning Based on Rete Algorithm

Towards Universal Dense Blocking for Entity Resolution

FROM CPU TO GPU: GPU-BASED ELECTROMAGNETIC COMPUTING (GPUECO)

Unsupervised Entity Resolution with Blocking and Graph Algorithms

Parallel Shooting and Bouncing Ray Method on Gpu Clusters for Analysis of Electromagnetic Scattering

GPU-Accelerated Rule Evaluation and Evolution

DyGA: A Hardware-Efficient Accelerator with Traffic-Aware Dynamic Scheduling for Graph Convolutional Networks.

PGAbB: A Block-Based Graph Processing Framework for Heterogeneous Platforms

Microsecond-scale Preemption for Concurrent GPU-accelerated DNN Inferences

Block-Parallel IDA* for GPUs (Extended Manuscript)

Fast Blocking of Householder Reflectors on Graphics Processors

Efficient Data Blocking and Skipping Framework Applying Heuristic Rules

Preemption-Aware Kernel Scheduling for GPUs

Modeling the Performance of 2.5D Blocking of 3D Stencil Code on GPUs

GraphPar: Efficient Workload-Aware Subgraph Matching System on Multiple GPUs

Framework to Accelerate Single-threaded Applications by Hyperblock Reformation on EDGE Architectures

Efficient GPU Spatial-Temporal Multitasking

Parallel Firewalls on General-Purpose Graphics Processing Units

Probabilistic Blocking with An Application to the Syrian Conflict

High Performance Computing Via a GPU