Abstract:Following the advances in technology scaling, graphics processing units (GPUs) incorporate an increasing amount of computing resources and it becomes difficult for a single GPU kernel to fully utilize the vast GPU resources. One solution to improve resource utilization is concurrent kernel execution (CKE). Early CKE mainly targets the leftover resources. However, it fails to optimize the resource utilization and does not provide fairness among concurrent kernels. Spatial multitasking assigns a subset of streaming multiprocessors (SMs) to each kernel. Although achieving better fairness, the resource underutilization within an SM is not addressed. Thus, intra-SM sharing has been proposed to issue thread blocks from different kernels to each SM. However, as shown in this study, the overall performance may be undermined in the intra-SM sharing schemes due to the severe interference among kernels. Specifically, as concurrent kernels share the memory subsystem, one kernel, even as computing-intensive, may starve from not being able to issue memory instructions in time. Besides, severe L1 D-cache thrashing and memory pipeline stalls caused by one kernel, especially a memory-intensive one, will impact other kernels, further hurting the overall performance. In this study, we investigate various approaches to overcome the aforementioned problems exposed in intra-SM sharing. We first highlight that cache partitioning techniques proposed for CPUs are not effective for GPUs. Then we propose two approaches to reduce memory pipeline stalls. The first is to balance memory accesses of concurrent kernels. The second is to limit the number of inflight memory instructions issued from individual kernels. Our evaluation shows that the proposed schemes significantly improve the weighted speedup of two state-of-the-art intra-SM sharing schemes, Warped-Slicer and SMK, by 24.6% and 27.2% on average, respectively, with lightweight hardware overhead.

Partitioned K-Exclusion Real-Time Locking Protocol Motivated by Multicore Multi-GPU Systems

PK-OMLP: an OMLP Based K-Exclusion Real-Time Locking Protocol for Multi-GPU Sharing under Partitioned Scheduling

Simultaneous Multikernel GPU: Multi-tasking throughput processors via fine-grained sharing

Simultaneous Multikernel: Fine-Grained Sharing of GPUs.

RTGPU: Real-Time GPU Scheduling of Hard Deadline Parallel Tasks With Fine-Grain Utilization

Lock-based synchronization for GPU architectures.

POSTER: Accelerate GPU Concurrent Kernel Execution by Mitigating Memory Pipeline Stalls

Parallel Transient Stability-Constrained Optimal Power Flow Using GPU as Coprocessor.

Unleashing the Power of Preemptive Priority-based Scheduling for Real-Time GPU Tasks

A Virtual Multi-Channel GPU Fair Scheduling Method for Virtual Machines.

Fair and Cache Blocking Aware Warp Scheduling for Concurrent Kernel Execution on GPU

Orchestrated Co-scheduling, Resource Partitioning, and Power Capping on CPU-GPU Heterogeneous Systems via Machine Learning

Kernel concurrency opportunities based on GPU benchmarks characterization

Novel GPU Data Partitioning Method to Overlap Communication and Computation

GPGPU Task Scheduling Technique for Reducing the Performance Deviation of Multiple GPGPU Tasks in RPC-Based GPU Virtualization Environments

Locality-aware Thread Block Design in Single and Multi-GPU Graph Processing

Preemption-Aware Kernel Scheduling for GPUs

A CPU-GPGPU Scheduler Based on Data Transmission Bandwidth of Workload

DPCP-p: A Distributed Locking Protocol for Parallel Real-Time Tasks

A perceptual and predictive batch-processing memory scheduling strategy for a CPU-GPU heterogeneous system

Exploiting the Task-Pipelined Parallelism of Stream Programs on Many-Core GPUs.