Abstract:Following the advances in technology scaling, graphics processing units (GPUs) incorporate an increasing amount of computing resources and it becomes difficult for a single GPU kernel to fully utilize the vast GPU resources. One solution to improve resource utilization is concurrent kernel execution (CKE). Early CKE mainly targets the leftover resources. However, it fails to optimize the resource utilization and does not provide fairness among concurrent kernels. Spatial multitasking assigns a subset of streaming multiprocessors (SMs) to each kernel. Although achieving better fairness, the resource underutilization within an SM is not addressed. Thus, intra-SM sharing has been proposed to issue thread blocks from different kernels to each SM. However, as shown in this study, the overall performance may be undermined in the intra-SM sharing schemes due to the severe interference among kernels. Specifically, as concurrent kernels share the memory subsystem, one kernel, even as computing-intensive, may starve from not being able to issue memory instructions in time. Besides, severe L1 D-cache thrashing and memory pipeline stalls caused by one kernel, especially a memory-intensive one, will impact other kernels, further hurting the overall performance. In this study, we investigate various approaches to overcome the aforementioned problems exposed in intra-SM sharing. We first highlight that cache partitioning techniques proposed for CPUs are not effective for GPUs. Then we propose two approaches to reduce memory pipeline stalls. The first is to balance memory accesses of concurrent kernels. The second is to limit the number of inflight memory instructions issued from individual kernels. Our evaluation shows that the proposed schemes significantly improve the weighted speedup of two state-of-the-art intra-SM sharing schemes, Warped-Slicer and SMK, by 24.6% and 27.2% on average, respectively, with lightweight hardware overhead.

Study on Optimization Techniques for Memory Accesses of CUDA Parallel Programs

An Experimental GPU Global Memory Performance Estimation and Optimization

Optimizing Non-Coalesced Memory Access for Irregular Applications with GPU Computing

CUDA Optimization Strategies for Compute- and Memory-Bound Neuroimaging Algorithms

ICCAD : U : Optimizing GPU Shared Memory Allocation in Automated Cto-CUDA Compilation

GPU Memory Optimization Through Program Restructuring Methods

A Survey of GPGPU Parallel Processing Architecture Performance Optimization

An Optimization Strategy for Improving Throughput of GPU Global Memory

Optimization method for multi-core last level Cache considering the memory access modes

Two Methods for Combining Original Memory Access Coalescing and Equivalent Memory Access Coalescing on GPGPU.

GPGPU Memory Estimation and Optimization Targeting OpenCL Architecture

Performance Evaluation and Optimization on Gpu

Benchmarking the GPU Memory at the Warp Level

An approach to accessing unified memory address space of heterogeneous kilo-cores system

Exploiting Memory Access Patterns to Improve Memory Performance in Data-Parallel Architectures

Workload Analysis for Typical GPU Programs Using CUPTI Interface

Program Optimization of Stencil Based Application on the GPU-Accelerated System

A Case Study of SWIM: Optimization of Memory Intensive Application on GPGPU

Research on Parallel Modern Optimization Algorithms Using GPU

Register and Thread Structure Optimization for GPUs

POSTER: Accelerate GPU Concurrent Kernel Execution by Mitigating Memory Pipeline Stalls