Abstract:The abstraction of a shared memory space over separate CPU and GPU memory domains has eased the burden of portability for many HPC codebases. However, users pay for ease of use provided by system-managed memory with a moderate-to-high performance overhead. NVIDIA Unified Virtual Memory (UVM) is currently the primary real-world implementation of such abstraction and offers a functionally equivalent testbed for in-depth performance study for both UVM and future Linux Heterogeneous Memory Management (HMM) compatible systems. The continued advocacy for UVM and HMM motivates improvement of the underlying system. We focus on UVM-based systems and investigate root causes of UVM overhead, a non-trivial task due to complex interactions of multiple hardware and software constituents and the desired cost granularity. In our prior work, we delved deeply into UVM system architecture and showed internal behaviors of page fault servicing in batches. We provided quantitative evaluation of batch handling for various applications under different scenarios, including prefetching and oversubscription. We revealed the driver workload depends on the interactions among application access patterns, GPU hardware constraints, and host OS components. Host OS components have significant overhead present across implementations, warranting close attention. This extension furthers our prior study in three aspects: fine-grain cost analysis and breakdown, extension to multiple GPUs, and investigation of platforms with different GPU-GPU interconnects. We take a top-down approach to quantitative batch analysis and uncover how constituent component costs accumulate and overlap governed by synchronous and asynchronous operations. Our multi-GPU analysis shows reduced cost of GPU-GPU batch workloads compared to CPU-GPU workloads. We further demonstrate that while specialized interconnects, NVLink, can improve batch cost, their benefits are limited by host OS software overhead and GPU oversubscription. This study serves as a proxy for future shared memory systems, such as those that interface with HMM, and the development of interconnects.

In-depth analyses of unified virtual memory system for GPU accelerated computing

Fine-Grain Quantitative Analysis of Demand Paging in Unified Virtual Memory

GPUVM: GPU-driven Unified Virtual Memory

UVMBench: A Comprehensive Benchmark Suite for Researching Unified Virtual Memory in GPUs

PVM: A Memory Management under Vertical Architecture

Oversubscribing GPU Unified Virtual Memory: Implications and Suggestions

HMM-V: Heterogeneous Memory Management for Virtualization

Harnessing Integrated CPU-GPU System Memory for HPC: a first look into Grace Hopper

Batch-Aware Unified Memory Management in GPUs for Irregular Workloads

A Virtual Multi-Channel GPU Fair Scheduling Method for Virtual Machines.

PILOT: a Runtime System to Manage Multi-tenant GPU Unified Memory Footprint

Improving Multi-Application Concurrency Support Within the GPU Memory System

Demystifying the Performance of HPC Scientific Applications on NVM-based Memory Systems

Hardware-Software Collaborative Tiered-Memory Management Framework for Virtualization

Performance Evaluation of Advanced Features in CUDA Unified Memory

Techniques for Shared Resource Management in Systems with Throughput Processors

Efficient Resource Sharing Through GPU Virtualization on Accelerated High Performance Computing Systems

Analyzing Memory Access on CPU-GPGPU Shared LLC Architecture

Optimizing GPU Virtualization with Address Mapping and Delayed Submission

Automatic BLAS Offloading on Unified Memory Architecture: A Study on NVIDIA Grace-Hopper

vCUDA: GPU accelerated high performance computing in virtual machines