Abstract:The many-body correlation function is a fundamental computation kernel in modern physics computing applications, e.g., Hadron Contractions in Lattice quantum chromodynamics (QCD). This kernel is both computation and memory intensive, involving a series of tensor contractions, and thus usually runs on accelerators like GPUs. Existing optimizations on many-body correlation mainly focus on individual tensor contractions (e.g., cuBLAS libraries and others). In contrast, this work discovers a new optimization dimension for many-body correlation by exploring the optimization opportunities among tensor contractions. More specifically, it targets general GPU architectures (both NVIDIA and AMD) and optimizes many-body correlation’s memory management by exploiting a set of memory allocation and communication redundancy elimination opportunities: first, GPU memory allocation redundancy : the intermediate output frequently occurs as input in the subsequent calculations; second, CPU-GPU communication redundancy : although all tensors are allocated on both CPU and GPU, many of them are used (and reused) on the GPU side only, and thus, many CPU/GPU communications (like that in existing Unified Memory designs) are unnecessary; third, GPU oversubscription: limited GPU memory size causes oversubscription issues, and existing memory management usually results in near-reuse data eviction, thus incurring extra CPU/GPU memory communications. Targeting these memory optimization opportunities, this article proposes MemHC, an optimized systematic GPU memory management framework that aims to accelerate the calculation of many-body correlation functions utilizing a series of new memory reduction designs. These designs involve optimizations for GPU memory allocation, CPU/GPU memory movement, and GPU memory oversubscription, respectively. More specifically, first, MemHC employs duplication-aware management and lazy release of GPU memories to corresponding host managing for better data reusability. Second, it implements data reorganization and on-demand synchronization to eliminate redundant (or unnecessary) data transfer. Third, MemHC exploits an optimized Least Recently Used (LRU) eviction policy called Pre-Protected LRU to reduce evictions and leverage memory hits. Additionally, MemHC is portable for various platforms including NVIDIA GPUs and AMD GPUs. The evaluation demonstrates that MemHC outperforms unified memory management by \( 2.18\times \) to \( 10.73\times \) . The proposed Pre-Protected LRU policy outperforms the original LRU policy by up to \( 1.36\times \) improvement. 1

Improving Oversubscribed GPU Memory Performance in the PyTorch Framework

Memory Optimization for Deep Networks

A Framework for Memory Oversubscription Management in Graphics Processing Units

Efficient Use of GPU Memory for Large-Scale Deep Learning Model Training

Performance Evaluation of Advanced Features in CUDA Unified Memory

Breaking the Memory Wall: A Study of I/O Patterns and GPU Memory Utilization for Hybrid CPU-GPU Offloaded Optimizers

Oversubscribing GPU Unified Virtual Memory: Implications and Suggestions

Efficient Memory Management for GPU-based Deep Learning Systems

BARM: A Batch-Aware Resource Manager for Boosting Multiple Neural Networks Inference on GPUs With Memory Oversubscription

HOME: A Holistic GPU Memory Management Framework for Deep Learning

MemHC: An Optimized GPU Memory Management Framework for Accelerating Many-body Correlation

PILOT: a Runtime System to Manage Multi-tenant GPU Unified Memory Footprint

SuperNeurons: Dynamic GPU Memory Management for Training Deep Neural Networks

TSPLIT: Fine-grained GPU Memory Management for Efficient DNN Training Via Tensor Splitting

Coordinated Page Prefetch and Eviction for Memory Oversubscription Management in GPUs

MegTaiChi: Dynamic Tensor-based Memory Management Optimization for DNN Training

Exploiting Memory Access Patterns to Improve Memory Performance in Data-Parallel Architectures

Improving Memory Utilization in Convolutional Neural Network Accelerators

ICCAD : U : Optimizing GPU Shared Memory Allocation in Automated Cto-CUDA Compilation

Optimizing GPU Memory Transactions for Convolution Operations

An Application-oblivious Memory Scheduling System for DNN Accelerators