Abstract:We propose overcoming the memory capacity limitation of GPUs with high-capacity Storage-Class Memory (SCM) and DRAM cache. By significantly increasing the memory capacity with SCM, the GPU can capture a larger fraction of the memory footprint than HBM for workloads that oversubscribe memory, achieving high speedups. However, the DRAM cache needs to be carefully designed to address the latency and BW limitations of the SCM while minimizing cost overhead and considering GPU's characteristics. Because the massive number of GPU threads can thrash the DRAM cache, we first propose an SCM-aware DRAM cache bypass policy for GPUs that considers the multi-dimensional characteristics of memory accesses by GPUs with SCM to bypass DRAM for data with low performance utility. In addition, to reduce DRAM cache probes and increase effective DRAM BW with minimal cost, we propose a Configurable Tag Cache (CTC) that repurposes part of the L2 cache to cache DRAM cacheline tags. The L2 capacity used for the CTC can be adjusted by users for adaptability. Furthermore, to minimize DRAM cache probe traffic from CTC misses, our Aggregated Metadata-In-Last-column (AMIL) DRAM cache organization co-locates all DRAM cacheline tags in a single column within a row. The AMIL also retains the full ECC protection, unlike prior DRAM cache's Tag-And-Data (TAD) organization. Additionally, we propose SCM throttling to curtail power and exploiting SCM's SLC/MLC modes to adapt to workload's memory footprint. While our techniques can be used for different DRAM and SCM devices, we focus on a Heterogeneous Memory Stack (HMS) organization that stacks SCM dies on top of DRAM dies for high performance. Compared to HBM, HMS improves performance by up to 12.5x (2.9x overall) and reduces energy by up to 89.3% (48.1% overall). Compared to prior works, we reduce DRAM cache probe and SCM write traffic by 91-93% and 57-75%, respectively.

Native DRAM Cache: Re-architecting DRAM as a Large-Scale Cache for Data Centers

Enabling Efficient Dynamic Resizing of Large DRAM Caches via A Hardware Consistent Hashing Mechanism

TDRAM: Tag-enhanced DRAM for Efficient Caching

Line-Coalescing DRAM Cache

Cooperatively Managing Dynamic Writeback and Insertion Policies in a Last-Level DRAM Cache.

Bi-Modal DRAM Cache: Improving Hit Rate, Hit Latency and Bandwidth

FASA-DRAM: Reducing DRAM Latency with Destructive Activation and Delayed Restoration

Exploring DRAM Cache Prefetching for Pooled Memory

CDAR-DRAM - an In-situ Charge Detection and Adaptive Data Restoration DRAM Architecture for Performance and Energy Efficiency Improvement.

DR DRAM: Accelerating Memory-Read-Intensive Applications

Trade-off between Hit Rate and Hit Latency for Optimizing DRAM Cache

HyGain: High Performance, Energy-Efficient Hybrid Gain Cell based Cache Hierarchy

Enabling the Adoption of Processing-in-Memory: Challenges, Mechanisms, Future Research Directions

Architecting On-Chip Interconnects for Stacked 3D STT-RAM Caches in CMPs

Banshee: Bandwidth-Efficient DRAM Caching Via Software/Hardware Cooperation

A Cache Reconfiguration Approach for Saving Leakage and Refresh Energy in Embedded DRAM Caches

Bandwidth-Effective DRAM Cache for GPUs with Storage-Class Memory

Read-Tuned STT-RAM and eDRAM Cache Hierarchies for Throughput and Energy Enhancement

CWFP: Novel Collective Writeback and Fill Policy for Last-Level DRAM Cache.

Understanding the Memory Behavior of Emerging Multi-core Workloads

Replacing 6T SRAMs with 3T1D DRAMs in the L1 Data Cache to Combat Process Variability