Abstract:Deep learning (DL)-based personalized recommendation systems consume the major resources in modern AI data centers. The embedding layers with large memory capacity requirement and high bandwidth demand have been identified as the bottleneck of personalized recommendation inference. To mitigate the memory bandwidth bottleneck, near-memory processing (NMP) would be an effective solution which utilizes the through-silicon via (TSV) bandwidth within 3D-stacked DRAMs. However, existing NMP architectures suffer from the limited memory bandwidth caused by hard-to-scale TSVs. To overcome this obstacle, integrating the compute-logic near memory banks becomes a promising but challenging solution, since large memory capacity requirement limits the use of 3D-stacked DRAMs and irregular memory accesses lead to poor data locality, heavy TSV data traffic and low bank-level bandwidth utilization. To address this problem, we propose RecPIM, the first in-memory processing system for personalized recommendation inference using near-bank architecture based on 3D-stacked memory. From the hardware perspective, we introduce a heterogeneous memory system combined with 3D-stacked DRAM and DIMMs to accommodate large embedding tables and provide high bandwidth. By integrating processing logic units near memory banks on DRAM dies, our architecture can exploit the enormous bank-level bandwidth which is much higher than TSV bandwidth. Then, we integrate a small scratchpad memory to exploit the unique data reusability of DL-based personalized recommendation systems. Furthermore, we adopt a unidirectional data communication scheme to avoid additional cross-vault data transfer. From the software perspective, we present a customized programming model to facilitate memory management and task offloading. To reduce the data communication through TSVs and enhance the utilization of bank-level bandwidth, we develop an efficient data mapping scheme by partitioning the vector into smaller subvectors. Experimental results show that RecPIM achieves up to 2.58x speedup and 49.8% energy saving for data movement over the state-of-the-art NMP solution.

Parana: A Parallel Neural Architecture Considering Thermal Problem of 3D Stacked Memory

DaDianNao: A Machine-Learning Supercomputer

Parapim: A Parallel Processing-In-Memory Accelerator For Binary-Weight Deep Neural Networks

NicePIM: Design Space Exploration for Processing-In-Memory DNN Accelerators with 3D-Stacked-DRAM

Accelerating Neural Network Inference with Processing-in-DRAM: From the Edge to the Cloud

Hardware Architecture and Software Stack for PIM Based on Commercial DRAM Technology : Industrial Product

The study of memory architectures for 3D chip multi-processors

Temperature-Aware DRAM Cache Management—Relaxing Thermal Constraints in 3-D Systems

HuNT: Exploiting Heterogeneous PIM Devices to Design a 3-D Manycore Architecture for DNN Training

Memory-Computing Decoupling: A DNN Multitasking Accelerator with Adaptive Data Arrangement.

Accelerating Neural Network Training with Processing-in-Memory GPU

Design of 3D DRAM and Its Application in 3D Integrated Multi-Core Computing Systems

Optimizing power efficiency for 3D stacked GPU-in-memory architecture

Flash Memory Array for Efficient Implementation of Deep Neural Networks

Improving power-performance via hybrid cache for chip many cores based on neural network prediction technique

Computing Utilization Enhancement for Chiplet-based Homogeneous Processing-in-Memory Deep Learning Processors

Enabling the Adoption of Processing-in-Memory: Challenges, Mechanisms, Future Research Directions

An Energy-Efficient Near-Data Processing Accelerator for DNNs that Optimizes Data Accesses

A 3D SoC Design for H.264 Application with On-Chip DRAM Stacking

RecPIM: Efficient In-Memory Processing for Personalized Recommendation Inference Using Near-Bank Architecture

A Hybrid Computing-In-Memory Architecture by Monolithic 3D Integration of BEOL CNT/IGZO-based CFET Logic and Analog RRAM