Abstract:While reduction in feature size makes computation cheaper in terms of latency, area, and power consumption, performance of emerging data-intensive applications is determined by data movement. These trends have introduced the concept of scalability as reaching a desirable performance per unit cost by using as few number of units as possible. Many proposals have moved compute closer to the memory. However, these efforts ignored maintaining a balance between bandwidth and compute rate of an architecture, with those of applications, which is a key principle in designing scalable large systems. This paper proposes the use of memory slices, a modular building block for scalable memory systems integrated with compute, in which performance scales with memory size (and volume of data). The slice architecture utilizes a programmable memory interface feeding a systolic compute engine with high reuse rate. The modularity feature of slice-based systems is exploited with a partitioning and data mapping strategy across allocated memory slices where training performance scales with the data size. These features enable shifting the most pressure to cheap compute units rather than expensive memory accesses or transfers via interconnection network. An application of the memory slices to a scale-out memory system is accelerating the training of recurrent, convolutional, and hybrid neural networks (RNNs and RNNs+CNN) that are forming cloud workloads. The results of our cycle-level simulations show that memory slices exhibits a superlinear speedup when the number of slices increases. Furthermore, memory slices improve power efficiency to 747 GFLOPs/J for training LSTMs. While our current evaluation uses memory slices with 3D packaging, a major value is that slices can also be constructed with a variety of packaging options, for example with DDR-based memory units.

Memory Layers at Scale

CNN with large memory layers

Ultra-Sparse Memory Network

$\text{Memory}^3$: Language Modeling with Explicit Memory

MemoryFormer: Minimize Transformer Computation by Removing Fully-Connected Layers

Augmenting Language Models with Long-Term Memory

Layer-Condensed KV Cache for Efficient Inference of Large Language Models

A Model or 603 Exemplars: Towards Memory-Efficient Class-Incremental Learning

MemLLM: Finetuning LLMs to Use An Explicit Read-Write Memory

Memory Slices: A Modular Building Block for Scalable, Intelligent Memory Systems

Needle in the Haystack for Memory Based Large Language Models

A Method for Building Large Language Models with Predefined KV Cache Capacity

Scaling Memory-Augmented Neural Networks with Sparse Reads and Writes

MEMORYLLM: Towards Self-Updatable Large Language Models

Memorizing Transformers

Memory Augmented Language Models through Mixture of Word Experts

ML-Fusion: Determining Memory Levels for Data Reuse Between DNN Layers

Layer-Centric Memory Reuse and Data Migration for Extreme-Scale Deep Learning on Many-Core Architectures

Memory Is All You Need: An Overview of Compute-in-Memory Architectures for Accelerating Large Language Model Inference

XKV: Personalized KV Cache Memory Reduction for Long-Context LLM Inference