Abstract:Transformer neural networks, driven by self-attention mechanisms, are core components of foundational and Large Language Models. In generative transformers, self-attention uses cache memory to store token projections, avoiding recomputation at each time step. However, GPU-stored projections must be loaded into SRAM for each new generation step, causing latency and energy bottlenecks for long sequences. In this work, we propose a fast and energy-efficient hardware implementation of self-attention using analog in-memory computing based on gain cell memories. Volatile gain cell memories can be efficiently written to store new tokens during sequence generation, while performing analog signed weight multiplications to compute the dot-products required for self-attention. We implement Sliding Window Attention, which keeps memory of a finite set of past steps. A charge-to-pulse converter for array readout eliminates the need for analog-to-digital conversion between self-attention stages. Using a co-designed initialization algorithm to adapt pre-trained weights to gain cell non-idealities, we achieve NLP performance comparable to ChatGPT-2 with minimal training iterations, despite hardware constraints. Our end-to-end hardware design includes digital controls, estimating area, latency, and energy. The system reduces attention latency by up to two orders of magnitude and energy consumption by up to five orders compared to GPUs, marking a significant step toward ultra-fast, low-power sequence generation in Large Language Models.

Augmenting Self-attention with Persistent Memory

Is Attention All What You Need? -- An Empirical Investigation on Convolution-Based Active Memory and Self-Attention

Memorization in Attention-only Transformers

GMAT: Global Memory Augmentation for Transformers

Augmenting Language Models with Long-Term Memory

An Evolved Universal Transformer Memory

Memory-Augmenting Decoder-Only Language Models through Encoders (Student Abstract)

Predictive Attention Transformer: Improving Transformer with Attention Map Prediction

Analog In-Memory Computing Attention Mechanism for Fast and Energy-Efficient Large Language Models

Beyond Scaling Laws: Understanding Transformer Performance with Associative Memory

TransformerFAM: Feedback attention is working memory

Global memory transformer for processing long documents

Extended Mind Transformers

Cached Transformers: Improving Transformers with Differentiable Memory Cache

Remembering Transformer for Continual Learning

MemoryFormer: Minimize Transformer Computation by Removing Fully-Connected Layers

Memorization Capacity of Multi-Head Attention in Transformers

AttMEMO : Accelerating Transformers with Memoization on Big Memory Systems

LaMemo: Language Modeling with Look-Ahead Memory

TRAMS: Training-free Memory Selection for Long-range Language Modeling

On Difficulties of Attention Factorization through Shared Memory