Abstract:Computing-in-memory (CIM) is an attractive approach for energy-efficient neural network (NN) processors. Attention mechanisms shows great performance in NLP and CV by capturing contextual knowledge from the entire tokens (X). The attention mechanism is essentially a content-based similarity search by computing attention probabilities (P) and final attention results (Att). For P, first, the query (Q) and the key (K) are computed by X and weight matrices $(\text{W}_{Q}, \text{W}_{K})$ respectively. Then, Q is multiplied by $\text{K}^{T}($ QxK $^{T})$ for the attention score (S). Finally, P is computed by Softmax-activating S. For Att, V is obtained by multiplying X and a weight matrix $(\text{W}_{V})$, and then, Att is computed by multiplying P and $\text{V}(\text{P}\times \text{V})$. As shown in Fig. 1, previous CIM chips face several challenges for P and Att computing [1, 2]. First, CIM shows great advantages only if multiplying a fixed matrix. But in P and Att computing, $\text{W}_{Q}$, $\text{W}_{K}, \text{W}_{V}$ are fixed, involving in 15% computations in Longformer. Thus, most computations mismatch the traditional paradigm of CIM. Second, in QxK $^{T}$, 34.7% of the computations are redundant as many near-zeros from Softmax become zero after quantization. Third, CIM macros perform inner product naturally. For Att, V is generated row-by-row (i.e., token-wise), but in $\text{P}\times \text{V}$, a column of V is left-multiplied by P (i.e., tokenacross). Only when V has been fully generated, can CIM macros perform $\text{P}\times \text{V}$. Thus, Att computing cannot be fully pipelined, reducing system throughput. This paper presents a processor named AttCIM that solves these issues with three key features: 1) A correlative CIM ring (CRCIMR) to avoid it to load dynamically generated matrices. 2) A Softmax-based speculate unit (SSU) to eliminate redundant computations in $\text{Q}\times \text{K}^{T}$. 3) A dataflow-reshaped digital-assisted CIM-array (DRCIMA) to achieve fully pipelined computations in $\text{P}\times \text{V}$.

A 6.54-to-26.03 TOPS/W Computing-In-Memory RNN Processor Using Input Similarity Optimization and Attention-based Context-breaking with Output Speculation

An 8-Bit in Resistive Memory Computing Core with Regulated Passive Neuron and Bitline Weight Mapping

A 3.89-Gops/mw Scalable Recurrent Neural Network Processor with Improved Efficiency on Memory and Computation

A 28nm 57.6TOPS/W Attention-based NN Processor with Correlative Computing-in-Memory Ring and Dataflow-reshaped Digital-assisted Computing-in-Memory Array

A 41.7TOPS/W@INT8 Computing-in-Memory Processor with Zig-Zag Backbone-Systolic CIM and Block/Self-Gating CAM for NN/Recommendation Applications

An Energy-Efficient Computing-in-Memory NN Processor with Set-Associate Blockwise Sparsity and Ping-Pong Weight Update

A 5.1pJ/Neuron 127.3us/Inference RNN-based Speech Recognition Processor using 16 Computing-in-Memory SRAM Macros in 65nm CMOS

STICKER-IM: A 65 nm Computing-in-Memory NN Processor Using Block-Wise Sparsity Optimization and Inter/Intra-Macro Data Reuse

TT@CIM: A Tensor-Train In-Memory-Computing Processor Using Bit-Level-Sparsity Optimization and Variable Precision Quantization

14.3 A 65nm Computing-in-Memory-Based CNN Processor with 2.9-to-35.8tops/w System Energy Efficiency Using Dynamic-Sparsity Performance-Scaling Architecture and Energy-Efficient Inter/Intra-Macro Data Reuse.

A Twin-8T SRAM Computation-in-Memory Unit-Macro for Multibit CNN-Based AI Edge Processors

A Low-Power In-Memory Multiplication and Accumulation Array with Modified Radix-4 Input and Canonical Signed Digit Weights

A Dual-Wordline 6T SRAM Computing-In-Memory Macro Featuring Full Signed Multi-Bit Computation for Lightweight Networks

A 28 Nm RRAM-Based 81.1 TOPS/mm2/bit Compute-In-Memory Macro with Uniform and Linear 64 Read Channels under 512 4-Bit Inputs

Tempo-CIM: A RRAM Compute-in-Memory Neuromorphic Accelerator with Area-Efficient LIF Neuron and Split-Train-Merged-Inference Algorithm for Edge AI Applications.

A 2.75-to-75.9tops/w Computing-in-Memory NN Processor Supporting Set-Associate Block-Wise Zero Skipping and Ping-Pong CIM with Simultaneous Computation and Weight Updating.

14.3 A 65nm Computing-in-Memory-Based CNN Processor with 2.9-to-35.8 TOPS/W System Energy Efficiency Using Dynamic-Sparsity Performance-Scaling Architecture and Energy …

A 28-Nm RRAM Computing-in-Memory Macro Using Weighted Hybrid 2T1R Cell Array and Reference Subtracting Sense Amplifier for AI Edge Inference

15.3 A 351TOPS/W and 372.4GOPS Compute-in-Memory SRAM Macro in 7nm FinFET CMOS for Machine-Learning Applications

An XOR-10T SRAM computing-in-memory macro with current MAC operations and time-to-digital conversion for BNN edge processors

A Heterogeneous Microprocessor for Intermittent AI Inference Using Nonvolatile-SRAM-based Compute-In-Memory