Abstract:Computing-in-memory (CIM) is an attractive approach for energy-efficient neural network (NN) processors. Attention mechanisms shows great performance in NLP and CV by capturing contextual knowledge from the entire tokens (X). The attention mechanism is essentially a content-based similarity search by computing attention probabilities (P) and final attention results (Att). For P, first, the query (Q) and the key (K) are computed by X and weight matrices $(\text{W}_{Q}, \text{W}_{K})$ respectively. Then, Q is multiplied by $\text{K}^{T}($ QxK $^{T})$ for the attention score (S). Finally, P is computed by Softmax-activating S. For Att, V is obtained by multiplying X and a weight matrix $(\text{W}_{V})$, and then, Att is computed by multiplying P and $\text{V}(\text{P}\times \text{V})$. As shown in Fig. 1, previous CIM chips face several challenges for P and Att computing [1, 2]. First, CIM shows great advantages only if multiplying a fixed matrix. But in P and Att computing, $\text{W}_{Q}$, $\text{W}_{K}, \text{W}_{V}$ are fixed, involving in 15% computations in Longformer. Thus, most computations mismatch the traditional paradigm of CIM. Second, in QxK $^{T}$, 34.7% of the computations are redundant as many near-zeros from Softmax become zero after quantization. Third, CIM macros perform inner product naturally. For Att, V is generated row-by-row (i.e., token-wise), but in $\text{P}\times \text{V}$, a column of V is left-multiplied by P (i.e., tokenacross). Only when V has been fully generated, can CIM macros perform $\text{P}\times \text{V}$. Thus, Att computing cannot be fully pipelined, reducing system throughput. This paper presents a processor named AttCIM that solves these issues with three key features: 1) A correlative CIM ring (CRCIMR) to avoid it to load dynamically generated matrices. 2) A Softmax-based speculate unit (SSU) to eliminate redundant computations in $\text{Q}\times \text{K}^{T}$. 3) A dataflow-reshaped digital-assisted CIM-array (DRCIMA) to achieve fully pipelined computations in $\text{P}\times \text{V}$.

ETCIM: an Error-Tolerant Digital-CIM Processor with Redundancy-Free Repair and Run-Time MAC and Cell Error Correction

An Overview of Computing-in-Memory Interfaces

CSA-CiM: Enhancing Multi-Functional Computing-in-Memory with Configurable Sense Amplifiers

Towards Error Correction for Computing in Racetrack Memory

Eliminating Timing Errors Through Collaborative Design to Maximize the Throughput

On Error Correction for Nonvolatile Processing-In-Memory

Accuracy Optimization with the Framework of Non-Volatile Computing-In-Memory Systems

34.3 A 22nm 64kb Lightning-Like Hybrid Computing-in-Memory Macro with a Compressed Adder Tree and Analog-Storage Quantizers for Transformer and CNNs.

A 4-Bit Calibration-Free Computing-In-Memory Macro with 3T1C Current-Programed Dynamic-Cascode Multi-Level-Cell Edram

Efficient Error-Correcting-Code Mechanism for High-Throughput Memristive Processing-in-Memory

Error Detection and Correction Codes for Safe In-Memory Computations

An Energy-Efficient Microprocessor Using Multilevel Error Correction For Timing Error Tolerance

A 28nm 16.9-300TOPS/W Computing-in-Memory Processor Supporting Floating-Point NN Inference/Training with Intensive-CIM Sparse-Digital Architecture

A 28nm 57.6TOPS/W Attention-based NN Processor with Correlative Computing-in-Memory Ring and Dataflow-reshaped Digital-assisted Computing-in-Memory Array

A 28-nm Floating-Point Computing-in-Memory Processor Using Intensive-CIM Sparse-Digital Architecture

An RRAM-Based Digital Computing-in-Memory Macro with Dynamic Voltage Sense Amplifier and Sparse-Aware Approximate Adder Tree

C 2 IM: A Compact Computing-In-Memory Unit of 10 Transistors with Standard 6T SRAM

ReDCIM: Reconfigurable Digital Computing- in -Memory Processor with Unified FP/INT Pipeline for Cloud AI Acceleration

34.7 A 28nm 2.4Mb/mm<sup>2</sup> 6.9 - 16.3TOPS/mm<sup>2</sup> eDRAM-LUT-Based Digital-Computing-in-Memory Macro with In-Memory Encoding and Refreshing

A Fully Bit-Flexible Computation in Memory Macro Using Multi-Functional Computing Bit Cell and Embedded Input Sparsity Sensing