Abstract:Transformer models have achieved impressive performance in various artificial intelligence (AI) applications. However, the high cost of computation and memory footprint make its inference inefficient. Although digital compute-in-memory (CIM) is a promising hardware architecture with high accuracy, Transformer’s attention mechanism raises three challenges in the access and computation of CIM: 1) the attention computation involving Query and Key results in massive data movement and under-utilization in CIM macros; 2) the attention computation involving Possibility and Value exhibits plenty of dynamic bit-level sparsity, resulting in redundant bit-serial CIM operations; and 3) the restricted data reload bandwidth in CIM macros results in a significant decrease in performance for large Transformer models. To address these challenges, we design a CIM accelerator called CIM Transformer (CIMFormer) with three corresponding features. First, the token-pruning-aware attention reformulation (TPAR) is a technique that adjusts attention computations according to the token-pruning ratio. This reformulation reduces the real-time access to and under-utilization of CIM macros. Second, the principal possibility gather-scatter scheduler (PPGSS) gathers the possibilities with greater effective bit-width as concurrent inputs to CIM macros, enhancing the efficiency of bit-serial CIM operations. Third, the systolic X $\mid$ W-CIM macro array efficiently handles the execution of large Transformer models that exceed the storage capacity of the on-chip CIM macros. Fabricated in a 28-nm technology, CIMFormer achieves a peak energy efficiency of 15.71 TOPS/W, with an over 1.46 $\times$ improvement compared with the state-of-the-art Transformer accelerator at an equivalent situation.

TSA-TICER: A Two-Stage TICER Acceleration Framework for Model Order Reduction

P-TICER: An Effective Parallel TICER Acceleration Method for Model Order Reduction

High-Dimensional Extension of the TICER Algorithm

Error Analysis on the TICER Approximation

A Time Constant Estimation Method for Block RC Circuits with Application to Power Grid Analysis

TurboMOR: an Efficient Model Order Reduction Technique for RC Networks with Many Ports

A Broadband Enhanced Structure-Preserving Reduced-Order Interconnect Macromodeling Method for Large-Scale Equation Sets of Transient Interconnect Circuit Problems

A new efficient CAD tool for reduction of large RC networks based on PACT algorithm

Model Order Reduction for Large Scale RC Networks Based on Optimal Elimination

Realizable Reduction of Multi-Port RCL Networks by Block Elimination

Modeling of Through-Silicon Capacitor and Its Applications for the Optimization of Power Distribution Network in 3-D Integrated Circuits

Order Reduction and Rapid Calculation for Multimodule Linear Circuits

ISPT-Net: A Noval Transient Backward-Stepping Reduction Policy by Irregular Sequential Prediction Transformer

Transistor Compact Model Based on Multigradient Neural Network and Its Application in SPICE Circuit Simulations for Gate-All-Around Si Cold Source FETs

TIGER: an Efficient Timing-Driven Global Router for Gate Array and Standard Cell Layout Design

Sparse Implicit Projection (SIP) for Reduction of General Many-Terminal Networks

Model Reduction Method Based on Selective Clustering Ensemble Algorithm and Theory of Constraints in Semiconductor Wafer Fabrication.

Transient Stability Assessment Combined Model Framework Based on Cost‐sensitive Method

Structure-Preserving Model Order Reduction for Thermal Analysis

CIMFormer: A Systolic CIM-Array-Based Transformer Accelerator with Token-Pruning-Aware Attention Reformulating and Principal Possibility Gathering

MulTCIM: Digital Computing-in-Memory-Based Multimodal Transformer Accelerator With Attention-Token-Bit Hybrid Sparsity