Abstract:Transformer networks have outperformed recurrent and convolutional neural networks in terms of accuracy in various sequential tasks. However, memory and compute bottlenecks prevent transformer networks from scaling to long sequences due to their high execution time and energy consumption. Different neural attention mechanisms have been proposed to lower computational load but still suffer from the memory bandwidth bottleneck. In-memory processing can help alleviate memory bottlenecks by reducing the transfer overhead between the memory and compute units, thus allowing transformer networks to scale to longer sequences. We propose an in-memory transformer network accelerator (iMTransformer) that uses a combination of crossbars and content-addressable memories to accelerate transformer networks. We accelerate transformer networks by (1) computing in-memory, thus minimizing the memory transfer overhead, (2) caching reusable parameters to reduce the number of operations, and (3) exploiting the available parallelism in the attention mechanism computation. To reduce energy consumption, the following techniques are introduced: (1) a configurable attention selector is used to choose different sparse attention patterns, (2) a content-addressable memory aided locality sensitive hashing helps to filter the number of sequence elements by their importance, and (3) FeFET-based crossbars are used to store projection weights while CMOS-based crossbars are used as an attentional cache to store attention scores for later reuse. Using a CMOS-FeFET hybrid iMTransformer introduced a significant energy improvement compared to the CMOS-only iMTransformer. The CMOS-FeFET hybrid iMTransformer achieved an 8.96× delay improvement and 12.57× energy improvement for the Vanilla transformers compared to the GPU baseline at a sequence length of 512. Implementing BERT using CMOS-FeFET hybrid iMTransformer achieves 13.71× delay improvement and 8.95× delay improvement compared to the GPU baseline at sequence length of 512. The hybrid iMTransformer also achieves a throughput of 2.23 K samples/sec and 124.8 samples/s/W using the MLPerf benchmark using BERT-large and SQuAD 1.1 dataset, an 11× speedup and 7.92× energy improvement compared to the GPU baseline.

IceFormer: Accelerated Inference with Long-Sequence Transformers on CPUs

DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models

Hardware-Software Co-Design of an In-Memory Transformer Network Accelerator

Transformer Acceleration with Dynamic Sparse Attention

Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inference

Benchmarking and In-depth Performance Study of Large Language Models on Habana Gaudi Processors

Enhancing Long Sequence Input Processing in FPGA-Based Transformer Accelerators Through Attention Fusion

Adaptive Multi-Resolution Attention with Linear Complexity

Fourier Transformer: Fast Long Range Modeling by Removing Sequence Redundancy with FFT Operator

AttMEMO : Accelerating Transformers with Memoization on Big Memory Systems

Expanding the Prediction Capacity in Long Sequence Time-Series Forecasting

An Efficient Sparse Inference Software Accelerator for Transformer-based Language Models on CPUs

Flash Inference: Near Linear Time Inference for Long Convolution Sequence Models and Beyond

A Length Adaptive Algorithm-Hardware Co-design of Transformer on FPGA Through Sparse Attention and Dynamic Pipelining

Improving Computation and Memory Efficiency for Real-world Transformer Inference on GPUs

An Algorithm-Hardware Co-Optimized Framework for Accelerating N:M Sparse Transformers

X-Former: In-Memory Acceleration of Transformers

Energon: Toward Efficient Acceleration of Transformers Using Dynamic Sparse Attention.

Inference Performance Optimization for Large Language Models on CPUs

Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting

Ayaka: A Versatile Transformer Accelerator with Low-Rank Estimation and Heterogeneous Dataflow