Abstract:The attention mechanism is becoming increasingly popular in Natural Language Processing (NLP) applications, showing superior performance than convolutional and recurrent architectures. However, attention becomes the compution bottleneck because of its quadratic computational complexity to input length, complicated data movement and low arithmetic intensity. Moreover, existing NN accelerators mainly focus on optimizing convolutional or recurrent models, and cannot efficiently support attention. In this paper, we present SpAtten, an efficient algorithm-architecture co-design that leverages token sparsity, head sparsity, and quantization opportunities to reduce the attention computation and memory access. Inspired by the high redundancy of human languages, we propose the novel cascade token pruning to prune away unimportant tokens in the sentence. We also propose cascade head pruning to remove unessential heads. Cascade pruning is fundamentally different from weight pruning since there is no trainable weight in the attention mechanism, and the pruned tokens and heads are selected on the fly. To efficiently support them on hardware, we design a novel top-k engine to rank token and head importance scores with high throughput. Furthermore, we propose progressive quantization that first fetches MSBs only and performs the computation; if the confidence is low, it fetches LSBs and recomputes the attention outputs, trading computation for memory reduction. Extensive experiments on 30 benchmarks show that, on average, SpAtten reduces DRAM access by 10.0x with no accuracy loss, and achieves 1.6x, 3.0x, 162x, 347x speedup, and 1,4x, 3.2x, 1193x, 4059x energy savings over A3 accelerator, MNNFast accelerator, TITAN Xp GPU, Xeon CPU, respectively.

What problem does this paper attempt to address?

The problem this paper attempts to address is the computational inefficiency of the attention mechanism in natural language processing (NLP) applications. Specifically, the attention mechanism becomes a computational bottleneck due to its quadratic computational complexity with respect to input length, complex data movement, and low arithmetic intensity. Existing neural network accelerators primarily optimize for convolutional or recurrent models and cannot efficiently support the attention mechanism. Therefore, the paper proposes an efficient algorithm-architecture co-design method called SpAtten, which reduces attention computation and memory access by leveraging token sparsity, head sparsity, and quantization opportunities. ### Main Issues: 1. **Computational Inefficiency**: The computational complexity of the attention mechanism is quadratic with respect to the input length, leading to inefficiency. 2. **Frequent Memory Access**: The attention mechanism requires a large amount of memory access, especially during the generation phase, where data movement takes up most of the time. 3. **Inapplicability of Existing Accelerators**: Existing neural network accelerators are primarily optimized for convolutional and recurrent models and cannot effectively support the attention mechanism. ### Solutions: 1. **Cascaded Token Pruning**: By evaluating the importance of tokens and removing unimportant ones, computation and memory access are reduced. 2. **Cascaded Head Pruning**: By evaluating the importance of heads and removing unimportant ones, computation and memory access are further reduced. 3. **Progressive Quantization**: By adjusting the bit-width of different heads and layers based on the attention probability distribution, memory access is reduced. ### Experimental Results: - In 30 benchmarks, SpAtten reduced DRAM access by an average of 10.0 times without loss of accuracy. - Compared to the A3 accelerator, MNNFast accelerator, TITAN Xp GPU, and Xeon CPU, SpAtten achieved speedups of 1.6 times, 3.0 times, 162 times, and 347 times, respectively, and energy savings of 1.4 times, 3.2 times, 1193 times, and 4059 times, respectively. Through these methods, SpAtten effectively addresses the computational inefficiency of the attention mechanism in NLP applications, enabling its deployment on resource-constrained devices.

SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruning

SAC: Accelerating and Structuring Self-Attention Via Sparse Adaptive Connection.

DTQAtten: Leveraging Dynamic Token-based Quantization for Efficient Attention Architecture

Sanger: A Co-Design Framework for Enabling Sparse Attention Using Reconfigurable Architecture.

CPSAA: Accelerating Sparse Attention using Crossbar-based Processing-In-Memory Architecture

Sparse Attention Acceleration with Synergistic In-Memory Pruning and On-Chip Recomputation

Efficient Sparse Attention needs Adaptive Token Release

Multilayer Dataflow: Orchestrate Butterfly Sparsity to Accelerate Attention Computation

Data-Informed Global Sparseness in Attention Mechanisms for Deep Neural Networks

S2-Attention: Hardware-Aware Context Sharding Among Attention Heads

COSA:Co-Operative Systolic Arrays for Multi-head Attention Mechanism in Neural Network Using Hybrid Data Reuse and Fusion Methodologies.

SparseAccelerate: Efficient Long-Context Inference for Mid-Range GPUs

SNP: Structured Neuron-level Pruning to Preserve Attention Scores

Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformers

DTATrans: Leveraging Dynamic Token-Based Quantization with Accuracy Compensation Mechanism for Efficient Transformer Architecture.

Post-Training Sparse Attention with Double Sparsity

SampleAttention: Near-Lossless Acceleration of Long Context LLM Inference with Adaptive Structured Sparse Attention

Adaptive Sparse and Monotonic Attention for Transformer-based Automatic Speech Recognition

Stochastic Spiking Attention: Accelerating Attention with Stochastic Computing in Spiking Networks

Accelerating Attention through Gradient-Based Learned Runtime Pruning