Abstract:Vision Transformers (ViTs) have recently made a splash in computer vision domain and achieved state-of-the-art in many vision tasks. Nevertheless, due to their vast model size and high computational costs, rare transformer-based models are adopted in real-world applications. Since the computational costs of attention operation is the square of the input size, some compression methods for the Multi-Head Self-Attention (MHSA) module have been proposed, reducing its FLOPs successfully but almost without parameters reduction. Meanwhile, the number of parameters and computational costs in the Feed-Forward Network (FFN) module exceeds the MHSA larger, while its compression technologies have not been delved deeper. Consequently, we focus our insight on the compression of FFN layer and present a pruning method named Multi-Dimension Compression of Feed-Forward Network in Vision Transformers(MCF), which greatly reduces the model’s parameters and computational costs. Firstly, we identify the critical elements in the output of the FFN module and then employ them to guide the irregular sparsity of this layer, recognizing insignificant elements of FFN layer that have less impact on the output. Successively, to discard the insignificant elements, we transform the irregular sparsity into regular sparsity and prune them, thus reducing the models’ parameters and getting a substantial speed-up during inference. Extensive results on ImageNet-1K validate the effectiveness of our proposed method, which obtains significant parameters and computational costs reduction with almost unimpaired generalization. For example, we compress DeiT-Tiny with 42% reduction in FLOPs and 33% reduction in parameters, almost without losing accuracy on the ImageNet dataset. Further, we verify the effectiveness of our method in the downstream task, using the pruned DeiT-Small as the backbone for the object detection task on the COCO dataset, gaining revenue without compromising its performance.

DP-FFN: Block-Based Dynamic Pooling for Accelerating Feed-Forward Layers in Transformers

Multi-Dimension Compression of Feed-Forward Network in Vision Transformers

AccTFM: an Effective Intra-Layer Model Parallelization Strategy for Training Large-Scale Transformer-Based Models.

An Efficient Two-Stage Pipelined Compute-in-Memory Macro for Accelerating Transformer Feed-Forward Networks

Accelerating Transformer Pre-training with 2:4 Sparsity

FACT: FFN-Attention Co-optimized Transformer Architecture with Eager Correlation Prediction.

FFSplit: Split Feed-Forward Network For Optimizing Accuracy-Efficiency Trade-off in Language Model Inference

Mixed-TD: Efficient Neural Network Accelerator with Layer-Specific Tensor Decomposition

MoEfication: Transformer Feed-forward Layers Are Mixtures of Experts

Smartformer: An Intelligent Model Compression Framework for Transformer

Hardware Accelerator for Multi-Head Attention and Position-Wise Feed-Forward in the Transformer

FTRANS: Energy-Efficient Acceleration of Transformers using FPGA

FPDeep: Scalable Acceleration of CNN Training on Deeply-Pipelined FPGA Clusters

An Algorithm-Hardware Co-Optimized Framework for Accelerating N:M Sparse Transformers

Exploring Efficient Partial Differential Equation Solution Using Speed Galerkin Transformer

Accelerating Framework of Transformer by Hardware Design and Model Compression Co-Optimization

A Unified Approximation Framework for Deep Neural Networks.

FP-DNN: an Automated Framework for Mapping Deep Neural Networks Onto FPGAs with RTL-HLS Hybrid Templates

Fast-FNet: Accelerating Transformer Encoder Models via Efficient Fourier Layers

Investigating the Role of Feed-Forward Networks in Transformers Using Parallel Attention and Feed-Forward Net Design