Abstract:Vision Transformers (ViTs) have recently made a splash in computer vision domain and achieved state-of-the-art in many vision tasks. Nevertheless, due to their vast model size and high computational costs, rare transformer-based models are adopted in real-world applications. Since the computational costs of attention operation is the square of the input size, some compression methods for the Multi-Head Self-Attention (MHSA) module have been proposed, reducing its FLOPs successfully but almost without parameters reduction. Meanwhile, the number of parameters and computational costs in the Feed-Forward Network (FFN) module exceeds the MHSA larger, while its compression technologies have not been delved deeper. Consequently, we focus our insight on the compression of FFN layer and present a pruning method named Multi-Dimension Compression of Feed-Forward Network in Vision Transformers(MCF), which greatly reduces the model’s parameters and computational costs. Firstly, we identify the critical elements in the output of the FFN module and then employ them to guide the irregular sparsity of this layer, recognizing insignificant elements of FFN layer that have less impact on the output. Successively, to discard the insignificant elements, we transform the irregular sparsity into regular sparsity and prune them, thus reducing the models’ parameters and getting a substantial speed-up during inference. Extensive results on ImageNet-1K validate the effectiveness of our proposed method, which obtains significant parameters and computational costs reduction with almost unimpaired generalization. For example, we compress DeiT-Tiny with 42% reduction in FLOPs and 33% reduction in parameters, almost without losing accuracy on the ImageNet dataset. Further, we verify the effectiveness of our method in the downstream task, using the pruned DeiT-Small as the backbone for the object detection task on the COCO dataset, gaining revenue without compromising its performance.

Multi-Stream Single Network: Efficient Compressed Video Action Recognition With a Single Multi-Input Multi-Output Network

Motion Guided Token Compression for Efficient Masked Video Modeling

Multi-Dimension Compression of Feed-Forward Network in Vision Transformers

Joint Feature Optimization and Fusion for Compressed Action Recognition

Compressed Video Action Recognition with Dual-Stream and Dual-Modal Transformer

MTRFN: Multiscale Temporal Receptive Field Network for Compressed Video Action Recognition at Edge Servers

F2D-SIFPNet: a Frequency 2D Slow-I-Fast-P Network for Faster Compressed Video Action Recognition

Human Action Recognition Based on Three-Stream Network with Frame Sequence Features

Multi-Fiber Networks for Video Recognition

Multi-head attention-based two-stream EfficientNet for action recognition

Frequency Enhancement Network for Efficient Compressed Video Action Recognition

Multi-scale Spatiotemporal Information Fusion Network for Video Action Recognition

Physical Knowledge Driven Multi-scale Temporal Receptive Field Network for Compressed Video Action Recognition

M^3Net: Multi-view Encoding, Matching, and Fusion for Few-shot Fine-grained Action Recognition

Multi-modality Fusion Network for Action Recognition.

M$^3$Net: Multi-view Encoding, Matching, and Fusion for Few-shot Fine-grained Action Recognition

MSF-Net: A Multilevel Spatiotemporal Feature Fusion Network Combines Attention for Action Recognition

Efficient dual attention SlowFast networks for video action recognition

Mixed Resolution Network with Hierarchical Motion Modeling for Efficient Action Recognition

Two-Stream 3-D Convnet Fusion for Action Recognition in Videos with Arbitrary Size and Length

Multi-Stream Multi-Class Fusion of Deep Networks for Video Classification.