Abstract:Vision Transformers (ViTs) have recently made a splash in computer vision domain and achieved state-of-the-art in many vision tasks. Nevertheless, due to their vast model size and high computational costs, rare transformer-based models are adopted in real-world applications. Since the computational costs of attention operation is the square of the input size, some compression methods for the Multi-Head Self-Attention (MHSA) module have been proposed, reducing its FLOPs successfully but almost without parameters reduction. Meanwhile, the number of parameters and computational costs in the Feed-Forward Network (FFN) module exceeds the MHSA larger, while its compression technologies have not been delved deeper. Consequently, we focus our insight on the compression of FFN layer and present a pruning method named Multi-Dimension Compression of Feed-Forward Network in Vision Transformers(MCF), which greatly reduces the model’s parameters and computational costs. Firstly, we identify the critical elements in the output of the FFN module and then employ them to guide the irregular sparsity of this layer, recognizing insignificant elements of FFN layer that have less impact on the output. Successively, to discard the insignificant elements, we transform the irregular sparsity into regular sparsity and prune them, thus reducing the models’ parameters and getting a substantial speed-up during inference. Extensive results on ImageNet-1K validate the effectiveness of our proposed method, which obtains significant parameters and computational costs reduction with almost unimpaired generalization. For example, we compress DeiT-Tiny with 42% reduction in FLOPs and 33% reduction in parameters, almost without losing accuracy on the ImageNet dataset. Further, we verify the effectiveness of our method in the downstream task, using the pruned DeiT-Small as the backbone for the object detection task on the COCO dataset, gaining revenue without compromising its performance.

DecomVQANet: Decomposing visual question answering deep network via tensor decomposition and regression

Simple and Effective Visual Question Answering in a Single Modality

Overcoming Language Priors In Vqa Via Decomposed Linguistic Representations

Deep Attention Neural Tensor Network For Visual Question Answering

Multi-Dimension Compression of Feed-Forward Network in Vision Transformers

Exploring Question Decomposition for Zero-Shot VQA

A lightweight Transformer-based visual question answering network with Weight-Sharing Hybrid Attention

Dual-decoder transformer network for answer grounding in visual question answering

Compressing And Debiasing Vision-Language Pre-Trained Models for Visual Question Answering

CLVIN: Complete language-vision interaction network for visual question answering

Visual Question Decomposition on Multimodal Large Language Models

Deep Convolutional Neural Network Compression Method: Tensor Ring Decomposition with Variational Bayesian Approach

Deep Residual Weight-Sharing Attention Network with Low-Rank Attention for Visual Question Answering.

Convolutional Neural Network Compression Based on Low-Rank Decomposition

Learning Sparse Mixture of Experts for Visual Question Answering

TinyVQA: Compact Multimodal Deep Neural Network for Visual Question Answering on Resource-Constrained Devices

Compositional Memory for Visual Question Answering

Advancing Vietnamese Visual Question Answering with Transformer and Convolutional Integration

Beyond Bilinear: Generalized Multimodal Factorized High-Order Pooling for Visual Question Answering

Structured Semantic Representation for Visual Question Answering.

Visual-Textual Semantic Alignment Network for Visual Question Answering