Abstract:Vision Transformer (ViT) has performed remarkably in various computer vision tasks. Nonetheless, affected by the massive amount of parameters, ViT usually suffers from serious overfitting problems with a relatively limited number of training samples. In addition, ViT generally demands heavy computing resources, which limit its deployment on resource-constrained devices. As a type of model-compression method, model binarization is potentially a good choice to solve the above problems. Compared with the full-precision one, the model with the binarization method replaces complex tensor multiplication with simple bit-wise binary operations and represents full-precision model parameters and activations with only 1-bit ones, which potentially solves the problem of model size and computational complexity, respectively. In this paper, we investigate a binarized ViT model. Empirically, we observe that the existing binarization technology designed for Convolutional Neural Networks (CNN) cannot migrate well to a ViT's binarization task. We also find that the decline of the accuracy of the binary ViT model is mainly due to the information loss of the Attention module and the Value vector. Therefore, we propose a novel model binarization technique, called Group Superposition Binarization (GSB), to deal with these issues. Furthermore, in order to further improve the performance of the binarization model, we have investigated the gradient calculation procedure in the binarization process and derived more proper gradient calculation equations for GSB to reduce the influence of gradient mismatch. Then, the knowledge distillation technique is introduced to alleviate the performance degradation caused by model binarization. Analytically, model binarization can limit the parameter's search space during parameter updates while training a model. Therefore, the binarization process can actually play an implicit regularization role and help solve the problem of overfitting in the case of insufficient training data. Experiments on three datasets with limited numbers of training samples demonstrate that the proposed GSB model achieves state-of-the-art performance among the binary quantization schemes and exceeds its full-precision counterpart on some indicators. Code and models are available at: https://github.com/IMRL/GSB-Vision-Transformer.

BFD: Binarized Frequency-enhanced Distillation for Vision Transformer

ViTKD: Feature-based Knowledge Distillation for Vision Transformers

Bi-ViT: Pushing the Limit of Vision Transformer Quantization

ViTKD: Practical Guidelines for ViT Feature Knowledge Distillation

Quantized Feature Distillation for Network Quantization

Learning Efficient Vision Transformers via Fine-Grained Manifold Distillation

Patient teacher can impart locality to improve lightweight vision transformer on small dataset

Cumulative Spatial Knowledge Distillation for Vision Transformers

DDViT: Double-Level Fusion Domain Adapter Vision Transformer (Student Abstract)

GSB: Group superposition binarization for vision transformer with limited training samples

Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformer

DFD: Distillng the Feature Disparity Differently for Detectors

DeiT-LT Distillation Strikes Back for Vision Transformer Training on Long-Tailed Datasets

Task-Balanced Distillation for Object Detection

FViT: A Focal Vision Transformer with Gabor Filter

BiViT: Extremely Compressed Binary Vision Transformers

DeViT: Decomposing Vision Transformers for Collaborative Inference in Edge Devices

AMD: Automatic Multi-step Distillation of Large-scale Vision Models

CF-ViT: A General Coarse-to-Fine Method for Vision Transformer

ViR:the Vision Reservoir

DearKD: Data-Efficient Early Knowledge Distillation for Vision Transformers