Abstract:Vision Transformer (ViT) has performed remarkably in various computer vision tasks. Nonetheless, affected by the massive amount of parameters, ViT usually suffers from serious overfitting problems with a relatively limited number of training samples. In addition, ViT generally demands heavy computing resources, which limit its deployment on resource-constrained devices. As a type of model-compression method, model binarization is potentially a good choice to solve the above problems. Compared with the full-precision one, the model with the binarization method replaces complex tensor multiplication with simple bit-wise binary operations and represents full-precision model parameters and activations with only 1-bit ones, which potentially solves the problem of model size and computational complexity, respectively. In this paper, we investigate a binarized ViT model. Empirically, we observe that the existing binarization technology designed for Convolutional Neural Networks (CNN) cannot migrate well to a ViT's binarization task. We also find that the decline of the accuracy of the binary ViT model is mainly due to the information loss of the Attention module and the Value vector. Therefore, we propose a novel model binarization technique, called Group Superposition Binarization (GSB), to deal with these issues. Furthermore, in order to further improve the performance of the binarization model, we have investigated the gradient calculation procedure in the binarization process and derived more proper gradient calculation equations for GSB to reduce the influence of gradient mismatch. Then, the knowledge distillation technique is introduced to alleviate the performance degradation caused by model binarization. Analytically, model binarization can limit the parameter's search space during parameter updates while training a model. Therefore, the binarization process can actually play an implicit regularization role and help solve the problem of overfitting in the case of insufficient training data. Experiments on three datasets with limited numbers of training samples demonstrate that the proposed GSB model achieves state-of-the-art performance among the binary quantization schemes and exceeds its full-precision counterpart on some indicators. Code and models are available at: https://github.com/IMRL/GSB-Vision-Transformer.

Progressive Stochastic Binarization of Deep Networks

Loss-aware Binarization of Deep Networks.

Embarrassingly Simple Binarization for Deep Single Imagery Super-Resolution Networks

Network Binarization via Contrastive Learning

AdaBin: Improving Binary Neural Networks with Adaptive Binary Sets

Efficient and Robust Mixed-Integer Optimization Methods for Training Binarized Deep Neural Networks

Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1

Gradient Corrected Approximation for Binary Neural Networks.

Signed Binarization: Unlocking Efficiency Through Repetition-Sparsity Trade-Off

Taming Binarized Neural Networks and Mixed-Integer Programs

Binarized Weight Error Networks With a Transition Regularization Term

Learning Discrete Weights and Activations Using the Local Reparameterization Trick

Iterative Training: Finding Binary Weight Deep Neural Networks with Layer Binarization

Adjustable Bounded Rectifiers: Towards Deep Binary Representations

A NOVEL STOCHASTIC BINARY NEURAL NETWORK

Understanding Neural Network Binarization with Forward and Backward Proximal Quantizers

Non-convex Bayesian Learning via Stochastic Gradient Markov Chain Monte Carlo

Distribution-sensitive Information Retention for Accurate Binary Neural Network

GSB: Group superposition binarization for vision transformer with limited training samples

Partially Stochastic Infinitely Deep Bayesian Neural Networks

Binarizing by Classification: is Soft Function Really Necessary?