Knowledge Distillation in Vision Transformers: A Critical Review

Gousia Habib,Tausifa Jan Saleem,Brejesh Lall

2024-02-10

Abstract:In Natural Language Processing (NLP), Transformers have already revolutionized the field by utilizing an attention-based encoder-decoder model. Recently, some pioneering works have employed Transformer-like architectures in Computer Vision (CV) and they have reported outstanding performance of these architectures in tasks such as image classification, object detection, and semantic segmentation. Vision Transformers (ViTs) have demonstrated impressive performance improvements over Convolutional Neural Networks (CNNs) due to their competitive modelling capabilities. However, these architectures demand massive computational resources which makes these models difficult to be deployed in the resource-constrained applications. Many solutions have been developed to combat this issue, such as compressive transformers and compression functions such as dilated convolution, min-max pooling, 1D convolution, etc. Model compression has recently attracted considerable research attention as a potential remedy. A number of model compression methods have been proposed in the literature such as weight quantization, weight multiplexing, pruning and Knowledge Distillation (KD). However, techniques like weight quantization, pruning and weight multiplexing typically involve complex pipelines for performing the compression. KD has been found to be a simple and much effective model compression technique that allows a relatively simple model to perform tasks almost as accurately as a complex model. This paper discusses various approaches based upon KD for effective compression of ViT models. The paper elucidates the role played by KD in reducing the computational and memory requirements of these models. The paper also presents the various challenges faced by ViTs that are yet to be resolved.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper primarily aims to address several key issues faced by Vision Transformers (ViTs) in computer vision tasks:

1. **High resource consumption**: Although ViTs perform excellently in classification, detection, and segmentation tasks, their computational and memory demands are high, making it difficult to deploy them on resource-constrained devices.

2. **Weak inductive bias**: Compared to Convolutional Neural Networks (CNNs), ViTs lack inductive biases such as translation invariance, resulting in poor generalization when data is insufficient.

3. **Quadratic complexity of the attention mechanism**: The attention mechanism in ViTs has quadratic complexity, leading to a large number of model parameters and high computational costs, making it challenging to apply on resource-constrained devices.

To address these issues, the paper focuses on the application of Knowledge Distillation (KD) techniques in compressing ViTs, aiming to improve model efficiency without sacrificing performance by distilling the knowledge of large teacher models into smaller student models. Specifically, the paper explores different types of distillation methods, including response-based, feature-based, and relation-based knowledge distillation, and discusses the advantages and challenges of these methods in compressing ViTs. Additionally, the paper analyzes various loss functions and their roles in knowledge distillation to further optimize the compression effect.