Abstract:Vision Transformers (ViTs) have achieved significant advancement in computer vision tasks due to their powerful modeling capacity. However, their performance notably degrades when trained with insufficient data due to lack of inherent inductive biases. Distilling knowledge and inductive biases from a Convolutional Neural Network (CNN) teacher has emerged as an effective strategy for enhancing the generalization of ViTs on limited datasets. Previous approaches to Knowledge Distillation (KD) have pursued two primary paths: some focused solely on distilling the logit distribution from CNN teacher to ViT student, neglecting the rich semantic information present in intermediate features due to the structural differences between them. Others integrated feature distillation along with logit distillation, yet this introduced alignment operations that limits the amount of knowledge transferred due to mismatched architectures and increased the computational overhead. To this end, this paper presents Hybrid Data-efficient Knowledge Distillation (HDKD) paradigm which employs a CNN teacher and a hybrid student. The choice of hybrid student serves two main aspects. First, it leverages the strengths of both convolutions and transformers while sharing the convolutional structure with the teacher model. Second, this shared structure enables the direct application of feature distillation without any information loss or additional computational overhead. Additionally, we propose an efficient light-weight convolutional block named Mobile Channel-Spatial Attention (MBCSA), which serves as the primary convolutional block in both teacher and student models. Extensive experiments on two medical public datasets showcase the superiority of HDKD over other state-of-the-art models and its computational efficiency. Source code at: <a class="link-external link-https" href="https://github.com/omarsherif200/HDKD" rel="external noopener nofollow">this https URL</a>

Cross-domain visual prompting with spatial proximity knowledge distillation for histological image classification

Knowledge Distillation in Histology Landscape by Multi-Layer Features Supervision

Pixel Distillation: A New Knowledge Distillation Scheme for Low-Resolution Image Recognition

Dataset Distillation for Histopathology Image Classification

Efficient knowledge distillation for liver CT segmentation using growing assistant network

VDKD: A ViT-Based Student-Teacher Knowledge Distillation for Multi-Texture Class Anomaly Detection

Attention to detail: inter-resolution knowledge distillation

SSD-KD: A Self-supervised Diverse Knowledge Distillation Method for Lightweight Skin Lesion Classification Using Dermoscopic Images

Image Distillation for Safe Data Sharing in Histopathology

Pixel Distillation: Cost-flexible Distillation Across Image Sizes and Heterogeneous Networks

CrossKD: Cross-Head Knowledge Distillation for Object Detection

Histokt: Cross Knowledge Transfer in Computational Pathology

Efficient knowledge distillation for hybrid models: A vision transformer‐convolutional neural network to convolutional neural network approach for classifying remote sensing images

Dynamic Knowledge Distillation with Cross-Modality Knowledge Transfer

HDKD: Hybrid Data-Efficient Knowledge Distillation Network for Medical Image Classification

DSP-KD: Dual-Stage Progressive Knowledge Distillation for Skin Disease Classification

Bridging Cross-task Protocol Inconsistency for Distillation in Dense Object Detection

Efficient Biomedical Instance Segmentation via Knowledge Distillation

Enhancing Weakly-Supervised Histopathology Image Segmentation with Knowledge Distillation on MIL-Based Pseudo-Labels

MSKD: Structured knowledge distillation for efficient medical image segmentation

Low-Resolution Object Recognition with Cross-Resolution Relational Contrastive Distillation