Abstract:Vision Transformers (ViTs) have achieved significant advancement in computer vision tasks due to their powerful modeling capacity. However, their performance notably degrades when trained with insufficient data due to lack of inherent inductive biases. Distilling knowledge and inductive biases from a Convolutional Neural Network (CNN) teacher has emerged as an effective strategy for enhancing the generalization of ViTs on limited datasets. Previous approaches to Knowledge Distillation (KD) have pursued two primary paths: some focused solely on distilling the logit distribution from CNN teacher to ViT student, neglecting the rich semantic information present in intermediate features due to the structural differences between them. Others integrated feature distillation along with logit distillation, yet this introduced alignment operations that limits the amount of knowledge transferred due to mismatched architectures and increased the computational overhead. To this end, this paper presents Hybrid Data-efficient Knowledge Distillation (HDKD) paradigm which employs a CNN teacher and a hybrid student. The choice of hybrid student serves two main aspects. First, it leverages the strengths of both convolutions and transformers while sharing the convolutional structure with the teacher model. Second, this shared structure enables the direct application of feature distillation without any information loss or additional computational overhead. Additionally, we propose an efficient light-weight convolutional block named Mobile Channel-Spatial Attention (MBCSA), which serves as the primary convolutional block in both teacher and student models. Extensive experiments on two medical public datasets showcase the superiority of HDKD over other state-of-the-art models and its computational efficiency. Source code at: <a class="link-external link-https" href="https://github.com/omarsherif200/HDKD" rel="external noopener nofollow">this https URL</a>

Efficient image classification through collaborative knowledge distillation: A novel AlexNet modification approach

DCCD: Reducing Neural Network Redundancy Via Distillation

Teacher-student collaborative knowledge distillation for image classification

Pixel Distillation: A New Knowledge Distillation Scheme for Low-Resolution Image Recognition

Fine-grained Image Classification Based on Knowledge Distillation.

Coordinate Attention Guided Dual-Teacher Adaptive Knowledge Distillation for Image Classification

Simplified Knowledge Distillation for Deep Neural Networks Bridging the Performance Gap with a Novel Teacher–Student Architecture

Lightweight High-Resolution Remote Sensing Scene Classification via Adaptive Enhanced Knowledge Distillation.

Tolerant Self-Distillation for Image Classification

ERKT-Net: Implementing Efficient and Robust Knowledge Distillation for Remote Sensing Image Classification

Knowledge Distillation with Feature Maps for Image Classification

A Unified Asymmetric Knowledge Distillation Framework for Image Classification

Relay Knowledge Distillation for Efficiently Boosting the Performance of Shallow Networks

Online Knowledge Distillation via Collaborative Learning

Interactive Knowledge Distillation for image classification

HDKD: Hybrid Data-Efficient Knowledge Distillation Network for Medical Image Classification

Semantic Knowledge Distillation for Onboard Satellite Earth Observation Image Classification

CES-KD: Curriculum-based Expert Selection for Guided Knowledge Distillation

Strengthening Attention: Knowledge Distillation Via Cross-Layer Feature Fusion for Image Classification

Improved Knowledge Distillation via Teacher Assistant

Adaptive Cross-Architecture Mutual Knowledge Distillation