Abstract:Knowledge distillation is commonly employed to compress neural networks, reducing the inference costs and memory footprint. In the scenario of homogenous architecture, feature-based methods have been widely validated for their effectiveness. However, in scenarios where the teacher and student models are of heterogeneous architectures, the inherent differences in feature representation significantly degrade the performance of these methods. Recent studies have highlighted that low-frequency components constitute the majority of image features. Motivated by this, we propose a Low-Frequency Components-based Contrastive Knowledge Distillation (LFCC) framework that significantly enhances the performance of feature-based distillation between heterogeneous architectures. Specifically, we designe a set of multi-scale low-pass filters to extract the low-frequency components of intermediate features from both the teacher and student models, aligning them in a compact space to overcome architectural disparities. Moreover, leveraging the intrinsic pairing characteristic of the teacher-student framework, we design an innovative sample-level contrastive learning framework that adeptly restructures the constraints of within-sample feature similarity and between-sample feature divergence into a contrastive learning task. This strategy enables the student model to capitalize on intra-sample feature congruence while simultaneously enhancing the discrimination of features among disparate samples. Consequently, our LFCC framework accurately captures the commonalities in feature representation across heterogeneous architectures. Extensive evaluations and empirical analyses across three architectures (CNNs, Transformers, and MLPs) demonstrate that LFCC achieves superior performance on the challenging benchmarks of ImageNet-1K and CIFAR-100. All codes will be publicly available.

Wasserstein Contrastive Representation Distillation: Supplementary Material

Wasserstein Contrastive Representation Distillation

DCCD: Reducing Neural Network Redundancy Via Distillation

Collaborative Distillation for Ultra-Resolution Universal Style Transfer

Contrastive Representation Distillation

Teacher-Student Complementary Sample Contrastive Distillation

DCD: Discriminative and Consistent Representation Distillation

Complementary Relation Contrastive Distillation

CKD: Contrastive Knowledge Distillation from A Sample-wise Perspective

DistilCSE: Effective Knowledge Distillation For Contrastive Sentence Embeddings

Feature Distillation With Guided Adversarial Contrastive Learning

Relational Representation Distillation

Contrastive Distillation on Intermediate Representations for Language Model Compression

Adversarial Contrastive Distillation with Adaptive Denoising

On Representation Knowledge Distillation for Graph Neural Networks

Aligning in a Compact Space: Contrastive Knowledge Distillation between Heterogeneous Architectures

Hybrid mix-up contrastive knowledge distillation

TC<SUP>3</SUP>KD: Knowledge distillation via teacher-student cooperative curriculum customization

In Defense of Feature Mimicking for Knowledge Distillation.

DisWOT: Student Architecture Search for Distillation WithOut Training

Supplementary Material forCo-Transport for Class-Incremental Learning