Abstract:Knowledge distillation is commonly employed to compress neural networks, reducing the inference costs and memory footprint. In the scenario of homogenous architecture, feature-based methods have been widely validated for their effectiveness. However, in scenarios where the teacher and student models are of heterogeneous architectures, the inherent differences in feature representation significantly degrade the performance of these methods. Recent studies have highlighted that low-frequency components constitute the majority of image features. Motivated by this, we propose a Low-Frequency Components-based Contrastive Knowledge Distillation (LFCC) framework that significantly enhances the performance of feature-based distillation between heterogeneous architectures. Specifically, we designe a set of multi-scale low-pass filters to extract the low-frequency components of intermediate features from both the teacher and student models, aligning them in a compact space to overcome architectural disparities. Moreover, leveraging the intrinsic pairing characteristic of the teacher-student framework, we design an innovative sample-level contrastive learning framework that adeptly restructures the constraints of within-sample feature similarity and between-sample feature divergence into a contrastive learning task. This strategy enables the student model to capitalize on intra-sample feature congruence while simultaneously enhancing the discrimination of features among disparate samples. Consequently, our LFCC framework accurately captures the commonalities in feature representation across heterogeneous architectures. Extensive evaluations and empirical analyses across three architectures (CNNs, Transformers, and MLPs) demonstrate that LFCC achieves superior performance on the challenging benchmarks of ImageNet-1K and CIFAR-100. All codes will be publicly available.

Model Sparsification for Communication-Efficient Multi-Party Learning Via Contrastive Distillation in Image Classification

Online Knowledge Distillation via Multi-branch Diversity Enhancement

Multi-Party Dual Learning

Device-Performance-Driven Heterogeneous Multiparty Learning for Arbitrary Images

MP-FedCL: Multi-Prototype Federated Contrastive Learning for Edge Intelligence

MP-FedCL: Multiprototype Federated Contrastive Learning for Edge Intelligence

ADDS: Adaptive Differentiable Sampling for Robust Multi-Party Learning

Aligning in a Compact Space: Contrastive Knowledge Distillation between Heterogeneous Architectures

FedDiff: Diffusion Model Driven Federated Learning for Multi-Modal and Multi-Clients

Heterogeneous Multi-party Learning with Data-driven Network Sampling

A Generalization Theory of Cross-Modality Distillation with Contrastive Learning

Multi-teacher Contrastive Knowledge Inversion for Data-Free Distillation

CDFKD-MFS: Collaborative Data-free Knowledge Distillation Via Multi-level Feature Sharing

Hybrid Distillation: Connecting Masked Autoencoders with Contrastive Learners

Towards Optimal Customized Architecture for Heterogeneous Federated Learning with Contrastive Cloud-Edge Model Decoupling

FedMD: Heterogenous Federated Learning via Model Distillation

Online Knowledge Distillation Via Mutual Contrastive Learning for Visual Recognition

Is Contrastive Distillation Enough for Learning Comprehensive 3D Representations?

Federated Learning Scheme Based on Secure Multi-party Computation and Differential Privacy

Fine-grained Image-to-LiDAR Contrastive Distillation with Visual Foundation Models

Federated Learning Model Aggregation in Heterogenous Aerial and Space Networks