Abstract:Knowledge distillation has emerged as a highly effective method for bridging the representation discrepancy between large-scale models and lightweight models. Prevalent approaches involve leveraging appropriate metrics to minimize the divergence or distance between the knowledge extracted from the teacher model and the knowledge learned by the student model. Centered Kernel Alignment (CKA) is widely used to measure representation similarity and has been applied in several knowledge distillation methods. However, these methods are complex and fail to uncover the essence of CKA, thus not answering the question of how to use CKA to achieve simple and effective distillation properly. This paper first provides a theoretical perspective to illustrate the effectiveness of CKA, which decouples CKA to the upper bound of Maximum Mean Discrepancy~(MMD) and a constant term. Drawing from this, we propose a novel Relation-Centered Kernel Alignment~(RCKA) framework, which practically establishes a connection between CKA and MMD. Furthermore, we dynamically customize the application of CKA based on the characteristics of each task, with less computational source yet comparable performance than the previous methods. The extensive experiments on the CIFAR-100, ImageNet-1k, and MS-COCO demonstrate that our method achieves state-of-the-art performance on almost all teacher-student pairs for image classification and object detection, validating the effectiveness of our approaches. Our code is available in

An Overview of Kernel Alignment and Its Applications

Generalized Convexity-Based Inexact Projection Method for Multiple Kernel Learning

A Kernel Function Optimization And Selection Algorithm Based On Cost Function Maximization

Cross-validatory Framework for Optimal Parameter Estimation of KPCA and KPLS Models

Kernel Alignment for Unsupervised Feature Selection via Matrix Factorization

A Theory of Neural Tangent Kernel Alignment and Its Influence on Training

Kernel optimization-based discriminant analysis for face recognition

Adaptive Sequence Learning and Applications for Multi-Scale Kernel Method

A General Kernelization Framework for Learning Algorithms Based on Kernel PCA.

Nonlinear Process Monitoring Using Improved Kernel Principal Component Analysis

Kernel alignment unsupervised discriminative dimensionality reduction

Learning a Data-Dependent Kernel Function for KPCA-based Nonlinear Process Monitoring

Adaptive Kernel Principal Component Analysis with Unsupervised Learning of Kernels

Feature Selection in a Kernel Space

Learning the kernel parameters in kernel minimum distance classifier

Generalized Semisupervised Self-Optimizing Kernel Model for Quality-Related Industrial Process Monitoring.

Rapid and brief communication: Learning the kernel parameters in kernel minimum distance classifier

Designing Ensemble Learning Algorithms Using Kernel Methods

Rethinking Centered Kernel Alignment in Knowledge Distillation

TAKES: a fast method to select features in the kernel space.

On the Essence and Prospect: An Investigation of Alignment Approaches for Big Models