Abstract:Transfer learning, or domain adaptation, is concerned with machine learning problems in which training and testing data come from possibly different probability distributions. In this work, we give an information-theoretic analysis of the generalization error and excess risk of transfer learning algorithms. Our results suggest, perhaps as expected, that the Kullback-Leibler (KL) divergence $D(\mu\|\mu')$ plays an important role in the characterizations where $\mu$ and $\mu'$ denote the distribution of the training data and the testing data, respectively. Specifically, we provide generalization error and excess risk upper bounds for learning algorithms where data from both distributions are available in the training phase. Recognizing that the bounds could be sub-optimal in general, we provide improved excess risk upper bounds for a certain class of algorithms, including the empirical risk minimization (ERM) algorithm, by making stronger assumptions through the \textit{central condition}. To demonstrate the usefulness of the bounds, we further extend the analysis to the Gibbs algorithm and the noisy stochastic gradient descent method. We then generalize the mutual information bound with other divergences such as $\phi$-divergence and Wasserstein distance, which may lead to tighter bounds and can handle the case when $\mu$ is not absolutely continuous with respect to $\mu'$. Several numerical results are provided to demonstrate our theoretical findings. Lastly, to address the problem that the bounds are often not directly applicable in practice due to the absence of the distributional knowledge of the data, we develop an algorithm (called InfoBoost) that dynamically adjusts the importance weights for both source and target data based on certain information measures. The empirical results show the effectiveness of the proposed algorithm.

Towards the Fundamental Limits of Knowledge Transfer over Finite Domains

Knowledge Distillation Based on Transformed Teacher Matching

Knowledge Distillation in Wide Neural Networks: Risk Bound, Data Efficiency and Imperfect Teacher

Analysis of Knowledge Transfer in Kernel Regime

An Information-Theoretic Analysis for Transfer Learning: Error Bounds and Applications

Boosting Knowledge Distillation Via Intra-class Logit Distribution Smoothing

Exploring Dark Knowledge under Various Teacher Capacities and Addressing Capacity Mismatch

Logit Standardization in Knowledge Distillation

Respecting Transfer Gap in Knowledge Distillation

Zero-shot Knowledge Transfer via Adversarial Belief Matching

Knowledge Distillation Under Ideal Joint Classifier Assumption

Knowledge Distillation as Semiparametric Inference

Towards Understanding Knowledge Distillation

On the Generalization for Transfer Learning: An Information-Theoretic Analysis

Distilling Knowledge via Intermediate Classifiers

Asymmetric Temperature Scaling Makes Larger Networks Teach Well Again

Knowledge distillation with insufficient training data for regression

On the Efficacy of Knowledge Distillation

Faculty Distillation with Optimal Transport

Knowledge Distillation in Generations: More Tolerant Teachers Educate Better Students