Abstract:Transfer learning, or domain adaptation, is concerned with machine learning problems in which training and testing data come from possibly different probability distributions. In this work, we give an information-theoretic analysis of the generalization error and excess risk of transfer learning algorithms. Our results suggest, perhaps as expected, that the Kullback-Leibler (KL) divergence $D(\mu\|\mu')$ plays an important role in the characterizations where $\mu$ and $\mu'$ denote the distribution of the training data and the testing data, respectively. Specifically, we provide generalization error and excess risk upper bounds for learning algorithms where data from both distributions are available in the training phase. Recognizing that the bounds could be sub-optimal in general, we provide improved excess risk upper bounds for a certain class of algorithms, including the empirical risk minimization (ERM) algorithm, by making stronger assumptions through the \textit{central condition}. To demonstrate the usefulness of the bounds, we further extend the analysis to the Gibbs algorithm and the noisy stochastic gradient descent method. We then generalize the mutual information bound with other divergences such as $\phi$-divergence and Wasserstein distance, which may lead to tighter bounds and can handle the case when $\mu$ is not absolutely continuous with respect to $\mu'$. Several numerical results are provided to demonstrate our theoretical findings. Lastly, to address the problem that the bounds are often not directly applicable in practice due to the absence of the distributional knowledge of the data, we develop an algorithm (called InfoBoost) that dynamically adjusts the importance weights for both source and target data based on certain information measures. The empirical results show the effectiveness of the proposed algorithm.

Transfer Learning Beyond Bounded Density Ratios

An Empirical Study of Scaling Laws for Transfer

Scaling Laws for Transfer

A General Class of Transfer Learning Regression without Implementation Cost

Limits of Model Selection under Transfer Learning

A More Unified Theory of Transfer Learning

Bayesian Transfer Learning

The Common Intuition to Transfer Learning Can Win or Lose: Case Studies for Linear Regression

Statistical mechanics of transfer learning in fully-connected networks in the proportional limit

Features are fate: a theory of transfer learning in high-dimensional regression

On the Generalization for Transfer Learning: An Information-Theoretic Analysis

Understanding the Transferability of Representations via Task-Relatedness

Towards the Fundamental Limits of Knowledge Transfer over Finite Domains

Transfer Learning with Dynamic Distribution Adaptation

An Information-Theoretic Analysis for Transfer Learning: Error Bounds and Applications

Transfer Learning for Latent Variable Network Models

Transfer learning with high dimensional composite quantile regression

The Limits of Transfer Reinforcement Learning with Latent Low-rank Structure

Transfer Learning for High-dimensional Quantile Regression with Distribution Shift

Estimation and inference for transfer learning with high-dimensional quantile regression