Abstract:Domain generalization (DG) aims to generalize the knowledge learned from multiple source domains to unseen target domains. Existing DG techniques can be subsumed under two broad categories, i.e., domain-invariant representation learning and domain manipulation. Nevertheless, it is extremely difficult to explicitly augment or generate the unseen target data. And when source domain variety increases, developing a domain-invariant model by simply aligning more domain-specific information becomes more challenging. In this paper, we propose a simple yet effective method for domain generalization, named Knowledge Distillation based Domain-invariant Representation Learning (KDDRL), that learns domain-invariant representation while encouraging the model to maintain domain-specific features, which recently turned out to be effective for domain generalization. To this end, our method incorporates multiple auxiliary student models and one student leader model to perform a two-stage distillation. In the first-stage distillation, each domain-specific auxiliary student treats the ensemble of other auxiliary students' predictions as a target, which helps to excavate the domain-invariant representation. Also, we present an error removal module to prevent the transfer of faulty information by eliminating incorrect predictions compared to the true labels. In the second-stage distillation, the student leader model with domain-specific features combines the domain-invariant representation learned from the group of auxiliary students to make the final prediction. Extensive experiments and in-depth analysis on popular DG benchmark datasets demonstrate that our KDDRL significantly outperforms the current state-of-the-art methods.

Robust Transfer Learning of High-Dimensional Generalized Linear Model

Transfer Learning Under High-Dimensional Generalized Linear Models

Estimation and Inference for High-Dimensional Generalized Linear Models with Knowledge Transfer

Transfer Learning for High Dimensional Robust Regression

Knowledge Distillation-based Domain-invariant Representation Learning for Domain Generalization

Robust transfer learning for high-dimensional quantile regression model with linear constraints

Weighted likelihood transfer learning for high-dimensional generalized linear models

Heterogeneous Transfer Learning for Building High-Dimensional Generalized Linear Models with Disparate Datasets

Estimation and inference for transfer learning with high-dimensional quantile regression

TransFusion: Covariate-Shift Robust Transfer Learning for High-Dimensional Regression

Transfer learning with high dimensional composite quantile regression

Shape-restricted transfer learning analysis for generalized linear regression model

Transfer Learning for High-Dimensional Quantile Regression Via Convolution Smoothing

Transfer Learning for High-dimensional Quantile Regression via Convolution Smoothing

Learning Transferrable and Interpretable Representations for Domain Generalization

AdaTrans: Feature-wise and Sample-wise Adaptive Transfer Learning for High-dimensional Regression

Features are fate: a theory of transfer learning in high-dimensional regression

The Common Intuition to Transfer Learning Can Win or Lose: Case Studies for Linear Regression

Robust multiple subspaces transfer for heterogeneous domain adaptation

Transfer Learning for High-dimensional Quantile Regression with Distribution Shift

Domain Generalization for Domain-Linked Classes