Abstract:Domain generalization (DG) aims to generalize the knowledge learned from multiple source domains to unseen target domains. Existing DG techniques can be subsumed under two broad categories, i.e., domain-invariant representation learning and domain manipulation. Nevertheless, it is extremely difficult to explicitly augment or generate the unseen target data. And when source domain variety increases, developing a domain-invariant model by simply aligning more domain-specific information becomes more challenging. In this paper, we propose a simple yet effective method for domain generalization, named Knowledge Distillation based Domain-invariant Representation Learning (KDDRL), that learns domain-invariant representation while encouraging the model to maintain domain-specific features, which recently turned out to be effective for domain generalization. To this end, our method incorporates multiple auxiliary student models and one student leader model to perform a two-stage distillation. In the first-stage distillation, each domain-specific auxiliary student treats the ensemble of other auxiliary students' predictions as a target, which helps to excavate the domain-invariant representation. Also, we present an error removal module to prevent the transfer of faulty information by eliminating incorrect predictions compared to the true labels. In the second-stage distillation, the student leader model with domain-specific features combines the domain-invariant representation learned from the group of auxiliary students to make the final prediction. Extensive experiments and in-depth analysis on popular DG benchmark datasets demonstrate that our KDDRL significantly outperforms the current state-of-the-art methods.

Gradient Estimation for Unseen Domain Risk Minimization with Pre-Trained Models

Knowledge Distillation-based Domain-invariant Representation Learning for Domain Generalization

Domain-Specific Risk Minimization for Domain Generalization

Domain-Specific Risk Minimization for Out-of-Distribution Generalization

Model-Based Domain Generalization

GradCa: Generalizing to Unseen Domains Via Gradient Calibration

Domain Generalization Guided by Gradient Signal to Noise Ratio of Parameters

PGrad: Learning Principal Gradients For Domain Generalization

Domain Generalization via Gradient Surgery

Privacy-Preserving Constrained Domain Generalization via Gradient Alignment

DomainDrop: Suppressing Domain-Sensitive Channels for Domain Generalization

Domain Generalization Guided by Large-Scale Pre-Trained Priors

Towards Unsupervised Domain Generalization

Domain Generalization using Pretrained Models without Fine-tuning

Domain Agnostic Conditional Invariant Predictions for Domain Generalization

Domain Generalization via Progressive Layer-wise and Channel-wise Dropout

Domain Adaptive Transfer Learning with Specialist Models

Rethinking the Evaluation Protocol of Domain Generalization.

Learning Sample Difficulty from Pre-trained Models for Reliable Prediction

Learning to Optimize Domain Specific Normalization for Domain Generalization

Shape Guided Gradient Voting for Domain Generalization