Abstract:Domain generalization (DG) aims to generalize the knowledge learned from multiple source domains to unseen target domains. Existing DG techniques can be subsumed under two broad categories, i.e., domain-invariant representation learning and domain manipulation. Nevertheless, it is extremely difficult to explicitly augment or generate the unseen target data. And when source domain variety increases, developing a domain-invariant model by simply aligning more domain-specific information becomes more challenging. In this paper, we propose a simple yet effective method for domain generalization, named Knowledge Distillation based Domain-invariant Representation Learning (KDDRL), that learns domain-invariant representation while encouraging the model to maintain domain-specific features, which recently turned out to be effective for domain generalization. To this end, our method incorporates multiple auxiliary student models and one student leader model to perform a two-stage distillation. In the first-stage distillation, each domain-specific auxiliary student treats the ensemble of other auxiliary students' predictions as a target, which helps to excavate the domain-invariant representation. Also, we present an error removal module to prevent the transfer of faulty information by eliminating incorrect predictions compared to the true labels. In the second-stage distillation, the student leader model with domain-specific features combines the domain-invariant representation learned from the group of auxiliary students to make the final prediction. Extensive experiments and in-depth analysis on popular DG benchmark datasets demonstrate that our KDDRL significantly outperforms the current state-of-the-art methods.

Invariant and Sufficient Supervised Representation Learning

IRLSG: Invariant Representation Learning for Single-Domain Generalization in Medical Image Segmentation

Learning Domain-Invariant Relationship with Instrumental Variable for Domain Generalization

Knowledge Distillation-based Domain-invariant Representation Learning for Domain Generalization

On Learning Invariant Representation for Domain Adaptation

The Causal Structure of Domain Invariant Supervised Representation Learning

Domain Generalization Via Conditional Invariant Representations

Learning Fair Invariant Representations under Covariate and Correlation Shifts Simultaneously

Domain Generalization In Robust Invariant Representation

Invariant and Transportable Representations for Anti-Causal Domain Shifts

DIRL: Domain-Invariant Representation Learning for Generalizable Semantic Segmentation

Learning Invariant Representations and Risks for Semi-supervised Domain Adaptation

Domain Invariant Representation Learning with Domain Density Transformations

Deep Representation Learning for Domain Generalization with Information Bottleneck Principle

When Invariant Representation Learning Meets Label Shift: Insufficiency and Theoretical Insights

Respecting Domain Relations: Hypothesis Invariance for Domain Generalization

Self-Supervised Representation Learning From Multi-Domain Data

Provable Domain Generalization via Invariant-Feature Subspace Recovery

Discriminative-Invariant Representation Learning for Unbiased Recommendation.

Learning Domain Invariant Representations for Generalizable Person Re-Identification

Invariant Shape Representation Learning For Image Classification