Abstract:Domain generalization (DG) aims to generalize the knowledge learned from multiple source domains to unseen target domains. Existing DG techniques can be subsumed under two broad categories, i.e., domain-invariant representation learning and domain manipulation. Nevertheless, it is extremely difficult to explicitly augment or generate the unseen target data. And when source domain variety increases, developing a domain-invariant model by simply aligning more domain-specific information becomes more challenging. In this paper, we propose a simple yet effective method for domain generalization, named Knowledge Distillation based Domain-invariant Representation Learning (KDDRL), that learns domain-invariant representation while encouraging the model to maintain domain-specific features, which recently turned out to be effective for domain generalization. To this end, our method incorporates multiple auxiliary student models and one student leader model to perform a two-stage distillation. In the first-stage distillation, each domain-specific auxiliary student treats the ensemble of other auxiliary students' predictions as a target, which helps to excavate the domain-invariant representation. Also, we present an error removal module to prevent the transfer of faulty information by eliminating incorrect predictions compared to the true labels. In the second-stage distillation, the student leader model with domain-specific features combines the domain-invariant representation learned from the group of auxiliary students to make the final prediction. Extensive experiments and in-depth analysis on popular DG benchmark datasets demonstrate that our KDDRL significantly outperforms the current state-of-the-art methods.

Learning Domain-Invariant Representations from Text for Domain Generalization.

Knowledge Distillation-based Domain-invariant Representation Learning for Domain Generalization

TDG: Text-guided Domain Generalization

Learning Domain-Invariant Relationship with Instrumental Variable for Domain Generalization

A Sentence Speaks a Thousand Images: Domain Generalization through Distilling CLIP with Language Guidance

Deep Representation Learning for Domain Generalization with Information Bottleneck Principle

Domain Generalization by Learning and Removing Domain-specific Features

Crossing the Gap: Domain Generalization for Image Captioning

Domain Generalization Via Conditional Invariant Representations

DIRL: Domain-Invariant Representation Learning for Generalizable Semantic Segmentation

Choosing Wisely and Learning Deeply: Selective Cross-Modality Distillation via CLIP for Domain Generalization

Domain Generalization for Domain-Linked Classes

Clustering-based Image-Text Graph Matching for Domain Generalization

Category-Stitch Learning for Union Domain Generalization

Language-Aware Domain Generalization Network for Cross-Scene Hyperspectral Image Classification

Domain generalization based on domain-specific adversarial learning

WIDIn: Wording Image for Domain-Invariant Representation in Single-Source Domain Generalization

Towards Reliable Domain Generalization: A New Dataset and Evaluations

Domain Generalization Through the Lens of Angular Invariance

Learning Generalizable Models via Disentangling Spurious and Enhancing Potential Correlations

Rethinking Domain Adaptation and Generalization in the Era of CLIP