Abstract:Domain generalization (DG) aims to generalize the knowledge learned from multiple source domains to unseen target domains. Existing DG techniques can be subsumed under two broad categories, i.e., domain-invariant representation learning and domain manipulation. Nevertheless, it is extremely difficult to explicitly augment or generate the unseen target data. And when source domain variety increases, developing a domain-invariant model by simply aligning more domain-specific information becomes more challenging. In this paper, we propose a simple yet effective method for domain generalization, named Knowledge Distillation based Domain-invariant Representation Learning (KDDRL), that learns domain-invariant representation while encouraging the model to maintain domain-specific features, which recently turned out to be effective for domain generalization. To this end, our method incorporates multiple auxiliary student models and one student leader model to perform a two-stage distillation. In the first-stage distillation, each domain-specific auxiliary student treats the ensemble of other auxiliary students' predictions as a target, which helps to excavate the domain-invariant representation. Also, we present an error removal module to prevent the transfer of faulty information by eliminating incorrect predictions compared to the true labels. In the second-stage distillation, the student leader model with domain-specific features combines the domain-invariant representation learned from the group of auxiliary students to make the final prediction. Extensive experiments and in-depth analysis on popular DG benchmark datasets demonstrate that our KDDRL significantly outperforms the current state-of-the-art methods.

LFME: A Simple Framework for Learning from Multiple Experts in Domain Generalization

LFME: A Simple Framework for Learning from Multiple Experts in Domain Generalization

FedDGP: Disentangling Global and Personal Models for Federated Learning

Knowledge Distillation-based Domain-invariant Representation Learning for Domain Generalization

MCKD: Mutually Collaborative Knowledge Distillation for Federated Domain Adaptation and Generalization

Sparse Fusion Mixture-of-Experts are Domain Generalizable Learners

Learning from Multiple Experts: Self-paced Knowledge Distillation for Long-tailed Classification

Mixture-of-Experts Learner for Single Long-Tailed Domain Generalization

Multiple Domain-Adversarial Ensemble Learning for Domain Generalization

Domain Generalization Guided by Large-Scale Pre-Trained Priors

Rethinking Multi-domain Generalization with A General Learning Objective

More is Better: A Novel Multi-view Framework for Domain Generalization

MVDG: A Unified Multi-view Framework for Domain Generalization

A Unified Personalized Federated Learning Framework Ensuring Domain Generalization

MLDGG: Meta-Learning for Domain Generalization on Graphs

Federated Learning with Domain Generalization

Learning to Generalize: Meta-Learning for Domain Generalization

Contrastive-enhanced Domain Generalization with Federated Learning

Personalized Federated Learning with Domain Generalization

Label-Efficient Domain Generalization Via Collaborative Exploration and Generalization.

Seeking Consistent Flat Minima for Better Domain Generalization via Refining Loss Landscapes