Abstract:Unsupervised domain adaptation (UDA) aims to alleviate domain shifts by transferring relevant domain information from a fully labeled source domain to an unknown target domain. Although significant progress has been made recently, the use of UDA in real-world applications is still limited owing to low-resource computers and privacy issues. To further extend the flexibility of the target model, this study attempts to address a new setting of UDA, where only a source model is provided, and target models with various network architectures can be learned according to the deployment environment. Therefore, we propose a novel two-stage Cross-domain Knowledge Distillation method via Teacher-Student Mutual Learning, termed CdKD-TSML, which enables the peer networks to employ pseudo labels assigned by one another as supplemental supervision. In the first stage, to depress the inevitable noise in hard pseudo labels generated by the self-supervised clustering procedure, we further propose to softly refine the pseudo labels with mutual learning, where networks and label predictions are online optimized cooperatively by distilling knowledge from each other. In the second stage, we jointly fine-tune the distilled models via a cooperative consistency learning strategy, which selects pseudo-labeled samples from the two peers to update the networks, thereby enhancing the generalization of the models. We theoretically demonstrate that the target error of CdKD-TSML is minimized by simultaneously decreasing the pseudo-label noise and alleviating the sample selection bias. Experiments on three challenging UDA datasets demonstrate that CdKD-TSML yields superior results compared to other state-of-the-art methods, proving its effectiveness in this novel setting.

Teacher-Student Consistency For Multi-Source Domain Adaptation

Transformer-Based Multi-Source Domain Adaptation Without Source Data.

Target-Discriminability-Induced Multi-Source-Free Domain Adaptation

Multi-Source Domain Adaptation for Object Detection with Prototype-based Mean-teacher

Multi-Source domain adaptation via supervised contrastive learning and confident consistency regularization

Teacher-Student Mutual Learning for efficient source-free unsupervised domain adaptation

Multi-Target Domain Adaptation with Collaborative Consistency Learning

Focus on Your Target: A Dual Teacher-Student Framework for Domain-adaptive Semantic Segmentation

Towards Better Stability and Adaptability: Improve Online Self-Training for Model Adaptation in Semantic Segmentation

Co-Teaching for Unsupervised Domain Adaptation and Expansion

Teacher-Student Competition for Unsupervised Domain Adaptation

Multi-Source Domain Adaptation with Mixture of Joint Distributions

Teacher-Student Asynchronous Learning with Multi-Source Consistency for Facial Landmark Detection

Collaborative Multi-source Domain Adaptation Through Optimal Transport

Mdalu: Multi-Source Domain Adaptation and Label Unification with Partial Datasets

Multi-Level Consistency Learning for Source-Free Model Adaptation

Multi-Source Domain Adaptation with Collaborative Learning for Semantic Segmentation

Balanced Teacher for Source-free Object Detection

Enhancing Multi-Source Open-Set Domain Adaptation through Nearest Neighbor Classification with Self-Supervised Vision Transformer

Adaptive Mutual Learning for Unsupervised Domain Adaptation

Multi-source Domain Adaptation for Visual Sentiment Classification