Abstract:Unsupervised domain adaptation (UDA) enables a learning machine to adapt from a labeled source domain to an unlabeled domain under the distribution shift. Thanks to the strong representation ability of deep neural networks, recent remarkable achievements in UDA resort to learning domain-invariant features. Intuitively, the hope is that a good feature representation, together with the hypothesis learned from the source domain, can generalize well to the target domain. However, the learning processes of domain-invariant features and source hypothesis inevitably involve domain-specific information that would degrade the generalizability of UDA models on the target domain. In this paper, motivated by the lottery ticket hypothesis that only partial parameters are essential for generalization, we find that only partial parameters are essential for learning domain-invariant information and generalizing well in UDA. Such parameters are termed transferable parameters. In contrast, the other parameters tend to fit domain-specific details and often fail to generalize, which we term as untransferable parameters. Driven by this insight, we propose Transferable Parameter Learning (TransPar) to reduce the side effect brought by domain-specific information in the learning process and thus enhance the memorization of domain-invariant information. Specifically, according to the distribution discrepancy degree, we divide all parameters into transferable and untransferable ones in each training iteration. We then perform separate updates rules for the two types of parameters. Extensive experiments on image classification and regression tasks (keypoint detection) show that TransPar outperforms prior arts by non-trivial margins. Moreover, experiments demonstrate that TransPar can be integrated into the most popular deep UDA networks and be easily extended to handle any data distribution shift scenarios.

Uniform Priors for Data-Efficient Transfer

Trust-aware Conditional Adversarial Domain Adaptation with Feature Norm Alignment.

Transferable Normalization: Towards Improving Transferability of Deep Neural Networks

Towards Understanding the Transferability of Deep Representations

Learning Transferable Features with Deep Adaptation Networks

Features are fate: a theory of transfer learning in high-dimensional regression

How transferable are features in deep neural networks?

Geometric Understanding of Discriminability and Transferability for Visual Domain Adaptation

Adversarially-Trained Deep Nets Transfer Better: Illustration on Image Classification

Learning Transferable Parameters for Unsupervised Domain Adaptation

Transferability in Deep Learning: A Survey

Learning explicitly transferable representations for domain adaptation

Make the Most of Your Data: Changing the Training Data Distribution to Improve In-distribution Generalization Performance

Deep Neural Network Models Trained With A Fixed Random Classifier Transfer Better Across Domains

Domain-Agnostic Priors for Semantic Segmentation Under Unsupervised Domain Adaptation and Domain Generalization

Diverse Imagenet Models Transfer Better

Generalizing to any diverse distribution: uniformity, gentle finetuning and rebalancing

FedUV: Uniformity and Variance for Heterogeneous Federated Learning

Predictive Modeling With an Adaptive Unsupervised Broad Transfer Algorithm

Similar norm more transferable: Rethinking feature norms discrepancy in adversarial domain adaptation

The Utility of Feature Reuse: Transfer Learning in Data-Starved Regimes