Can We Evaluate Domain Adaptation Models Without Target-Domain Labels?

Jianfei Yang,Hanjie Qian,Yuecong Xu,Kai Wang,Lihua Xie

2024-02-18

Abstract:Unsupervised domain adaptation (UDA) involves adapting a model trained on a label-rich source domain to an unlabeled target domain. However, in real-world scenarios, the absence of target-domain labels makes it challenging to evaluate the performance of UDA models. Furthermore, prevailing UDA methods relying on adversarial training and self-training could lead to model degeneration and negative transfer, further exacerbating the evaluation problem. In this paper, we propose a novel metric called the \textit{Transfer Score} to address these issues. The proposed metric enables the unsupervised evaluation of UDA models by assessing the spatial uniformity of the classifier via model parameters, as well as the transferability and discriminability of deep representations. Based on the metric, we achieve three novel objectives without target-domain labels: (1) selecting the best UDA method from a range of available options, (2) optimizing hyperparameters of UDA models to prevent model degeneration, and (3) identifying which checkpoint of UDA model performs optimally. Our work bridges the gap between data-level UDA research and practical UDA scenarios, enabling a realistic assessment of UDA model performance. We validate the effectiveness of our metric through extensive empirical studies on UDA datasets of different scales and imbalanced distributions. The results demonstrate that our metric robustly achieves the aforementioned goals.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

### Problems the Paper Aims to Solve

This paper aims to address the issue of how to effectively evaluate Unsupervised Domain Adaptation (UDA) models in the absence of labels in the target domain. Specifically, the paper proposes a new metric called "Transfer Score" to tackle the following three key challenges:

1. **Selecting the Best UDA Method**: Choosing the best-performing UDA method from multiple available options.
2. **Optimizing Hyperparameters**: Adjusting the hyperparameters of the UDA model to prevent negative transfer and enhance performance.
3. **Determining the Optimal Checkpoint**: Identifying the best training epoch of the UDA model to avoid overfitting.

By achieving these goals, this work bridges the gap between data-level UDA research and practical applications, making it possible to evaluate the performance of UDA models in real-world scenarios. Additionally, the paper demonstrates the effectiveness of the proposed metric through extensive experiments.