Abstract:Domain adaptation aims to reduce the model degradation on the target domain caused by the domain shift between the source and target domains. Although encouraging performance has been achieved by combining cognitive learning with the self-training paradigm, they suffer from ambiguous scenarios caused by scale, illumination, or overlapping when deploying deterministic embedding. To address these issues, we propose probabilistic proto-typical pixel contrast (PPPC), a universal adaptation framework that models each pixel embedding as a probability via multivariate Gaussian distribution to fully exploit the uncertainty within them, eventually improving the representation quality of the model. In addition, we derive prototypes from probability estimation posterior probability estimation which helps to push the decision boundary away from the ambiguity points. Moreover, we employ an efficient method to compute similarity between distributions, eliminating the need for sampling and reparameterization, thereby significantly reducing computational overhead. Further, we dynamically select the ambiguous crops at the image level to enlarge the number of boundary points involved in contrastive learning, which benefits the establishment of precise distributions for each category. Extensive experimentation demonstrates that PPPC not only helps to address ambiguity at the pixel level, yielding discriminative representations but also achieves significant improvements in both synthetic-to-real and day-to-night adaptation tasks. It surpasses the previous state-of-the-art (SOTA) by +5.2% mIoU in the most challenging daytime-to-nighttime adaptation scenario, exhibiting stronger generalization on other unseen datasets. The code and models are available at <a class="link-external link-https" href="https://github.com/DarlingInTheSV/Probabilistic-Prototypical-Pixel-Contrast" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: in the semantic segmentation task in the domain adaptation scenario, how to reduce the problem of model performance degradation caused by the domain shift between the source domain and the target domain. Specifically, although the existing methods that combine contrastive learning and self - training paradigms have achieved certain success, they perform poorly in handling ambiguous scenarios caused by factors such as scale, illumination or overlap, especially when deploying deterministic embedding. To address these problems, the authors propose a new method - Probabilistic Prototypical Pixel Contrast (PPPC), aiming to fully utilize the uncertainty in each pixel embedding by modeling the probability distribution of each pixel embedding, thereby improving the quality of the model representation. ### Specific Problem Description 1. **Domain Shift Problem**: When a model migrates from a labeled source domain to an unlabeled target domain, due to the differences between the two domains (such as different data distributions), the performance of the model on the target domain often degrades. 2. **Ambiguous Scenario Problem**: Existing methods are not effective in handling ambiguous scenarios caused by factors such as scale, illumination or overlap. Especially when using deterministic embedding, it is easy to generate wrong pseudo - labels, causing the decision boundary to deviate from the correct direction. 3. **Computational Efficiency Problem**: Traditional contrastive learning methods have a large computational cost when dealing with pixel - level contrast, and it is difficult to be efficiently applied to large - scale datasets. ### Solution To solve the above problems, the PPPC framework proposes the following improvements: - **Probabilistic Embedding**: Model each pixel embedding as a multivariate Gaussian distribution \(p(z_i|x_i)=\mathcal{N}(z_i; \mu_i,\Sigma_i)\) to explicitly express uncertainty. - **Probabilistic Prototype Construction**: Derive prototypes from pixel embeddings through posterior probability estimation, helping to move the decision boundary away from ambiguous points. - **Efficient Similarity Metric**: Adopt the Expected Likelihood Kernel (ELK) to calculate the similarity between distributions, avoiding the need for sampling and re - parameterization, and significantly reducing the computational cost. - **Dynamic Selection of Ambiguous Regions**: Dynamically select image - level ambiguous regions in a global manner, increasing the number of boundary points involved in contrastive learning, which helps to better handle ambiguous categories. ### Main Contributions 1. Propose a general probability adaptation framework PPPC, which can fully utilize the uncertainty information of each pixel embedding and significantly improve the segmentation performance in ambiguous scenarios. 2. Design efficient modules and loss functions to ensure that the model performance is significantly improved without significantly increasing the computational cost and GPU memory usage. 3. Conduct extensive experiments on three typical UDA tasks. The results show that PPPC has achieved excellent performance on multiple benchmark datasets, especially in the most challenging day - to - night adaptation task, with an mIoU improvement of 5.2%. Through these improvements, PPPC not only solves the shortcomings of existing methods in ambiguous scenarios but also improves the generalization ability and robustness of the model.

Reducing Semantic Ambiguity In Domain Adaptive Semantic Segmentation Via Probabilistic Prototypical Pixel Contrast

ADeLA: Automatic Dense Labeling with Attention for Viewpoint Shift in Semantic Segmentation

See Clearer at Night: Towards Robust Nighttime Semantic Segmentation Through Day-Night Image Conversion

Unsupervised Pixel-Wise Weighted Adversarial Domain Adaptation

A Prototypical Knowledge Oriented Adaptation Framework for Semantic Segmentation

Pixel-Level Cycle Association: A New Perspective for Domain Adaptive Semantic Segmentation

SePiCo: Semantic-Guided Pixel Contrast for Domain Adaptive Semantic Segmentation.

Prototypical Contrast Adaptation for Domain Adaptive Semantic Segmentation

Combining Pixel-Level and Structure-Level Adaptation for Semantic Segmentation

Contrast, Stylize and Adapt: Unsupervised Contrastive Learning Framework for Domain Adaptive Semantic Segmentation

A Domain Adaptive Semantic Segmentation Method Using Contrastive Learning and Data Augmentation

Pixel-Level Domain Adaptation: A New Perspective for Enhancing Weakly Supervised Semantic Segmentation

SPCL: A New Framework for Domain Adaptive Semantic Segmentation via Semantic Prototype-based Contrastive Learning

Towards Source-free Domain Adaptive Semantic Segmentation via Importance-aware and Prototype-contrast Learning

Causal Prototype-inspired Contrast Adaptation for Unsupervised Domain Adaptive Semantic Segmentation of High-resolution Remote Sensing Imagery

Semantic Distribution-aware Contrastive Adaptation for Semantic Segmentation

Domain Adaptation for Remote Sensing Image Semantic Segmentation: An Integrated Approach of Contrastive Learning and Adversarial Learning

Exploiting Diverse Characteristics and Adversarial Ambivalence for Domain Adaptive Segmentation

Domain Adaptive Semantic Segmentation Via Regional Contrastive Consistency Regularization

Unsupervised Domain Adaptive Semantic Segmentation Based on Clip-Guided Prototypical Contrastive Learning

Prompting to Adapt Foundational Segmentation Models