Abstract:Unsupervised domain adaptation aims to leverage labeled data from a source domain to learn a classifier for an unlabeled target domain. Amongst its many variants, open set domain adaptation (OSDA) is perhaps the most challenging one, as it further assumes the presence of unknown classes in the target domain. In this paper, we study OSDA with a particular focus on enriching its ability to traverse across larger domain gaps, and we show that existing state-of-the-art methods suffer a considerable performance drop in the presence of larger domain gaps, especially on a new dataset (PACS) that we re-purposed for OSDA. Exploring this is pivotal for OSDA as with increasing domain shift, identifying unknown samples in the target domain becomes harder for the model, thus making negative transfer between source and target domains more challenging. Accordingly, we propose a Mutual-to-Separate (MTS) framework to address the larger domain gaps. Essentially we design two networks – (a) Sample Separation Network (SSN): which is trained to learn a hyperplane for separating unknown samples from known ones, and (b) Distribution Matching Network (DMN): which is trained to maximise domain confusion between source and target domains without unknown samples under the guidance of the SSN. The key insight lies in how we exploit the mutually beneficial information between these two networks. On closer observation, we see that SSN can reveal which samples in the target domain belong to the unknown class by instance weighting whereas, DMN pushes apart the samples that most likely belong to the unknown class in the target domain, which in turn reduces the difficulty of SSN in identifying unknown samples. It follows that (a) and (b) will mutually supervise each other and alternate until convergence, which can better align the source and target domains in the shared label space. Extensive experiments on five datasets (Office-31, Office-Home, PACS, VisDA, and mini _DomainNet) demonstrate the efficiency of the proposed method. Detailed ablation experiments also validate the effectiveness of each component and the generality of the proposed framework. Codes are available at: https://github.com/PRIS-CV/Mutual-to-Separate.

Towards Multimodal Open-Set Domain Generalization and Adaptation through Self-supervision

SimMMDG: A Simple and Effective Framework for Multi-modal Domain Generalization

Adaptive Open Set Recognition with Multi-modal Joint Metric Learning.

Self-Supervised Multimodal Learning: A Survey

Multimodal Unsupervised Domain Generalization by Retrieving Across the Modality Gap

Deep Multi-Modality Adversarial Networks for Unsupervised Domain Adaptation

Achieving Cross Modal Generalization with Multimodal Unified Representation.

Multisource Collaborative Domain Generalization for Cross-Scene Remote Sensing Image Classification

Open-set domain adaptation for scene classification using multi-adversarial learning

Learning Class and Domain Augmentations for Single-Source Open-Domain Generalization

Mind the Gap: Open Set Domain Adaptation Via Mutual-to-Separate Framework

Balanced Multimodal Learning via On-the-fly Gradient Modulation

Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysis

Enhancing Multi-Source Open-Set Domain Adaptation through Nearest Neighbor Classification with Self-Supervised Vision Transformer

Learning Unseen Modality Interaction

Multi-Task Multi-Modal Self-Supervised Learning for Facial Expression Recognition

Multi-Modal Domain Generalization for Cross-Scene Hyperspectral Image Classification

Multimodal Unsupervised Domain Adaptation for Remote Sensing Image Segmentation

Dense Multimodal Alignment for Open-Vocabulary 3D Scene Understanding

Cluster, Split, Fuse, and Update: Meta-Learning for Open Compound Domain Adaptive Semantic Segmentation

COSMo: CLIP Talks on Open-Set Multi-Target Domain Adaptation