Abstract:Cross-modal retrieval (e.g., query a given image to obtain a semantically similar sentence, and vice versa) is an important but challenging task, as the heterogeneous gap and inconsistent distributions exist between different modalities. The dominant approaches struggle to bridge the heterogeneity by capturing the common representations among heterogeneous data in a constructed subspace which can reflect the semantic closeness. However, insufficient consideration is taken into the fact that learned latent representations are actually heavily entangled with those semantic-unrelated features, which obviously further compounds the challenges of cross-modal retrieval. To alleviate the difficulty, this work makes an assumption that the data are jointly characterized by two independent features: semantic-shared and semantic-unrelated representations. The former presents characteristics of consistent semantics shared by different modalities, while the latter reflects the characteristics with respect to the modality yet unrelated to semantics, such as background, illumination, and other low-level information. Therefore, this paper aims to disentangle the shared semantics from the entangled features, andthus the purer semantic representation can promote the closeness of paired data. Specifically, this paper designs a novel Semantics Disentangling approach for Cross-Modal Retrieval (termed as SDCMR) to explicitly decouple the two different features based on variational auto-encoder. Next, the reconstruction is performed by exchanging shared semantics to ensure the learning of semantic consistency. Moreover, a dual adversarial mechanism is designed to disentangle the two independent features via a pushing-and-pulling strategy. Comprehensive experiments on four widely used datasets demonstrate the effectiveness and superiority of the proposed SDCMR method by achieving a new bar on performance when compared against 15 state-of-the-art methods.

Detach and Enhance: Learning Disentangled Cross-modal Latent Representation for Efficient Face-Voice Association and Matching

Learning Discriminative Joint Embeddings for Efficient Face and Voice Association.

Hearing like Seeing: Improving Voice-Face Interactions and Associations via Adversarial Deep Semantic Matching Network

Disentangled Representation Learning for Cross-Modal Biometric Matching.

Hearing Like Seeing

Taking a Part for the Whole: An Archetype-agnostic Framework for Voice-Face Association

An Efficient Momentum Framework for Face-Voice Association Learning.

Disentanglement for Discriminative Visual Recognition

Facial Landmark Disentangled Network with Variational Autoencoder

EFT: Expert Fusion Transformer for Voice-Face Association Learning.

Cross-domain Facial Expression Recognition Via Disentangling Identity Representation

Disentangled Noisy Correspondence Learning

Unsupervised Voice-Face Representation Learning by Cross-Modal Prototype Contrast

Exploiting Multi-modal Fusion for Robust Face Representation Learning with Missing Modality

Contrastive Learning-based Chaining-Cluster for Multilingual Voice-Face Association

Fuse after Align: Improving Face-Voice Association Learning via Multimodal Encoder

Heterogeneous Face Recognition with Attention-guided Feature Disentangling.

Exploring Robust Face-Voice Matching in Multilingual Environments

Semantics Disentangling for Cross-Modal Retrieval

Learning Disentangled Speech Representations with Contrastive Learning and Time-Invariant Retrieval

Cross-VAE: Towards Disentangling Expression from Identity for Human Faces