Abstract:Cross-modal retrieval (e.g., query a given image to obtain a semantically similar sentence, and vice versa) is an important but challenging task, as the heterogeneous gap and inconsistent distributions exist between different modalities. The dominant approaches struggle to bridge the heterogeneity by capturing the common representations among heterogeneous data in a constructed subspace which can reflect the semantic closeness. However, insufficient consideration is taken into the fact that learned latent representations are actually heavily entangled with those semantic-unrelated features, which obviously further compounds the challenges of cross-modal retrieval. To alleviate the difficulty, this work makes an assumption that the data are jointly characterized by two independent features: semantic-shared and semantic-unrelated representations. The former presents characteristics of consistent semantics shared by different modalities, while the latter reflects the characteristics with respect to the modality yet unrelated to semantics, such as background, illumination, and other low-level information. Therefore, this paper aims to disentangle the shared semantics from the entangled features, andthus the purer semantic representation can promote the closeness of paired data. Specifically, this paper designs a novel Semantics Disentangling approach for Cross-Modal Retrieval (termed as SDCMR) to explicitly decouple the two different features based on variational auto-encoder. Next, the reconstruction is performed by exchanging shared semantics to ensure the learning of semantic consistency. Moreover, a dual adversarial mechanism is designed to disentangle the two independent features via a pushing-and-pulling strategy. Comprehensive experiments on four widely used datasets demonstrate the effectiveness and superiority of the proposed SDCMR method by achieving a new bar on performance when compared against 15 state-of-the-art methods.

Multimodal Disentanglement Variational AutoEncoders for Zero-Shot Cross-Modal Retrieval

Learning Disentangled Representation for Cross-Modal Retrieval with Deep Mutual Information Estimation.

Zero-shot Cross-modal Retrieval by Assembling AutoEncoder and Generative Adversarial Network

Learning Cross-Aligned Latent Embeddings for Zero-Shot Cross-Modal Retrieval

Incomplete Cross-modal Retrieval with Dual-Aligned Variational Autoencoders.

Semantics Disentangling for Cross-Modal Retrieval

Instance-Level Semantic Alignment for Zero-Shot Cross-Modal Retrieval

Learning Explicit and Implicit Latent Common Spaces for Audio-Visual Cross-Modal Retrieval

Cross-modal Semantic Autoencoder with Embedding Consensus

Variational Autoencoder with CCA for Audio-Visual Cross-Modal Retrieval

Multi-grained Encoding and Joint Embedding Space Fusion for Video and Text Cross-Modal Retrieval

Learning Aligned Cross-Modal Representation for Generalized Zero-Shot Classification

Zero-shot Learning with Regularized Cross-Modality Ranking.

Inter-Modality Fusion Based Attention for Zero-Shot Cross-Modal Retrieval.

Audio-Visual Cross-Modal Generation with Multimodal Variational Generative Model

Cross-Modal Retrieval Based on Full-Modal Autoencoder with Generative Adversarial Mechanism

Correlated Features Synthesis and Alignment for Zero-shot Cross-modal Retrieval

A Discriminative Cross-Aligned Variational Autoencoder for Zero-Shot Learning

Multi-dimensional Alignment Via Variational Autoencoders for Generalized Zero-Shot and Few-Shot Learning.

Achieving Cross Modal Generalization with Multimodal Unified Representation.

Learning Controlled Semantic Embedding for Cross-Modal Retrieval.