Abstract:Cross-modal retrieval (e.g., query a given image to obtain a semantically similar sentence, and vice versa) is an important but challenging task, as the heterogeneous gap and inconsistent distributions exist between different modalities. The dominant approaches struggle to bridge the heterogeneity by capturing the common representations among heterogeneous data in a constructed subspace which can reflect the semantic closeness. However, insufficient consideration is taken into the fact that learned latent representations are actually heavily entangled with those semantic-unrelated features, which obviously further compounds the challenges of cross-modal retrieval. To alleviate the difficulty, this work makes an assumption that the data are jointly characterized by two independent features: semantic-shared and semantic-unrelated representations. The former presents characteristics of consistent semantics shared by different modalities, while the latter reflects the characteristics with respect to the modality yet unrelated to semantics, such as background, illumination, and other low-level information. Therefore, this paper aims to disentangle the shared semantics from the entangled features, andthus the purer semantic representation can promote the closeness of paired data. Specifically, this paper designs a novel Semantics Disentangling approach for Cross-Modal Retrieval (termed as SDCMR) to explicitly decouple the two different features based on variational auto-encoder. Next, the reconstruction is performed by exchanging shared semantics to ensure the learning of semantic consistency. Moreover, a dual adversarial mechanism is designed to disentangle the two independent features via a pushing-and-pulling strategy. Comprehensive experiments on four widely used datasets demonstrate the effectiveness and superiority of the proposed SDCMR method by achieving a new bar on performance when compared against 15 state-of-the-art methods.

Semantically Supervised Maximal Correlation for Cross-Modal Retrieval

Semantic Consistency Hashing for Cross-Modal Retrieval

Discrete Cross-Modal Hashing for Efficient Multimedia Retrieval

Efficient Discrete Supervised Hashing for Large-scale Cross-modal Retrieval

Self-supervised Correlation Learning for Cross-Modal Retrieval

Analyzing semantic correlation for cross-modal retrieval

Deep Supervised Cross-Modal Retrieval

Adversarial Learning-Based Semantic Correlation Representation for Cross-Modal Retrieval

Learning Semantic Correlations for Cross-Media Retrieval.

Robust Cross-Modal Remote Sensing Image Retrieval Via Maximal Correlation Augmentation

Learning Explicit and Implicit Latent Common Spaces for Audio-Visual Cross-Modal Retrieval

Federated learning for supervised cross-modal retrieval

A General Framework For Incomplete Cross-Modal Retrieval With Missing Labels And Missing Modalities

Adversarial Cross-Modal Retrieval via Learning and Transferring Single-Modal Similarities

Asymmetric Supervised Consistent and Specific Hashing for Cross-Modal Retrieval

Large-Scale Supervised Multimodal Hashing with Semantic Correlation Maximization

Adaptive Marginalized Semantic Hashing for Unpaired Cross-Modal Retrieval

Adversarial Cross-Modal Retrieval

Deep supervised fused similarity hashing for cross-modal retrieval

Understanding Visual-Auditory Correlation from Heterogeneous Features for Cross-Media Retrieval

Semantics Disentangling for Cross-Modal Retrieval