Abstract:Nowadays, cross-modal retrieval plays an important role to flexibly find useful information across different modalities of data. Effectively measuring the similarity between different modalities of data is the key of cross-modal retrieval. Different modalities such as image and text have imbalanced and complementary relationship, and they contain unequal amount of information when describing the same semantics. For example, images often contain more details that cannot be demonstrated by textual descriptions and vice versa. Existing works based on Deep Neural Network (DNN) mostly construct one common space for different modalities, to find the latent alignments between them, which lose their exclusive modality-specific characteristics. Therefore, we propose modality-specific cross-modal similarity measurement (MCSM) approach by constructing the independent semantic space for each modality, which adopts an endto- end framework to directly generate modality-specific crossmodal similarity without explicit common representation. For each semantic space, modality-specific characteristics within one modality are fully exploited by recurrent attention network, while the data of another modality is projected into this space with attention based joint embedding, which utilizes the learned attention weights for guiding the fine-grained cross-modal correlation learning, and captures the imbalanced and complementary relationship between different modalities. Finally, the complementarity between the semantic spaces for different modalities is explored by adaptive fusion of the modality-specific cross-modal similarities to perform cross-modal retrieval. Experiments on the widely-used Wikipedia, Pascal Sentence, MS-COCO datasets as well as our constructed large-scale XMediaNet dataset verify the effectiveness of our proposed approach, outperforming 9 stateof- the-art methods.

A New Approach to Cross-Modal Retrieval

Semantic Consistency Hashing for Cross-Modal Retrieval

Towards Improving Canonical Correlation Analysis for Cross-modal Retrieval.

Adversarial Learning-Based Semantic Correlation Representation for Cross-Modal Retrieval

Analyzing semantic correlation for cross-modal retrieval

Learning Semantic Correlations for Cross-Media Retrieval.

Image Retrieval Approach Based on Sparse Canonical Correlation Analysis

End-to-End Cross-Modality Retrieval with CCA Projections and Pairwise Ranking Loss

Learning Explicit and Implicit Latent Common Spaces for Audio-Visual Cross-Modal Retrieval

Adversarial Cross-Modal Retrieval

Hybrid Deep Neural Network-Based Cross-Modal Image and Text Retrieval Method for Large-Scale Data.

Correlated Features Synthesis and Alignment for Zero-shot Cross-modal Retrieval

Cross-Modal Learning Based on Semantic Correlation and Multi-Task Learning for Text-Video Retrieval

Modality-Specific Cross-Modal Similarity Measurement With Recurrent Attention Network

Cross-Modal Learning With Images, Texts And Their Semantics

Semantically Supervised Maximal Correlation for Cross-Modal Retrieval

Adversarial Cross-Modal Retrieval via Learning and Transferring Single-Modal Similarities

Learning Shared Semantic Space with Correlation Alignment for Cross-Modal Event Retrieval

A Deep Semantic Alignment Network for the Cross-Modal Image-Text Retrieval in Remote Sensing

Rethinking Label-Wise Cross-Modal Retrieval from A Semantic Sharing Perspective

CCL: Cross-modal Correlation Learning with Multi-grained Fusion by Hierarchical Network.