Abstract:Most cross-modal retrieval methods assume the multi-modal training data is complete and has a one-to-one correspondence. However, in the real world, multi-modal data generally suffers from missing modality information due to the uncertainty of data collection and storage processes, which limits the practical application of existing cross-modal retrieval methods. Although some solutions have been proposed to generate the missing modality data using a single pseudo sample, this may lead to incomplete semantic restoration and sub-optimal retrieval results due to the limited semantic information it provides. To address this challenge, this article proposes an Incomplete Cross-Modal Retrieval with Deep Correlation Transfer (ICMR-DCT) method that can robustly model incomplete multi-modal data and dynamically capture the adjacency semantic correlation for cross-modal retrieval. Specifically, we construct intra-modal graph attention-based auto-encoder to learn modality-invariant representations by performing semantic reconstruction through intra-modality adjacency correlation mining. Then, we design dual cross-modal alignment constraints to project multi-modal representations into a common semantic space, thus bridging the heterogeneous modality gap and enhancing the discriminability of the common representation. We further introduce semantic preservation to enhance adjacency semantic information and achieve cross-modal semantic correlation. Moreover, we propose a nearest-neighbor weighting integration strategy with cross-modal correlation transfer to generate the missing modality data according to inter-modality mapping relations and adjacency correlations between each sample and its neighbors, which improves the robustness of our method against incomplete multi-modal training data. Extensive experiments on three widely tested benchmark datasets demonstrate the superior performance of our method in cross-modal retrieval tasks under both complete and incomplete retrieval scenarios. Our used datasets and source codes are available at https://github.com/shidan0122/DCT.git .

Cluster-sensitive Structured Correlation Analysis for Web cross-modal retrieval

Analyzing semantic correlation for cross-modal retrieval

Combining Link And Content Correlation Learning For Cross-Modal Retrieval In Social Multimedia

Modeling Intra- and Inter-Pair Correlation Via Heterogeneous High-Order Preserving for Cross-Modal Retrieval

Self-supervised Correlation Learning for Cross-Modal Retrieval

Multimedia Feature Mapping and Correlation Learning for Cross-Modal Retrieval

Adversarial Learning-Based Semantic Correlation Representation for Cross-Modal Retrieval

Cross-Modal Subspace Clustering Via Deep Canonical Correlation Analysis.

Cross-Modal Correlation Learning by Adaptive Hierarchical Semantic Aggregation

CCL: Cross-modal Correlation Learning with Multi-grained Fusion by Hierarchical Network.

Semisupervised Cross-Media Retrieval By Distance-Preserving Correlation Learning And Multi-Modal Manifold Regularization

Learning Semantic Correlations for Cross-Media Retrieval.

Cross-Modality Retrieval by Joint Correlation Learning

Attention-Sharing Correlation Learning For Cross-Media Retrieval

Learning Explicit and Implicit Latent Common Spaces for Audio-Visual Cross-Modal Retrieval

Incomplete Cross-Modal Retrieval with Deep Correlation Transfer

Progressive Cross-Media Correlation Learning.

Cross-media Retrieval Based on Semi-Supervised Regularization and Correlation Learning

Understanding Visual-Auditory Correlation from Heterogeneous Features for Cross-Media Retrieval

Semantically Supervised Maximal Correlation for Cross-Modal Retrieval

Exploiting Correlation Consensus: Towards Subspace Clustering For Multi-Modal Data