Abstract:Most cross-modal retrieval methods assume the multi-modal training data is complete and has a one-to-one correspondence. However, in the real world, multi-modal data generally suffers from missing modality information due to the uncertainty of data collection and storage processes, which limits the practical application of existing cross-modal retrieval methods. Although some solutions have been proposed to generate the missing modality data using a single pseudo sample, this may lead to incomplete semantic restoration and sub-optimal retrieval results due to the limited semantic information it provides. To address this challenge, this article proposes an Incomplete Cross-Modal Retrieval with Deep Correlation Transfer (ICMR-DCT) method that can robustly model incomplete multi-modal data and dynamically capture the adjacency semantic correlation for cross-modal retrieval. Specifically, we construct intra-modal graph attention-based auto-encoder to learn modality-invariant representations by performing semantic reconstruction through intra-modality adjacency correlation mining. Then, we design dual cross-modal alignment constraints to project multi-modal representations into a common semantic space, thus bridging the heterogeneous modality gap and enhancing the discriminability of the common representation. We further introduce semantic preservation to enhance adjacency semantic information and achieve cross-modal semantic correlation. Moreover, we propose a nearest-neighbor weighting integration strategy with cross-modal correlation transfer to generate the missing modality data according to inter-modality mapping relations and adjacency correlations between each sample and its neighbors, which improves the robustness of our method against incomplete multi-modal training data. Extensive experiments on three widely tested benchmark datasets demonstrate the superior performance of our method in cross-modal retrieval tasks under both complete and incomplete retrieval scenarios. Our used datasets and source codes are available at https://github.com/shidan0122/DCT.git .

HGR Maximal Correlation Augmented Cross-Modal Remote Sensing Retrieval

Robust Cross-Modal Remote Sensing Image Retrieval Via Maximal Correlation Augmentation

A Deep Cross-Modality Hashing Network for SAR and Optical Remote Sensing Images Retrieval

Dual Modality Collaborative Learning for Cross-Source Remote Sensing Retrieval

Remote Sensing Cross-Modal Text-Image Retrieval Based on Global and Local Information

Cross-modal Retrieval with Dual Optimization

Hierarchical Set-to-Set Representation for 3-D Cross-Modal Retrieval

Modeling Intra- and Inter-Pair Correlation Via Heterogeneous High-Order Preserving for Cross-Modal Retrieval

MCRN: A Multi-source Cross-modal Retrieval Network for Remote Sensing

Semantically Supervised Maximal Correlation for Cross-Modal Retrieval

Deep Multiscale Fine-Grained Hashing for Remote Sensing Cross-Modal Retrieval

Heterogeneous memory enhanced graph reasoning network for cross-modal retrieval

Analyzing semantic correlation for cross-modal retrieval

Similarity Graph-correlation Reconstruction Network for unsupervised cross-modal hashing

Cross-Modal Pre-Aligned Method with Global and Local Information for Remote-Sensing Image and Text Retrieval

Cross-Modality Retrieval by Joint Correlation Learning

Asymmetric Correlation Quantization Hashing for Cross-modal Retrieval

A New Approach to Cross-Modal Retrieval

A Lightweight Multi-Scale Crossmodal Text-Image Retrieval Method in Remote Sensing

Deep Multi-Graph Hierarchical Enhanced Semantic Representation for Cross-Modal Retrieval

Incomplete Cross-Modal Retrieval with Deep Correlation Transfer