Abstract:Most cross-modal retrieval methods assume the multi-modal training data is complete and has a one-to-one correspondence. However, in the real world, multi-modal data generally suffers from missing modality information due to the uncertainty of data collection and storage processes, which limits the practical application of existing cross-modal retrieval methods. Although some solutions have been proposed to generate the missing modality data using a single pseudo sample, this may lead to incomplete semantic restoration and sub-optimal retrieval results due to the limited semantic information it provides. To address this challenge, this article proposes an Incomplete Cross-Modal Retrieval with Deep Correlation Transfer (ICMR-DCT) method that can robustly model incomplete multi-modal data and dynamically capture the adjacency semantic correlation for cross-modal retrieval. Specifically, we construct intra-modal graph attention-based auto-encoder to learn modality-invariant representations by performing semantic reconstruction through intra-modality adjacency correlation mining. Then, we design dual cross-modal alignment constraints to project multi-modal representations into a common semantic space, thus bridging the heterogeneous modality gap and enhancing the discriminability of the common representation. We further introduce semantic preservation to enhance adjacency semantic information and achieve cross-modal semantic correlation. Moreover, we propose a nearest-neighbor weighting integration strategy with cross-modal correlation transfer to generate the missing modality data according to inter-modality mapping relations and adjacency correlations between each sample and its neighbors, which improves the robustness of our method against incomplete multi-modal training data. Extensive experiments on three widely tested benchmark datasets demonstrate the superior performance of our method in cross-modal retrieval tasks under both complete and incomplete retrieval scenarios. Our used datasets and source codes are available at https://github.com/shidan0122/DCT.git .

Robust Cross-Modal Remote Sensing Image Retrieval Via Maximal Correlation Augmentation

HGR Maximal Correlation Augmented Cross-Modal Remote Sensing Retrieval

A Deep Cross-Modality Hashing Network for SAR and Optical Remote Sensing Images Retrieval

Dual Modality Collaborative Learning for Cross-Source Remote Sensing Retrieval

Content-Based High-Resolution Remote Sensing Image Retrieval Via Unsupervised Feature Learning And Collaborative Affinity Metric Fusion

Remote Sensing Cross-Modal Text-Image Retrieval Based on Global and Local Information

Cross-Modal Remote Sensing Image-Audio Retrieval with Adaptive Learning for Aligning Correlation

Deep Multiscale Fine-Grained Hashing for Remote Sensing Cross-Modal Retrieval

Semantically Supervised Maximal Correlation for Cross-Modal Retrieval

A Lightweight Multi-Scale Crossmodal Text-Image Retrieval Method in Remote Sensing

Enhancing Remote Sensing Image Retrieval: A Hierarchical Approach Integrating Visual and Semantic Similarities

Incomplete Cross-Modal Retrieval with Deep Correlation Transfer

MCRN: A Multi-source Cross-modal Retrieval Network for Remote Sensing

Modeling Intra- and Inter-Pair Correlation Via Heterogeneous High-Order Preserving for Cross-Modal Retrieval

Supervised Contrastive Learning Based on Fusion of Global and Local Features for Remote Sensing Image Retrieval

Multimedia Feature Mapping and Correlation Learning for Cross-Modal Retrieval

Consistency Center-Based Deep Cross-Modal Hashing for Multisource Remote Sensing Image Retrieval

Asymmetric Correlation Quantization Hashing for Cross-modal Retrieval

Adversarial Cross-Modal Retrieval

A Deep Semantic Alignment Network for the Cross-Modal Image-Text Retrieval in Remote Sensing

Soft Contrastive Cross-Modal Retrieval