Abstract:Most cross-modal retrieval methods assume the multi-modal training data is complete and has a one-to-one correspondence. However, in the real world, multi-modal data generally suffers from missing modality information due to the uncertainty of data collection and storage processes, which limits the practical application of existing cross-modal retrieval methods. Although some solutions have been proposed to generate the missing modality data using a single pseudo sample, this may lead to incomplete semantic restoration and sub-optimal retrieval results due to the limited semantic information it provides. To address this challenge, this article proposes an Incomplete Cross-Modal Retrieval with Deep Correlation Transfer (ICMR-DCT) method that can robustly model incomplete multi-modal data and dynamically capture the adjacency semantic correlation for cross-modal retrieval. Specifically, we construct intra-modal graph attention-based auto-encoder to learn modality-invariant representations by performing semantic reconstruction through intra-modality adjacency correlation mining. Then, we design dual cross-modal alignment constraints to project multi-modal representations into a common semantic space, thus bridging the heterogeneous modality gap and enhancing the discriminability of the common representation. We further introduce semantic preservation to enhance adjacency semantic information and achieve cross-modal semantic correlation. Moreover, we propose a nearest-neighbor weighting integration strategy with cross-modal correlation transfer to generate the missing modality data according to inter-modality mapping relations and adjacency correlations between each sample and its neighbors, which improves the robustness of our method against incomplete multi-modal training data. Extensive experiments on three widely tested benchmark datasets demonstrate the superior performance of our method in cross-modal retrieval tasks under both complete and incomplete retrieval scenarios. Our used datasets and source codes are available at https://github.com/shidan0122/DCT.git .

Discriminative Canonical Correlation Analysis with Missing Samples

Regularized Canonical Correlation Analysis with Unlabeled Data

A Novel Method of Combined Feature Extraction for Recognition

Sparse canonical correlation analysis for recognition

Discriminative Multiple Canonical Correlation Analysis for Information Fusion

Multimodal Emotion Recognition Using Deep Canonical Correlation Analysis

Two-Dimensional Discriminant Canonical Correlation Analysis for Face Recognition

A Novel Discriminant Minimum Class Locality Preserving Canonical Correlation Analysis and Its Applications

Sparse Additive Discriminant Canonical Correlation Analysis for Multiple Features Fusion

A Dynamic Discriminative Canonical Correlation Analysis via Adaptive Weight Scheme

Sparse Representation Based Discriminative Canonical Correlation Analysis for Face Recognition

Two-dimensional canonical correlation analysis and its application in small sample size face recognition

Identifiable Shared Component Analysis of Unpaired Multimodal Mixtures

Exploring Shared Subspace and Joint Sparsity for Canonical Correlation Analysis.

Towards Improving Canonical Correlation Analysis for Cross-modal Retrieval.

Multimodal Emotion Recognition Using Deep Generalized Canonical Correlation Analysis with an Attention Mechanism

Composite Nonlinear Multiset Canonical Correlation Analysis for Multiview Feature Learning and Recognition

Tensor Canonical Correlation Analysis for Multi-View Dimension Reduction

Sufficient Canonical Correlation Analysis

Incomplete Cross-Modal Retrieval with Deep Correlation Transfer

Canonical correlation analysis based on local sparse representation and linear discriminative analysis