Abstract:Information retrieval among different modalities becomes a significant issue with many promising applications. However, inconsistent feature representation of various multimedia data causes the "heterogeneity gap" among various modalities, which is a challenge in cross-modal retrieval. For bridging the "heterogeneity gap," the popular methods attempt to project the original data into a common representation space, which needs great fitting ability of the model. To address the above issue, we propose a novel Graph Representation Learning (GRL) method for bridging the heterogeneity gap, which does not project the original feature into an aligned representation space but adopts a cross-modal graph to link different modalities. The GRL approach consists of two subnetworks, Feature Transfer Learning Network (FTLN) and Graph Representation Learning Network (GRLN). Firstly, FTLN model finds a latent space for each modality, where the cosine similarity is suitable to describe their similarity. Then, we build a cross-modal graph to reconstruct the original data and their relationships. Finally, we abandon the features in the latent space and turn into embedding the graph vertexes into a common representation space directly. During the process, the proposed Graph Representation Learning method bypasses the most challenging issue by utilizing a cross-modal graph as a bridge to link the "heterogeneity gap" among different modalities. This attempt utilizes a cross-modal graph as an intermediary agent to bridge the "heterogeneity gap" in cross-modal retrieval, which is simple but effective. Extensive experiment results on six widely-used datasets indicate that the proposed GRL outperforms other state-of-the-art cross-modal retrieval methods.

Heterogeneous Constraint Propagation with Constrained Sparse Representation

Exhaustive and Efficient Constraint Propagation: A Graph-Based Learning Approach and Its Applications

Exhaustive and Efficient Constraint Propagation: A Semi-Supervised Learning Perspective and Its Applications

Pairwise Constraint Propagation on Multi-View Data.

Symmetric Graph Regularized Constraint Propagation.

Unified Constraint Propagation on Multi-View Data.

Heterogeneous Graph Sparsification for Efficient Representation Learning

Pairwise Constraint Propagation with Dual Adversarial Manifold Regularization

Matrix Completion for Cross-view Pairwise Constraint Propagation

Jointprop: Joint Semi-supervised Learning for Entity and Relation Extraction with Heterogeneous Graph-based Propagation

Weighted Graph-structured Semantics Constraint Network for Cross-Modal Retrieval

Joint Dictionary Learning and Semantic Constrained Latent Subspace Projection for Cross-Modal Retrieval.

Graph Embedding with Constraints

Multi-Label Transfer Learning with Sparse Representation

Deep Multi-Graph Hierarchical Enhanced Semantic Representation for Cross-Modal Retrieval

Graph-based Semi-Supervised Learning by Mixed Label Propagation with a Soft Constraint.

Cross-modality Correlation Propagation for Cross-Media Retrieval

Bridging Multimedia Heterogeneity Gap Via Graph Representation Learning for Cross-Modal Retrieval.

Learning Discriminative Representations for Semantic Cross Media Retrieval

Neural Embedding Propagation on Heterogeneous Networks

How to Bridge Spatial and Temporal Heterogeneity in Link Prediction? A Contrastive Method