Abstract:Fine-grained image-text retrieval has been a hot research topic to bridge the vision and languages, and its main challenge is how to learn the semantic correspondence across different modalities. The existing methods mainly focus on learning the global semantic correspondence or intramodal relation correspondence in separate data representations, but which rarely consider the intermodal relation that interactively provide complementary hints for fine-grained semantic correlation learning. To address this issue, we propose a relation-aggregated cross-graph (RACG) model to explicitly learn the fine-grained semantic correspondence by aggregating both intramodal and intermodal relations, which can be well utilized to guide the feature correspondence learning process. More specifically, we first build semantic-embedded graph to explore both fine-grained objects and their relations of different media types, which aim not only to characterize the object appearance in each modality, but also to capture the intrinsic relation information to differentiate intramodal discrepancies. Then, a cross-graph relation encoder is newly designed to explore the intermodal relation across different modalities, which can mutually boost the cross-modal correlations to learn more precise intermodal dependencies. Besides, the feature reconstruction module and multihead similarity alignment are efficiently leveraged to optimize the node-level semantic correspondence, whereby the relation-aggregated cross-modal embeddings between image and text are discriminatively obtained to benefit various image-text retrieval tasks with high retrieval performance. Extensive experiments evaluated on benchmark datasets quantitatively and qualitatively verify the advantages of the proposed framework for fine-grained image-text retrieval and show its competitive performance with the state of the arts.

Image Retrieval Based on Augmented Relational Graph Representation

Image Retrieval Algorithm Based on Enhanced Relational Graph.

Incremental semi-supervised subspace learning for image retrieval.

Bayesian network based semantic image classification with attributed relational graph

Image retrieval based on incremental subspace learning

Model Semantic Relations with Extended Attributes

Learning A Semantic Space from User'S Relevance Feedback for Image Retrieval

Image Retrieval Based on Fuzzy Semantic Relevance Matrix

Semi-supervised Dimensionality Reduction for Image Retrieval

Visual Relations Augmented Cross-modal Retrieval

Improving Supervised Cross-modal Retrieval with Semantic Graph Embedding

Modeling Image Data for Effective Indexing and Retrieval in Large General Image Databases.

Relation-Aggregated Cross-Graph Correlation Learning for Fine-Grained Image–Text Retrieval

A unified optimization based learning method for image retrieval

Feedback-Based image retrieval using probabilistic hypergraph ranking augmented by ant colony algorithm

Learning Dual Semantic Relations with Graph Attention for Image-Text Matching

Enhancing Relevance Feedback in Image Retrieval Using Unlabeled Data

Weak Augmentation Guided Relational Self-Supervised Learning

An Image Retrieval Method Based On Collaborative Filtering

Exploiting Unlabeled Data In Content-Based Image Retrieval

A Graphic-Theoretic Model for Incremental Relevance Feedback in Image Retrieval