Abstract:Fine-grained image-text retrieval has been a hot research topic to bridge the vision and languages, and its main challenge is how to learn the semantic correspondence across different modalities. The existing methods mainly focus on learning the global semantic correspondence or intramodal relation correspondence in separate data representations, but which rarely consider the intermodal relation that interactively provide complementary hints for fine-grained semantic correlation learning. To address this issue, we propose a relation-aggregated cross-graph (RACG) model to explicitly learn the fine-grained semantic correspondence by aggregating both intramodal and intermodal relations, which can be well utilized to guide the feature correspondence learning process. More specifically, we first build semantic-embedded graph to explore both fine-grained objects and their relations of different media types, which aim not only to characterize the object appearance in each modality, but also to capture the intrinsic relation information to differentiate intramodal discrepancies. Then, a cross-graph relation encoder is newly designed to explore the intermodal relation across different modalities, which can mutually boost the cross-modal correlations to learn more precise intermodal dependencies. Besides, the feature reconstruction module and multihead similarity alignment are efficiently leveraged to optimize the node-level semantic correspondence, whereby the relation-aggregated cross-modal embeddings between image and text are discriminatively obtained to benefit various image-text retrieval tasks with high retrieval performance. Extensive experiments evaluated on benchmark datasets quantitatively and qualitatively verify the advantages of the proposed framework for fine-grained image-text retrieval and show its competitive performance with the state of the arts.

Relation-guided network for image-text retrieval

Deep Relation Embedding for Cross-Modal Retrieval.

Learning Aligned Image-Text Representations Using Graph Attentive Relational Network

Cross-modal Graph Matching Network for Image-text Retrieval

Visual Global-Salient-Guided Network for Remote Sensing Image-Text Retrieval

Building Joint Relationship Attention Network for Image-Text Generation.

Visual Relations Augmented Cross-modal Retrieval

Relation-Aggregated Cross-Graph Correlation Learning for Fine-Grained Image–Text Retrieval

Context‐aware relation enhancement and similarity reasoning for image‐text retrieval

Learning Relationship-Enhanced Semantic Graph for Fine-Grained Image-Text Matching

A Mutually Textual and Visual Refinement Network for Image-Text Matching

Learning from Different Text-Image Pairs: A Relation-enhanced Graph Convolutional Network for Multimodal NER

Scene Graph Based Fusion Network For Image-Text Retrieval

Multilateral Semantic Relations Modeling for Image Text Retrieval

Text-Image Matching for Cross-Modal Remote Sensing Image Retrieval via Graph Neural Network

GA-SRN: Graph Attention Based Text-Image Semantic Reasoning Network for Fine-Grained Image Classification and Retrieval

Dual Relation Network for Scene Text Recognition

Dual-Level Representation Enhancement on Characteristic and Context for Image-Text Retrieval

Image-text interaction graph neural network for image-text sentiment analysis

Relational Graph-Bridged Image-Text Interaction: A Novel Method for Multi-Modal Relation Extraction

Attention-Based Relation Reasoning Network for Video-Text Retrieval.