Abstract:Fine-grained image-text retrieval has been a hot research topic to bridge the vision and languages, and its main challenge is how to learn the semantic correspondence across different modalities. The existing methods mainly focus on learning the global semantic correspondence or intramodal relation correspondence in separate data representations, but which rarely consider the intermodal relation that interactively provide complementary hints for fine-grained semantic correlation learning. To address this issue, we propose a relation-aggregated cross-graph (RACG) model to explicitly learn the fine-grained semantic correspondence by aggregating both intramodal and intermodal relations, which can be well utilized to guide the feature correspondence learning process. More specifically, we first build semantic-embedded graph to explore both fine-grained objects and their relations of different media types, which aim not only to characterize the object appearance in each modality, but also to capture the intrinsic relation information to differentiate intramodal discrepancies. Then, a cross-graph relation encoder is newly designed to explore the intermodal relation across different modalities, which can mutually boost the cross-modal correlations to learn more precise intermodal dependencies. Besides, the feature reconstruction module and multihead similarity alignment are efficiently leveraged to optimize the node-level semantic correspondence, whereby the relation-aggregated cross-modal embeddings between image and text are discriminatively obtained to benefit various image-text retrieval tasks with high retrieval performance. Extensive experiments evaluated on benchmark datasets quantitatively and qualitatively verify the advantages of the proposed framework for fine-grained image-text retrieval and show its competitive performance with the state of the arts.

Text correlation measurement method and system based on fine-grained matching signal

An adaptive method for text domain similarity calculation

A Matching Algorithm Based On Correlations Between Matrices Of Reference Differences Combining Disparities

Text Matching As Image Recognition

A Survey on Deep Text Matching

Improving Text Matching with Semantic Dependency Graph via Message Passing Neural Network

An Adaptive Correlation Filtering Method for Text-Based Person Search

Uyghur Text Matching in Graphic Images for Biomedical Semantic Analysis.

Semantic Similarity Computing Model Based on Multi Model Fine-Grained Nonlinear Fusion

Semantic Correlation Network Based Text Clustering

Bridging the gap: dual perception attention and local-global similarity fusion for cross-modal image-text matching

Cross-Graph Attention Enhanced Multi-Modal Correlation Learning for Fine-Grained Image-Text Retrieval

Short text matching model with multiway semantic interaction based on multi-granularity semantic embedding

Chinese long text similarity calculation of semantic progressive fusion based on Bert

Multi-granularity Correlation Refinement for Semantic Correspondence

Dual Semantic Relationship Attention Network for Image-Text Matching

Relation-Aggregated Cross-Graph Correlation Learning for Fine-Grained Image–Text Retrieval

A survey on the techniques, applications, and performance of short text semantic similarity

Correlation Coefficients and Semantic Textual Similarity

Show Your Faith: Cross-Modal Confidence-Aware Network for Image-Text Matching.

A novel locality-sensitive hashing relational graph matching network for semantic textual similarity measurement