Abstract:Cross-modal image-text matching has attracted considerable interest in both computer vision and natural language processing communities. The main issue of image-text matching is to learn the compact cross-modal representations and the correlation between image and text representations. However, the image-text matching task has two major challenges. First, the current image representation methods focus on the semantic information and disregard the spatial position relations between image regions. Second, most existing methods pay little attention to improving textual representation which plays a significant role in image-text matching. To address these issues, we designed a decipherable cross-modal multi-relationship aware reasoning network (CMRN) for image-text matching. In particular, a new method is proposed to extract multi-relationship and to learn the correlations between image regions, including two kinds of visual relations: the geometric position relation and semantic interaction. In addition, images are processed as graphs, and a novel spatial relation encoder is introduced to perform reasoning on the graphs by employing a graph convolutional network (GCN) with attention mechanism. Thereafter, a contextual text encoder based on Bidirectional Encoder Representations from Transformers is adopted to learn distinctive textual representations. To verify the effectiveness of the proposed model, extensive experiments were conducted on two public datasets, namely MSCOCO and Flickr30K. The experimental results show that CMRN achieved superior performance when compared with state-of-the-art methods. On Flickr30K, the proposed method outperforms state-of-the-art methods more than 7.4% in text retrieval from image query, and 5.0% relatively in image retrieval with text query (based on Recall@1). On MSCOCO, the performance reaches 73.9% for text retrieval and 60.4% for image retrieval (based on Recall@1).

Dual Relation-Aware Synergistic Attention Network for Image-Text Matching

Learning Dual Semantic Relations with Graph Attention for Image-Text Matching

Select & Re-Rank: Effectively and Efficiently Matching Multimodal Data with Dynamically Evolving Attention

Dual Semantic Relationship Attention Network for Image-Text Matching

Bridging the gap: dual perception attention and local-global similarity fusion for cross-modal image-text matching

Bi-directional Spatial-Semantic Attention Networks for Image-Text Matching.

Multi-Modal Memory Enhancement Attention Network for Image-Text Matching

Contextual Relation Guided Image-Text Matching

Asymmetric Polysemous Reasoning for Image-Text Matching

Dual-path Rare Content Enhancement Network for Image and Text Matching

Cross-modal Multi-Relationship Aware Reasoning for Image-Text Matching

Dual Relation Network for Scene Text Recognition

Similarity Reasoning and Filtration for Image-Text Matching

Reference-Aware Adaptive Network for Image-Text Matching

Dual Relation Alignment for Composed Image Retrieval

DSGSR - Dynamic Semantic Generation and Similarity Reasoning for Image-Text Matching.

Saliency-Guided Attention Network for Image-Sentence Matching

Expressing Objects just like Words: Recurrent Visual Embedding for Image-Text Matching

Visual-Semantic Matching by Exploring High-Order Attention and Distraction

Decoupled Cross-Modal Phrase-Attention Network for Image-Sentence Matching