Abstract:Cross-modal image-text matching has attracted considerable interest in both computer vision and natural language processing communities. The main issue of image-text matching is to learn the compact cross-modal representations and the correlation between image and text representations. However, the image-text matching task has two major challenges. First, the current image representation methods focus on the semantic information and disregard the spatial position relations between image regions. Second, most existing methods pay little attention to improving textual representation which plays a significant role in image-text matching. To address these issues, we designed a decipherable cross-modal multi-relationship aware reasoning network (CMRN) for image-text matching. In particular, a new method is proposed to extract multi-relationship and to learn the correlations between image regions, including two kinds of visual relations: the geometric position relation and semantic interaction. In addition, images are processed as graphs, and a novel spatial relation encoder is introduced to perform reasoning on the graphs by employing a graph convolutional network (GCN) with attention mechanism. Thereafter, a contextual text encoder based on Bidirectional Encoder Representations from Transformers is adopted to learn distinctive textual representations. To verify the effectiveness of the proposed model, extensive experiments were conducted on two public datasets, namely MSCOCO and Flickr30K. The experimental results show that CMRN achieved superior performance when compared with state-of-the-art methods. On Flickr30K, the proposed method outperforms state-of-the-art methods more than 7.4% in text retrieval from image query, and 5.0% relatively in image retrieval with text query (based on Recall@1). On MSCOCO, the performance reaches 73.9% for text retrieval and 60.4% for image retrieval (based on Recall@1).

Multi-view and region reasoning semantic enhancement for image-text retrieval

Image-Text Embedding Learning Via Visual and Textual Semantic Reasoning.

Visual Semantic Reasoning for Image-Text Matching

Multilateral Semantic Relations Modeling for Image Text Retrieval

Visual and textual fusion for semantically supervised region-based retrieval

Exploiting Visual Semantic Reasoning for Video-Text Retrieval

Semantic enhancement and multi-level alignment network for cross-modal retrieval

Visual & textual fusion for region retrieval: from both fuzzy matching and bayesian reasoning aspects.

Context‐aware relation enhancement and similarity reasoning for image‐text retrieval

Commonsense-Guided Semantic and Relational Consistencies for Image-Text Retrieval

Semantic Completion: Enhancing Image-Text Retrieval with Information Extraction and Compression

RFE-SRN: Image-text similarity reasoning network based on regional feature enhancement

Semantic Completion and Filtration for Image–Text Retrieval

Scene-text aware cross-modal retrieval based on semantic matching (ChinaMM2024)

Semantic Sensitive Region Retrieval Using Keyword-Integrated Bayesian Reasoning

Cross-modal Multi-Relationship Aware Reasoning for Image-Text Matching

Joint Intra & Inter-Grained Reasoning: A New Look into Semantic Consistency of Image-Text Retrieval

Multi-Modal Reasoning Graph for Scene-Text Based Fine-Grained Image Classification and Retrieval

Using Visual Dictionary to Associate Semantic Objects in Region-Based Image Retrieval

Feature First: Advancing Image-Text Retrieval Through Improved Visual Features

Context-Aware Multi-View Summarization Network for Image-Text Matching