Abstract:A novel context‐aware relation enhancement and similarity reasoning model is proposed to achieve precise image‐text retrieval, which conducts both intra‐modal relation enhancement and inter‐modal similarity reasoning while considering the global‐context information. Image‐text retrieval is a fundamental yet challenging task, which aims to bridge a semantic gap between heterogeneous data to achieve precise measurements of semantic similarity. The technique of fine‐grained alignment between cross‐modal features plays a key role in various successful methods that have been proposed. Nevertheless, existing methods cannot effectively utilise intra‐modal information to enhance feature representation and lack powerful similarity reasoning to get a precise similarity score. Intending to tackle these issues, a context‐aware Relation Enhancement and Similarity Reasoning model, called RESR, is proposed, which conducts both intra‐modal relation enhancement and inter‐modal similarity reasoning while considering the global‐context information. For intra‐modal relation enhancement, a novel context‐aware graph convolutional network is introduced to enhance local feature representations by utilising relation and global‐context information. For inter‐modal similarity reasoning, local and global similarity features are exploited by the bidirectional alignment of image and text, and the similarity reasoning is implemented among multi‐granularity similarity features. Finally, refined local and global similarity features are adaptively fused to get a precise similarity score. The experimental results show that our effective model outperforms some state‐of‐the‐art approaches, achieving average improvements of 2.5% and 6.3% in R@sum on the Flickr30K and MS‐COCO dataset.

Bridging Semantic Gaps in Information Retrieval : Context-Based Approaches

Research on high-level semantic image retrieval

An Overview of Semantics Processing in Content-Based 3D Model Retrieval

Bridging the Semantic Gap Between Image Contents and Tags

Data-Driven Approach for Bridging the Cognitive Gap in Image Retrieval

Bridging the Gap between Local Semantic Concepts and Bag of Visual Words for Natural Scene Image Retrieval

Image Retrieval Based on Fuzzy Semantic Relevance Matrix

Modeling Image Data for Effective Indexing and Retrieval in Large General Image Databases.

Content-based Image Retrieval and the Semantic Gap in the Deep Learning Era

GIS Semantics-Based Approach of Remote Sensing Image Retrieval

Semantic Context Transfer Across Heterogeneous Sources for Domain Adaptive Video Search.

AN APPROACH TO SEMANTICS-BASED IMAGE RETRIEVAL AND BROWSING

ImageSaker : A Semantic-based Image Retrieval System Refining with Concept Model

Using Context to Improve the Evaluation of Information Retrieval Systems

A survey of content-based image retrieval with high-level semantics

Object-Based Image Retrieval Using Spatial Context

Context‐aware relation enhancement and similarity reasoning for image‐text retrieval

Exploiting Multi-Context Analysis in Semantic Image Classification

A Semantics-based Information Retrieval Model

A framework for contextual information retrieval from the WWW

Exploring Inter-Concept Relationship with Context Space for Semantic Video Indexing