Abstract:Image and text retrieval is a crucial topic in the fields of language and vision. The key to successful Image-Text retrieval is achieving accurate cross-modal representation and capturing essential correlations between image-sentence or words-regions. While existing work has designed intricate interactions to capture these correlations, challenges remain due to inadequate feature representations, such as insufficient text descriptions of image and ambiguous region representations. To address these challenges, we propose a novel approach, multi-view and region reasoning semantic enhancement, for image and text retrieval, which aims to enhance the semantic representation of features from both textual and visual modalities. Specifically, considering that an image can have multiple corresponding texts from different perspectives, with each text describing a single view, we devise a multi-view textual semantic enhancement module. This module takes advantage of the positive textual cues provided by corresponding image to make up for the limited knowledge in single-text views and produce a comprehensive image-based textual representation. Then, to address the semantic diversity of an image, we design a region reasoning semantic enhancement module that employs a graph structure to integrate both semantic and spatial reasoning knowledge from different regions, thereby clarifying the semantics of image regions and enhancing the overall semantic understanding of these areas. Extensive experiments and analyses demonstrate the superior performance of the proposed method on the Flickr30K and MSCOCO datasets, validating the effectiveness of the proposed solution.

Image Location Inference by Multi-Saliency Enhancement

Efficient Classification Using Salient Regions

Image Location Estimation by Salient Region Matching

Color boosted visual saliency detection and its application to image classification

Fast Image Enhancement Using Multi-Scale Saliency Extraction in Infrared Imagery

Detect saliency to understand a photo

Region of Interests Extraction Based on Visual Saliency for Remote Sensing Image

How Important is Location Information in Saliency Detection of Natural Images

How Important is Location in Saliency Detection

Fine-Grained Image Classification Via Spatial Saliency Extraction.

Salient Object Extraction Based on Region Saliency Ratio

Region of Interest Extraction Via Common Salient Feature Analysis and Feedback Reinforcement Strategy for Remote Sensing Images

Common Regions of Interest Extraction Based on Saliency Statistic Analysis for Multiple Remote Sensing Images.

Multi-view and region reasoning semantic enhancement for image-text retrieval

Salient region detection : Integrate both global and local cues

Salient Object Detection Via Google Image Retrieval

Region-of-Interest Detection via Superpixel-to-Pixel Saliency Analysis for Remote Sensing Image

Visual saliency detection by integrating spatial position prior of object with background cues

Img2Loc: Revisiting Image Geolocalization using Multi-modality Foundation Models and Image-based Retrieval-Augmented Generation

Location Sensitive Image Retrieval and Tagging

Multi-Camera Saliency.