Abstract:As a fundamental topic in bridging the gap between vision and language, cross-modal retrieval purposes to obtain the correspondences' relationship between fragments, i.e., subregions in images and words in texts. Compared with earlier methods that focus on learning the visual semantic embedding from images and sentences to the shared embedding space, the existing methods tend to learn the correspondences between words and regions via cross-modal attention. However, such attention-based approaches invariably result in semantic misalignment between subfragments for two reasons: 1) without modeling the relationship between subfragments and the semantics of the entire images or sentences, it will be hard for such approaches to distinguish images or sentences with multiple same semantic fragments and 2) such approaches focus attention evenly on all subfragments, including nonvisual words and a lot of redundant regions, which also will face the problem of semantic misalignment. To solve these problems, this article proposes a bidirectional correct attention network (BCAN), which introduces a novel concept of the relevance between subfragments and the semantics of the entire images or sentences and designs a novel correct attention mechanism by modeling the local and global similarity between images and sentences to correct the attention weights focused on the wrong fragments. Specifically, we introduce a concept about the semantic relationship between subfragments and entire images or sentences and use this concept to solve the semantic misalignment from two aspects. In our correct attention mechanism, we design two independent units to correct the weight of attention focused on the wrong fragments. Global correct unit (GCU) with modeling the global similarity between images and sentences into the attention mechanism to solve the semantic misalignment problem caused by focusing attention on relevant subfragments in irrelevant pairs (RI) and the local correct unit (LCU) consider the difference in the attention weights between fragments among two steps to solve the semantic misalignment problem caused by focusing attention on irrelevant subfragments in relevant pairs (IR). Extensive experiments on large-scale MS-COCO and Flickr30K show that our proposed method outperforms all the attention-based methods and is competitive to the state-of-the-art. Our code and pretrained model are publicly available at: https://github.com/liuyyy111/BCAN.

Multi-step Self-attention Network for Cross-modal Retrieval Based on a Limited Text Space.

Cross‐modal retrieval with dual multi‐angle self‐attention

Semantic enhancement and multi-level alignment network for cross-modal retrieval

Modality-Specific Cross-Modal Similarity Measurement With Recurrent Attention Network

Context-Aware Attention Network for Image-Text Retrieval

A Deep Semantic Alignment Network for the Cross-Modal Image-Text Retrieval in Remote Sensing

Scene-text aware cross-modal retrieval based on semantic matching (ChinaMM2024)

BCAN: Bidirectional Correct Attention Network for Cross-Modal Retrieval

Unsupervised hash retrieval based on multiple similarity matrices and text self-attention mechanism

Adversarial Cross-Modal Retrieval

Multi-task hierarchical convolutional network for visual-semantic cross-modal retrieval

CSMA-CNER:Multi-modal Chinese NER Task with Cross- and Self-Modality Attention

Deep Supervised Dual Cycle Adversarial Network for Cross-Modal Retrieval

A Lightweight Multi-Scale Crossmodal Text-Image Retrieval Method in Remote Sensing

TCAN: Text-oriented Cross Attention Network for Multimodal Sentiment Analysis

Cross-Graph Attention Enhanced Multi-Modal Correlation Learning for Fine-Grained Image-Text Retrieval

Cross-Modal Learning Based on Semantic Correlation and Multi-Task Learning for Text-Video Retrieval

SCA-PVNet: Self-and-Cross Attention Based Aggregation of Point Cloud and Multi-View for 3D Object Retrieval

Cross-modal retrieval by an end to end way

Multi-Scale Context Attention Network for Image Retrieval.

Boosting cross-modal retrieval in remote sensing via a novel unified attention network