Abstract:Remote sensing (RS) image-text retrieval (RSITR) aims to retrieve relevant texts (RS images) based on the content of a given RS image (text). Existing methods are used to employing the convolutional neural network (CNN) and recurrent neural network (RNN) as encoders to learn visual and textual features for retrieval. Although feasible, the global information hidden in different data does not receive the attention it deserves. To mitigate this problem, transformers have been introduced. Nevertheless, the complexity of RS images present challenges in directly introducing Transformer-based architectures to multimodal learning in RS scenes, particularly in visual feature extraction and cross-modal interaction. In addition, the textual captions are always simpler than the complex RS images, leading to a semantic description appearing in different images. This typical false-negative (FN) sample problem increases the difficulty of RSITR tasks. To address the above limitations, we propose a new RSITR model named prior-experience-based RS vision-language (PERSVL). First, the specific visual and text encoders are used to extract features from RS images and texts. Also, a high-level feature complement (HFC) module is developed based on the self-attention mechanism (SAM) for the visual encoder to explore the complex contents from RS images fully. Second, a dual-branch multimodal fusion encoder (DBMFE) is designed to complete the cross-modal learning. It comprises a dual-branch multimodal interaction (DBMI) module and a branch fusion module. DBMI is designed to fully explore the relationships between different modalities, enriching visual and textual features. The branch fusion module integrates the cross-modal features and utilizes a classification head to generate matching scores for retrieval. Finally, a learning from prior experiences (LPEs) module is designed to reduce the influence of FN samples by analyzing the historical data produced in the model training process. Experiments are conducted on three popular datasets, and the positive results show that our PERSVL model achieves superior performance compared with previous methods. By integrating the advantages of natural language and RS images, our PERSVL can be applied in various applications, such as environmental monitoring, disaster evaluation, and urban planning. Our source codes are available at: https://github.com/TangXu-Group/Cross-modal-remote-sensing-image-and-text-retrieval-models/tree/main/PERSVL.

PECR:Parameter-Efficient Transfer Learning with Cross-Modal Representation Learning for Remote Sensing Visual Question Answering

PERS: Parameter-Efficient Multimodal Transfer Learning for Remote Sensing Visual Question Answering

RSAdapter: Adapting Multimodal Models for Remote Sensing Visual Question Answering

Overcoming Language Priors In Vqa Via Decomposed Linguistic Representations

Multi-Modal Fusion Transformer for Visual Question Answering in Remote Sensing

LiT-4-RSVQA: Lightweight Transformer-based Visual Question Answering in Remote Sensing

Parameter-Efficient Transfer Learning for Remote Sensing Image-Text Retrieval

RSMoDM: Multimodal Momentum Distillation Model for Remote Sensing Visual Question Answering

Prior-Experience-Based Vision-Language Model for Remote Sensing Image-Text Retrieval

Parameter-Efficient Transfer Learning for Medical Visual Question Answering

Enhancing Remote Sensing Visual Question Answering: A Mask-Based Dual-Stream Feature Mutual Attention Network

Pre-training Model Based on Parallel Cross-Modality Fusion Layer

Text-Guided Coarse-to-Fine Fusion Network for Robust Remote Sensing Visual Question Answering

Large Vision-Language Models for Remote Sensing Visual Question Answering

A multi-scale contextual attention network for remote sensing visual question answering

Parameter-Efficient Cross-lingual Transfer of Vision and Language Models via Translation-based Alignment

Visual Question Answering in Remote Sensing with Cross-Attention and Multimodal Information Bottleneck

Context-aware Multi-level Question Embedding Fusion for visual question answering

Segmentation-guided Attention for Visual Question Answering from Remote Sensing Images

Attribute-Prompting Multi-Modal Object Reasoning Transformer for Remote Sensing Visual Grounding