Abstract:In recent years, there has been a growing interest in remote sensing image–text cross-modal retrieval due to the rapid development of space information technology and the significant increase in the volume of remote sensing image data. Remote sensing images have unique characteristics that make the cross-modal retrieval task challenging. Firstly, the semantics of remote sensing images are fine-grained, meaning they can be divided into multiple basic units of semantic expression. Different combinations of basic units of semantic expression can generate diverse text descriptions. Additionally, these images exhibit variations in resolution, color, and perspective. To address these challenges, this paper proposes a multi-task guided fusion encoder (MTGFE) based on the multimodal fusion encoding method, the progressiveness of which has been proved in the cross-modal retrieval of natural images. By jointly training the model with three tasks: image–text matching (ITM), masked language modeling (MLM), and the newly introduced multi-view joint representations contrast (MVJRC), we enhance its capability to capture fine-grained correlations between remote sensing images and texts. Specifically, the MVJRC task is designed to improve the model's consistency in joint representation expression and fine-grained correlation, particularly for remote sensing images with significant differences in resolution, color, and angle. Furthermore, to address the computational complexity associated with large-scale fusion models and improve retrieval efficiency, this paper proposes a retrieval filtering method, which achieves higher retrieval efficiency while minimizing accuracy loss. Extensive experiments were conducted on four public datasets to evaluate the proposed method, and the results validate its effectiveness.

Enhancing the RAG Retrieval Engine Through Multi-Encoder Fusion

Fine-Grained Cross-Modal Retrieval with Triple-Streamed Memory Fusion Transformer Encoder

Blended RAG: Improving RAG (Retriever-Augmented Generation) Accuracy with Semantic Search and Hybrid Query-Based Retrievers

Improving Natural Language Understanding with Computation-Efficient Retrieval Augmentation

CoFE-RAG: A Comprehensive Full-chain Evaluation Framework for Retrieval-Augmented Generation with Enhanced Data Diversity

A Fusion Encoder with Multi-Task Guidance for Cross-Modal Text–Image Retrieval in Remote Sensing

Multi-channel Encoder for Neural Machine Translation

Multi-Head RAG: Solving Multi-Aspect Problems with LLMs

RAG-Fusion: a New Take on Retrieval-Augmented Generation

LongRAG: Enhancing Retrieval-Augmented Generation with Long-context LLMs

M-RAG: Reinforcing Large Language Model Performance through Retrieval-Augmented Generation with Multiple Partitions

GNN-encoder: Learning a Dual-encoder Architecture Via Graph Neural Networks for Dense Passage Retrieval

R^2AG: Incorporating Retrieval Information into Retrieval Augmented Generation

Don't Forget to Connect! Improving RAG with Graph-based Reranking

Multi-grained Encoding and Joint Embedding Space Fusion for Video and Text Cross-Modal Retrieval

MBA-RAG: a Bandit Approach for Adaptive Retrieval-Augmented Generation through Question Complexity

Towards Optimizing a Retrieval Augmented Generation using Large Language Model on Academic Data

IM-RAG: Multi-Round Retrieval-Augmented Generation Through Learning Inner Monologues

Toward Optimal Search and Retrieval for RAG

MemoRAG: Moving towards Next-Gen RAG Via Memory-Inspired Knowledge Discovery