Abstract:Vision-language models have demonstrated impressive capabilities in associating images and text by pretraining on extensive image-text paired data. The paradigm of continual pretraining followed by fine-tuning has become prevailing for boosting performance in domain-specific tasks under constrained computation resources. Benefiting from the superior generalization abilities of foundation models, the demands for computational resources and extensive data corpora have been significantly reduced. Nonetheless, it is crucial to tailor the model for the characteristics of downstream tasks to mitigate the misalignment between the pretraining pretext tasks and actual applications of interest. In this study, we utilize a CLIP-based model that has been continually pretrained on the 5 million image-text dataset in the remote sensing field as the foundation model, focusing on cross-modal image-text retrieval tasks. We introduce an efficient framework called remote sensing image-text retrieval fine-grained fine-tuning (RSITR-FFT), which refines the feature space by introducing fine-grained word-region alignment and incorporating consistency constraint regularization terms in the learning objectives. The fine-grained alignment aims for precise word-region correspondence beyond classical global-level image-text matching, while the consistency regularization encourages geometric coherence between the image and text modalities. Remarkably, our method achieves observable performance improvements while requiring far fewer fine-tuning samples—about 10 000, in contrast to the 400 million and 5 million samples used during the CLIP's initial pretraining and GeoRSCLIP's continual pretraining stages, respectively. We perform quantitative evaluation on RSICD, NWPU-Captions, and UCM-Captions datasets to demonstrate the effectiveness of RSITR-FFT. We further showcase its realistic application on the high-resolution remote sensing imagery through the qualitative visualization experiments on the FAIR1M-1.0 dataset. The code and models are available at https://github.com/d1x1u/RSITR-FFT.

Text-Guided Knowledge Transfer for Remote Sensing Image-Text Retrieval

Knowledge-aware Text-Image Retrieval for Remote Sensing Images

Knowledge-Aided Momentum Contrastive Learning for Remote-Sensing Image Text Retrieval

Visual Global-Salient-Guided Network for Remote Sensing Image-Text Retrieval

Fine-Grained Information Supplementation and Value-Guided Learning for Remote Sensing Image-Text Retrieval

CLIP-based Synergistic Knowledge Transfer for Text-based Person Retrieval

Remote Sensing Cross-Modal Text-Image Retrieval Based on Global and Local Information

Spatial–Channel Attention Transformer With Pseudo Regions for Remote Sensing Image-Text Retrieval

PIR: Remote Sensing Image-Text Retrieval with Prior Instruction Representation Learning

A Fusion Encoder with Multi-Task Guidance for Cross-Modal Text–Image Retrieval in Remote Sensing

Knowledge Transfer and Domain Adaptation for Fine-Grained Remote Sensing Image Segmentation

TextRS: Deep Bidirectional Triplet Network for Matching Text to Remote Sensing Images

A Lightweight Multi-Scale Crossmodal Text-Image Retrieval Method in Remote Sensing

SIRS: Multitask Joint Learning for Remote Sensing Foreground-Entity Image–Text Retrieval

GIS Semantics-Based Approach of Remote Sensing Image Retrieval

Exploring Text-Guided Single Image Editing for Remote Sensing Images

A Deep Semantic Alignment Network for the Cross-Modal Image-Text Retrieval in Remote Sensing

Composed Image Retrieval for Remote Sensing

RSITR-FFT: Efficient Fine-Grained Fine-Tuning Framework With Consistency Regularization for Remote Sensing Image-Text Retrieval

Cross-Modal Pre-Aligned Method with Global and Local Information for Remote-Sensing Image and Text Retrieval