Abstract:The essence of improving the effect of cross-modal image–text retrieval (CIR) lies in the finer-grained modeling of homogeneous features between modalities. However, in remote sensing (RS) scenarios, existing methods usually apply the image–sentence granular feature alignment paradigm, bringing significant difficulties to the fine-grained representation of homogeneous features between modalities. Besides, more complex background noise and extreme scale ranges of foreground targets are hard to distinguish, causing the feature mottle problem. To address the above issues, we propose a novel Semantic-guided Image–text Retrieval framework with Segmentation (SIRS). It is a multitask joint learning framework for plug-and-play and end-to-end training RS CIR models efficiently, including semantic-guided spatial attention (SSA) and adaptive multiscale weighting (AMW) modules. First, SSA introduces a background reconstruction (BR) branch based on noise perception and a semantic segmentation (SS) branch based on pixel-level prediction. It explores a joint learning strategy that concisely filters background noise and refines foreground features considerably. Second, AMW performs multiscale weighting on various layers of feature map output by the encoder, effectively improving the learning efficiency of foreground targets at different scales. It is worth mentioning that SIRS outputs combination results with image and segmentation mask, which is not available in other methods. Based on the RSITMD dataset, we complete the SS annotation RSITMD-SS to verify the performance of the proposed method. Sufficient and complete experiments verify the effectiveness of the proposed method. With SIRS, the mainstream SVP and CLIP-based methods improve about 7 mR and derive segmentation prediction with acceptable computational cost optionally. The code and associated dataset will be available at https://github.com/StarBurstStream0/SIRS.

Dual Modality Collaborative Learning for Cross-Source Remote Sensing Retrieval

A Deep Cross-Modality Hashing Network for SAR and Optical Remote Sensing Images Retrieval

MCRN: A Multi-source Cross-modal Retrieval Network for Remote Sensing

A Discriminative Distillation Network for Cross-Source Remote Sensing Image Retrieval

Cross-Source Image Retrieval Based on Ensemble Learning and Knowledge Distillation for Remote Sensing Images.

Exploring Uni-Modal Feature Learning on Entities and Relations for Remote Sensing Cross-Modal Text-Image Retrieval

SIRS: Multi-task Joint Learning for Remote Sensing Foreground-entity Image-text Retrieval

SIRS: Multitask Joint Learning for Remote Sensing Foreground-Entity Image–Text Retrieval

A Lightweight Multi-Scale Crossmodal Text-Image Retrieval Method in Remote Sensing

Integrating Multisubspace Joint Learning With Multilevel Guidance for Cross-Modal Retrieval of Remote Sensing Images

HCMSL: Hybrid Cross-modal Similarity Learning for Cross-modal Retrieval

A Jointly Guided Deep Network for Fine-Grained Cross-Modal Remote Sensing Text–Image Retrieval

Integrating Multi-subspace Joint Learning with Multi-level Guidance for Cross-Modal Retrieval of Remote Sensing Images

HGR Maximal Correlation Augmented Cross-Modal Remote Sensing Retrieval

Supervised Contrastive Learning Based on Fusion of Global and Local Features for Remote Sensing Image Retrieval

Robust Cross-Modal Remote Sensing Image Retrieval Via Maximal Correlation Augmentation

CMR-net: A cross modality reconstruction network for multi-modality remote sensing classification

Coordinated and Specific Restricted Boltzmann Machine for Cross-Modal Retrieval

A Collaborative Correlation-Matching Network for Multimodality Remote Sensing Image Classification.

Hybrid DAER Based Cross-modal Retrieval Exploiting Deep Representation Learning

Cross-Modal Contrastive Learning for Remote Sensing Image Classification.