Abstract:The essence of improving the effect of cross-modal image-text retrieval (CIR) lies in the finer-grained modeling of homogeneous features between modalities. However, in remote sensing (RS) scenarios, existing methods usually apply the image-sentence granular feature alignment paradigm, bringing significant difficulties to the fine-grained representation of homogeneous features between modalities. Besides, more complex background noise and extreme scale ranges of foreground targets are hard to distinguish, causing the feature mottle problem. To address the above issues, we propose a novel Semantic-guided Image-text Retrieval framework with Segmentation (SIRS). It is a multitask joint learning framework for plug-and-play and end-to-end training RS CIR models efficiently, including semantic-guided spatial attention (SSA) and adaptive multi-scale weighting (AMW) modules. First, SSA introduces a background reconstruction (BR) branch based on noise perception and a semantic segmentation (SS) branch based on pixel-level prediction. It explores a joint learning strategy that concisely filters background noise and refines foreground features considerably. Second, AMW performs multiscale weighting on various layers of feature map output by the encoder, effectively improving the learning efficiency of foreground targets at different scales. It is worth mentioning that SIRS outputs combination results with image and segmentation mask, which is not available in other methods. Based on the RSITMD dataset, we complete the SS annotation RSITMD-SS to verify the performance of the proposed method. Sufficient and complete experiments verify the effectiveness of the proposed method. With SIRS, the main-stream SVP and CLIP-based methods improve about 7 mR and derive segmentation prediction with acceptable computational cost optionally. The code and associated dataset will be available at https://github.com/StarBurstStream0/SIRS.

A Discriminative Distillation Network for Cross-Source Remote Sensing Image Retrieval

Cross-Source Image Retrieval Based on Ensemble Learning and Knowledge Distillation for Remote Sensing Images.

Dual Modality Collaborative Learning for Cross-Source Remote Sensing Retrieval

A Discriminative Feature Learning Approach with Distinguishable Distance Metrics for Remote Sensing Image Classification and Retrieval

Learning Source-Invariant Deep Hashing Convolutional Neural Networks for Cross-Source Remote Sensing Image Retrieval.

A Deep Cross-Modality Hashing Network for SAR and Optical Remote Sensing Images Retrieval

A Jointly Guided Deep Network for Fine-Grained Cross-Modal Remote Sensing Text–Image Retrieval

Cross-view Matching Neural Network for Remote Sensing Images.

Supervised Contrastive Learning Based on Fusion of Global and Local Features for Remote Sensing Image Retrieval

A Lightweight Multi-Scale Crossmodal Text-Image Retrieval Method in Remote Sensing

MCRN: A Multi-source Cross-modal Retrieval Network for Remote Sensing

Wide-Context Attention Network for Remote Sensing Image Retrieval.

Intra- and Inter-source Interactive Representation Learning Network for Remote Sensing Images Classification

Proxy-Based Rotation Invariant Deep Metric Learning for Remote Sensing Image Retrieval

Consistency- and dependence-guided knowledge distillation for object detection in remote sensing images

Intra- and Intersource Interactive Representation Learning Network for Remote Sensing Images Classification

Duplex-Hierarchy Representation Learning for Remote Sensing Image Classification

Distribution-Independent Domain Generalization for Multisource Remote Sensing Classification

Dual Contrastive Network for Few-Shot Remote Sensing Image Scene Classification

Adaptive Discriminative Regions Learning Network for Remote Sensing Scene Classification

SIRS: Multi-task Joint Learning for Remote Sensing Foreground-entity Image-text Retrieval