Abstract:The essence of improving the effect of cross-modal image–text retrieval (CIR) lies in the finer-grained modeling of homogeneous features between modalities. However, in remote sensing (RS) scenarios, existing methods usually apply the image–sentence granular feature alignment paradigm, bringing significant difficulties to the fine-grained representation of homogeneous features between modalities. Besides, more complex background noise and extreme scale ranges of foreground targets are hard to distinguish, causing the feature mottle problem. To address the above issues, we propose a novel Semantic-guided Image–text Retrieval framework with Segmentation (SIRS). It is a multitask joint learning framework for plug-and-play and end-to-end training RS CIR models efficiently, including semantic-guided spatial attention (SSA) and adaptive multiscale weighting (AMW) modules. First, SSA introduces a background reconstruction (BR) branch based on noise perception and a semantic segmentation (SS) branch based on pixel-level prediction. It explores a joint learning strategy that concisely filters background noise and refines foreground features considerably. Second, AMW performs multiscale weighting on various layers of feature map output by the encoder, effectively improving the learning efficiency of foreground targets at different scales. It is worth mentioning that SIRS outputs combination results with image and segmentation mask, which is not available in other methods. Based on the RSITMD dataset, we complete the SS annotation RSITMD-SS to verify the performance of the proposed method. Sufficient and complete experiments verify the effectiveness of the proposed method. With SIRS, the mainstream SVP and CLIP-based methods improve about 7 mR and derive segmentation prediction with acceptable computational cost optionally. The code and associated dataset will be available at https://github.com/StarBurstStream0/SIRS.

Cross-Source Image Retrieval Based on Ensemble Learning and Knowledge Distillation for Remote Sensing Images.

A Discriminative Distillation Network for Cross-Source Remote Sensing Image Retrieval

Dual Modality Collaborative Learning for Cross-Source Remote Sensing Retrieval

Learning Source-Invariant Deep Hashing Convolutional Neural Networks for Cross-Source Remote Sensing Image Retrieval.

Cross-view Matching Neural Network for Remote Sensing Images.

Content-Based High-Resolution Remote Sensing Image Retrieval Via Unsupervised Feature Learning And Collaborative Affinity Metric Fusion

Intra- and Inter-source Interactive Representation Learning Network for Remote Sensing Images Classification

Feature Selection for Cross-Scene Hyperspectral Image Classification Using Cross-Domain ReliefF

Intra- and Intersource Interactive Representation Learning Network for Remote Sensing Images Classification

A Lightweight Multi-Scale Crossmodal Text-Image Retrieval Method in Remote Sensing

Delving into Deep Representations for Remote Sensing Image Retrieval

MCRN: A Multi-source Cross-modal Retrieval Network for Remote Sensing

Supervised Contrastive Learning Based on Fusion of Global and Local Features for Remote Sensing Image Retrieval

A Deep Cross-Modality Hashing Network for SAR and Optical Remote Sensing Images Retrieval

SIRS: Multi-task Joint Learning for Remote Sensing Foreground-entity Image-text Retrieval

Cross-Dimension Attention Guided Self-Supervised Remote Sensing Single-Image Super-Resolution

On Gleaning Knowledge from Cross Domains by Sparse Subspace Correlation Analysis for Hyperspectral Image Classification.

A Fast and Accurate Method for Remote Sensing Image-Text Retrieval Based on Large Model Knowledge Distillation

Learning Low Dimensional Convolutional Neural Networks for High-Resolution Remote Sensing Image Retrieval

Multi-Source Remote Sensing Data Cross Scene Classification Based on Multi-Graph Matching

SIRS: Multitask Joint Learning for Remote Sensing Foreground-Entity Image–Text Retrieval