Abstract:The essence of improving the effect of cross-modal image–text retrieval (CIR) lies in the finer-grained modeling of homogeneous features between modalities. However, in remote sensing (RS) scenarios, existing methods usually apply the image–sentence granular feature alignment paradigm, bringing significant difficulties to the fine-grained representation of homogeneous features between modalities. Besides, more complex background noise and extreme scale ranges of foreground targets are hard to distinguish, causing the feature mottle problem. To address the above issues, we propose a novel Semantic-guided Image–text Retrieval framework with Segmentation (SIRS). It is a multitask joint learning framework for plug-and-play and end-to-end training RS CIR models efficiently, including semantic-guided spatial attention (SSA) and adaptive multiscale weighting (AMW) modules. First, SSA introduces a background reconstruction (BR) branch based on noise perception and a semantic segmentation (SS) branch based on pixel-level prediction. It explores a joint learning strategy that concisely filters background noise and refines foreground features considerably. Second, AMW performs multiscale weighting on various layers of feature map output by the encoder, effectively improving the learning efficiency of foreground targets at different scales. It is worth mentioning that SIRS outputs combination results with image and segmentation mask, which is not available in other methods. Based on the RSITMD dataset, we complete the SS annotation RSITMD-SS to verify the performance of the proposed method. Sufficient and complete experiments verify the effectiveness of the proposed method. With SIRS, the mainstream SVP and CLIP-based methods improve about 7 mR and derive segmentation prediction with acceptable computational cost optionally. The code and associated dataset will be available at https://github.com/StarBurstStream0/SIRS.

BoSR: A CNN-based aurora image retrieval method.

A lightweight and stochastic depth residual attention network for remote sensing scene classification

Parallel-Connected Residual Channel Attention Network for Remote Sensing Image Super-Resolution

RSCNN: A CNN-Based Method to Enhance Low-Light Remote-Sensing Images

An integrated aurora image retrieval system: AuroraEye

Learning Low Dimensional Convolutional Neural Networks for High-Resolution Remote Sensing Image Retrieval

Retrieval Across Optical and SAR Images with Deep Neural Network.

A Learnable Blur Kernel For Remote Sensing Image Retrieval

Fast Object Retrieval Using Direct Spatial Matching

Orientation-Aware Deep Neural Network for Real Image Super-Resolution.

CRNet: Channel-Enhanced Remodeling-Based Network for Salient Object Detection in Optical Remote Sensing Images

SGR: An Improved Point-Based Method for Remote Sensing Object Detection via Dual-Domain Alignment Saliency-Guided RepPoints

ASNet: Adaptive Semantic Network Based on Transformer–CNN for Salient Object Detection in Optical Remote Sensing Images

A Remote-Sensing Image-Retrieval Model Based On An Ensemble Neural Networks

A Transformer and Visual Foundation Model-Based Method for Cross-View Remote Sensing Image Retrieval

A Novel Region-Based Image Registration Method for Multisource Remote Sensing Images Via CNN

High-Resolution Remote Sensing Image Retrieval Based on CNNs from a Dimensional Perspective.

Spatial–Channel Attention Transformer With Pseudo Regions for Remote Sensing Image-Text Retrieval

SIRS: Multitask Joint Learning for Remote Sensing Foreground-Entity Image–Text Retrieval

Non-Locally up-Down Convolutional Attention Network for Remote Sensing Image Super-Resolution

A Lightweight Multi-Scale Crossmodal Text-Image Retrieval Method in Remote Sensing