Abstract:Cross-modal remote sensing image-text retrieval (CMRSITR) aims to extract comprehensive information from diverse modalities. The primary challenge in this field is developing effective mappings between visual and textual modalities to a shared latent space. Existing approaches generally focus on utilizing pretrained unimodal models to independently extract features from each modality. However, these techniques often fall short of achieving the critical alignment necessary for effective cross-modal matching. These techniques predominantly concentrate on the extraction of features and alignment at an instance level, suggesting potential areas for enhancement. To address these limitations, we introduce the masked interaction inferring and aligning (MIIA) framework, utilizing dynamic contrastive learning (DCL). This framework is adept at discerning the intricate relationships between local visual-textual tokens, thereby significantly bolstering the congruence of global image-text pairings without relying on additional prior supervision. Initially, we devise a masked interaction inferring (MII) module, which fosters token-level interplays through a novel masked visual-language (VL) modeling approach. Following this, we implement a cross-modal DCL mechanism, which is instrumental in capturing and aligning semantic correlations between images and texts more effectively. Finally, to ensure the comprehensive matching of visual and textual embeddings, we introduce a unique technique known as bidirectional distribution matching (BDM). This method is designed to minimize the Kullback-Leibler (KL) divergence between the distributions of image-text similarity, computed using the negative queues in momentum contrast learning. Comprehensive experiments performed on well-established public datasets consistently validate the state-of-the-art performance of MIIA methods in the CMRSITR task.

Iterative Uni-modal and Cross-modal Clustered Contrastive Learning for Image-text Retrieval

Cross-modal Contrastive Learning for Generalizable and Efficient Image-text Retrieval

Efficient Token-Guided Image-Text Retrieval With Consistent Multimodal Contrastive Training

Dual-Level Cross-Modal Contrastive Clustering

CODER: Coupled Diversity-Sensitive Momentum Contrastive Learning for Image-Text Retrieval

Contrastive Cross-Modal Knowledge Sharing Pre-training for Vision-Language Representation Learning and Retrieval

Improving Cross-Modal Image-Text Retrieval With Teacher-Student Learning

Image–Text Matching Model Based on CLIP Bimodal Encoding

COTS: Collaborative Two-Stream Vision-Language Pre-Training Model for Cross-Modal Retrieval

Image–Text Cross-Modal Retrieval with Instance Contrastive Embedding

Clustering swap prediction for image-text pre-training

GrowCLIP: Data-aware Automatic Model Growing for Large-scale Contrastive Language-Image Pre-training

COOKIE: Contrastive Cross-Modal Knowledge Sharing Pre-training for Vision-Language Representation

AsCL: An Asymmetry-sensitive Contrastive Learning Method for Image-Text Retrieval with Cross-Modal Fusion

X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrieval

Unified Contrastive Learning in Image-Text-Label Space

Vision-Language Pre-Training with Triple Contrastive Learning

Dual-Modal Attention-Enhanced Text-Video Retrieval with Triplet Partial Margin Contrastive Learning

ERNIE-ViL 2.0: Multi-view Contrastive Learning for Image-Text Pre-training

Masking-Based Cross-Modal Remote Sensing Image–Text Retrieval via Dynamic Contrastive Learning

VT-CLIP: Enhancing Vision-Language Models with Visual-guided Texts