Abstract:The coordinated utilization of diverse types of satellite sensors provides a more comprehensive view of the Earth’s surface. However, due to the significant heterogeneity across modalities and the scarcity of high-quality labels, most existing methods face bottlenecks in the underutilization of massive unlabeled multimodal satellite data, making it challenging to understand the scene comprehensively. To this end, we propose a multi-modal unified representation learning framework (MURLF) based on masked image modeling for remote sensing images, aiming to make better use of massive unlabeled multi-modal remote sensing data. MURLF leverages the consistency and complementarity relationships among modalities to extract both common and distinctive features, mitigating the challenges faced by encoders due to significant heterogeneity across various data types. Additionally, MURLF employs multi-level masking independently across different modalities, utilizing visual tokens both within the same modality and across modalities to jointly recover masked pixels as the pretext task, facilitating comprehensive cross-modal information interaction. Furthermore, we design a pre-selected sensor-specific feature extractor (PSFE) to exploit the heterogeneous characteristics of various data sources, thereby extracting discriminative features. By integrating the multi-stage PSFE with the ViT backbone, MURLF can naturally extract multimodal hierarchical representations for downstream tasks, fully preserving valuable information from each modality. The proposed MURLF is not restricted to multi-modal inputs but also supports single-modal inputs during the fine-tuning stage, significantly broadening the framework’s application. Extensive experiments across multiple tasks demonstrate the superiority of the proposed MURLF compared to several advanced multi-modal models. The code will be released soon.

Learning Unified Sparse Representations For Multi-Modal Data

Learning Disentangled Representation for Cross-Modal Retrieval with Deep Mutual Information Estimation.

Multiple Kernel Visual-Auditory Representation Learning for Retrieval

Universal Vision-Language Dense Retrieval: Learning A Unified Representation Space for Multi-Modal Retrieval

Joint Feature Selection and Subspace Learning for Cross-Modal Retrieval

Joint Dictionary Learning and Semantic Constrained Latent Subspace Projection for Cross-Modal Retrieval.

Learning Explicit and Implicit Latent Common Spaces for Audio-Visual Cross-Modal Retrieval

Domain Separation Network for Cross-Modal Retrieval.

A Multi-Modal Unified Representation Learning Framework with Masked Image Modeling for Remote Sensing Images

Multimodal sparse representation learning and applications

Multi-class Joint Subspace Learning for Cross-Modal Retrieval.

Effective Deep Learning-Based Multi-Modal Retrieval

Semantic Pre-alignment and Ranking Learning with Unified Framework for Cross-modal Retrieval

Cross-Modal Retrieval Using Multiordered Discriminative Structured Subspace Learning.

Task-Dependent and Query-Dependent Subspace Learning for Cross-Modal Retrieval

Deep Multi-Graph Hierarchical Enhanced Semantic Representation for Cross-Modal Retrieval

Learning Sufficient Scene Representation for Unsupervised Cross-Modal Retrieval.

Joint Latent Subspace Learning and Regression for Cross-Modal Retrieval

Weakly-paired Deep Dictionary Learning for Cross-Modal Retrieval

Adaptively Unified Semi-supervised Learning for Cross-Modal Retrieval

UCSL: Toward Unsupervised Common Subspace Learning for Cross-Modal Image Classification