Abstract:Joint classification of multi-modal remote sensing images has achieved great success thanks to complementary advantages of multi-modal images. However, modality absence is a common dilemma in real world caused by imaging conditions, which leads to a breakdown of most classification methods that rely on complete modalities. Existing approaches either learn shared representations or train specific models for each absence case so that they commonly confront the difficulty of balancing the complementary advantages of the modalities and scalability of the absence case. In this paper, we propose a language-guided visual prompt compensation network (LVPCnet) to achieve joint classification in case of arbitrary modality absence using a unified model that simultaneously considers modality complementarity. It embeds missing modality-specific knowledge into visual prompts to guide the model in capturing complete modal information from available ones for classification. Specifically, a language-guided visual feature decoupling stage (LVFD-stage) is designed to extract shared and specific modal feature from multi-modal images, establishing a complementary representation model of complete modalities. Subsequently, an absence-aware visual prompt compensation stage (VPC-stage) is proposed to learn visual prompts containing missing modality-specific knowledge through cross-modal representation alignment, further guiding the complementary representation model to reconstruct modality-specific features for missing modalities from available ones based on the learned prompts. The proposed VPC-stage entails solely training visual prompts to perceive missing information without retraining the model, facilitating effective scalability to arbitrary modal missing scenarios. Systematic experiments conducted on three public datasets have validated the effectiveness of the proposed approach.

Joint Classification of Hyperspectral Image and LiDAR Data Based on Spectral Prompt Tuning

Efficient Prompt Tuning of Large Vision-Language Model for Fine-Grained Ship Classification

Joint Classification of Hyperspectral and LiDAR Data Based on Position-Channel Cooperative Attention Network

Joint Classification of Hyperspectral Images and LiDAR Data Based on Dual-Branch Transformer

LMPT: Prompt Tuning with Class-Specific Embedding Loss for Long-tailed Multi-Label Visual Recognition

Visual Prompt Tuning for Few-Shot Text Classification.

Mutual Prompt Leaning for Vision Language Models

Joint Classification of Hyperspectral and LiDAR Data Based on Adaptive Gating Mechanism and Learnable Transformer

Learnable Prompts-Based Transformers for Domain Generalization of Hyperspectral Image Classification

MixPrompt: Enhancing Generalizability and Adversarial Robustness for Vision-Language Models Via Prompt Fusion

Progressive Multi-modal Conditional Prompt Tuning

MPT4LM: Multi-Modal Prompt Tuning Makes Pre-Trained Large Language Models Better Vision-Language Learners

Multimodal Semantic Collaborative Classification for Hyperspectral Images and LiDAR Data

Ternary Modality Contrastive Learning for Hyperspectral and LiDAR Data Classification

LSPT: Long-term Spatial Prompt Tuning for Visual Representation Learning

TAI++: Text as Image for Multi-Label Image Classification by Co-Learning Transferable Prompt

MePT: Multi-Representation Guided Prompt Tuning for Vision-Language Model

MuDPT: Multi-modal Deep-symphysis Prompt Tuning for Large Pre-trained Vision-Language Models

Joint spectral-spatial hyperspectral classification based on transfer learning (SSTL) from red-green-blue (RGB) images

Language-Guided Visual Prompt Compensation for Multi-Modal Remote Sensing Image Classification with Modality Absence

CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models