Abstract:Multimodal (e.g., RGB-Depth/RGB-Thermal) fusion has shown great potential for improving semantic segmentation in complex scenes (e.g., indoor/low-light conditions). Existing approaches often fully fine-tune a dual-branch encoder-decoder framework with a complicated feature fusion strategy for achieving multimodal semantic segmentation, which is training-costly due to the massive parameter updates in feature extraction and fusion. To address this issue, we propose a surprisingly simple yet effective dual-prompt learning network (dubbed DPLNet) for training-efficient multimodal (e.g., RGB-D/T) semantic segmentation. The core of DPLNet is to directly adapt a frozen pre-trained RGB model to multimodal semantic segmentation, reducing parameter updates. For this purpose, we present two prompt learning modules, comprising multimodal prompt generator (MPG) and multimodal feature adapter (MFA). MPG works to fuse the features from different modalities in a compact manner and is inserted from shadow to deep stages to generate the multi-level multimodal prompts that are injected into the frozen backbone, while MPG adapts prompted multimodal features in the frozen backbone for better multimodal semantic segmentation. Since both the MPG and MFA are lightweight, only a few trainable parameters (3.88M, 4.4% of the pre-trained backbone parameters) are introduced for multimodal feature fusion and learning. Using a simple decoder (3.27M parameters), DPLNet achieves new state-of-the-art performance or is on a par with other complex approaches on four RGB-D/T semantic segmentation datasets while satisfying parameter efficiency. Moreover, we show that DPLNet is general and applicable to other multimodal tasks such as salient object detection and video semantic segmentation. Without special design, DPLNet outperforms many complicated models. Our code will be available at <a class="link-external link-http" href="http://github.com/ShaohuaDong2021/DPLNet" rel="external noopener nofollow">this http URL</a>.

MultiPLe: Multilingual Prompt Learning for Relieving Semantic Confusions in Few-shot Event Detection

Meta-Learning with Dynamic-Memory-Based Prototypical Network for Few-Shot Event Detection.

Multi-channels Prototype Contrastive Learning with Condition Adversarial Attacks for Few-shot Event Detection

Collaborate SLM and LLM with latent answers for event detection

Few-shot Event Detection: An Empirical Study and a Unified View

Multi-Modal Few-Shot Object Detection with Meta-Learning-Based Cross-Modal Prompting

Span-based few-shot event detection via aligning external knowledge

DiPEx: Dispersing Prompt Expansion for Class-Agnostic Object Detection

Learning Event Representations for Zero-Shot Detection via Dual-Contrastive Prompting

Mixture-of-Prompt-Experts for Multi-modal Semantic Understanding

Relational Prompt-based Pre-trained Language Models for Social Event Detection

Unified-modal Salient Object Detection via Adaptive Prompt Learning

Few-Shot Object Detection With Multilevel Information Interaction for Optical Remote Sensing Images

Zero- and Few-Shot Event Detection via Prompt-Based Meta Learning

HPL-ESS: Hybrid Pseudo-Labeling for Unsupervised Event-based Semantic Segmentation

Few-Shot Joint Multimodal Entity-Relation Extraction via Knowledge-Enhanced Cross-modal Prompt Model

Attention-optimized vision-enhanced prompt learning for few-shot multi-modal sentiment analysis

Improving Low-Resource Chinese Event Detection with Multi-task Learning.

Multi-Modal Attentive Prompt Learning for Few-shot Emotion Recognition in Conversations

APSeg: Auto-Prompt Network for Cross-Domain Few-Shot Semantic Segmentation

Efficient Multimodal Semantic Segmentation via Dual-Prompt Learning