Abstract:Previous multi-modal transformers for RGB-D salient object detection (SOD) generally directly connect all patches from two modalities to model cross-modal correlation and perform multi-modal combination without differentiation, which can lead to confusing and inefficient fusion. Instead, we disentangle the cross-modal complementarity from two views to reduce cross-modal fusion ambiguity: 1) Context disentanglement. We argue that modeling long-range dependencies across modalities as done before is uninformative due to the severe modality gap. Differently, we propose to disentangle the cross-modal complementary contexts to intra-modal self-attention to explore global complementary understanding, and spatial-aligned inter-modal attention to capture local cross-modal correlations, respectively. 2) Representation disentanglement. Unlike previous undifferentiated combination of cross-modal representations, we find that cross-modal cues complement each other by enhancing common discriminative regions and mutually supplement modal-specific highlights. On top of this, we divide the tokens into consistent and private ones in the channel dimension to disentangle the multi-modal integration path and explicitly boost two complementary ways. By progressively propagate this strategy across layers, the proposed Disentangled Feature Pyramid module (DFP) enables informative cross-modal cross-level integration and better fusion adaptivity. Comprehensive experiments on a large variety of public datasets verify the efficacy of our context and representation disentanglement and the consistent improvement over state-of-the-art models. Additionally, our cross-modal attention hierarchy can be plug-and-play for different backbone architectures (both transformer and CNN) and downstream tasks, and experiments on a CNN-based model and RGB-D semantic segmentation verify this generalization ability.

Cross-modal Deformable DETR for RGB-D Object Detection

3D-SSD: Learning Hierarchical Features from RGB-D Images for Amodal 3D Object Detection

Multi-scale Cross-Modal Transformer Network for RGB-D Object Detection

Depth Cue Enhancement and Guidance Network for RGB-D Salient Object Detection

DE-CrossDet: Divisible and Extensible Crossline Representation for Object Detection

Cross-Modal Weighting Network for RGB-D Salient Object Detection

Disentangled Cross-Modal Transformer for RGB-D Salient Object Detection and Beyond

PR-Deformable DETR: DETR for Remote Sensing Object Detection

Cross-modality Discrepant Interaction Network for RGB-D Salient Object Detection

Cross-Modal Fusion and Progressive Decoding Network for RGB-D Salient Object Detection

Cross Teaching Between Single-Spectral and Multi-Spectral Detection Transformers for Remote Sensing Object Detection

RGB-D Salient Object Detection Method Based on Multi-Modal Fusion and Contour Guidance

Cross-modal refined adjacent-guided network for RGB-D salient object detection

Lightweight Cross-Modal Information Mutual Reinforcement Network for RGB-T Salient Object Detection

Cross-modal and multi-level feature refinement network for RGB-D salient object detection

GM-DETR: Generalized Muiltispectral DEtection TRansformer with Efficient Fusion Encoder for Visible-Infrared Detection

RGB-D Salient Object Detection with Cross-Modality Modulation and Selection

RGBD Salient Object Detection via Disentangled Cross-modal Fusion

Cross-Modal Attentional Context Learning for RGB-D Object Detection

RGB-D Salient Object Detection Based on Cross-Modal and Cross-Level Feature Fusion

EM-Trans: Edge-Aware Multimodal Transformer for RGB-D Salient Object Detection