Abstract:Previous multi-modal transformers for RGB-D salient object detection (SOD) generally directly connect all patches from two modalities to model cross-modal correlation and perform multi-modal combination without differentiation, which can lead to confusing and inefficient fusion. Instead, we disentangle the cross-modal complementarity from two views to reduce cross-modal fusion ambiguity: 1) Context disentanglement. We argue that modeling long-range dependencies across modalities as done before is uninformative due to the severe modality gap. Differently, we propose to disentangle the cross-modal complementary contexts to intra-modal self-attention to explore global complementary understanding, and spatial-aligned inter-modal attention to capture local cross-modal correlations, respectively. 2) Representation disentanglement. Unlike previous undifferentiated combination of cross-modal representations, we find that cross-modal cues complement each other by enhancing common discriminative regions and mutually supplement modal-specific highlights. On top of this, we divide the tokens into consistent and private ones in the channel dimension to disentangle the multi-modal integration path and explicitly boost two complementary ways. By progressively propagate this strategy across layers, the proposed Disentangled Feature Pyramid module (DFP) enables informative cross-modal cross-level integration and better fusion adaptivity. Comprehensive experiments on a large variety of public datasets verify the efficacy of our context and representation disentanglement and the consistent improvement over state-of-the-art models. Additionally, our cross-modal attention hierarchy can be plug-and-play for different backbone architectures (both transformer and CNN) and downstream tasks, and experiments on a CNN-based model and RGB-D semantic segmentation verify this generalization ability.

RGBD Salient Object Detection via Disentangled Cross-modal Fusion

RGB-D Salient Object Detection with Cross-Modality Modulation and Selection

CNN-Based RGB-D Salient Object Detection: Learn, Select, and Fuse

Discriminative feature fusion for RGB-D salient object detection

Disentangled Cross-Modal Transformer for RGB-D Salient Object Detection and Beyond

RGB-D Salient Object Detection Based on Cross-Modal and Cross-Level Feature Fusion

RGB-D salient object detection via cross-modal joint feature extraction and low-bound fusion loss

Feature interaction and two-stage cross-modal fusion for RGB-D salient object detection

Modality-Induced Transfer-Fusion Network for RGB-D and RGB-T Salient Object Detection

Cross-Modal Fusion and Progressive Decoding Network for RGB-D Salient Object Detection

Cross-Modality Double Bidirectional Interaction and Fusion Network for RGB-T Salient Object Detection

Deep RGB-D Saliency Detection with Depth-Sensitive Attention and Automatic Multi-Modal Fusion

Unified Information Fusion Network for Multi-Modal RGB-D and RGB-T Salient Object Detection

Cross-modality Discrepant Interaction Network for RGB-D Salient Object Detection

Enabling modality interactions for RGB-T salient object detection

Discriminative unimodal feature selection and fusion for RGB-D salient object detection

Employing Bilinear Fusion and Saliency Prior Information for RGB-D Salient Object Detection

Attention-guided cross-modal multiple feature aggregation network for RGB-D salient object detection

Cross-modal and multi-level feature refinement network for RGB-D salient object detection

Middle-Level Feature Fusion for Lightweight RGB-D Salient Object Detection

Global-prior-guided fusion network for salient object detection