Abstract:Previous multi-modal transformers for RGB-D salient object detection (SOD) generally directly connect all patches from two modalities to model cross-modal correlation and perform multi-modal combination without differentiation, which can lead to confusing and inefficient fusion. Instead, we disentangle the cross-modal complementarity from two views to reduce cross-modal fusion ambiguity: 1) Context disentanglement. We argue that modeling long-range dependencies across modalities as done before is uninformative due to the severe modality gap. Differently, we propose to disentangle the cross-modal complementary contexts to intra-modal self-attention to explore global complementary understanding, and spatial-aligned inter-modal attention to capture local cross-modal correlations, respectively. 2) Representation disentanglement. Unlike previous undifferentiated combination of cross-modal representations, we find that cross-modal cues complement each other by enhancing common discriminative regions and mutually supplement modal-specific highlights. On top of this, we divide the tokens into consistent and private ones in the channel dimension to disentangle the multi-modal integration path and explicitly boost two complementary ways. By progressively propagate this strategy across layers, the proposed Disentangled Feature Pyramid module (DFP) enables informative cross-modal cross-level integration and better fusion adaptivity. Comprehensive experiments on a large variety of public datasets verify the efficacy of our context and representation disentanglement and the consistent improvement over state-of-the-art models. Additionally, our cross-modal attention hierarchy can be plug-and-play for different backbone architectures (both transformer and CNN) and downstream tasks, and experiments on a CNN-based model and RGB-D semantic segmentation verify this generalization ability.

Multi-scale Cross-Modal Transformer Network for RGB-D Object Detection

A Transformer-Based Object Detector with Coarse-Fine Crossing Representations

Cross-Modality Fusion Transformer for Multispectral Object Detection

Disentangled Cross-Modal Transformer for RGB-D Salient Object Detection and Beyond

M2RNet: Multi-modal and Multi-scale Refined Network for RGB-D Salient Object Detection

Cross-modal Deformable DETR for RGB-D Object Detection

Cross Modal Transformer: Towards Fast and Robust 3D Object Detection

Multi-Task Foreground-Aware Network with Depth Completion for Enhanced RGB-D Fusion Object Detection Based on Transformer

MMNet: Multi-Stage and Multi-Scale Fusion Network for RGB-D Salient Object Detection

Global Guided Cross-Modal Cross-Scale Network for RGB-D Salient Object Detection

Transformer-based Network for RGB-D Saliency Detection

Unifying convolution and transformer: a dual stage network equipped with cross-interactive multi-modal feature fusion and edge guidance for RGB-D salient object detection

Multi-Modality Task Cascade for 3D Object Detection

Multi-level cross-modal interaction network for RGB-D salient object detection

EM-Trans: Edge-Aware Multimodal Transformer for RGB-D Salient Object Detection

TANet: Transformer-based Asymmetric Network for RGB-D Salient Object Detection

Discriminative Cross-Modal Transfer Learning and Densely Cross-Level Feedback Fusion for RGB-D Salient Object Detection

: Edge-Aware Multimodal Transformer for RGB-D Salient Object Detection

GroupTransNet: Group transformer network for RGB-D salient object detection

Modality-Induced Transfer-Fusion Network for RGB-D and RGB-T Salient Object Detection

Discriminative feature fusion for RGB-D salient object detection