Abstract:Previous multi-modal transformers for RGB-D salient object detection (SOD) generally directly connect all patches from two modalities to model cross-modal correlation and perform multi-modal combination without differentiation, which can lead to confusing and inefficient fusion. Instead, we disentangle the cross-modal complementarity from two views to reduce cross-modal fusion ambiguity: 1) Context disentanglement. We argue that modeling long-range dependencies across modalities as done before is uninformative due to the severe modality gap. Differently, we propose to disentangle the cross-modal complementary contexts to intra-modal self-attention to explore global complementary understanding, and spatial-aligned inter-modal attention to capture local cross-modal correlations, respectively. 2) Representation disentanglement. Unlike previous undifferentiated combination of cross-modal representations, we find that cross-modal cues complement each other by enhancing common discriminative regions and mutually supplement modal-specific highlights. On top of this, we divide the tokens into consistent and private ones in the channel dimension to disentangle the multi-modal integration path and explicitly boost two complementary ways. By progressively propagate this strategy across layers, the proposed Disentangled Feature Pyramid module (DFP) enables informative cross-modal cross-level integration and better fusion adaptivity. Comprehensive experiments on a large variety of public datasets verify the efficacy of our context and representation disentanglement and the consistent improvement over state-of-the-art models. Additionally, our cross-modal attention hierarchy can be plug-and-play for different backbone architectures (both transformer and CNN) and downstream tasks, and experiments on a CNN-based model and RGB-D semantic segmentation verify this generalization ability.

Task-aware Disentanglement for Object Detection

HTD: Heterogeneous Task Decoupling for Two-Stage Object Detection

Task-Specific Context Decoupling for Object Detection

Disentangle Your Dense Object Detector

DDOD: Dive Deeper into the Disentanglement of Object Detector

Decoupled DETR: Spatially Disentangling Localization and Classification for Improved End-to-End Object Detection

Feature difference for single-shot object detection

Task-Balanced Distillation for Object Detection

Decouple and Align Classification and Regression in One-Stage Object Detection

TCD: Task-Collaborated Detector for Oriented Objects in Remote Sensing Images

Vector-Decomposed Disentanglement for Domain-Invariant Object Detection

Cross Domain Object Detection by Target-Perceived Dual Branch Distillation

Task-specific Inconsistency Alignment for Domain Adaptive Object Detection

Instance-Invariant Domain Adaptive Object Detection via Progressive Disentanglement

HEAD: HEtero-Assists Distillation for Heterogeneous Object Detectors

Feature Disentanglement Network: Multi-Object Tracking Needs More Differentiated Features

Dense Crosstalk Feature Aggregation for Classification and Localization in Object Detection

Disentangled Cross-Modal Transformer for RGB-D Salient Object Detection and Beyond

DETRs with Collaborative Hybrid Assignments Training

Task-Aligned Oriented Object Detection in Remote Sensing Images

Rethinking the Misalignment Problem in Dense Object Detection