Abstract:The use of complementary information, namely depth or thermal information, has shown its benefits to salient object detection (SOD) during recent years. However, the RGB-D or RGB-T SOD problems are currently only solved independently, and most of them directly extract and fuse raw features from backbones. Such methods can he easily restricted by low-quality modality data and redundant cross-modal features. In this work, a unified end-to-end framework is designed to simultaneously analyze RCB-D and RGB-T SOD tasks. Specifically, to effectively tackle multi-modal features, we propose a novel multi-stage and multi-scale fusion network (MMNet), which consists of a cross-modal multi-stage fusion module (CMFM) and a bi-directional multi-scale decoder (BMD). Similar to the visual color stage doctrine in the human visual system (HVS), the proposed CMFM aims to explore important feature representations in feature response stage, and integrate them into cross-modal features in adversarial combination stage. Moreover, the proposed BMD learns the combination of multilevel cross-modal fused features to capture both local and global information of salient objects, and can further boost the multimodal SOD performance. The proposed unified cross-modality feature analysis framework based on two-stage and multi-scale information fusion can be used for diverse multi-modal SOD tasks. Comprehensive experiments (similar to 92K image-pairs) demonstrate that the proposed method consistently outperforms the other 21 state-of-the-art methods on nine benchmark datasets. This validates that our proposed method can work well on diverse multi-modal SOD tasks with good generalization and robustness, and provides a good multi-modal SOD benchmark.

Multimodal Data Curation via Object Detection and Filter Ensembles

Leveraging Image-Text Similarity and Caption Modification for the DataComp Challenge: Filtering Track and BYOD Track

Exploring Data Augmentation for Multi-Modality 3D Object Detection

Informative Data Selection With Uncertainty for Multimodal Object Detection

Unified Information Fusion Network for Multi-Modal RGB-D and RGB-T Salient Object Detection

Uni$^2$Det: Unified and Universal Framework for Prompt-Guided Multi-dataset 3D Detection

Towards Unified 3D Object Detection via Algorithm and Data Unification

CNN-Based RGB-D Salient Object Detection: Learn, Select, and Fuse

Enhancing 3D object detection through multi-modal fusion for cooperative perception

Informative Data Selection with Uncertainty for Multi-modal Object Detection

Cross-Modality 3D Object Detection

MECOM: A Meta-Completion Network for Fine-Grained Recognition With Incomplete Multi-Modalities

Data curation via joint example selection further accelerates multimodal learning

MLF-DET: Multi-Level Fusion for Cross-Modal 3D Object Detection

Filter Fusion: Camera-LiDAR Filter Fusion for 3D Object Detection with a Robust Fused Head

CML-MOTS: Collaborative Multi-task Learning for Multi-Object Tracking and Segmentation

Multimodal Collaboration Networks for Geospatial Vehicle Detection in Dense, Occluded, and Large-Scale Events

Scaling Multi-Camera 3D Object Detection through Weak-to-Strong Eliciting

Towards A Weakly Supervised Framework for 3D Point Cloud Object Detection and Annotation

Multimodal Deep Representation Learning for Video Classification

Combining Multi-Representation for Multimedia Event Detection Using Co-Training