Abstract:Image Salient Object Detection (SOD) is a fundamental research topic in the area of computer vision. Recently, the multimodal information in RGB, Depth (D), and Thermal (T) modalities has been proven to be beneficial to the SOD. However, existing methods are only designed for RGB-D or RGB-T SOD, which may limit the utilization in various modalities, or just finetuned on specific datasets, which may bring about extra computation overhead. These defects can hinder the practical deployment of SOD in real-world applications. In this paper, we propose an end-to-end Unified Triplet Decoder Network, dubbed UTDNet, for both RGB-T and RGB-D SOD tasks. The intractable challenges for the unified multimodal SOD are mainly two-fold, i.e., (1) accurately detecting and segmenting salient objects, and (2) preferably via a single network that fits both RGB-T and RGB-D SOD. First, to deal with the former challenge, we propose the multi-scale feature extraction unit to enrich the discriminative contextual information, and the efficient fusion module to explore cross-modality complementary information. Then, the multimodal features are fed to the triplet decoder, where the hierarchical deep supervision loss further enable the network to capture distinctive saliency cues. Second, as to the latter challenge, we propose a simple yet effective continual learning method to unify multimodal SOD. Concretely, we sequentially train multimodal SOD tasks by applying Elastic Weight Consolidation (EWC) regularization with the hierarchical loss function to avoid catastrophic forgetting without inducing more parameters. Critically, the triplet decoder separates task-specific and task-invariant information, making the network easily adaptable to multimodal SOD tasks. Extensive comparisons with 26 recently proposed RGB-T and RGB-D SOD methods demonstrate the superiority of the proposed UTDNet.

Unified Information Fusion Network for Multi-Modal RGB-D and RGB-T Salient Object Detection

MMNet: Multi-Stage and Multi-Scale Fusion Network for RGB-D Salient Object Detection

UMINet: a unified multi-modality interaction network for RGB-D and RGB-T salient object detection

Modality-Induced Transfer-Fusion Network for RGB-D and RGB-T Salient Object Detection

Cross-Modality Double Bidirectional Interaction and Fusion Network for RGB-T Salient Object Detection

Multi-modality information refinement fusion network for RGB-D salient object detection

Lightweight Cross-Modal Information Mutual Reinforcement Network for RGB-T Salient Object Detection

MFCINet: multi-level feature and context information fusion network for RGB-D salient object detection

Compensated Attention Feature Fusion and Hierarchical Multiplication Decoder Network for RGB-D Salient Object Detection

Multi-level cross-modal interaction network for RGB-D salient object detection

Cross-Modal Fusion and Progressive Decoding Network for RGB-D Salient Object Detection

HFMDNet: Hierarchical Fusion and Multilevel Decoder Network for RGB-D Salient Object Detection

MSEDNet: Multi-scale fusion and edge-supervised network for RGB-T salient object detection

Feature interaction and two-stage cross-modal fusion for RGB-D salient object detection

Learning Adaptive Fusion Bank for Multi-modal Salient Object Detection

Attention-guided cross-modal multiple feature aggregation network for RGB-D salient object detection

Interactive Context-Aware Network for RGB-T Salient Object Detection

Middle-Level Feature Fusion for Lightweight RGB-D Salient Object Detection

MFFNet: Multi-modal Feature Fusion Network for V-D-T Salient Object Detection

Lightweight Multi-modal Representation Learning for RGB Salient Object Detection

UTDNet: A unified triplet decoder network for multimodal salient object detection