Abstract:Image Salient Object Detection (SOD) is a fundamental research topic in the area of computer vision. Recently, the multimodal information in RGB, Depth (D), and Thermal (T) modalities has been proven to be beneficial to the SOD. However, existing methods are only designed for RGB-D or RGB-T SOD, which may limit the utilization in various modalities, or just finetuned on specific datasets, which may bring about extra computation overhead. These defects can hinder the practical deployment of SOD in real-world applications. In this paper, we propose an end-to-end Unified Triplet Decoder Network, dubbed UTDNet, for both RGB-T and RGB-D SOD tasks. The intractable challenges for the unified multimodal SOD are mainly two-fold, i.e., (1) accurately detecting and segmenting salient objects, and (2) preferably via a single network that fits both RGB-T and RGB-D SOD. First, to deal with the former challenge, we propose the multi-scale feature extraction unit to enrich the discriminative contextual information, and the efficient fusion module to explore cross-modality complementary information. Then, the multimodal features are fed to the triplet decoder, where the hierarchical deep supervision loss further enable the network to capture distinctive saliency cues. Second, as to the latter challenge, we propose a simple yet effective continual learning method to unify multimodal SOD. Concretely, we sequentially train multimodal SOD tasks by applying Elastic Weight Consolidation (EWC) regularization with the hierarchical loss function to avoid catastrophic forgetting without inducing more parameters. Critically, the triplet decoder separates task-specific and task-invariant information, making the network easily adaptable to multimodal SOD tasks. Extensive comparisons with 26 recently proposed RGB-T and RGB-D SOD methods demonstrate the superiority of the proposed UTDNet.

KCDNet: Multimodal Object Detection in Modal Information Imbalance Scenes

C2BG-Net: Cross-modality and cross-scale balance network with global semantics for multi-modal 3D object detection

Improving Multispectral Pedestrian Detection by Addressing Modality Imbalance Problems

Unified Information Fusion Network for Multi-Modal RGB-D and RGB-T Salient Object Detection

Knowledge Synergy Learning for Multi-Modal Tracking

UMINet: a unified multi-modality interaction network for RGB-D and RGB-T salient object detection

Adaptive Modality Balanced Online Knowledge Distillation for Brain-Eye-Computer based Dim Object Detection

Revisiting Modality Imbalance In Multimodal Pedestrian Detection

Cross-Modality Knowledge Distillation Network for Monocular 3D Object Detection

Informative Data Selection With Uncertainty for Multimodal Object Detection

Imbalance Knowledge-Driven Multi-modal Network for Land-Cover Semantic Segmentation Using Images and LiDAR Point Clouds

Imbalance knowledge-driven multi-modal network for land-cover semantic segmentation using aerial images and LiDAR point clouds

Enhanced Multimodal Representation Learning with Cross-modal KD.

UTDNet: A unified triplet decoder network for multimodal salient object detection

Consistency- and dependence-guided knowledge distillation for object detection in remote sensing images

Learning a Dynamic Cross-Modal Network for Multispectral Pedestrian Detection

Heterogeneous Object-level Aircraft Change Detection Via Cross-modal Interaction and Imbalanced Learning

Learn to Combine Modalities in Multimodal Deep Learning

CM2-Net: Continual Cross-Modal Mapping Network for Driver Action Recognition

A Dimensional Structure based Knowledge Distillation Method for Cross-Modal Learning