Abstract:The complementary properties exhibited upon RGB-T data involve context complementarity as well as content complementarity. During cross-modal feature fusion, most existing RGB-T semantic segmentation methods are dedicated to highlighting the exploitation of content-complementary information. Unfortunately, these methods usually overlook the excavation of cross-modal context-complementary information ( i.e ., the contextual dependencies among different regions that only exist in one certain modality data) or try to exploit such cross-modal context-complementary information in an implicit way, yielding fragmentary semantic segmentation results. To remedy this problem, in this paper, a novel Cross-modal Context- and Content-Complementarity Network (C 4 Net) is presented for RGB-T semantic segmentation, in which both the cross-modal context-complementary information and the cross-modal content-complementary information are fully excavated and exploited during cross-modal feature fusion. Specifically, a Context-Complementary Information Aggregation (CxCIA) module is carefully designed, in which the cross-modal context-complementary information is explicitly excavated by measuring the discrepancies between contextual dependencies from different modality data. Then, such cross-modal context-complementary information is further exploited to enhance the original RGB and thermal contextual dependencies for boosting the integrity of objects in the fused features. In the meantime, a Content-Complementary Information Aggregation (CnCIA) module is presented, which highlights the utilization of cross-modal content-complementary information from a multi-scale perspective. Furthermore, an MLP-based Multi-level Feature Interaction (MFI) decoder is presented, in which the semantic gaps among different levels of fused features are mitigated by establishing the interactions of multi-level fused features along spatial and channel dimensions. Comprehensive experimental results on several public datasets demonstrate that our proposed C 4 Net surpasses other state-of-the-art models.

MS-IRTNet: Multistage Information Interaction Network for RGB-T Semantic Segmentation

NLFNet: Non-Local Fusion Towards Generalized Multimodal Semantic Segmentation Across RGB-Depth, Polarization, and Thermal Images

CIGF-Net: Cross-Modality Interaction and Global-Feature Fusion for RGB-T Semantic Segmentation

ACNET: Attention Based Network to Exploit Complementary Features for RGBD Semantic Segmentation.

Mitigating Modality Discrepancies for RGB-T Semantic Segmentation

Interactive Efficient Multi-Task Network for RGB-D Semantic Segmentation

MMSMCNet: Modal Memory Sharing and Morphological Complementary Networks for RGB-T Urban Scene Semantic Segmentation

MMNet: Multi-modal multi-stage network for RGB-T image semantic segmentation

Residual Spatial Fusion Network for RGB-Thermal Semantic Segmentation

ABMDRNet: Adaptive-weighted Bi-directional Modality Difference Reduction Network for RGB-T Semantic Segmentation

Mask-guided Modality Difference Reduction Network for RGB-T Semantic Segmentation

Multispectral Fusion Transformer Network for RGB-Thermal Urban Scene Semantic Segmentation

RGB-T Semantic Segmentation with Location, Activation, and Sharpening

MEFNET: Multi-expert fusion network for RGB-Thermal semantic segmentation

Channel and Spatial Relation-Propagation Network for RGB-Thermal Semantic Segmentation

IAFFNet: Illumination-Aware Feature Fusion Network for All-Day RGB-Thermal Semantic Segmentation of Road Scenes.

C4Net: Excavating Cross-modal Context- and Content-Complementarity for RGB-T Semantic Segmentation

MTANet: Multitask-Aware Network with Hierarchical Multimodal Fusion for RGB-T Urban Scene Understanding

Context-Aware Interaction Network for RGB-T Semantic Segmentation

Multi-branch Differential Bidirectional Fusion Network for RGB-T Semantic Segmentation

Dual-branch deep cross-modal interaction network for semantic segmentation with thermal images