Abstract:Combining color (RGB) images with thermal images can facilitate semantic segmentation of poorly lit urban scenes. However, for RGB-thermal (RGB-T) semantic segmentation, most existing models address cross-modal feature fusion by focusing only on exploring the samples while neglecting the connections between different samples. Additionally, although the importance of boundary, binary, and semantic information is considered in the decoding process, the differences and complementarities between different morphological features are usually neglected. In this paper, we propose a novel RGB-T semantic segmentation network, called MMSMCNet, based on modal memory fusion and morphological multiscale assistance to address the aforementioned problems. For this network, in the encoding part, we used SegFormer for feature extraction of bimodal inputs. Next, our modal memory sharing module implements staged learning and memory sharing of sample information across modal multiscales. Furthermore, we constructed a decoding union unit comprising three decoding units in a layer-by-layer progression that can extract two different morphological features according to the information category and realize the complementary utilization of multiscale cross-modal fusion information. Each unit contains a contour positioning module based on detail information, a skeleton positioning module with deep features as the primary input, and a morphological complementary module for mutual reinforcement of the first two types of information and construction of semantic information. Based on this, we constructed a new supervision strategy, that is, a multi-unit-based complementary supervision strategy. Extensive experiments using two standard datasets showed that MMSMCNet outperformed related state-of-the-art methods. The code is available at: https://github.com/2021nihao/MMSMCNet.

On Exploring Shape and Semantic Enhancements for RGB-X Semantic Segmentation

Semantic Reconstruction based on RGB Image and Sparse Depth

An RGB-D Fusion Based Semantic Segmentation Algorithm Based on Neighborhood Metric Relations

Spatial-information Guided Adaptive Context-aware Network for Efficient RGB-D Semantic Segmentation

A Novel Semantic Segmentation Algorithm for RGB-D Images Based on Non-Symmetry and Anti-Packing Pattern Representation Model

A Deep Semantic Segmentation Network with Semantic and Contextual Refinements

Residual Spatial Fusion Network for RGB-Thermal Semantic Segmentation

Bi-directional Cross-Modality Feature Propagation with Separation-and-Aggregation Gate for RGB-D Semantic Segmentation

MMSMCNet: Modal Memory Sharing and Morphological Complementary Networks for RGB-T Urban Scene Semantic Segmentation

Sigma: Siamese Mamba Network for Multi-Modal Semantic Segmentation

Self-Enhanced Feature Fusion for RGB-D Semantic Segmentation

Self-Learned Feature Reconstruction and Offset-Dilated Feature Fusion for Real-Time Semantic Segmentation

Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation

Interactive Efficient Multi-Task Network for RGB-D Semantic Segmentation

Mitigating Modality Discrepancies for RGB-T Semantic Segmentation

Boosting Semantic Segmentation from the Perspective of Explicit Class Embeddings

SSA-Seg: Semantic and Spatial Adaptive Pixel-level Classifier for Semantic Segmentation

Semantic Image Segmentation with Improved Position Attention and Feature Fusion

Semantic-Guided Representation Enhancement for Self-supervised Monocular Trained Depth Estimation

Delving into Shape-aware Zero-shot Semantic Segmentation

Object-Enhanced Semantic Segmentation Model for High-Resolution Remote Sensing Images