Abstract:Combining color (RGB) images with thermal images can facilitate semantic segmentation of poorly lit urban scenes. However, for RGB-thermal (RGB-T) semantic segmentation, most existing models address cross-modal feature fusion by focusing only on exploring the samples while neglecting the connections between different samples. Additionally, although the importance of boundary, binary, and semantic information is considered in the decoding process, the differences and complementarities between different morphological features are usually neglected. In this paper, we propose a novel RGB-T semantic segmentation network, called MMSMCNet, based on modal memory fusion and morphological multiscale assistance to address the aforementioned problems. For this network, in the encoding part, we used SegFormer for feature extraction of bimodal inputs. Next, our modal memory sharing module implements staged learning and memory sharing of sample information across modal multiscales. Furthermore, we constructed a decoding union unit comprising three decoding units in a layer-by-layer progression that can extract two different morphological features according to the information category and realize the complementary utilization of multiscale cross-modal fusion information. Each unit contains a contour positioning module based on detail information, a skeleton positioning module with deep features as the primary input, and a morphological complementary module for mutual reinforcement of the first two types of information and construction of semantic information. Based on this, we constructed a new supervision strategy, that is, a multi-unit-based complementary supervision strategy. Extensive experiments using two standard datasets showed that MMSMCNet outperformed related state-of-the-art methods. The code is available at: https://github.com/2021nihao/MMSMCNet.

Clothes Color Segmentation Based on Channel Attention Module

ACNET: Attention Based Network to Exploit Complementary Features for RGBD Semantic Segmentation.

Triple Attention Network for Clothing Parsing

BiSeNet-oriented context attention model for image semantic segmentation

Class Semantic Enhancement Network for Semantic Segmentation

Learning Cross-Channel Representations for Semantic Segmentation

Method for Coloring Night-vision Imagery Based on Multispectral Semantic Segmentation.

Semantic segmentation of remote sensing images based on dual‐channel attention mechanism

FFENet: Frequency-Spatial Feature Enhancement Network for Clothing Classification

MMSMCNet: Modal Memory Sharing and Morphological Complementary Networks for RGB-T Urban Scene Semantic Segmentation

Semantic Image Segmentation Based On Attentions To Intra Scales And Inner Channels

Human-Centric Clothing Segmentation via Deformable Semantic Locality-Preserving Network

Texture and Shape Biased Two-Stream Networks for Clothing Classification and Attribute Recognition

SECBNet: Semantic Segmentation Enhanced Color Balance Network for Optical Satellite Images

Channel and spatial attention based deep object co-segmentation

A Semantic Segmentation Network Exploiting Multiple Cues and Channel Exchanging

A Bintree Energy Approach for Colour Image Segmentation Using Adaptive Channel Selection

BMSeNet: Multiscale Context Pyramid Pooling and Spatial Detail Enhancement Network for Real-Time Semantic Segmentation

Semantic Locality-Aware Deformable Network for Clothing Segmentation

A Segmentation Method of Color Images Based on Improved PCNN

Bimodal SegNet: Instance Segmentation Fusing Events and RGB Frames for Robotic Grasping