Abstract:Depth information can contribute to the semantic segmentation of scenes from red–green–blue (RGB) images. Therefore, the amount of information that can be obtained from RGB and RGB-depth (RGB-D) images is significantly greater for this task. However, RGB and RGB-D modalities are different in terms of object representation. Features that are extracted from these modalities and fused effectively are key to scene semantic segmentation. In addition, complete segmentation requires the fusion of multiscale features to unify global information. However, existing approaches primarily use multiscale features for sequential integration. This study introduces a cross-modal and progressive feature fusion network (CMPFFNet) for semantic segmentation of indoor scenes in RGB-D images. First, a multimodal adaptive alignment fusion (MAAF) module based on an attention mechanism is introduced. This module aligns the two modal channels by additive attention and then computes the spatial similarity between the two modalities based on the dot product to incorporate the complementary information of the depth modality into the RGB modality. In addition, a reverse attention augmentation (RAA) module is introduced to augment the more abstract high-level features for two adjacent multilevel features using the concrete semantic information of the lower-level features in them. After augmenting the extracted multilevel features, a multilevel feature progressive fusion (MFPF) module is deployed; this module sequentially fuses the neighboring two features progressively with emphasis on the spatial semantics. The network uses the Segformer network with high performance as a backbone in multiple computer vision tasks to enhance the segmentation capability. Experimental results obtained from two publicly available datasets of indoor scenes reveal that the proposed CMPFFNet outperforms existing models in semantic segmentation of indoor scenes of RGB-D images. Note to Practitioners—This study introduces a cross-modal and progressive feature fusion network (CMPFFNet) for indoor scene semantic segmentation in RGB-D images. The complementary information of the depth modality is incorporated into the RGB modality in both channel and spatial forms to form a discriminative representation for easy segmentation. A multilevel feature aggregation decoder is proposed to predict the results of semantic segmentation of scenes. The network uses the Segformer network with high performance as a backbone in multiple computer vision tasks to enhance the segmentation capability.

HAFFseg: RGB-Thermal semantic segmentation network with hybrid adaptive feature fusion strategy

NLFNet: Non-Local Fusion Towards Generalized Multimodal Semantic Segmentation Across RGB-Depth, Polarization, and Thermal Images

Retinal Vessel Segmentation Via Cross-attention Feature Fusion

SFAF-MA: Spatial Feature Aggregation and Fusion With Modality Adaptation for RGB-Thermal Semantic Segmentation

CCAFFMNet: Dual-spectral semantic segmentation network with channel-coordinate attention feature fusion module

IAFFNet: Illumination-Aware Feature Fusion Network for All-Day RGB-Thermal Semantic Segmentation of Road Scenes.

Residual Spatial Fusion Network for RGB-Thermal Semantic Segmentation

Multispectral Fusion Transformer Network for RGB-Thermal Urban Scene Semantic Segmentation

FEANet: Feature-Enhanced Attention Network for RGB-Thermal Real-time Semantic Segmentation

HAPNet: Toward Superior RGB-Thermal Scene Parsing via Hybrid, Asymmetric, and Progressive Heterogeneous Feature Fusion

An RGB-D Fusion Based Semantic Segmentation Algorithm Based on Neighborhood Metric Relations

Complementarity-aware cross-modal feature fusion network for RGB-T semantic segmentation

DHFNet: Decoupled Hierarchical Fusion Network for RGB-T dense prediction tasks

FuseSeg: Semantic Segmentation of Urban Scenes Based on RGB and Thermal Data Fusion

A RGB-Thermal Image Segmentation Method Based on Parameter Sharing and Attention Fusion for Safe Autonomous Driving

MEFNET: Multi-expert fusion network for RGB-Thermal semantic segmentation

A Feature Divide-and-Conquer Network for RGB-T Semantic Segmentation

Rgb-t semantic segmentation based on cross-operational fusion attention in autonomous driving scenario

CMPFFNet: Cross-Modal and Progressive Feature Fusion Network for RGB-D Indoor Scene Semantic Segmentation

MFAFNet: A Lightweight and Efficient Network with Multi-Level Feature Adaptive Fusion for Real-Time Semantic Segmentation

AMFuse: Add–Multiply-Based Cross-Modal Fusion Network for Multi-Spectral Semantic Segmentation