Abstract:Depth information can contribute to the semantic segmentation of scenes from red–green–blue (RGB) images. Therefore, the amount of information that can be obtained from RGB and RGB-depth (RGB-D) images is significantly greater for this task. However, RGB and RGB-D modalities are different in terms of object representation. Features that are extracted from these modalities and fused effectively are key to scene semantic segmentation. In addition, complete segmentation requires the fusion of multiscale features to unify global information. However, existing approaches primarily use multiscale features for sequential integration. This study introduces a cross-modal and progressive feature fusion network (CMPFFNet) for semantic segmentation of indoor scenes in RGB-D images. First, a multimodal adaptive alignment fusion (MAAF) module based on an attention mechanism is introduced. This module aligns the two modal channels by additive attention and then computes the spatial similarity between the two modalities based on the dot product to incorporate the complementary information of the depth modality into the RGB modality. In addition, a reverse attention augmentation (RAA) module is introduced to augment the more abstract high-level features for two adjacent multilevel features using the concrete semantic information of the lower-level features in them. After augmenting the extracted multilevel features, a multilevel feature progressive fusion (MFPF) module is deployed; this module sequentially fuses the neighboring two features progressively with emphasis on the spatial semantics. The network uses the Segformer network with high performance as a backbone in multiple computer vision tasks to enhance the segmentation capability. Experimental results obtained from two publicly available datasets of indoor scenes reveal that the proposed CMPFFNet outperforms existing models in semantic segmentation of indoor scenes of RGB-D images. Note to Practitioners—This study introduces a cross-modal and progressive feature fusion network (CMPFFNet) for indoor scene semantic segmentation in RGB-D images. The complementary information of the depth modality is incorporated into the RGB modality in both channel and spatial forms to form a discriminative representation for easy segmentation. A multilevel feature aggregation decoder is proposed to predict the results of semantic segmentation of scenes. The network uses the Segformer network with high performance as a backbone in multiple computer vision tasks to enhance the segmentation capability.

IEFM and IDS: Enhancing 3D Environment Perception Via Information Encoding in Indoor Point Cloud Semantic Segmentation

Elevation Information-Guided Multimodal Fusion Robust Framework for Remote Sensing Image Segmentation

SIESEF-FusionNet: Spatial Inter-correlation Enhancement and Spatially-Embedded Feature Fusion Network for LiDAR Point Cloud Semantic Segmentation

PointMS: Semantic Segmentation for Point Cloud Based on Multi-scale Directional Convolution

3D Semantic Segmentation Using Deep Learning for Large-Scale Indoor Point Cloud

Instance-Aware Embedding for Point Cloud Instance Segmentation

Investigate Indistinguishable Points in Semantic Segmentation of 3D Point Cloud

EFDCNet: Encoding Fusion and Decoding Correction Network for RGB-D Indoor Semantic Segmentation

An RGB-D Fusion Based Semantic Segmentation Algorithm Based on Neighborhood Metric Relations

MSIDA-Net: Point Cloud Semantic Segmentation via Multi-Spatial Information and Dual Adaptive Blocks

Selecting Optimal Combination of Data Channels for Semantic Segmentation in City Information Modelling (CIM)

AEFF-SSC: An Attention-Enhanced Feature Fusion for 3D Semantic Scene Completion

CMPFFNet: Cross-Modal and Progressive Feature Fusion Network for RGB-D Indoor Scene Semantic Segmentation

Interactive Efficient Multi-Task Network for RGB-D Semantic Segmentation

CONSTRUCTION OF A DUAL-TASK MODEL FOR INDOOR SCENE RECOGNITION AND SEMANTIC SEGMENTATION BASED ON POINT CLOUDS

IMENet: Joint 3D Semantic Scene Completion and 2D Semantic Segmentation through Iterative Mutual Enhancement

SIE: infrared and visible image fusion based on scene information embedding

EPMF: Efficient Perception-Aware Multi-Sensor Fusion for 3D Semantic Segmentation

BEMF-Net: Semantic Segmentation of Large-Scale Point Clouds via Bilateral Neighbor Enhancement and Multi-Scale Fusion

MsVFE and V-SIAM: Attention-based multi-scale feature interaction and fusion for outdoor LiDAR semantic segmentation

A Cyclic Information–Interaction Model for Remote Sensing Image Segmentation