Abstract:Depth information can contribute to the semantic segmentation of scenes from red–green–blue (RGB) images. Therefore, the amount of information that can be obtained from RGB and RGB-depth (RGB-D) images is significantly greater for this task. However, RGB and RGB-D modalities are different in terms of object representation. Features that are extracted from these modalities and fused effectively are key to scene semantic segmentation. In addition, complete segmentation requires the fusion of multiscale features to unify global information. However, existing approaches primarily use multiscale features for sequential integration. This study introduces a cross-modal and progressive feature fusion network (CMPFFNet) for semantic segmentation of indoor scenes in RGB-D images. First, a multimodal adaptive alignment fusion (MAAF) module based on an attention mechanism is introduced. This module aligns the two modal channels by additive attention and then computes the spatial similarity between the two modalities based on the dot product to incorporate the complementary information of the depth modality into the RGB modality. In addition, a reverse attention augmentation (RAA) module is introduced to augment the more abstract high-level features for two adjacent multilevel features using the concrete semantic information of the lower-level features in them. After augmenting the extracted multilevel features, a multilevel feature progressive fusion (MFPF) module is deployed; this module sequentially fuses the neighboring two features progressively with emphasis on the spatial semantics. The network uses the Segformer network with high performance as a backbone in multiple computer vision tasks to enhance the segmentation capability. Experimental results obtained from two publicly available datasets of indoor scenes reveal that the proposed CMPFFNet outperforms existing models in semantic segmentation of indoor scenes of RGB-D images. Note to Practitioners—This study introduces a cross-modal and progressive feature fusion network (CMPFFNet) for indoor scene semantic segmentation in RGB-D images. The complementary information of the depth modality is incorporated into the RGB modality in both channel and spatial forms to form a discriminative representation for easy segmentation. A multilevel feature aggregation decoder is proposed to predict the results of semantic segmentation of scenes. The network uses the Segformer network with high performance as a backbone in multiple computer vision tasks to enhance the segmentation capability.

Automatic curtain wall frame detection based on deep learning and cross-modal feature fusion

Automatic segmentation of curtain wall frame using a context collaboration pyramid network

Curtain wall frame segmentation using a dual-flow aggregation network: Application to robot pose estimation

Window Detection in Facades Using Heatmap Fusion

People Flow Detection Algorithm Based on a Multiencoder-Classifier Cotraining Architecture for FMCW Radar.

Deep learning for detecting building façade elements from images considering prior knowledge

Research on Non-Destructive Testing of Curtain Wall Hangings Based on Efficient Channel Attention Vision Transformer Network

A Multi-channel Input Framework for Structure Displacement Response Prediction Using Convolutional Neural Network

ASFD: Automatic and Scalable Face Detector

Window Detection In Facade Imagery: A Deep Learning Approach Using Mask R-CNN

A new real-time image semantic segmentation framework based on a lightweight deep convolutional encoder-decoder architecture for robotic environment sensing

A Feature Prefusion and Mask-Guided Network for Camera Decoration Defect Detection

CMPFFNet: Cross-Modal and Progressive Feature Fusion Network for RGB-D Indoor Scene Semantic Segmentation

A cross-channel multi-scale gated fusion network for recognizing construction and demolition waste from high-resolution remote sensing images

Deep-Learning-Based Automated Building Construction Progress Monitoring for Prefabricated Prefinished Volumetric Construction

Automatic Pixel-Level Crack Detection on Dam Surface Using Deep Convolutional Network

Attention-Based Multiscale Feature Fusion for Efficient Surface Defect Detection

AFMCT: adaptive fusion module based on cross-modal transformer block for 3D object detection

Adjacent-Level Feature Cross-Fusion With 3-D CNN for Remote Sensing Image Change Detection

CSFuser: A Cascade Siamese Fusion Architecture for RGB-Infrared Object Detection

Deep Learning-Based RGB-D Fusion for Multimodal Condition Assessment of Civil Infrastructure