Abstract:The 3D scene understanding is mainly considered as a crucial requirement in computer vision and robotics applications. One of the high-level tasks in 3D scene understanding is semantic segmentation of RGB-Depth images. With the availability of RGB-D cameras, it is desired to improve the accuracy of the scene understanding process by exploiting the depth features along with the appearance features. As depth images are independent of illumination, they can improve the quality of semantic labeling alongside RGB images. Consideration of both common and specific features of these two modalities improves the performance of semantic segmentation. One of the main problems in RGB-Depth semantic segmentation is how to fuse or combine these two modalities to achieve more advantages of each modality while being computationally efficient. Recently, the methods that encounter deep convolutional neural networks have reached the state-of-the-art results by early, late, and middle fusion strategies. In this paper, an efficient encoder-decoder model with the attention-based fusion block is proposed to integrate mutual influences between feature maps of these two modalities. This block explicitly extracts the interdependences among concatenated feature maps of these modalities to exploit more powerful feature maps from RGB-Depth images. The extensive experimental results on three main challenging datasets of NYU-V2, SUN RGB-D, and Stanford 2D-3D-Semantic show that the proposed network outperforms the state-of-the-art models with respect to computational cost as well as model size. Experimental results also illustrate the effectiveness of the proposed lightweight attention-based fusion model in terms of accuracy.

DSEM-NeRF: Multimodal feature fusion and global-local attention for enhanced 3D scene reconstruction

Attention-based Multi-modal Fusion Network for Semantic Scene Completion.

M^2DNeRF: Multi-Modal Decomposition NeRF with 3D Feature Fields

${M^2D}$NeRF: Multi-Modal Decomposition NeRF with 3D Feature Fields

DSSMNeRF: Depth Self-supervised MVS NeRF

NeRF-DetS: Enhanced Adaptive Spatial-wise Sampling and View-wise Fusion Strategies for NeRF-based Indoor Multi-view 3D Object Detection

Semantic Reconstruction based on RGB Image and Sparse Depth

AM2FNet: Attention-based Multiscale & Multi-modality Fused Network

Sem2NeRF: Converting Single-View Semantic Masks to Neural Radiance Fields

NeRF-Det++: Incorporating Semantic Cues and Perspective-aware Depth Supervision for Indoor Multi-View 3D Detection

3D Reconstruction of Neural Radiation Field Based on Improved Multiple Layer Perceptron

GP-NeRF: Generalized Perception NeRF for Context-Aware 3D Scene Understanding

MMNeRF: Multi-Modal and Multi-View Optimized Cross-Scene Neural Radiance Fields

S$^3$-NeRF: Neural Reflectance Field from Shading and Shadow under a Single Viewpoint

3D Former: Monocular Scene Reconstruction with 3D SDF Transformers

Semantic Is Enough: Only Semantic Information For NeRF Reconstruction

DistillNeRF: Perceiving 3D Scenes from Single-Glance Images by Distilling Neural Fields and Foundation Model Features

MM-NeRF: Large-Scale Scene Representation with Multi-Resolution Hash Grid and Multi-View Priors Features

IMENet: Joint 3D Semantic Scene Completion and 2D Semantic Segmentation through Iterative Mutual Enhancement

Multi-Modal Attention-based Fusion Model for Semantic Segmentation of RGB-Depth Images

Drone-NeRF: Efficient NeRF based 3D scene reconstruction for large-scale drone survey