Abstract:Deep learning techniques are increasingly applied to point cloud semantic segmentation, where single-modal point cloud often suffers from accuracy-limiting confusion phenomena. Moreover, some networks with image and LiDAR data lack an efficient fusion mechanism, and the occlusion of images may do harm to the segmentation accuracy of a point cloud. To overcome the above issues, we propose the integration of multi-modal data to enhance network performance, addressing the shortcomings of existing feature-fusion strategies that neglect crucial information and struggle with matching modal features effectively. This paper introduces the Multi-View Guided Point Cloud Semantic Segmentation Model (MVG-Net), which extracts multi-scale and multi-level features and contextual data from urban aerial images and LiDAR, and then employs a multi-view image feature-aggregation module to capture highly correlated texture information with the spatial and channel attentions of point-wise image features. Additionally, it incorporates a fusion module that uses image features to instruct point cloud features for stressing key information. We present a new dataset, WK2020, which combines multi-view oblique aerial images with LiDAR point cloud to validate segmentation efficacy. Our method demonstrates superior performance, especially in building segmentation, achieving an F1 score of 94.6% on the Vaihingen Dataset—the highest among the methods evaluated. Furthermore, MVG-Net surpasses other networks tested on the WK2020 Dataset. Compared to backbone network for single point modality, our model achieves overall accuracy improvement of 5.08%, average F1 score advancement of 6.87%, and mean Intersection over Union (mIoU) betterment of 7.9%.

Multi-View PointNet for 3D Scene Understanding

MVPNet: A multi-scale voxel-point adaptive fusion network for point cloud semantic segmentation in urban scenes

Multi-View Vision Fusion Network: Can 2D Pre-Trained Model Boost 3D Point Cloud Data-Scarce Learning?

MVP-Net: Multiple View Pointwise Semantic Segmentation of Large-Scale Point Clouds

Multiview Fusion Driven 3-D Point Cloud Semantic Segmentation Based on Hierarchical Transformer

MPVNN: Multi-resolution Point-Voxel Non-parametric Network for 3D Point Cloud Processing

PVNet: A Joint Convolutional Network of Point Cloud and Multi-View for 3D Shape Recognition.

A Multi-Level Semantic Fusion VoteNet for 3D Object Detection on Point Clouds

Multi-view Vision-Prompt Fusion Network: Can 2D Pre-trained Model Boost 3D Point Cloud Data-scarce Learning?

Multi Voxel-Point Neurons Convolution (MVPConv) for Fast and Accurate 3D Deep Learning

MFFNet: Multimodal Feature Fusion Network for Point Cloud Semantic Segmentation

Towards Deeper and Better Multi-view Feature Fusion for 3D Semantic Segmentation

Multi Point-Voxel Convolution (MPVConv) for Deep Learning on Point Clouds

MVX-Net: Multimodal VoxelNet for 3D Object Detection

MVPN: Multi-View Prototype Network for 3D Shape Recognition

PVI-Net: Point-Voxel-Image Fusion for Semantic Segmentation of Point Clouds in Large-Scale Autonomous Driving Scenarios

MANet: Multimodal Attention Network based Point- View fusion for 3D Shape Recognition

MM-Point: Multi-View Information-Enhanced Multi-Modal Self-Supervised 3D Point Cloud Understanding

MVG-Net: LiDAR Point Cloud Semantic Segmentation Network Integrating Multi-View Images

Visibility-Aware Point-Based Multi-View Stereo Network

Multi-feature Fusion VoteNet for 3D Object Detection