Abstract:LiDAR and camera fusion techniques are promising for achieving 3D object detection in autonomous driving. Most multi-modal 3D object detection frameworks integrate semantic knowledge from 2D images into 3D LiDAR point clouds to enhance detection accuracy. Nevertheless, the restricted resolution of 2D feature maps impedes accurate re-projection and often induces a pronounced boundary-blurring effect, which is primarily attributed to erroneous semantic segmentation. To well handle this limitation, we propose a general multi-modal fusion framework Multi-Sem Fusion (MSF) to fuse the semantic information from both the 2D image and 3D points scene parsing results. Specifically, we employ 2D/3D semantic segmentation methods to generate the parsing results for 2D images and 3D point clouds. The 2D semantic information is further reprojected into the 3D point clouds with calibration parameters. To handle the misalignment between the 2D and 3D parsing results, we propose an Adaptive Attention-based Fusion (AAF) module to fuse them by learning an adaptive fusion score. Then the point cloud with the fused semantic label is sent to the following 3D object detectors. Furthermore, we propose a Deep Feature Fusion (DFF) module to aggregate deep features at different levels to boost the final detection performance. The effectiveness of the framework has been verified on two public large-scale 3D object detection benchmarks by comparing them with different baselines. The experimental results show that the proposed fusion strategies can significantly improve the detection performance compared to the methods using only point clouds and the methods using only 2D semantic information. Most importantly, the proposed approach significantly outperforms other approaches and sets state-of-the-art results on the nuScenes testing benchmark.

Robust 3D Semantic Segmentation Based on Multi-Phase Multi-Modal Fusion for Intelligent Vehicles

Robust 3D Semantic Segmentation Method Based on Multi-Modal Collaborative Learning

Accurate 3D Semantic Segmentation of Point Clouds for Intelligent Vehicles Based on Multi-view Edge Guidance and Fusion

A Multi-phase Camera-LiDAR Fusion Network for 3D Semantic Segmentation with Weak Supervision

Accurate 3-D Semantic Segmentation of Point Clouds for Intelligent Vehicles Based on Multiview Edge Guidance and Fusion

CLFusion:3D Semantic Segmentation Based on Camera and Lidar Fusion

Scene-Adaptive 3D Semantic Segmentation Based on Multi-Level Boundary-Semantic-Enhancement for Intelligent Vehicles

LIF-Seg: LiDAR and Camera Image Fusion for 3D LiDAR Semantic Segmentation

Scene-Adaptive 3-D Semantic Segmentation Method Based on Multiphase Edge Enhancement for Intelligent Vehicles

Perception-Aware Multi-Sensor Fusion for 3D LiDAR Semantic Segmentation.

Improved 3D Semantic Segmentation Model Based on RGB Image and LiDAR Point Cloud Fusion for Automantic Driving

Image Fusion Via Mutual Information Maximization for Semantic Segmentation in Autonomous Vehicles

MSeg3D: Multi-modal 3D Semantic Segmentation for Autonomous Driving

Multi-Sem Fusion: Multimodal Semantic Fusion for 3D Object Detection

Multi-sensor Fusion Network for Unstructured Scene Segmentation with Surface Normal Incorporated

A Semantic-aided Fusion Localization Method for Intelligent Vehicles in GNSS-denied Dynamic Environments

LiDAR-Based Real-Time Panoptic Segmentation via Spatiotemporal Sequential Data Fusion

RGB and LiDAR Fusion-based 3D Semantic Segmentation for Autonomous Driving

Multi-Sem Fusion: Multimodal Semantic Fusion for 3-D Object Detection

Revisiting Multi-modal 3D Semantic Segmentation in Real-world Autonomous Driving

Sem-Aug: Improving Camera-LiDAR Feature Fusion With Semantic Augmentation for 3D Vehicle Detection