Abstract:LiDAR and camera fusion techniques are promising for achieving 3D object detection in autonomous driving. Most multi-modal 3D object detection frameworks integrate semantic knowledge from 2D images into 3D LiDAR point clouds to enhance detection accuracy. Nevertheless, the restricted resolution of 2D feature maps impedes accurate re-projection and often induces a pronounced boundary-blurring effect, which is primarily attributed to erroneous semantic segmentation. To well handle this limitation, we propose a general multi-modal fusion framework Multi-Sem Fusion (MSF) to fuse the semantic information from both the 2D image and 3D points scene parsing results. Specifically, we employ 2D/3D semantic segmentation methods to generate the parsing results for 2D images and 3D point clouds. The 2D semantic information is further reprojected into the 3D point clouds with calibration parameters. To handle the misalignment between the 2D and 3D parsing results, we propose an Adaptive Attention-based Fusion (AAF) module to fuse them by learning an adaptive fusion score. Then the point cloud with the fused semantic label is sent to the following 3D object detectors. Furthermore, we propose a Deep Feature Fusion (DFF) module to aggregate deep features at different levels to boost the final detection performance. The effectiveness of the framework has been verified on two public large-scale 3D object detection benchmarks by comparing them with different baselines. The experimental results show that the proposed fusion strategies can significantly improve the detection performance compared to the methods using only point clouds and the methods using only 2D semantic information. Most importantly, the proposed approach significantly outperforms other approaches and sets state-of-the-art results on the nuScenes testing benchmark.

Inference Fusion with Associative Semantics for Unseen Object Detection

Multi-spectral Image Fusion for Moving Object Detection

Multi-Modality Image Fusion and Object Detection Based on Semantic Information

Semantic-aligned Fusion Transformer for One-shot Object Detection

Diverse Generation while Maintaining Semantic Coordination: A Diffusion-Based Data Augmentation Method for Object Detection

MetaFusion: Infrared and Visible Image Fusion Via Meta-Feature Embedding from Object Detection

Multi-Sem Fusion: Multimodal Semantic Fusion for 3-D Object Detection

Visual Relationship Detection with Multimodal Fusion and Reasoning

An object detection algorithm based on infrared-visible dual modal feature fusion

From Text to Pixels: A Context-Aware Semantic Synergy Solution for Infrared and Visible Image Fusion

Multi-Sem Fusion: Multimodal Semantic Fusion for 3D Object Detection

DualTeacher: Bridging Coexistence of Unlabelled Classes for Semi-supervised Incremental Object Detection

Adaptive Multilevel Fusion Refinement Network for Object Detection in Remote Sensing Images

Multi-Semantic Interactive Learning for Object Detection

CONDA: Condensed Deep Association Learning for Co-Salient Object Detection

Enhanced Object Detection via Fusion With Prior Beliefs from Image Classification

DetMatch: Two Teachers are Better than One for Joint 2D and 3D Semi-Supervised Object Detection

FM-Fusion: Instance-Aware Semantic Mapping Boosted by Vision-Language Foundation Models

Enhancing 3D object detection through multi-modal fusion for cooperative perception

Towards Unified Object Detection And Semantic Segmentation

Rethinking the Late Fusion of LiDAR-Camera Based 3D Object Detection