Abstract:Accurate 3D object detection is vital for autonomous driving since it facilitates accurate perception of the environment through multiple sensors. Although cameras can capture detailed color and texture features, they have limitations regarding depth information. Additionally, they can struggle under adverse weather or lighting conditions. In contrast, LiDAR sensors offer robust depth information but lack the visual detail for precise object classification. This work presents a multimodal fusion model that improves 3D object detection by combining the benefits of LiDAR and camera sensors to address these challenges. This model processes camera images and LiDAR point cloud data into a voxel-based representation, further refined by encoder networks to enhance spatial interaction and reduce semantic ambiguity. The proposed multiresolution attention module and integration of discrete wavelet transform and inverse discrete wavelet transform to the image backbone improve the feature extraction capability. This approach enhances the fusion of LiDAR depth information with the camera's textural and color detail. The model also incorporates a transformer decoder network with self-attention and cross-attention mechanisms, fostering robust and accurate detection through global interaction between identified objects and encoder features. Furthermore, the proposed network is refined with advanced optimization techniques, including pruning and Quantization-Aware Training (QAT), to maintain a competitive performance while significantly decreasing the need for memory and computational resources. Performance evaluations on the nuScenes dataset show that the optimized model architecture offers competitive results and significantly improves operational efficiency and effectiveness in multimodal fusion 3D object detection.

CR-DINO: A Novel Camera-Radar Fusion 2D Object Detection Model Based On Transformer

Mask-RadarNet: Enhancing Transformer With Spatial-Temporal Semantic Context for Radar Object Detection in Autonomous Driving

Combing Transformer with Convolutional Neural Networks for Radar Object Detection in Millimeter-Wave Imaging Systems

TransCAR: Transformer-based Camera-And-Radar Fusion for 3D Object Detection

Camera-Radar Fusion with Radar Channel Extension and Dual-CBAM-FPN for Object Detection

CRAFT: Camera-Radar 3D Object Detection with Spatio-Contextual Fusion Transformer

Radar-Image Fusion Transformer for Object Detection with Uncertainty Estimation

RCDPT: Radar-Camera fusion Dense Prediction Transformer

RCTrans: Radar-Camera Transformer via Radar Densifier and Sequential Decoder for 3D Object Detection

Radar-camera Fusion for 3D Object Detection with Aggregation Transformer

TL-4DRCF:A Two-Level 4D Radar-Camera Fusion Method for Object Detection in Adverse Weather

Temporal-enhanced Radar and Camera Fusion for Object Detection

MVFusion: Multi-View 3D Object Detection with Semantic-aligned Radar and Camera Fusion

Radar Enlighten the Dark: Enhancing Low-Visibility Perception for Automated Vehicles with Camera-Radar Fusion

CramNet: Camera-Radar Fusion with Ray-Constrained Cross-Attention for Robust 3D Object Detection

TransRadar: Adaptive-Directional Transformer for Real-Time Multi-View Radar Semantic Segmentation

Transformer-Based Optimized Multimodal Fusion for 3D Object Detection in Autonomous Driving

RCM-Fusion: Radar-Camera Multi-Level Fusion for 3D Object Detection

LRCFormer: lightweight transformer based radar-camera fusion for 3D target detection

CRT-Fusion: Camera, Radar, Temporal Fusion Using Motion Information for 3D Object Detection

Two-Stage Feature Attention Fusion for Radar-Camera 3D Object Detection