Abstract:The field of autonomous vehicles (AVs) predominantly leverages multi-modal integration of LiDAR and camera data to achieve better performance compared to using a single modality. However, the fusion process encounters challenges in detecting distant objects due to the disparity between the high resolution of cameras and the sparse data from LiDAR. Insufficient integration of global perspectives with local-level details results in sub-optimal fusion <a class="link-external link-http" href="http://performance.To" rel="external noopener nofollow">this http URL</a> address this issue, we have developed an innovative two-stage fusion process called Quantum Inverse Contextual Vision Transformers (Q-ICVT). This approach leverages adiabatic computing in quantum concepts to create a novel reversible vision transformer known as the Global Adiabatic Transformer (GAT). GAT aggregates sparse LiDAR features with semantic features in dense images for cross-modal integration in a global form. Additionally, the Sparse Expert of Local Fusion (SELF) module maps the sparse LiDAR 3D proposals and encodes position information of the raw point cloud onto the dense camera feature space using a gating point fusion approach. Our experiments show that Q-ICVT achieves an mAPH of 82.54 for L2 difficulties on the Waymo dataset, improving by 1.88% over current state-of-the-art fusion methods. We also analyze GAT and SELF in ablation studies to highlight the impact of Q-ICVT. Our code is available at <a class="link-external link-https" href="https://github.com/sanjay-810/Qicvt" rel="external noopener nofollow">this https URL</a> Q-ICVT

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to improve the performance of 3D object detection through multimodal fusion in autonomous vehicles (AVs). Specifically, the paper addresses the following issues: 1. **Challenges in long - distance object detection**: Due to the difference between the high resolution of camera images and the sparsity of LiDAR point - cloud data, it is difficult to effectively detect long - distance objects during the fusion process. 2. **Inadequate combination of global perspective and local details**: Existing multimodal fusion methods perform poorly in integrating the global perspective and local details, resulting in sub - optimal fusion performance. To solve these problems, the authors propose an innovative two - stage fusion process, called **Quantum Inverse Contextual Vision Transformers (Q - ICVT)**. The main contributions of Q - ICVT include: 1. **Introducing the Global Adiabatic Transformer (GAT) based on the concept of adiabatic computing**: - GAT aims to align sparse voxelized features with dense image features through a global point - to - point attention mechanism. - Use the reversibility of adiabatic computing to improve the global context and effectively handle the interaction between LiDAR point - cloud (\(G_V\in\mathbb{R}^{H_V\times W_V\times C_V}\)) and image data (\(G_I\in\mathbb{R}^{H_I\times W_I\times C_I}\)). 2. **Developing the local fusion module Sparse Expert of Local Fusion (SELF)**: - The SELF module maps sparse LiDAR 3D proposal features to the dense camera feature space through the gated point fusion method and encodes the position information of the original point cloud. - Use the Mixture of Experts (MoE) model to capture long - term dependencies in heterogeneous sequence data, thereby fusing LiDAR and image data more effectively. Through these innovations, Q - ICVT achieves an mAPH (mean Average Precision) of 82.54 at the L2 difficulty level on the Waymo dataset, which is 1.88% higher than the current state - of - the - art fusion methods. In addition, the authors also conducted ablation experiments to evaluate the impact of each component, demonstrating the importance of the GAT and SELF modules for the overall performance improvement. In summary, this paper aims to overcome the limitations of existing methods in long - distance object detection and global - local feature fusion by introducing new fusion techniques, thereby significantly improving the 3D object detection performance in autonomous vehicles.

Quantum Inverse Contextual Vision Transformers (Q-ICVT): A New Frontier in 3D Object Detection for AVs

AYDIV: Adaptable Yielding 3D Object Detection via Integrated Contextual Vision Transformer

PVT-SSD: Single-Stage 3D Object Detector with Point-Voxel Transformer

3D Vehicle Detection Using Cheap LiDAR and Camera Sensors.

FusionViT: Hierarchical 3D Object Detection via LiDAR-Camera Vision Transformer Fusion

VIMI: Vehicle-Infrastructure Multi-view Intermediate Fusion for Camera-based 3D Object Detection

ContextualFusion: Context-Based Multi-Sensor Fusion for 3D Object Detection in Adverse Operating Conditions

End-to-End Multi-View Fusion for 3D Object Detection in LiDAR Point Clouds

Transformer-Based Optimized Multimodal Fusion for 3D Object Detection in Autonomous Driving

IFTR: An Instance-Level Fusion Transformer for Visual Collaborative Perception

VISTA: Boosting 3D Object Detection via Dual Cross-VIew SpaTial Attention

RangeViT: Towards Vision Transformers for 3D Semantic Segmentation in Autonomous Driving

LIFT: Learning 4D LiDAR Image Fusion Transformer for 3D Object Detection

TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers

CI3D: Context Interaction for Dynamic Objects and Static Map Elements in 3D Driving Scenes.

HM-ViT: Hetero-modal Vehicle-to-Vehicle Cooperative perception with vision transformer

SGDViT: Saliency-Guided Dynamic Vision Transformer for UAV Tracking

Multi-View Fusion of Sensor Data for Improved Perception and Prediction in Autonomous Driving

Enhanced Parking Perception by Multi-Task Fisheye Cross-view Transformers

Emerging Trends in Autonomous Vehicle Perception: Multimodal Fusion for 3D Object Detection