Abstract:Vision-centric Bird's Eye View (BEV) perception, encompassing object detection and map segmentation, plays a pivotal role in providing crucial 3D environmental information for autonomous driving decisions. However, due to the inherent absence of depth information in 2D images, the conversion of perspective views to BEV poses challenges and hinders the performance of camera-based BEV perception in comparison to methods equipped with depth sensors. In this research paper, we propose an innovative approach that integrates depth estimation into camera-based BEV perception. By employing a depth estimation network, the method enhances the transformation of 2D-3D features. Specifically, our method consists of a depth estimation branch and a BEV perception branch. The input image is fed into the shared image encoder to extract multi-scale features. In the depth estimation branch, these features are utilized to generate a depth map through the depth decoder, which, in combination with sequential images and relative pose information, forms the basis for reprojection photometric error, guiding and supervising the branch. To address the challenge of scale ambiguity in monocular depth estimation, we incorporate ground-truth trajectory information collected by an IMU to constrain the predicted depth values, ensuring that the predicted depth is scale-aware. In the BEV perception branch, the afore-mentioned multi-scale features are projected into 3D space along the perspective rays, with the assistance of depth information derived from the depth estimation branch. Subsequently, the 3D features are collapsed along the vertical axis to generate BEV features, which are further input into a task-specific head after feature extraction. Experimental results on the nuScenes dataset demonstrate that our proposed method effectively enhances the performance of BEV-based object detection and map semantic segmentation by 2.8 % and 2.2 %, respectively.

CoDRMA: Collaborative Depth Refinement Via Dual-Mask and Dual-Attention for Bird’s Eye View Collaborative 3D Object Detection

Collaboration Helps Camera Overtake LiDAR in 3D Detection

BEVDepth: Acquisition of Reliable Depth for Multi-view 3D Object Detection

KAN-RCBEVDepth: A multi-modal fusion algorithm in object detection for autonomous driving

UVCPNet: A UAV-Vehicle Collaborative Perception Network for 3D Object Detection

OA-BEV: Bringing Object Awareness to Bird's-Eye-View Representation for Multi-Camera 3D Object Detection

A Streamlined Framework for Bev-Based 3d Object Detection with Prior Masking

CoBEV: Elevating Roadside 3D Object Detection with Depth and Height Complementarity

BEVDistill: Cross-Modal BEV Distillation for Multi-View 3D Object Detection

BEV-LGKD: A Unified LiDAR-Guided Knowledge Distillation Framework for Multi-View BEV 3D Object Detection

Instance-aware Multi-Camera 3D Object Detection with Structural Priors Mining and Self-Boosting Learning

Co-Fix3D: Enhancing 3D Object Detection with Collaborative Refinement

Bridging the View Disparity Between Radar and Camera Features for Multi-Modal Fusion 3D Object Detection

BEV-LGKD: A Unified LiDAR-Guided Knowledge Distillation Framework for BEV 3D Object Detection

Depth-Assisted Camera-Based Bird's Eye View Perception for Autonomous Driving

ODM3D: Alleviating Foreground Sparsity for Semi-Supervised Monocular 3D Object Detection

Enhance the 3D Object Detection With 2D Prior

RCBEVDet: Radar-camera Fusion in Bird's Eye View for 3D Object Detection

A Versatile Multi-View Framework for LiDAR-based 3D Object Detection with Guidance from Panoptic Segmentation