Abstract:Vision-centric Bird's Eye View (BEV) perception, encompassing object detection and map segmentation, plays a pivotal role in providing crucial 3D environmental information for autonomous driving decisions. However, due to the inherent absence of depth information in 2D images, the conversion of perspective views to BEV poses challenges and hinders the performance of camera-based BEV perception in comparison to methods equipped with depth sensors. In this research paper, we propose an innovative approach that integrates depth estimation into camera-based BEV perception. By employing a depth estimation network, the method enhances the transformation of 2D-3D features. Specifically, our method consists of a depth estimation branch and a BEV perception branch. The input image is fed into the shared image encoder to extract multi-scale features. In the depth estimation branch, these features are utilized to generate a depth map through the depth decoder, which, in combination with sequential images and relative pose information, forms the basis for reprojection photometric error, guiding and supervising the branch. To address the challenge of scale ambiguity in monocular depth estimation, we incorporate ground-truth trajectory information collected by an IMU to constrain the predicted depth values, ensuring that the predicted depth is scale-aware. In the BEV perception branch, the afore-mentioned multi-scale features are projected into 3D space along the perspective rays, with the assistance of depth information derived from the depth estimation branch. Subsequently, the 3D features are collapsed along the vertical axis to generate BEV features, which are further input into a task-specific head after feature extraction. Experimental results on the nuScenes dataset demonstrate that our proposed method effectively enhances the performance of BEV-based object detection and map semantic segmentation by 2.8 % and 2.2 %, respectively.

360BEV: Panoramic Semantic Mapping for Indoor Bird’s-Eye View

OneBEV: Using One Panoramic Image for Bird's-Eye-View Semantic Mapping

Semi-Supervised Learning for Visual Bird's Eye View Semantic Segmentation

Can We PASS Beyond the Field of View? Panoramic Annular Semantic Segmentation for Real-World Surrounding Perception

Semi-Supervised Learning for Visual Bird's Eye View Semantic Segmentation

Aerial-PASS: Panoramic Annular Scene Segmentation in Drone Videos

RSBEV: Multi-view Collaborative Segmentation of 3D Remote Sensing Scenes with Bird’s-Eye-View Representation

Bird's-Eye-View Panoptic Segmentation Using Monocular Frontal View Images

BEV-Seg: Bird's Eye View Semantic Segmentation Using Geometry and Semantic Point Cloud

SG-BEV: Satellite-Guided BEV Fusion for Cross-View Semantic Segmentation

GAFB-Mapper: Ground Aware Forward-Backward View Transformation for Monocular BEV Semantic Mapping

Focus on BEV: Self-calibrated Cycle View Transformation for Monocular Birds-Eye-View Segmentation

Object-aware Semantic Mapping of Indoor Scenes Using Octomap

SkyEye: Self-Supervised Bird's-Eye-View Semantic Mapping Using Monocular Frontal View Images

Depth-Assisted Camera-Based Bird's Eye View Perception for Autonomous Driving

Trans4Map: Revisiting Holistic Bird's-Eye-View Mapping from Egocentric Images to Allocentric Semantics with Vision Transformers

Bird’s-Eye View Semantic Segmentation and Voxel Semantic Segmentation Based on Frustum Voxel Modeling and Monocular Camera

M$^2$BEV: Multi-Camera Joint 3D Detection and Segmentation with Unified Birds-Eye View Representation

BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation

Improving Bird's Eye View Semantic Segmentation by Task Decomposition