Abstract:Depth estimation, visual odometry (VO), and bird's-eye-view (BEV) scene layout estimation present three critical tasks for driving scene perception, which is fundamental for motion planning and navigation in autonomous driving. Though they are complementary to each other, prior works usually focus on each individual task and rarely deal with all three tasks together. A naive way is to accomplish them independently in a sequential or parallel manner, but there are many drawbacks, i.e., 1) the depth and VO results suffer from the inherent scale ambiguity issue; 2) the BEV layout is directly predicted from the front-view image without using any depth-related information, although the depth map contains useful geometry clues for inferring scene layouts. In this paper, we address these issues by proposing a novel joint perception framework named JPerceiver, which can simultaneously estimate scale-aware depth and VO as well as BEV layout from a monocular video sequence. It exploits the cross-view geometric transformation (CGT) to propagate the absolute scale from the road layout to depth and VO based on a carefully-designed scale loss. Meanwhile, a cross-view and cross-modal transfer (CCT) module is devised to leverage the depth clues for reasoning road and vehicle layout through an attention mechanism. JPerceiver can be trained in an end-to-end multi-task learning way, where the CGT scale loss and CCT module promote inter-task knowledge transfer to benefit feature learning of each task. Experiments on Argoverse, Nuscenes and KITTI show the superiority of JPerceiver over existing methods on all the above three tasks in terms of accuracy, model size, and inference speed. The code and models are available at~\href{<a class="link-external link-https" href="https://github.com/sunnyHelen/JPerceiver" rel="external noopener nofollow">this https URL</a>}{<a class="link-external link-https" href="https://github.com/sunnyHelen/JPerceiver" rel="external noopener nofollow">this https URL</a>}.

YoloDepth: Yolo with Monocular Depth Estimation for Object Distance Measurement

A Robust Monocular Depth Estimation Framework Based on Light-Weight ERF-Pspnet for Day-Night Driving Scenes

A Depth Estimation Framework Based on Unsupervised Learning and Cross-Modal Translation

A Fusion Method Aiming at Environmental Perception of Autonomous Vehicle Based on Visual Scheme

Spatio-Temporal Fusion of LiDAR and Camera Data for Omnidirectional Depth Perception

Multi-Task Environmental Perception Methods for Autonomous Driving

YOLOMH: you only look once for multi-task driving perception with high efficiency

YOLOP: You Only Look Once for Panoptic Driving Perception

Depth Estimation Matters Most: Improving Per-Object Depth Estimation for Monocular 3D Detection and Tracking

JPerceiver: Joint Perception Network for Depth, Pose and Layout Estimation in Driving Scenes

YOLO-Vehicle-Pro: A Cloud-Edge Collaborative Framework for Object Detection in Autonomous Driving under Adverse Weather Conditions

MobileYOLO: Real-Time Object Detection Algorithm in Autonomous Driving Scenarios

Pseudo-Mono for Monocular 3D Object Detection in Autonomous Driving

Object Detection with Depth Information in Road Scenes

Research on Multi-Task Perception Network of Traffic Scene Based on Feature Fusion1

Real-Time Monocular Human Depth Estimation and Segmentation on Embedded Systems

RealNet: Combining Optimized Object Detection with Information Fusion Depth Estimation Co-Design Method on IoT

Vision-Based Environmental Perception for Autonomous Driving

Learning Cross-Modality Interaction for Robust Depth Perception of Autonomous Driving

Learning Depth Estimation from Memory Infusing Monocular Cues: A Generalization Prediction Approach

A Real-Time Traffic Environmental Perception Algorithm Fusing Stereo Vision and Deep Network.