Abstract:Three-dimensional perception from multi-view cameras is a crucial component in autonomous driving systems, which involves multiple tasks like 3D object detection and bird's-eye-view (BEV) semantic segmentation. To improve perception precision, large image encoders, high-resolution images, and long-term temporal inputs have been adopted in recent 3D perception models, bringing remarkable performance gains. However, these techniques are often incompatible in training and inference scenarios due to computational resource constraints. Besides, modern autonomous driving systems prefer to adopt an end-to-end framework for multi-task 3D perception, which can simplify the overall system architecture and reduce the implementation complexity. However, conflict between tasks often arises when optimizing multiple tasks jointly within an end-to-end 3D perception model. To alleviate these issues, we present an end-to-end framework named HENet for multi-task 3D perception in this paper. Specifically, we propose a hybrid image encoding network, using a large image encoder for short-term frames and a small image encoder for long-term temporal frames. Then, we introduce a temporal feature integration module based on the attention mechanism to fuse the features of different frames extracted by the two aforementioned hybrid image encoders. Finally, according to the characteristics of each perception task, we utilize BEV features of different grid sizes, independent BEV encoders, and task decoders for different tasks. Experimental results show that HENet achieves state-of-the-art end-to-end multi-task 3D perception results on the nuScenes benchmark, including 3D object detection and BEV semantic segmentation. The source code and models will be released at <a class="link-external link-https" href="https://github.com/VDIGPKU/HENet" rel="external noopener nofollow">this https URL</a>.

SDAPNet: End-to-End Multi-task Simultaneous Detection and Prediction Network.

Learning Temporal Cues by Predicting Objects Move for Multi-camera 3D Object Detection

MotionNet: Joint Perception and Motion Prediction for Autonomous Driving Based on Bird's Eye View Maps

A Multi-Task Network Based on Dual-Neck Structure for Autonomous Driving Perception

BEVerse: Unified Perception and Prediction in Birds-Eye-View for Vision-Centric Autonomous Driving

AOP-Net: All-in-One Perception Network for Joint LiDAR-based 3D Object Detection and Panoptic Segmentation

A Multi-view 3D Vehicle Detection Method Based On Novel 3D Proposal Generation Method

Multitask Network for Joint Object Detection, Semantic Segmentation and Human Pose Estimation in Vehicle Occupancy Monitoring

TBP-Former: Learning Temporal Bird's-Eye-View Pyramid for Joint Perception and Prediction in Vision-Centric Autonomous Driving

Driving Scene Perception Network: Real-time Joint Detection, Depth Estimation and Semantic Segmentation

BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-View

HENet: Hybrid Encoding for End-to-end Multi-task 3D Perception from Multi-view Cameras

Towards Efficient 3D Object Detection in Bird's-Eye-View Space for Autonomous Driving: A Convolutional-Only Approach

Multi-View Fusion of Sensor Data for Improved Perception and Prediction in Autonomous Driving

MODNet: Moving Object Detection Network with Motion and Appearance for Autonomous Driving

Multi-View 3D Object Detection Network for Autonomous Driving

SA-BEV: Generating Semantic-Aware Bird's-Eye-View Feature for Multi-view 3D Object Detection

Multi-Task Foreground-Aware Network with Depth Completion for Enhanced RGB-D Fusion Object Detection Based on Transformer

MaskBEV: Towards A Unified Framework for BEV Detection and Map Segmentation

Real-time Joint Object Detection and Semantic Segmentation Network for Automated Driving