Abstract:Three-dimensional perception from multi-view cameras is a crucial component in autonomous driving systems, which involves multiple tasks like 3D object detection and bird's-eye-view (BEV) semantic segmentation. To improve perception precision, large image encoders, high-resolution images, and long-term temporal inputs have been adopted in recent 3D perception models, bringing remarkable performance gains. However, these techniques are often incompatible in training and inference scenarios due to computational resource constraints. Besides, modern autonomous driving systems prefer to adopt an end-to-end framework for multi-task 3D perception, which can simplify the overall system architecture and reduce the implementation complexity. However, conflict between tasks often arises when optimizing multiple tasks jointly within an end-to-end 3D perception model. To alleviate these issues, we present an end-to-end framework named HENet for multi-task 3D perception in this paper. Specifically, we propose a hybrid image encoding network, using a large image encoder for short-term frames and a small image encoder for long-term temporal frames. Then, we introduce a temporal feature integration module based on the attention mechanism to fuse the features of different frames extracted by the two aforementioned hybrid image encoders. Finally, according to the characteristics of each perception task, we utilize BEV features of different grid sizes, independent BEV encoders, and task decoders for different tasks. Experimental results show that HENet achieves state-of-the-art end-to-end multi-task 3D perception results on the nuScenes benchmark, including 3D object detection and BEV semantic segmentation. The source code and models will be released at <a class="link-external link-https" href="https://github.com/VDIGPKU/HENet" rel="external noopener nofollow">this https URL</a>.

Enhanced encoder–decoder architecture for visual perception multitasking of autonomous driving

Joint Semantic Understanding with a Multilevel Branch for Driving Perception

A Multi-Task Network Based on Dual-Neck Structure for Autonomous Driving Perception

Human Insights Driven Latent Space for Different Driving Perspectives: A Unified Encoder for Efficient Multi-Task Inference

YOLOMH: you only look once for multi-task driving perception with high efficiency

NeurAll: Towards a Unified Visual Perception Model for Automated Driving

Cutransnet: Transformers to Make Strong Encoders for Multi-Task Vision Perception of Autonomous Driving

HENet: Hybrid Encoding for End-to-end Multi-task 3D Perception from Multi-view Cameras

Multi-Task Visual Perception for Object Detection and Semantic Segmentation in Intelligent Driving

LiDAR-BEVMTN: Real-Time LiDAR Bird's-Eye View Multi-Task Perception Network for Autonomous Driving

Multi-Task Deep Learning Model for Autonomous Driving: Object Detection, Semantic Segmentation, and Depth Estimation

Detection-segmentation convolutional neural network for autonomous vehicle perception

Think Twice Before Driving: Towards Scalable Decoders for End-to-End Autonomous Driving

YOLOP: You Only Look Once for Panoptic Driving Perception

A panoramic driving perception fusion algorithm based on multi-task learning

Multi-task Panoramic Driving Perception Algorithm Based on Improved YOLOv5

BEVerse: Unified Perception and Prediction in Birds-Eye-View for Vision-Centric Autonomous Driving

Visual Exemplar Driven Task-Prompting for Unified Perception in Autonomous Driving

Semantic Segmentation and Depth Estimation of Urban Road Scene Images Using Multi-Task Networks

Real-time Joint Object Detection and Semantic Segmentation Network for Automated Driving

Mobip: a lightweight model for driving perception using MobileNet