Abstract:For autonomous driving, it is imperative to perform various high-computation image recognition tasks with high accuracy, utilizing diverse sensors to perceive the surrounding environment. Specifically, cameras are used to perform lane detection, object detection, and segmentation, and, in the absence of lidar, tasks extend to inferring 3D information through depth estimation, 3D object detection, 3D reconstruction, and SLAM. However, accurately processing all these image recognition operations in real-time for autonomous driving under constrained hardware conditions is practically unfeasible. In this study, considering the characteristics of image recognition tasks performed by these sensors and the given hardware conditions, we investigated MTL (multi-task learning), which enables parallel execution of various image recognition tasks to maximize their processing speed, accuracy, and memory efficiency. Particularly, this study analyzes the combinations of image recognition tasks for autonomous driving and proposes the MDO (multi-task decision and optimization) algorithm, consisting of three steps, as a means for optimization. In the initial step, a MTS (multi-task set) is selected to minimize overall latency while meeting minimum accuracy requirements. Subsequently, additional training of the shared backbone and individual subnets is conducted to enhance accuracy with the predefined MTS. Finally, both the shared backbone and each subnet undergo compression while maintaining the already secured accuracy and latency performance. The experimental results indicate that integrated accuracy performance is critically important in the configuration and optimization of MTL, and this integrated accuracy is determined by the ITC (inter-task correlation). The MDO algorithm was designed to consider these characteristics and construct multi-task sets with tasks that exhibit high ITC. Furthermore, the implementation of the proposed MDO algorithm, coupled with additional SSL (semi-supervised learning) based training, resulted in a significant performance enhancement. This advancement manifested as approximately a 12% increase in object detection mAP performance, a 15% improvement in lane detection accuracy, and a 27% reduction in latency, surpassing the results of previous three-task learning techniques like YOLOP and HybridNet.

Multi-task Learning with Attention for End-to-end Autonomous Driving

Multi-task Learning for Real-time Autonomous Driving Leveraging Task-adaptive Attention Generator

MultiNet: Multi-Modal Multi-Task Learning for Autonomous Driving

Brain Inspired Cognitive Model with Attention for Self-Driving Cars

Scaling Vision-based End-to-End Driving with Multi-View Attention Learning

Using External Attention in Vision-based Autonomous Driving

Explaining Autonomous Driving by Learning End-to-End Visual Attention

Conditional Affordance Learning for Driving in Urban Environments

Multi-Task Conditional Imitation Learning for Autonomous Navigation at Crowded Intersections

Multi-task Perception for Autonomous Driving

Multi-Task Learning in Autonomous Driving Scenarios Via Adaptive Feature Refinement Networks

Unsupervised Reinforcement Learning for Multi-Task Autonomous Driving: Expanding Skills and Cultivating Curiosity

Dynamic Conditional Imitation Learning for Autonomous Driving

Yaw-Guided Imitation Learning for Autonomous Driving in Urban Environments

Optimal Configuration of Multi-Task Learning for Autonomous Driving

End-to-end Driving Model Based on Deep Learning and Attention Mechanism

End-to-end Driving via Conditional Imitation Learning

Addressing Limitations of State-Aware Imitation Learning for Autonomous Driving

Deep learning and control algorithms of direct perception for autonomous driving

Multi-task Safe Reinforcement Learning for Navigating Intersections in Dense Traffic

Spatial-Temporal Multimodal End-to-End Autonomous Driving.