Abstract:Path planning is a significant and challenging task in the domain of automatic driving. Many applications, such as autonomous driving, robotic navigation and aircraft object tracking in complex and changing urban road scenes, need accurate and robust path planning by detecting obstacles in the forward direction. The traditional methods only rely on the path search method without considering the environmental factors, the vehicle path planning method cannot deal with the complex and changeable environment. To deal with above problems, we propose a perception-aware based multi-modal feature fusion approach that combines visual-inertial odometer (VIO) poses and semantic obstacles in the forward scene of vehicles to plan driving paths. The proposed method takes environment awareness as the guide and combines path search algorithm to realize path optimization task in complex environment. The proposed approach first uses a long short memory network (LSTM) to build a VIO that fuses visual and inertial data for pose estimation. To detect obstacles, the proposed method uses a segmentation model with a lightweight structure to extract semantic 3D landmarks. Finally, a path search strategy combining an A* algorithm and visual information is proposed to plan driving paths for intelligent vehicles. We estimate the proposed path planning method on assimilated scenes and public datasets (KITTI and Cityscapes) by using a micro controller (Jetson Xavier NX) installed on a small vehicle. We also show comparable results with path planning that only uses the greedy algorithm or heuristic algorithm without using visual information and show that our method is adequate in coping with different complex scenes.

Navigating an Automated Driving Vehicle via the Early Fusion of Multi-Modality

Multimodal End-to-End Autonomous Driving

Autonomous Driving with Human Guided Image Feature Extraction

Multi-Modal Sensor Fusion-Based Deep Neural Network for End-to-End Autonomous Driving With Scene Understanding

Multi-Modal Fusion Transformer for End-to-End Autonomous Driving

LiDAR-as-Camera for End-to-End Driving

Probabilistic End-to-End Vehicle Navigation in Complex Dynamic Environments with Multimodal Sensor Fusion

MMFN: Multi-Modal-Fusion-Net for End-to-End Driving

Exploiting Multi-Modal Fusion for Urban Autonomous Driving Using Latent Deep Reinforcement Learning

Autonomous Multi-Sensor Fusion Techniques for Environmental Perception in Self-Driving Vehicles

End-to-End Autonomous Driving With Semantic Depth Cloud Mapping and Multi-Agent

Multimodal Fusion Using Deep Learning Applied to Driver's Referencing of Outside-Vehicle Objects

M2DA: Multi-Modal Fusion Transformer Incorporating Driver Attention for Autonomous Driving

Multi-Modal Neural Feature Fusion for Automatic Driving Through Perception-Aware Path Planning

Towards Self-Supervised High Level Sensor Fusion

Enhanced Perception for Autonomous Driving Using Semantic and Geometric Data Fusion

Multimodal fusion for autonomous navigation via deep reinforcement learning with sparse rewards and hindsight experience replay

A Multimodal Perception-Driven Self Evolving Autonomous Ground Vehicle

Integrating Modular Pipelines with End-to-End Learning: A Hybrid Approach for Robust and Reliable Autonomous Driving Systems