Abstract:Automated driving desires better performance on tasks like motion planning and interacting with pedestrians in mixed-traffic environments. Deep learning algorithms can achieve high performance in these tasks with remarkable visual scene understanding and generalization abilities. However, when common scene-parsing methods are used to train end-to-end models, limitations of explainability in such algorithms inhibit their implementations in fully automated driving. The main challenges include algorithm performance deficiencies and inconsistencies, insufficient AI transparency, degraded user trust, and undermining human-AI interactions. This research aids the decision-making performance and transparency of automated driving systems by providing multi-modal explanations, especially when interacting with pedestrians. The proposed algorithm combines global visual features and interrelation features by parsing scene images as self-constructed graphs and using an attention-based module to capture the interrelationship among the ego-vehicle and other traffic-related objects. The output modules make decisions while simultaneously generating semantic text explanations. The results show that the fusion of the features from global frames and interrelational graphs improves decision-making and explanation predictions compared to two state-of-the-art benchmark algorithms. The interrelation module also enhances algorithm transparency by disclosing the visual attention used for decision-making. The importance of interrelation features on the two prediction tasks is further revealed along with the underlying mechanism of multitask learning on the datasets with hierarchical labels. The proposed model improves driving decision-making during pedestrian interactions with intelligible reasoning cues for building an appropriate mental model of automated driving performance for human users.

Driving behavior explanation with multi-level fusion

Multi-View Fusion of Sensor Data for Improved Perception and Prediction in Autonomous Driving

Sensor Fusion: Gated Recurrent Fusion to Learn Driving Behavior from Temporal Multimodal Data

Exploiting Multi-Modal Fusion for Urban Autonomous Driving Using Latent Deep Reinforcement Learning

Attention-Based Interrelation Modeling for Explainable Automated Driving

Multi-Modal Fusion Transformer for End-to-End Autonomous Driving

BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation

Leveraging Driver Attention for an End-to-End Explainable Decision-Making From Frontal Images

Gated Recurrent Fusion to Learn Driving Behavior from Temporal Multimodal Data

Multi-modal policy fusion for end-to-end autonomous driving

Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformer

CrossFuser: Multi-Modal Feature Fusion for End-to-End Autonomous Driving Under Unseen Weather Conditions

Driving Behavior Prediction Considering Cognitive Prior and Driving Context

NLE-DM: Natural-Language Explanations for Decision Making of Autonomous Driving Based on Semantic Scene Understanding

MMFN: Multi-Modal-Fusion-Net for End-to-End Driving

AEFusion: an Attention-Based Ensemble Learning Approach for BEV Fusion Perception in Autonomous Modular Buses

RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Learning in Multi-Modal Large Language Model

Multi-level Attention Fusion for Multimodal Driving Maneuver Recognition

Multi-Modality Cascaded Fusion Technology for Autonomous Driving