Abstract:Automated driving desires better performance on tasks like motion planning and interacting with pedestrians in mixed-traffic environments. Deep learning algorithms can achieve high performance in these tasks with remarkable visual scene understanding and generalization abilities. However, when common scene-parsing methods are used to train end-to-end models, limitations of explainability in such algorithms inhibit their implementations in fully automated driving. The main challenges include algorithm performance deficiencies and inconsistencies, insufficient AI transparency, degraded user trust, and undermining human-AI interactions. This research aids the decision-making performance and transparency of automated driving systems by providing multi-modal explanations, especially when interacting with pedestrians. The proposed algorithm combines global visual features and interrelation features by parsing scene images as self-constructed graphs and using an attention-based module to capture the interrelationship among the ego-vehicle and other traffic-related objects. The output modules make decisions while simultaneously generating semantic text explanations. The results show that the fusion of the features from global frames and interrelational graphs improves decision-making and explanation predictions compared to two state-of-the-art benchmark algorithms. The interrelation module also enhances algorithm transparency by disclosing the visual attention used for decision-making. The importance of interrelation features on the two prediction tasks is further revealed along with the underlying mechanism of multitask learning on the datasets with hierarchical labels. The proposed model improves driving decision-making during pedestrian interactions with intelligible reasoning cues for building an appropriate mental model of automated driving performance for human users.

MindReaD: Enhancing Pedestrian-Vehicle Interaction with Micro-Level Reasoning Data Annotation

Can Reasons Help Improve Pedestrian Intent Estimation? A Cross-Modal Approach

From Time to Space: Automatic Annotation of Unmarked Traffic Scene Based on Trajectory Data.

Cognitive Level-$k$ Meta-Learning for Safe and Pedestrian-Aware Autonomous Driving

Pedestrian Detection for Autonomous Vehicles Using Virtual-to-Real Augmentation

Reasoning Makes Good Annotators : an Automatic Task-specific Rules Distilling Framework for Low-resource Relation Extraction

OpenAnnotate2: Multi-Modal Auto-Annotating for Autonomous Driving

Diving Deeper Into Pedestrian Behavior Understanding: Intention Estimation, Action Prediction, and Event Risk Assessment

World knowledge-enhanced Reasoning Using Instruction-guided Interactor in Autonomous Driving

Reinforcing Pedestrian Parsing on Small Scale Dataset

Ambiguous Annotations: When is a Pedestrian not a Pedestrian?

PSI: A Pedestrian Behavior Dataset for Socially Intelligent Autonomous Car

TBD Pedestrian Data Collection: Towards Rich, Portable, and Large-Scale Natural Pedestrian Data

ARAIDA: Analogical Reasoning-Augmented Interactive Data Annotation

No Need to Sacrifice Data Quality for Quantity: Crowd-Informed Machine Annotation for Cost-Effective Understanding of Visual Data

Bifold and Semantic Reasoning for Pedestrian Behavior Prediction

Attention-Based Interrelation Modeling for Explainable Automated Driving

Proximity based automatic data annotation for autonomous driving

The Reasonable Crowd: Towards evidence-based and interpretable models of driving behavior

A Matter of Annotation: An Empirical Study on In Situ and Self-Recall Activity Annotations from Wearable Sensors

Robust Roadside Perception: an Automated Data Synthesis Pipeline Minimizing Human Annotation