Abstract:Autonomous driving technologies often use state-of-the-art artificial intelligence algorithms to understand the relationship between the vehicle and the external environment, to predict the changes of the environment, and then to plan and control the behaviors of the vehicle accordingly. The complexity of such technologies makes it challenging to evaluate the performance of autonomous driving systems and to find ways to improve them. The current approaches to evaluating such autonomous driving systems largely use a single score to indicate the overall performance of a system, but domain experts have difficulties in understanding how individual components or algorithms in an autonomous driving system may contribute to the score. To address this problem, we collaborate with domain experts on autonomous driving algorithms, and propose a visual evaluation method for autonomous driving. Our method considers the data generated in all components during the whole process of autonomous driving, including perception results, planning routes, prediction of obstacles, various controlling parameters, and evaluation of comfort. We develop a visual analytics workflow to integrate an evaluation mathematical model with adjustable parameters, support the evaluation of the system from the level of the overall performance to the level of detailed measures of individual components, and to show both evaluation scores and their contributing factors. Our implemented visual analytics system provides an overview evaluation score at the beginning and shows the animation of the dynamic change of the scores at each period. Experts can interactively explore the specific component at different time periods and identify related factors. With our method, domain experts not only learn about the performance of an autonomous driving system, but also identify and access the problematic parts of each component. Our visual evaluation system can be applied to the autonomous driving simulation system and used for various evaluation cases. The results of using our system in some simulation cases and the feedback from involved domain experts confirm the usefulness and efficiency of our method in helping people gain in-depth insight into autonomous driving systems.

Autonomy Evaluation Methods for the Autonomous Ground Vehicles

A review of recent advances, techniques, and control algorithms for automated guided vehicle systems

A Survey of Path Planning Algorithms for Autonomous Vehicles

An Improved Grey Relational Theory Evaluation Method: Considering the Comprehensive Performance of Autonomous Vehicles in Virtual Test

A survey on perception, localization, planning, and control of autonomous vehicles: challenges and solutions

Performance Evaluation of an Autonomous Vehicle Using Resilience Engineering

Visual Evaluation for Autonomous Driving

Design and Experimental Evaluation of a Hierarchical Controller for an Autonomous Ground Vehicle with Large Uncertainties

Autonomous Advanced Aerial Mobility -- An End-to-end Autonomy Framework for UAVs and Beyond

An Overview about Emerging Technologies of Autonomous Driving

An Evaluation Method for Automated Vehicles Combining Subjective and Objective Factors

Towards a Universal Evaluation Model for Careful and Competent Autonomous Driving

Reliability assessment of autonomous vehicles based on the safety control structure

Perception in autonomous ground vehicles

A taxonomy of validation strategies to ensure the safe operation of highly automated vehicles

Recent development in automatic guidance and autonomous vehicle for agriculture:A Review

Guidance and Control for Own Aircraft in the Autonomous Air Combat: A Historical Review and Future Prospects

Analysis Of Key Technologies in Autonomous Guided Vehicles (Agvs) Navigation Technology

Autonomous eVTOL: A Summary of Researches and Challenges

Towards the Unified Principles for Level 5 Autonomous Vehicles

Autonomous Ground Vehicles—Concepts and a Path to the Future