S2O: An Integrated Driving Decision-making Performance Evaluation Method Bridging Subjective Feeling to Objective Evaluation

Yuning Wang,Zehong Ke,Yanbo Jiang,Jinhao Li,Shaobing Xu,John M. Dolan,Jianqiang Wang

2024-09-23

Abstract:Autonomous driving decision-making is one of the critical modules towards intelligent transportation systems, and how to evaluate the driving performance comprehensively and precisely is a crucial challenge. A biased evaluation misleads and hinders decision-making modification and development. Current planning evaluation metrics include deviation from the real driver trajectory and objective driving experience indicators. The former category does not necessarily indicate good driving performance since human drivers also make errors and has been proven to be ineffective in interactive close-loop systems. On the other hand, existing objective driving experience models only consider limited factors, lacking comprehensiveness. And the integration mechanism of various factors relies on intuitive experience, lacking precision. In this research, we propose S2O, a novel integrated decision-making evaluation method bridging subjective human feeling to objective evaluation. First, modified fundamental models of four kinds of driving factors which are safety, time efficiency, comfort, and energy efficiency are established to cover common driving factors. Then based on the analysis of human rating distribution regularity, a segmental linear fitting model in conjunction with a complementary SVM segment classifier is designed to express human's subjective rating by objective driving factor terms. Experiments are conducted on the D2E dataset, which includes approximately 1,000 driving cases and 40,000 human rating scores. Results show that S2O achieves a mean absolute error of 4.58 to ground truth under a percentage scale. Compared with baselines, the evaluation error is reduced by 32.55%. Implementation on the SUMO platform proves the real-time efficiency of online evaluation, and validation on performance evaluation of three autonomous driving planning algorithms proves the feasibility.

Robotics,Human-Computer Interaction

What problem does this paper attempt to address?

The paper attempts to address the problem of how to comprehensively and accurately evaluate driving performance in the decision-making process of autonomous driving. Specifically, the existing evaluation methods have the following issues:

1. **Bias evaluation based on real driver trajectories**: This method cannot fully reflect good driving performance because human drivers also make mistakes, and it has been proven ineffective in interactive closed-loop systems.
2. **Existing objective driving experience models**: These models only consider limited factors, lack comprehensiveness, and the integration mechanism of various factors relies on intuitive experience, lacking precision.

Therefore, the paper proposes a new comprehensive decision evaluation method—S2O (Subjective to Objective), aiming to transform subjective human perceptions into objective evaluation results. S2O improves models of four driving factors (safety, time efficiency, comfort, and energy efficiency) and, combined with human rating data, designs a piecewise linear fitting model to express human subjective ratings. Experimental results show that S2O reduces evaluation errors by 32.55% compared to baseline methods and validates its effectiveness in real-time evaluation on the SUMO platform.