Abstract:Advanced Driver Assistance Systems (ADAS) and Automated Driving Systems (ADS) are expected to improve comfort, productivity and, most importantly, safety for all road users. To ensure that the systems are safe, rules and regulations describing the systems' approval and validation procedures are in effect in Europe. The UNECE Regulation 157 (R157) is one of those. Annex 3 of R157 describes two driver models, representing the performance of a "competent and careful" driver, which can be used as benchmarks to determine whether, in certain situations, a crash would be preventable by a human driver. However, these models have not been validated against human behavior in real safety-critical events. Therefore, this study uses counterfactual simulation to assess the performance of the two models when applied to 38 safety-critical cut-in near-crashes from the SHRP2 naturalistic driving study. The results show that the two computational models performed rather differently from the human drivers: one model showed a generally delayed braking reaction compared to the human drivers, causing crashes in three of the original near-crashes. The other model demonstrated, in general, brake onsets substantially earlier than the human drivers, possibly being overly sensitive to lateral perturbations. That is, the first model does not seem to behave as the competent and careful driver it is supposed to represent, while the second seems to be overly careful. Overall, our results show that, if models are to be included in regulations, they need to be substantially improved. We argue that achieving this will require better validation across the scenario types that the models are intended to cover (e.g., cut-in conflicts), a process which should include applying the models counterfactually to near-crashes and validating them against several different safety related metrics.

Predictive models of safety based on audit findings: Part 1: Model development and reliability.

Predictive models of safety based on audit findings: Part 2: Measurement of model validity

An audit-based prediction model for aviation maintenance safety

Safety of Airport Approach Prediction Using a Skill - and Rule-Based Pilot Model

Routes to failure: Analysis of 41 civil aviation accidents from the Republic of China using the human factors analysis and classification system

Modelling Causal Relationship Among Performance Shaping Factors Through Bayesian Network on Aviation Safety

Human error analysis of commercial aviation accidents: application of the Human Factors Analysis and Classification system (HFACS)

Analysis on coupling dynamic effect of human errors in aviation safety

Using HFACS to understand human error in railway dispatcher performance: a study of proactive safety inspection records

Human Factors Risk Assessment: an Integrated Method for Improving Safety in Clinical Use of Medical Devices

A Cautionary Tail: A Framework and Case Study for Testing Predictive Model Validity

Statistical Development and Validation of Clinical Prediction Models

Modeling and Simulation in an Aircraft Safety Design Based on a Hybrid AHP and FCA Algorithm

How audits fail according to accident investigations: A counterfactual logic analysis

Validation of human benchmark models for Automated Driving System approval: How competent and careful are they really?

Simulation Study on Civil Aviation Human Reliability Learning from Incidents Using System Dynamics

Testing the reliability and validity of risk assessment methods in Human Factors and Ergonomics

The development of audit detection risk assessment system: Using the fuzzy theory and audit risk model

A Comparison of Rule-Based and Machine Learning Models for Classification of Human Factors Aviation Safety Event Reports

Human Factors Analysis of Air Traffic Safety Based on HFACS-BN Model

Predicating the safety of airport approaches using a cognitive pilot model