Abstract:In order to ensure autonomous vehicles are safe for on-road deployment, simulation-based testing has become an integral complement to on-road testing. The rise in simulation testing and validation reflects a growing need to verify that AV behavior is consistent with desired outcomes even in edge case scenarios $-$ which may seldom or never appear in on-road testing data. This raises a critical question: to what extent are AV failures in simulation consistent with data collected from real-world testing? As a result of the gap between simulated and real sensor data (sim-to-real gap), failures in simulation can either be spurious (simulation- or simulator-specific issues) or relevant (safety-critical AV system issues). One possible method for validating if simulated time series failures are consistent with real world time series sensor data could involve retrieving instances of the failure scenario from a real-world time series dataset, in order to understand AV performance in these scenarios. Adopting this strategy, we propose a formal definition of what constitutes a match between a real-world labeled time series data item and a simulated scenario written from a fragment of the Scenic probabilistic programming language for simulation generation. With this definition of a match, we develop a querying algorithm that identifies the subset of a labeled time series dataset matching a given scenario. To allow this approach to be used to verify the safety of other cyber-physical systems (CPS), we present a definition and algorithm for matching scalable beyond the autonomous vehicles domain. Experiments demonstrate the precision and scalability of the algorithm for a set of challenging and uncommon time series scenarios identified from the nuScenes autonomous driving dataset. We include a full system implementation of the querying algorithm freely available for use across a wide range of CPS.

A Consistency Analysis Method for Traffic Sequence Charts

Automated Inconsistency Analysis of Real-Time Requirements: A Domain Expert Friendly Approach.

Navigating Dimensionality through State Machines in Automotive System Validation

Concretization of Abstract Traffic Scene Specifications Using Metaheuristic Search

Intersection Two-Vehicle Crash Scenario Specification for Automated Vehicle Safety Evaluation Using Sequence Analysis and Bayesian Networks

Scene-Extrapolation: Generating Interactive Traffic Scenarios

Fingerprint of a Traffic Scene: an Approach for a Generic and Independent Scene Assessment

Towards a Digital Highway Code using Formal Modelling and Verification of Timed Automata

Scenario-Based Consistency Verification of Component-Based Real-Time System Designs

A Scalable Approach to Detecting Safety Requirements Inconsistencies for Railway Systems

Querying Labeled Time Series Data with Scenario Programs

Towards a Completeness Argumentation for Scenario Concepts

Checking Component-Based Designs for Scenario-Based Specifications

ArChes -- Automatic generation of component fault trees from continuous function charts

Criticality Metrics for Automated Driving: A Review and Suitability Analysis of the State of the Art

Scenario-Based Safety Assessment Framework for Automated Vehicles

Scenario-Based Behavioral Nonexistent Consistency Checking for Cyber-Physical Systems

Specification-Guided Critical Scenario Identification for Automated Driving

A Concept for Semi-Automatic Configuration of Sufficiently Valid Simulation Setups for Automated Driving Systems

Finding Critical Scenarios for Automated Driving Systems: A Systematic Mapping Study