SOTIF-Compliant Scenario Generation Using Semi-Concrete Scenarios and Parameter Sampling

Lukas Birkemeyer,Julian Fuchs,Alessio Gambi,Ina Schaefer

2023-08-14

Abstract:The SOTIF standard (ISO 21448) requires scenario-based testing to verify and validate Advanced Driver Assistance Systems and Automated Driving Systems but does not suggest any practical way to do so effectively and efficiently. Existing scenario generation approaches either focus on exploring or exploiting the scenario space. This generally leads to test suites that cover many known cases but potentially miss edge cases or focused test suites that are effective but also contain less diverse scenarios. To generate SOTIF-compliant test suites that achieve higher coverage and find more faults, this paper proposes semi-concrete scenarios and combines them with parameter sampling to adequately balance scenario space exploration and exploitation. Semi-concrete scenarios enable combinatorial scenario generation techniques that systematically explore the scenario space, while parameter sampling allows for the exploitation of continuous parameters. Our experimental results show that the proposed concept can generate more effective test suites than state-of-the-art coverage-based sampling. Moreover, our results show that including a feedback mechanism to drive parameter sampling further increases test suites' effectiveness.

Software Engineering,Systems and Control

What problem does this paper attempt to address?

The paper primarily addresses the challenges of scenario testing in the verification and validation process of Advanced Driver Assistance Systems (ADAS) and Automated Driving Systems (ADS). Specifically, the paper aims to solve the following key issues:

1. **Meeting SOTIF Standard Requirements**: The SOTIF standard (ISO 21448) requires scenario-based testing to verify whether ADAS and ADS operate as expected within their specified Operational Design Domain (ODD). However, the standard does not provide practical and effective methods for scenario generation.

2. **Balancing Exploration and Exploitation**: Existing scenario generation methods either focus on exploration (covering known cases) or exploitation (discovering specific edge cases). This leads to two extreme situations: one where the test suite covers many known cases but may overlook edge cases, and another where the test suite is effective but lacks diversity in the scenarios.

To address the above issues, the paper proposes the following solutions:

- **Concept of Semi-Concrete Scenarios**: Semi-concrete scenarios lie between logical scenarios and concrete scenarios, allowing for combinatorial coverage of discrete parameters (exploration) while sampling continuous parameters (exploitation). This approach optimizes specific parameter values while ensuring the exploration of the scenario space.

- **Parameter Sampling Strategies**:
- **Parameter Range Sampling**: Randomly sampling continuous parameter values within the defined entire range.
- **Sub-Parameter Range Sampling**: Dividing the range of continuous parameter values into smaller sub-intervals and randomly selecting values from these sub-intervals.

- **Feedback Mechanism**: Introducing a feedback mechanism to guide the parameter sampling process, increasing the effectiveness of the test suite and reducing the occurrence of irrelevant scenarios.

Through experimental evaluation, the paper verifies that the method combining combinatorial testing techniques and parameter sampling can generate more effective test suites than existing technologies. In particular, sub-parameter range sampling and parameter sampling strategies with feedback mechanisms can significantly improve the effectiveness of the test suite in certain cases. These results indicate that the proposed concepts can effectively balance the exploration and exploitation of the scenario space, thereby enhancing the efficiency and effectiveness of SOTIF-compliant scenario testing.