A Methodology to Identify Relevant Use-Cases and Safety-Critical Scenarios

By Dr. Xizhe Zhang

The vision of the i4Driving Project to create a modular simulation library that captures the complexity of road traffic, managing uncertainties in human behaviours and driving scenarios. This approach will support stakeholders in developing, testing, and validating automated driving systems, ensuring safety in mixed traffic with Automated Vehicles (AVs). The i4Driving methodology paves the way for more robust and resilient Cooperative, connected and automated mobility (CCAM) systems, offering a proposition for AV licensing in the future.

i4Driving’s third innovation is to develop a methodology for identifying users, use cases, and scenarios for the project. The process involves interviews with relevant partners, representing various user groups such as Academia, Insurers, Test Centres, Regulators, Tier 1 suppliers, and more.

Identifying use cases and safety-critical scenarios

i4Driving’s third innovation is to develop a methodology to identify relevant use-cases and safety-critical scenarios. In the first stage of the project, we have used available data (such as accident data, insurance claim record etc.) and conducted interview to identify relevant use cases and perform scenario generation. One of the purposes for defining the use case is to create coherent objectives throughout the system/model development cycle, and to create logical and (ultimately) concrete scenarios.

These interviews define scenario design goals and gather input for model development. The inputs are analysed to extract clear Operational Design Domain (ODD – a term for a set of operating conditions for an automated system, often used in the field of AVs) and behaviour elements, ensuring alignment with the project objectives. Standardised ODD and behaviour tags, based on the ASAM OpenLABEL scenario tagging standard (this uses an ODD and behaviour based tagging hierarchy, together with a JSON based format to tag scenarios for database organisation) are used to facilitate the data collection process. This approach yielded a set of 61 representative scenarios fulfilling task requirements. The process followed the V model for system development, ensuring traceability and efficacy. Figure 1 below outlines the workflow to identify relevant use cases and safety critical scenarios. The four main sources include partners’ ODD and behaviour inputs, existing scenarios (such as Automated lane keeping system (ALKS) regulatory scenarios, outlined below), Miro board brainstorming exercise, and newly created scenarios (such as accident data generated scenarios). ODD and behaviour keywords from partners’ interviews and the Miro board exercised are merged to enhance existing scenarios. All scenarios are hosted in the Safety PoolTM database, allowing easy exchange and access for development and testing (via website and API).

Figure 1. Overview of the workflow

The consideration of ODD during the scenario derivation stage is vital for system development and testing. ODD defines the conditions under which the system is designed to safely operate. Scenarios and ODD both contain scenery and dynamic elements, but scenarios cover additional behavioural elements. Figure 2 below shows the relationship between scenarios and ODD boundaries. ODD represents the system operating boundary, scenarios specify attribute combinations either inside, on, or outside such a boundary. Scenarios help explore ODD boundaries and test edge-cases during the system development and testing phases.

Figure 2. Relation between ODD and scenarios

The Safety Pool™ Scenario Database (SPSD) supports Connected and Autonomous Vehicle (CAV) technology development with over a quarter million test scenarios. It enables scenario sharing, aids real-world and simulation testing, and is part of the CAM TestBedUK ecosystem. Scenarios are organised using ODD and behaviour-based tagging referencing ASAM OpenLABEL, stored in BSI Flex 1889, WMG’s SDL Level-2, and ASAM OpenScenario v1.1 and OpenDrive v1.7 formats. Safety PoolTM database utilises a mix of data-based and knowledge-driven scenario generation methods. There are eight methods contributing to ~250,000 live scenarios. Data-based approaches use accident and insurance records, as well as trial recorded data. Knowledge-based approaches involve expert insights, formal analysis, highway code rules, system ODD, and ontology-based approaches. These diverse methods ensure a broad coverage of scenarios for testing ADS technologies.

Figure 3. Scenario Database in relation to types of user organisations

Process Evaluation

From a validation point of view, the scenario generation methodology must be critically assessed to ensure safety-critical scenarios accurately represent the system’s ODD. Key assessment factors are:

  1. Coverage: The methodology ensures comprehensive scenario coverage by:
    • Drawing from multiple sources, including ODD and behaviour requirements identified by users for abstract scenarios, and utilising the existing scenario database for concrete scenarios. This approach fills any gaps in scenario identification.
    • Following an iterative process, allowing new users to contribute and extend coverage from existing users with the possibility of adding coverage metrics. 
  2. Scalability: The scenario generation method should accommodate new user scenario requests for different user groups or ODD. This prevents limitations and biases toward specific users. The methodology ensures the addition of new user sets without redundant scenarios.   
  3. Data Quality and Assurance: High-quality data, resembling real-world scenarios in simulations, is crucial. The methodology relies on trusted and standardised sources like GIDAS and Safety Pool to meet required quality standards for robust scenario development. 

Scenario Abstraction Levels and V Model

Menzel et. al proposed functional, logical, and concrete scenarios, this was later expanded by the addition of abstract scenario. Different formats (BSI Flex 1889, WMG SDL level 2, ASAM OpenX) cater to different abstract levels, which cover the V model development cycle for various target operations, as seen in Figure 4. The BSI Flex 1889 aims at the natural language level for non-technical audiences. WMG’s SDL level 2 suits stakeholders with varying expertise, offering logical level scenario descriptions. OpenSCENARIO1.x and OpenDRIVE lack human readability, however it enjoys a wide simulation tool execution support. WMG also develops and maintains an open-source tools to translate scenarios between formats, bridging high-level languages and low-level formats. Within the i4Driving project, we further extend the process prior to abstract scenarios to link with user, use cases, and ODD and behaviour requirements, as seen in Figure 4.

Figure 4. V model for system mapped to i4Driving T1.4 scope

Scenario input

Accident Database

The German accident data GIDAS, detailed for Hannover and Dresden since 1999, categorised into Car-to-Car, Car-to-Pedestrian, and Car-to-Cyclist crashes, is used as the first input source for scenario collection. Frequency analysis informs safety-critical scenario creation (Figure 5). Accident analysis considered conflicts scenarios since 2000 in urban areas. The data describes conflict scenarios causing accidents, not the accidents themselves. SAFE-UP project results further contributed to defining safety-critical scenarios for Car-to-Pedestrian and Car-to-Cyclist crashes, using GIDAS dataset as a source.

Figure 5. Derivation process of accident data based on the GIDAS database
Partners’ interviews

A series of 1-to-1 interview were conducted with project partners to explore the relevant use cases and extract the requirements for scenario generation. The outcomes from the interviews contain information gathered in three main groups, the representative end user, the use cases, and the requirements towards ODD and behaviour. An example is shown below (information was slightly adjusted to ensure confidentiality).

Figure 6. Example of an interview
The Miro Board Group Exercise

During Work Package 3 activities, a workshop exercise involved the project partners selecting ODD and behaviour keywords to create scenarios. The outcomes formed essential layers of i4Driving scenarios, as shown in Figure 7.

Figure 7. Individual scenario input based on the Miro board exercise
The UN Regulation No. 157 – ALKS

UN Regulation No. 157 mandates ALKS testing in 3 traffic disturbance scenarios: Cut-in, Cut-out, and Deceleration. Scenarios are based on situational variables and human/ALKS responses. Outcome includes GIDAS accident scenarios, 1-to-1 interview and Miro board requirements, merged scenarios, and ALKS regulation compliance.

The consolidated summary from all four sources outlines the ODD and behaviour requirements for scenario generation across various user groups, including research centres, certification departments, driving simulator providers, research institutes, and insurance companies. Scenarios cover diverse settings, from urban to highway environments, encompassing adverse weather conditions, interactions with pedestrians, cyclists, and other vehicles. The focus is on analysing human driving behaviour, testing ADS compliance with traffic rules, and comparing ADS and human driver behaviours. This holistic approach ensures a comprehensive and representative scenario set for effective testing of autonomous driving technologies.

Example Scenario (Visualisation & BSI Flex 1889 Description)

Figure 8 below provides an overview of an example scenario:

Figure 8. Scenario – Motorway cut-in with vehicle following

The scenario takes place in the early morning, under a dark lighting condition, and a clear cloud condition. 

Scenario description
  • There is no junction present.
  • There is 1 road, Road R1. Road R1 is a long straight motorway. There are 3 lanes on Road R1, Lane L1, Lane L2 and Lane L3. The travel direction between Lane L1 and Lane L2 is the same, the travel direction between Lane L2 and Lane L3 is the same. Lane L1 has broken line, Lane L2 has broken line, Lane L3 has broken line.
  • There are 3 vehicles, vehicle ego, vehicle V1 and vehicle V2. Vehicle ego is at Road R1 and Lane L3. Vehicle V1 is at Road R1 and Lane L2, and vehicle V1 is at front left of vehicle ego with a small distance. Vehicle V2 is at Road R1 and Lane L3, and vehicle V2 is at rear of vehicle ego with a normal distance.
  • When vehicle ego is driving, vehicle V1 drives away from vehicle ego at its front left with a distance, at a very high speed. And then vehicle V1 changes lane left cut-in towards vehicle ego at its front left with a distance, at a very high speed. Vehicle V2 drives towards vehicle ego at its rear with a distance, at a high speed.

Next steps

This chosen scenario set will be discussed in the subsequent work packages from experimental designs to model development, and model validation and demonstration in various use cases. They represent a diverse range of ODD and behaviour collected from the partner group, with an emphasis on accident analysis and regulatory aspect. The SafetyPoolTM scenario database is utilised for storing and exchanging the scenario among project partners.

Author Information

Dr. Xizhe Zhang (Jason), Principal Engineer (Simulation Lead) at the Verification and Validation Group, Intelligent Vehicles, WMG, University of Warwick

Leave a Comment

Your email address will not be published. Required fields are marked *