Background
The i4Driving project aims to extend current state-of-the-art microscopic traffic simulation tools to enable assessment of traffic safety by adding a cognitive layer on top of the traditional behavioural models. Added cognitive layers require calibration and furthermore, since the added cognitive layer and the added mechanisms are mainly based on human factors theories, they need to be empirically verified. One important aspect to capture is the heterogeneity in driving behaviour, especially in factors that affect risk for, or severity of, accidents (where the latter is probably even more difficult to assess using traffic simulation and is not addressed in this project). The necessary data collection for the verification of the implementation of the human factors theories and calibration of the complete model can be done, for example, by real-world naturalistic experiments, test track experiments, driving simulator experiments and more.
In i4Driving we use several approaches, including mining data from experiments already conducted (naturalistic, test track, driving simulator), as well as conducting test track and driving simulator experiments. As such, multiple and similar driving simulator studies were conducted at different sites, which is the basis for i4Driving’s fifth innovation, an approach for standardisation of experimentations on different driving simulators. The ambition is that participants at different driving simulator sites have a comparable driving experience and to share simulator resources and reuse parts between sites so that not every partner has to set up similar environments and situations.
Driving simulators are different
Driving simulators as tools for research are usually designed with a specific research interest in mind. Combining research interest with budget and other needs (for example available physical space), driving simulator designs are different and, to generalise, there is no driving simulator that is the best solution for all driving simulator studies. With simulators being different, running a simulator study in more than one simulator puts requirements on the study design and how the simulator hardware and software are used to present the driving environment.
Below in Figure 1, the variety of driving simulators at the i4Driving partners VTI, CTAG, UNINA, and TUM are shown.
Figure 1. Driving simulators at different sites among the i4driving partners CTAG, TUM, UNINA, and VTI.
Driving simulator studies
A driving simulator study generally encompasses multiple test participants who, upon arriving at the simulator facility, are prepared for their driving task. They proceed to drive in the simulator and are debriefed when finished. When running a simulator study in more than one simulator, it is important to give the test participants as equal an experience as possible, otherwise it will be problematic to compare data collected from the simulators. An important part of providing equal simulator experiences in different simulators are the standards.
Our envisioned workflow is to be able to select different standardszed simulator scenarios of interest from a database with plenty of scenarios and then be able to use these scenarios in one or more simulators of choice. This workflow is visualized in Figure 2, which also shows the need for comparable results and how standards are used. The vision is to enable comparable results, but it is of course important to note that driving simulators can be substantially different in terms of functionality and design and the results might still not be comparable, even if this workflow is strictly followed.
Figure 2. Envisioned simulator study design flow for selecting scenarios and simulators and then run the study.
With the ambition that simulator drivers’ experiences should be comparable, two approaches were focused on:
1. Driving simulator environments and scenarios specified using open standards so that they can be shared
2. In the experimental design, a calibration part is added
Overview of existing (driving simulator) scenario standards
The available driving simulators were quite different and the supported standards varied. Supported standards were collected with a focus on three areas: road description, scenario description, and traffic description.
The most established standard seems to be the road description standard OpenDRIVE which was supported by every site. In short, the OpenDRIVE format, developed by the ASAM (Association for Standardization of Automation and Measuring Systems), is a standard for the logical description of road networks. It is based on the XML (Extensible Markup Language) format and includes road and lane geometry as well as objects along the road. Manually writing OpenDRIVE XML files from scratch is possible, although the required effort makes it practically unfeasible. No standard was found for the visual representation of the road objects and surroundings.
Connected to the ASAM OpenDRIVE standard is the scenario standard ASAM OpenSCENARIO 1.x. While OpenDRIVE manages the static description of the road, OpenSCENARIO handles dynamic objects and events, thereby defining the actions of all elements, akin to a storyboard. This standard was not directly supported by all partners but could be implemented by most. Note that OpenSCENARIO provides a low-level description which is needed for a driving simulator. A more humanly readable high-level description can be used, for example the Scenario Definition Language (SDL), if the same low-level description is generated in every simulator. For simplicity, we focused on the low-level description.
For traffic description standards, there was no common ground between simulators. Two simulators used the proprietary traffic solution that came with the simulator system when buying it, one solution was connected to SUMO and another used inhouse developed software to manage traffic. This doesn’t say anything about which solution might be better than any other, but when running a driving simulator study including surrounding traffic, the traffic will be different at the different sites and behave differently.
Missing pieces of existing standards
Upon reviewing existing technologies and standards support at the different driving simulator sites, it was noted that there are multiple missing pieces concerning standards for road, scenarios and traffic. It was out of scope to address all these issues within the i4Driving project. Thus, we present our challenges using the standards.
Free or controlled traffic
During a driving simulator study scenario, everything is commonly controlled in detail, including surrounding vehicles, so that when analysing the data, the human driver behaviour is due to differences depending on the drivers and not difference in conditions. On the other hand, for traffic simulation, the vehicles are controlling themselves independently of each other and as such, cannot be guaranteed to get into certain positions/situations. These are two opposite ways of handling simulated traffic, and it is challenging to find software and standards with support for mixing both controlled and independent traffic. The current way to solve this is to have clear separation between traffic and scenario parts with as little interaction as possible.
Putting a scenarios puzzle
There is a desire to select scenarios from a database for the complete experimental design similarly to putting a puzzle[CB1] together. Early tests were based on the developed scenario test-cases from the Safety Pool™ Scenario Database[1]. In general, many scenarios focused on their situation only and did not contain any information about what happens before and after. While this is beneficial for batch simulations, this poses a challenge for a driving simulator study where context is needed so that the sequence of scenarios and in between driving feels natural for test participants. In the end new scenarios were created from scratch for the simulator study instead of modifying available scenarios.
Human participants and balanced order
Driving simulator studies with human participants need to fully or in part support a balanced order design. An OpenSCENARIO description is tightly connected to an OpenDRIVE road and moving a scenario from one road to another can be complex. Thus, for a scenario to be moved around, a complete road was constructed where specific locations could be used for a certain type of scenario. Parameters within OpenSCENARIO were used to move the scenarios and in total, creating a specific road and manually setting parameters for the scenarios required significant manual work.
Tool support for standards
A standard by itself doesn’t guarantee a smooth workflow and for a standard to provide a smoother workflow good tool support is often needed. For example, XML based standards usually handle large files which are practically impossible to write by hand. Lack of tool functionality can, in the worst case, render the usage of a standard impossible. Example of tools used in this work were RoadRunner[2] for creating OpenDRIVE roads and OpenSCENARIO scenarios and Esmini[3] to run OpenSCENARIO scenarios. Both these tools had shortcomings affecting the experimental design.
Graphic elements
An OpenDRIVE road description is missing the graphical representation of objects along the road, although it usually specifies an object name that can give a hint. Further, the surrounding landscape is often missing which is illustrated in Figure 3. Hence a driving simulator study can have different visual representations depending on the interpretation of the developer using the standard preparing the study and how they decide to add surroundings and other objects.
______________________________________________________
[1] https://www.safetypool.ai
[2] https://se.mathworks.com/products/roadrunner.html
[3] https://github.com/esmini/esmini
Figure 3. A road visualized in Esmini with (left) and without (right) its surrounding environment
Experiment design
There are many factors, such as driver traits, traffic, infrastructure, non-driving related tasks (NDRT), weather conditions, lighting conditions, other external factors, that ideally should be considered in the experimental design. The number of combinations of factors increase fast with every additional factor to consider but project budget and time do not allow to vary all combinations of all the factors. Varying more than two to three factors often gives too many combinations and leads to experiments that are too lengthy with low statistical power. Hence, there is a need to choose wisely which combinations are expected to be most relevant.
The approach taken was to design the experiments by variations around a set of base scenarios. In this way two to three factors could be varied systematically at the same time in one DSE and another combination of two to three factors in another DSE. Two different setups were considered:
1. Run a limited set of motorway and urban base scenarios at all four driving simulator sites.
2. Run a limited set of motorway base scenarios at two driving simulator sites and a limited set of urban base scenarios at the other two sites.
The advantage of the first approach is that each base scenario is driven by all participants at all sites, which implies capturing a larger heterogeneity. The disadvantage is that the number of non-driver trait related factors that can be varied are fewer. The advantage of the second approach is that it allows for more variations of non-driver trait related factors. It also allows for considering different driver traits for the different environments, which can be important since the important driver traits related to safety might vary between contexts. The disadvantage is that the total number of participants are divided into two groups. In i4Driving we have adopted the second approach, adding some common familiarisation, warm-up car-following tasks and final critical situations to be run in each DSE, as illustrated in Figure 4.
Figure 4. Illustration of overall setup with the five different parts (training, familiarization, warm-up, variants of the urban or motorway base scenarios, and the final critical scenario) of each DSE.
Slightly different approaches are used for the design of the motorway and urban experiments. The reason behind this is that the cognitive layer extended traffic simulation model is more mature for the motorway context and less so for urban areas and intersections. Hence, for the motorway context the functions and parameters to be calibrated are better established than for the urban context, while in the urban context there is a larger need to collect data, not only for calibration, but also for the model development.
Concluding remarks and future work
Designing driving simulator experiments is a non-trivial task and experiments using multiple simulators quickly increase in complexity. There are several standards that ease the process, but there are still significant shortcomings if a study is to be conducted in multiple simulators. Here we highlight where we experienced challenges and which efforts we made to overcome them. Even if every simulator site uses available standards, the simulator setups would most likely differ as parts not described by a standard are up to each site to implement, e.g. surrounding environment. Understanding that there are technical differences between simulators, the experiment design was adjusted to capture the main effects where simulator differences would affect the conclusions. In our setup, data from the familiarisation and warm-up parts can be used for the interpretation of the critical scenario that was run at different driving simulator facilities and in different settings (urban and motorway).
Key takeaways
• There are no perfect standards for everything, therefore this requires engineering and collaboration between partners
• Standardised scenarios are rarely plug-and-play
• Collaboration between different simulators generates unique knowledge in Europe
• INNO5 enables comparability, efficiency and confidence in the results
Figure 2. A road visualized in SCANER II (CTAG’s simulator).
 | CTAG_Simulator_Zones_of_Interest_Projectors |
 | CTAG_Simulator_Zones_of_Interest_Incabin |
  | CTAG_Simulator: Road examples (inside simulator) |
 | CTAG_FaceLab |