By Tobias Mascetta, Florian Finkeldei, Matthias Althoff
i4Driving Project Overview
i4Driving (Integrated 4D driver modelling under uncertainty) is an EU-funded project within the EU-horizon funding program for research and innovation in the European Union.
One of the horizon missions is finding key innovations for the topic cluster of climate, energy, and mobility. Within this cluster, one field of importance is the area of cooperative, connected, and automated mobility (CCAM).
Figure 1: Cluster 5 of the EU-Horizon key funding program.
The vision of i4Driving is to lay the foundation for a new industry-standard methodology to establish a realistic human road safety baseline for virtual assessment of CCAM systems.
The innovations of the project pave the way towards the long-term goal: a driving licence for autonomous vehicles (AV), trusted by regulatory bodies, industrial entities, and the public. Testing AV extensively in simulation is necessary, since testing them purely on the roads would require several tenths of millions of kilometres of test driving, rendering the procedure infeasible.
The two central pillars of innovation of the project are a simulation library for human driving and a cross-disciplinary methodology to account for the uncertainty in both human behaviours and use case circumstances. Experimental evidence is gathered to create realistic simulation models for (near) accident multi-driver scenarios.
Figure 2: Overview of the i4Driving project.
Following an interdisciplinary approach, the i4Driving project combines areas of expertise in traffic engineering, human factors and data and computer science from both academia and industry with partners in and outside the EU.
The i4Driving project aims to develop more realistic driver models by considering four general dimensions, hence the name i4Driving. These dimensions are:
- tactical driving, incorporating information from the front, the side, and the back of the vehicle,
- operational driving, incorporating stronger relaxations on the assumption of being collision-free,
- inter and intra driver heterogeneity, facilitating a disentanglement of human factor dynamics from ideal driving behaviour, and
- vehicle dynamics, facilitating a disentanglement between the driver and the vehicle.
Figure 4: The four dimensions of the driver model.
Along the research process, the i4Driving project aims to develop nine key innovations:
- Applying data mining of naturalistic and simulated datasets to identify causal relationships between external and human factors, and safety-critical driver behaviours at the level of specific driving situations.
- Augmenting existing driver models with a cognitive layer that incorporates the four dimensions described in Figure 3.
- A Methodology to identify relevant use-cases and critical scenarios in the available datasets.
- An Approach for automatically generating critical scenarios.
- An Approach for standardizing experiments on different driving simulators.
- A Methodology to robustly encode driver heterogeneity into probabilistic human behaviour models.
- Quantifying the limitations and assumptions of the developed models in terms of realism by applying Global Uncertainty Quantification and Sensitivity Analysis and in terms of plausibility and credibility by applying Sensitivity Auditing.
- Conducting Turing Tests between a real driver and the developed driver models in driving simulators.
- A Probabilistic framework to quantitatively validate the developed driver models at multiple scales.
INNOVATION 4 Description
i4Driving’s forth innovation develops an approach to automatically generate critical traffic scenarios. In summary, the generated scenarios tie into the rest of the project as follows:
The generated critical traffic scenarios, among others, will be used in the driving simulator experiments. These experiments are conducted to test several hypotheses about human factors in driving situations and driver behavior. The successful hypotheses will then be used to develop new and better driver models that are a first step towards realistic and trustworthy testing and verification simulations for AV.
The critical scenario generation uses a CommonRoad (Composable Benchmarks for Motion Planning on Roads) scenario as a starting point. Such a scenario consists of a lanelet network including traffic signs and traffic lights, static and dynamic obstacles, including non-agent vehicles, and one or multiple planning problems representing the agent vehicles.
Figure 5: Exemplary CommonRoad scenario with three planning problems and several non-agent vehicles. The green arrows indicate the three initial states. Each of these planning problems has the same orange goal region. The blue vehicles are dynamic obstacles that follow their path without control of the user. The scenario was generated from a real-world traffic recording in the USA.
The approach of INNO4 uses CommonRoad scenarios with multiple planning problems but no non-agent vehicles (i.e., dynamic obstacles). Our approach uses set-based predictions (i.e., reachability analysis), semantic specifications and specifications from linear-temporal-logic to analyse and modify the input scenario.
The general pipeline for INNO4 currently investigated is as follows:
Input: CommonRoad scenarios with several planning problems.
- Trajectory is optimized to enhance criticality for the vehicle under test (VUT).
- During the trajectory optimization step, the conflicted drivableareas are distributed in a way that increases criticality.
- During the reachable set computation, traffic rule compliance is asserted.
Output: Interactive CommonRoad scenarios with trajectories for the agents, which increase the criticality for the VUT.
The long-term aim of INNO4 is to build the pipeline real-time capable so that it can be used in online agent control in driving simulator studies.
2.1. CommonRoad Scenarios with Mutiple Planning Problems
CommonRoad is an open-source benchmark toolbox for autonomous vehicles, which offers a variety of urban and highway scenarios including merging, intersections, and roundabouts. Moreover, the CommonRoad toolbox provides several dataset converters from other state-of-the-art datasets to the CommonRoad format.
INNO4 provides an approach to convert the recorded trajectories of vehicles into planning problems consisting of an initial state and a goal state. One of these planning problems is used for the VUT, and the others as the agents of the scenario.
2.2. Specification Compliant Trajectory
INNO4 offers an easy-to-use specification interface that allows the user to only consider trajectory solutions that are compliant with the given specifications, e.g,“vehicle 1 should always be behind vehicle 2”.
The specification compliance is realized by first computing the reachable sets of each agent, then pruning non-compliant parts and, finally, ensuring the integrity of the solution by applying backward propagation to prune non-valid sets. Within these specification compliant sets, the trajectory for each agent is found solving a Quadratic Programming optimization problem. In the current implementation, drivable areas that have a conflicting overlap with another agent are allocated via a heuristic. The further development of the scenario synthesis, starting from a preliminary work [1] , enables the consideration of general traffic rules. In addition, the combination of reachability analysis and quadratic programming allows a considerable acceleration of the execution speed compared to the previously used mixed integer quadratic programming approach. We published this approach successfully at the peer-reviewed IEEE International Conference on Intelligent Transportation Systems 2023 [2].
2.3 Concept Overview
The basic approach of our software is to control other vehicles, called ‘attackers’ from hereon, to increase the criticality of the scenario dynamically by shrinking the drivable area of the VUT. A predictive scenario analysis is used as the basis for the criticality enhancement and the planning of the attackers’ trajectories. Both the replanning and the re-computation of the prediction are executed in periodic cycles.
The developed methodology combines reachability analysis with numerical optimisation to increase the criticality of situations via a predictive approach:
In a first step, the traffic rule compliant reachable sets of all vehicles are determined by applying reachability analysis. Reachability analysis determines the states a dynamic system can reach over a time horizon. The states are defined as the combinations of longitudinal and lateral position and velocity. The drivable area is the projection of the reachable set onto the position domain. During each replanning cycle, the reachable sets are computed for the entire cycle time horizon.
However, not all these reachable states comply with traffic rules. Those states that violate traffic rules are removed from the reachable sets by applying automaton-based model checking techniques. For an introduction to this approach in a single-agent context, please refer to [3].
The existing approaches can be used for a multi-agent problem by temporarily reducing it to a single agent problem with respect to the VUT. Herein, it is assumed that the attackers follow their reference path with a constant velocity for a short period of time. This assumption is valid since the multi-agent trajectories have a fast replanning-cycle. An approach for multi-agent traffic control conformity is given, e.g., in [3]. A recent approach to multi-agent rule-compliant traffic rule driving corridor generation generated from our group was accepted to this year’s IEEE Intelligent Vehicles Symposium] [4]
Once the traffic rule compliant reachable sets for all traffic participants have been computed, the trajectories for all attackers are determined. Firstly, the intersections of the VUT’s drivable area with the drivable areas of the attackers are examined and suitable target states are chosen via a heuristic algorithm. Secondly, the trajectories of the attackers are planned via convex optimization of a Quadratic Program (QP).
A common approach that significantly eases planning in autonomous driving is to switch between a global cartesian coordinate frame and a vehicle-dependent curvilinear coordinate frame (also known as Frenet-Frame) created along the reference path, usually given by the centerline of the lanelet, see, e.g. [2, 4]. The software of INNO4 also uses this approach, since one of the core benefits is that the reachable sets are convex polygons in the curvilinear frame. This is a necessary precondition for applying convex optimisation, as used in the QP described above.
At the end of the planning cycle, the inputs of the VUT are processed and collisions between the VUT and attackers are detected.
2.4. Criticality Enhancement
When identifying human driver models, one is especially interested in how humans react in edge-case situation with high criticality. However, these traffic situations seldom occur in real-world traffic, making it hard to use recorded datasets for this purpose. The approach implemented in this deliverable therefore aims at generating artificial traffic scenarios based on real-world situations by steering the surrounding vehicles in a way that continuously increases the criticality of traffic situations. So far, similar approaches have not been applicable to human driver studies as they were far away from being real-time capable. Our approach solves this by leveraging reachability analysis in combination with heuristic guiding of the attackers. The results of our approach are shown for several examples and instructions to use our open-source software are provided.
Examples
In the following examples, the VUT is always depicted in red, and the attackers are always depicted in blue. The goal regions are depicted as orange polygons. Both the original trajectory without criticality enhancement and the enhanced trajectories of all attacking vehicles are depicted as doted black lines. If collisions occur, they are shown as yellow circles. The drivable areas are translucent polygons.
For detailed examples with animated GIFs, please refer to the /examples folder of the repository included with this deliverable.
Merge Scenario with One Attacker
Figure 6: Merge scenario with one attacker.
In Figure 6, (A) shows the initial time step, (B) the unmodified scenario, and (C) and (D) scenarios with increased criticality. The algorithm increases the criticality by choosing a more dangerous entry manoeuvre, which cuts the VUT. Note that the VUT has an activated Merge Rule; therefore, only states are considered where the VUT does not violate the rule by moving to the right lane. In (D), the VUT planner is deliberately chosen poorly to show the collision detection.
Merge Scenario with Two Attackers
Figure 7: Merge scenario with two attackers.
In Figure 7, a similar basic scenario to the one above is used, but an additional attacker is placed behind the VUT. The entering vehicle also cuts the VUT, whilst the attacker from behind accelerates. Note that the attacker behind the VUT respects the Merge Rule by not entering the right lane. This example shows that the attackers also consider activated traffic rules.
Real-world Roundabout Scenario
Figure 8: Real-world roundabout scenario.
In Figure 8, (A) shows the unmodified scenario, whilst (B) and (C) show criticality-increased scenarios for different VUTs. In (B), the algorithm makes the attacker cut the VUT. In (C), the algorithm chooses to first pass the VUT safely and then suddenly decelerate in the roundabout, since the starting velocities of the VUT and the attacker were purposefully chosen so that it is impossible for the attacker to just produce criticality by cutting the VUT. This shows the predictive capabilities of the algorithm. This scenario was recorded on the German street B741 near Munich.
Passing Vehicles
Figure 9: Scenario with passing vehicles.
Regarding Figure 9, in (A) the algorithm chooses the attacker coming from the left to move into the middle of both lanes to increase criticality to a near-crash scenario. (B) shows the same criticality-enhanced scenarios a few time-steps later, where the previously attacking vehicle is back on its rightful lane and reaching its goal region. This exemplifies that the algorithm generates trajectories for the attackers that increase the criticality whilst still pursuing the goal from the unmodified scenario.
Real-world Merge Scenario
Figure 10: Criticality-enhanced real-world merge scenario.
In this scenario (Figure 10), the attackers chose to make a risky, near crash (rule compliant) merge to the right. The real-world scenario that this enhancement is based on was recorded on the German Autobahn between Aachen and Cologne. Note that although the original goal-region (in orange) was surpassed, the attackers continue trying to increase the criticality.
Bibliography
[1] M. Klischat and M. Althoff, “Synthesizing Traffic Scenarios from Formal Specifications for Testing Automated Vehicles,” 2020 IEEE Intelligent Vehicles Symposium (IV), Las Vegas, NV, USA, 2020, pp. 2065-2072, doi: 10.1109/IV47402.2020.9304617.
[2] F. Finkeldei and M. Althoff, “Synthesizing Traffic Scenarios from Formal Specifications Using Reachability Analysis,” 2023 IEEE International Conference on Intelligent Transportation Systems (ITSC), Bilbao, Spain.
[3] E. Irani Liu and M. Althoff, “Specification-Compliant Driving Corridors for Motion Planning of Automated Vehicles,” in IEEE Transactions on Intelligent Vehicles, doi: 10.1109/TIV.2023.3289580
[4] T. Mascetta, E. Irani Liu and M. Althoff “Rule-Compliant Multi-Agent Driving Corridor Generation using Reachable Sets and Combinatorial Negotiations”, in IEEE Intelligent Vehicles Symposium, 2024.