MECCA_20-01_v10_web Testing of Automated Driving Systems OND!EJ VACULÍN, MICHAEL GELLRICH, ROBERT MATAWA, STEFFEN WITSCHASS MECCA 01 2020 PAGE 7 10.14311/mecdc.2020.01.02 Testing of Automated Driving Systems OND!EJ VACULÍN, MICHAEL GELLRICH, ROBERT MATAWA, STEFFEN WITSCHASS 1. INTRODUCTION The modern cars were invented as purely mechanical systems more than 130 years ago. However, since introduction of Antilock Brake Systems in 1970s the computerization of the vehicle driving started and computers played more and more significant role in the vehicles. The systems such as Antilock Brakes, Traction Control or Stability Control interrupt the direct connection between the driver and vehicle with the objective to reduce the possible risk, either to avoid the collision or at least to reduce the collision velocity in potentially dangerous situations. In general, four groups of assistance systems can be recognized: 1. Comfort Systems such as Headlight or Rain Assistant – such systems take duties from drive, which are not directly connected with vehicle dynamic functions, 2. Information and Warning Systems such as Driver Alert, Lane Departure Warning or Traffic Sign Recognition Systems – such systems just inform driver about certain state of vehicle, driver or infrastructure, 3. Intervening Emergency Systems such as Stability Control or Automatic Emergency Braking – such systems take partial control over the vehicle in critical (near accident) situations, 4. Continuously Acting Systems such as Adaptive Cruise Control of Lane Keeping Assistance – such systems support driver in long time periods by taking part of his duties in standard situations. The increasing computational power together with reducing purchase costs and as well as availability of low cost efficient and reliable sensors allow the manufacturers to implement functions such as lane keeping assistance or emergency braking options even to low cost cars, which are called Advance Driver Assistance Systems (ADAS). The logical consequence of the ADAS development is a vehicle which is either partly or even fully able to take the driver’s duties. Such an automated driving (AD) vehicle will offer a co-pilot functions or drive even autonomously OND!EJ VACULÍN Technische Hochschule Ingolstadt, Esplanade 10, 85049 Ingolstadt, Germany, +420 776 637 352, E-mail: Ondrej.Vaculin@thi.de (Formerly TÜV SÜD Czech, Novodvorská 994/138, 142 21 Praha 4, Czech Republic) MICHAEL GELLRICH, ROBERT MATAWA, STEFFEN WITSCHASS TÜV SÜD Auto Service, TÜV SÜD Auto Service GmbH, 85748 Garching, Daimlerstraße 11, Germany, +49 89 32950-751, E-mail: Michael.Gellrich@tuv-sud.de, Robert.Matawa@tuev-sued.de, Steffen.Witschass@tuev-sued.de ABSTRACT The automated driving requires new testing approaches, which are more complex than the current testing systems. The complexity and requirements for accuracy is important, because of interconnection of virtual with physical testing. This paper presents a generic approach to testing of automated driving functions and demonstrates its implementation on measurement of two scenarios. KEY WORDS: AUTOMATED DRIVING, TESTING, TESTING SCENARIOS SHRNUTÍ Automatizované !ízení vy"aduje nové testovací p!ístupy, které jsou daleko komplexn#j$í ne" sou%asné testovací systémy. Komplexnost a po"adavky na p!esnost jsou d&le"ité z pohledu na propojení fyzického a virtuálního testování. Tento %lánek prezentuje obecn' p!ístup k testování funkcí automatizovaného !ízení a demonstruje jeho implementaci na m#!eních dvou scéná!&. KLÍ"OVÁ SLOVA: AUTOMATIZOVANÉ !ÍZENÍ, TESTOVÁNÍ, TESTOVACÍ SCÉNÁ!E TESTING OF AUTOMATED DRIVING SYSTEMS P!ijato k publikaci v roce 2018 Testing of Automated Driving Systems OND!EJ VACULÍN, MICHAEL GELLRICH, ROBERT MATAWA, STEFFEN WITSCHASS MECCA 01 2020 PAGE 8 TABLE 1: SAE levels of automated driving [1]. TABULKA 1: Úrovn# automatizace !ízení dle SAE [1]. SAE Level Name Narrative Definition Execution of Steering and Acceleration/ Deceleration Monitoring of Driving Environment Fallback Performance of Dynamic Driving Task System Capability (Driving Modes) Human driver monitors the driving environment 0 No Automation the full-time performance by the human driver of all aspects of the dynamic driving task, even when enhanced by warning or intervention systems Human driver Human driver Human driver n/a 1 Drive Assistance the driving mode-specific execution by a driver assistance system of either steering or acceleration/deceleration using information about the driving environment and with the expectation that the human driver performs all remaining aspects of the dynamic driving task Human driver and system Human driver Human driver Some driving modes 2 Partial Automation the driving mode-specific execution by one or more driver assistance systems of both steering and acceleration/deceleration using information about the driving environment and with the expectation that the human driver performs all remaining aspects of the dynamic driving task System Human driver Human driver Some driving modes Automated driving system monitors the driving environment 3 Conditional Automation the driving mode-specific performance by an automated driving system of all aspects of the dynamic driving task with the expectation that the human driver will respond appropriately to a request to intervene System System Human driver Some driving modes 4 High Automation the driving mode-specific performance by an automated driving system of all aspects of the dynamic driving task, even if a human driver does not respond appropriately to a request to intervene System System System Many driving modes 5 Full Automation the full-time performance by an automated driving system of all aspects of the dynamic driving task under all roadway and environmental conditions that can be managed by a human driver System System System All driving modes Testing of Automated Driving Systems OND!EJ VACULÍN, MICHAEL GELLRICH, ROBERT MATAWA, STEFFEN WITSCHASS MECCA 01 2020 PAGE 9 without a driver. According to SAE the development will be divided in several levels as indicated in Table 1. In order to drive automatically, the vehicles must take responsibility from a human driver to its IT systems and control algorithms. Despite it is expected that driverless vehicles will be able to reduce significantly the number of accidents and fatalities, some sources expect even 90% or more, the initial stages of AD implementation will be accompanied by increase of accidents due to heterogeneous traffic of driverless and human driven vehicles and the insufficient maturity of AD systems [2]. Currently the regulatory bodies and consumer organizations define for some vehicle categories couple of physical proving ground tests to assess the functionality of ADAS. However, the testing procedures based on physical testing seem to be insufficient to cover the all possible cases and thus to evaluate the effectiveness and safety. To cover the vast number of possible scenarios, simulation methods are the only feasible way [3]. However, the physical tests will be still needed for verification and validation of these simulation models and set-ups. The proper standardization and regulatory basis is important for all stakeholders. Since current regulations and inspection specifications are not sufficient or even not existing, several committees and project groups such as German project PEGASUS are developing new international regulations and standards. To be able to implement complicated scenarios on a proving ground, new testing approaches must be developed and implemented, in which traffic simulation vehicles (TSV) and soft crash targets (SCT) together with other entities define repeatable environment for testing of so-called Vehicle Under Test (VUT). Such tasks are being solved within couple of projects and in an ISO level in ISO/TC 22/SC 33/WG 16 (Active Safety Test Equipment). 2. TYPE APPROVAL OF ASSISTANCE SYSTEMS As usual, the development of standards and regulations is slower than the development of the technology itself. The ADAS functions are implicitly addressed by the UN ECE Regulations 13 and 79 with Annexes on electronic systems. However, current Regulation 79 explicitly defines the requirements for systems up to automatic parking, i.e. with low velocities. Further development of Regulation 79 is in progress. The vehicles of categories M2, M3, N2 and N3, i.e. trucks and buses with some exceptions are legislatively controlled by the Commission Regulation (EU) No. 351/2012 and 347/2012 to be equipped with The Regulation 130 defines the approval tests for Lane Departure Warning Systems for vehicles categories N2, M2, N3, M3, i.e. trucks and buses. The test set-up is shown in Figure 1. FIGURE 1: Test set-up according to UN ECE R 130 OBRÁZEK 1: Test setup dle p!edpisu EHK OSN 130 The Regulation 131 defines the timing and type of warning as well as automatic braking maneuver based on tests with a stationary and moving target, which represents a passenger car of category M1, class AA saloon. The initial velocity is defined to 80 km/h. The AEB system should at first warn the driver and if he does not react then to automatically brake the vehicle. The warning timings are different for various categories of vehicles, warning time for N2 is shorter than for M3. Further testing is focused on identification of failures and finally the driving in the gap between 2 parking vehicles, which are 4.5 meters side to side from each other as indicated in Figure 2. FIGURE 2: Test set-up according to UN ECE R 131 OBRÁZEK 2: Test setup dle p!edpisu EHK OSN 131 Testing of Automated Driving Systems OND!EJ VACULÍN, MICHAEL GELLRICH, ROBERT MATAWA, STEFFEN WITSCHASS MECCA 01 2020 PAGE 10 LDW and AEB functions since November 2015 [4, 5]. For AEB the deadline depends on the braking system and suspension. On the UN ECE level the Regulations 130 and 131 exist, which define the technical requirements and testing procedures. FIGURE 3: Scenario traffic jam tail end OBRÁZEK 3: Scéná! konec kolony FIGURE 4: Scenario lane change OBRÁZEK 4: Scéná! zm#na jízdního pruhu Two Soft Crash Targets (SCTs) were installed on the proving ground as indicated in Figure 5. SCT 1 was a balloon car from the EuroNCAP target for testing ADAS functions and SCT 2 was a model of a motorcyclist. SCT 2 TSV SCT 1 FIGURE 5: Soft Crash Targets OBRÁZEK 5: M#kké cíle Testing of Automated Driving Systems OND!EJ VACULÍN, MICHAEL GELLRICH, ROBERT MATAWA, STEFFEN WITSCHASS MECCA 01 2020 PAGE 11 3. CONSUMER TESTING Currently the consumer organizations are focused on Advanced Emergency Braking (AEB) and Lane Departure Warning (LDW). Consumer organization tests under the New Car Assessment Program (NCAP) [6], serves for the testing of AEB and LDW in passenger cars. Currently about nine different NCAP consumer organizations around the world exist, however not each has the AEB tests in its portfolio. The different NCAP test procedures demonstrate the heterogeneity of approaches in different countries and regions. While for example the US NCAP is based on tests drivers, the Euro NCAP uses driving and pedal robots along with accurate measurements of vehicle position. The advantages of the European approach are obvious: the higher accuracy in the position and higher repeatability the lower number of necessary tests to be performed. Euro NCAP currently tests AEB systems in three areas: AEB city, AEB inter-urban and AEB pedestrian. Despite the complete set of velocities, this method considers only single, limited representative scenarios without considering the driver’s behavior. This can be sufficient for consumer review to ensure comparability of different vehicles. However, for the vehicle safety and future type approval this is not enough, because realistic scenarios and driver and environmental influences are not included. 4. ENTITIES INVOLVED IN TESTING To be able to generate the testing scenarios for both virtual and physical testing the possible set-ups should have a common basis. The presented structure has been developed in the project PEGASUS. The so-called generic approach for proving ground tests [7] summarizes the participants of the tests and defines the following entities [8, 9]: 1. Test Object 2. Basic Route 3. Guidance Infrastructure 4. Temporary Adjustments 5. Stationary Objects 6. Mobile Objects 7. Environment The scenarios were performed in couple of runs in order to assess the repeatability and accuracy. The results achieved are presented in Figures 6 and 7. The first graph presents the velocity profile of the test; the second graph shows the lateral deviation. The results indicate that further development of the vehicle dynamic controllers is necessary in order to achieve the trajectory deviation better than +/- 0.1 m. FIGURE 6: Measured data – traffic jam tail end OBRÁZEK 6: M#!ení – konec kolony Testing of Automated Driving Systems OND!EJ VACULÍN, MICHAEL GELLRICH, ROBERT MATAWA, STEFFEN WITSCHASS MECCA 01 2020 PAGE 12 The testing scenario is then defined by a combination of different entities. In order to implement the generic procedure of the scenarios on the proving a control center must implement all the tasks. The entire system should complete measurement equipment to perform tests of highly automated functions. Two types of scenarios are recognized: (i) time invariant and (ii) time variant. The time invariant test is synchronized by a Traffic Simulation Vehicle. In the time variant case the test is synchronized by the Vehicle Under Tests. The time variant case means that the TVS and SCT trajectories must be modified dynamically. Very important feature of the control center is the wireless communication with low latency to all testing entities. All moving entities such as TSV, SCT and VUT must be equipped with a precise localization based e.g. on Real Time Kinematic (RTK) satellite navigation together with an inertial platform. Since the trajectory of TSVs should be controlled, the vehicle must be actuated. Either it is possible to use external actuators such as steering and pedal robots of the internal actuators, which are already available in the vehicle. The second solution is of advantage because no additional devices must be installed in the vehicle. The current implementation uses direct control of internal vehicle actuators for steering and throttle and indirect control of deceleration by a braking robot. The system is in development and in the next version, it is expected that also the braking function will be actuated directly without installation of a braking robot. 5. PROVING GROUND TESTS The first proving ground test has been performed in order to verify the implementation with selected test cases in real conditions. Two scenarios have been selected: 1. Traffic jam tail end 2. Lane change Traffic jam tail end (Figure 3) is defined in the following steps: a) SCTs in lanes 1 and 3 with v = 0 km/h represent the tail end of a traffic jam. b) TSV, followed by VUT with ~2 s distance, drives with ~80 km/h in lane 2. c) TSV decelerates ~123 m in front of SCTs with ~2 m/s( and stops beside the SCTs. Lane change (Figure 4) is defined in the following steps: a) SCTs in lanes 1 and 3 with v = 0 km/h represent the tail end of a traffic jam. FIGURE 7: Measured data – lane change OBRÁZEK 7: M#!ení – zm#na jízdního pruhu Testing of Automated Driving Systems OND!EJ VACULÍN, MICHAEL GELLRICH, ROBERT MATAWA, STEFFEN WITSCHASS MECCA 01 2020 PAGE 13 b) TSV, followed by VUT with ~2 s distance, drives with ~80 km/h in lane 2. c) TSV changes lane into lane 1 behind the SCTs and stops. VUT accelerates in lane 2. 6. CONCLUSION The proving ground testing will be a necessary part of the prove of effectiveness of the future automated driving functions. The complexity of the task requires to combine physical and virtual testing methods. The objective of the proving ground testing equipment is to deliver accurate and repeatable testing environment for the automated driving systems. The paper presented a generic approach and an example of its implementation into the real environment together with some preliminary results based on predefined time invariant scenarios. The results indicate that further development of the trajectory controllers must be performed in order to achieve the acceptable motion of the traffic simulation vehicle. Such complex systems as automated driving shall always be considered not only in terms of their effectiveness, but also functional safety and IT security issues are essential for the system overall rating. The system is in development and the next generation will include full direct control of the vehicle using vehicle actuators as well as time variant capability. Furthermore, it is intended to integrate more traffic simulation vehicles together with moving platforms with soft crash targets for more complex scenarios. ACKNOWLEDGEMENTS The author would like to acknowledge the support of project PEGASUS by German Federal Ministry of Economic Affairs and Energy. REFERENCES [1] SAE J3016 (2016). Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles, 2016-09-30. [2] Sivak, M., Schoettle, B. (2015). Road Safety with Self-Driving Vehicles: General Limitations and Road Sharing with Conventional Vehicles, University of Michigan, Transportation Research Institute, Report No. UMTRI-2015-2. [3] Vaculín, O., Wech, L., Steininger, U., Lewerenz, P., Prokop, P. (2016). Virtuelle Prüfungen als Ergänzung von realen Prüfungen für die Typgenehmigung von Fahrerassistenzsystemen. In: Kompaß, K. (ed.). Aktive Sicherheit und Automatisiertes Fahren Methodenentwicklung im Expertendialog, HdT Fachbuch Band 144, Essen, pp 76 – 87, ISBN 3816933653. [4] Commission Regulation (EU) No. 351/2012: type-approval requirements for the installation of lane departure warning systems in motor vehicles, 2012. [5] Commission Regulation (EU) No. 347/2012: type-approval requirements for certain categories of motor vehicles with regard to advanced emergency braking systems, 2012. [6] P. Seiniger (2016). NCAP Testing of Safety Related Driver Assistance Systems. In: SafetyUpDate 2016, Aschaffenburg, 2016. [7] Ponn, T., Vaculín, O., Gimm, K., Knake-Langhorst, S., Joganatham, R., Schuldt, F., Diermeyer, F., Abdellatif, H. (2017). Generischer Ansatz für Prüfgeländetests von hochautomatisierten Fahrzeugen in `8. Tagung Fahrerassistenz‘, München. [8] Bagschik, G., Menzel, T., Maurer, M. (2018). Ontology based Scene Creation for the Development of Automated Vehicles, https://arxiv.org/abs/1704.01006v4, 19 Jan 2018. [9] Schuldt, F. (2017). Ein Beitrag für den methodischen Test von automatisierten Fahrfunktionen mit Hilfe von virtuellen Umgebungen. Universitätsbibliothek Braunschweig, https:// doi.org/10.24355/dbbs.084-201704241210.