Acta Polytechnica CTU Proceedings https://doi.org/10.14311/APP.2023.41.0049 Acta Polytechnica CTU Proceedings 41:49–57, 2023 © 2023 The Author(s). Licensed under a CC-BY 4.0 licence Published by the Czech Technical University in Prague U SMART ZONE – CREATING HIGHLY REALISTIC VIRTUAL ENVIRONMENT FOR VEHICLE-IN-THE-LOOP SIMULATIONS Ondřej Piksaa,∗, Adam Orlickýa, Martin Schánob a Czech Technical University in Prague, Faculty of Transportation Sciences, Department of Vehicle Technology, Horská 3, 128 03 Prague 2, Czech Republic b Czech Technical University in Prague, Faculty of Transportation Sciences, Department of Forensic Experts in Transportation, Horská 3, 128 03 Prague 2, Czech Republic ∗ corresponding author: piksa@fd.cvut.cz Abstract. Developing Advanced Driver Assistance Systems (ADAS) and Autonomous Vehicles (AV) based on machine learning is generally an extensive and costly process. Due to the increasing complexity of autonomous systems, the need for extensive testing and validation arises. In recent years, computer simulation has been used for these purposes. Performing realistic simulations, especially for the purpose of computer vision-based systems, requires a high-quality, almost photorealistic virtual environment. This paper introduces U SMART ZONE, a high-fidelity virtual model of the Severní Terasa district in Ústí nad Labem, Czech Republic comprising of more than 7.5 km of drivable roads with a total area of approximately 1.4 km2 for human-in-the-loop and hardware-in-the-loop (HiL) simulations. Keywords: Virtual environment, human-in-the-loop, hardware-in-the-loop, vehicle simulation. 1. Introduction Autonomous driving is undoubtedly a technology that will, with its superiority, ultimately revolutionize the way we understand transportation and vehicle own- ership. Many companies have been trying to develop this innovative technology for more than a decade. However, as the development progresses, it is becom- ing clear that a long way to fully autonomous vehicles still awaits us. Urban traffic is significantly more com- plicated, primarily due to the number of unpredictable or difficult-to-predict phenomena. The development of sufficiently complex systems for navigating in such a chaotic environment proved to be an issue principally because of the costly and time-consuming nature of machine learning. In addition, regarding security and legislative restrictions, it is often impossible to test experimental vehicle systems in live traffic. The range of learned scenarios is thus limited. As the complexity of autonomous driving increases, so does the need for high-quality and comprehensive testing. Autonomous driving systems are tested mainly on closed polygons, in virtual reality, or in a combination thereof, i.e., in augmented reality. While closed polygons offer almost laboratory conditions for testing and the possibility of reconstructing exact situations, virtual reality is significantly less limiting in the amount and scope of tests, which are, nonetheless, confined by preparation time, computing power, and other factors. Research in recent years indicates the increasingly efficacious use of virtual simulations for learning or validating of machine-learning-based vehicle systems. The supply of specialized software for “in-the-loop” testing has expanded significantly over the past years. Based on our long-term understanding of vehicle simulations, predominantly in the human-in-the-loop domain, and our involvement in the commercial sector in developing Advanced Driver Assistance Systems (ADAS) and Autonomous Vehicles (AV), we have identified one element as yet untapped. That is the photorealism and authenticity of the virtual environ- ment. Virtual environments are still relatively simple and idealized. For complex ADAS and AV devel- opment and testing, improvements in the simulated environment are necessary to match the complexity of a general urban landscape. In addition, modern vehicle systems built on automatic image recognition systems are becoming progressively routine. And for these systems, it is critical in HiL learning to have a virtual landscape closely resembling the actual en- vironment. For this reason, we decided to create an accurate photorealistic model of a sizeable part of the city Ústí nad Labem in the Czech Republic for virtual simulations with the possibility of validation via parallel testing. 2. Literature review An extensive body of literature exists on developing simulation tools for HiL [1, 3]. Many of them con- sider their instruments highly realistic [2–6]. Kang et al. [7] presented an overview of a publicly available virtual testing environment. This section introduces some approaches using high-quality, almost photore- alistic computer-generated imagery. These computer- generated images work with objects with pre-assigned attributes. That in turn, gives the simulation tool the ability to display a semantic layer alongside the realistic scene, which can then be easily used for arti- ficial intelligence learning or validation. This chapter 49 https://doi.org/10.14311/APP.2023.41.0049 https://creativecommons.org/licenses/by/4.0/ https://www.cvut.cz/en O. Piksa, A. Orlický, M. Scháno Acta Polytechnica CTU Proceedings Figure 1. Real environment (left) and its virtual recreation (right) [1]. Figure 2. Simulation for testing Lane Keeping and Cooperative Adaptive Cruise Control [2]. presents how to create such environment and what constitutes a high-quality environment. The first ap- proach uses a direct reconstruction of the actual envi- ronment, for example, based on video footage. This method is utilized, for instance, by Gaidon et al. [1]. Firstly, they acquire Real-World data. Consequently, they semi-automatically create a photorealistic dy- namic 3D virtual world using off-the-shelf computer graphics engines and graphic assets (realistic 3D mod- els and materials). In their model, main roads are positioned semi-automatically, while secondary roads and background objects (trees, buildings, etc.) are placed manually (the authors indicate these steps could further be automated). The resulting scenario can be seen in Figure 1. They then make minor adjust- ments to these virtual models, yielding a significantly broader range of scenarios for machine learning than they would get by simply annotating the underlying data. Alternative approach operates with augmented re- ality. That is mainly used to validate already learned systems by extending the potential of closed test poly- gons. Authors use a virtual environment to generate road traffic. That includes vehicles, pedestrians, cy- clists, or trains. Or they use it to add signs, obstacles and other static objects [4, 10, 11]. Unfortunately, none of the mentioned-above authors on mixed reality testing showed the visual quality of the virtual ele- ments. A different approach uses completely virtual systems. An example of such a system is a CAV HiL simulator [2]. This simulator was employed for the development and testing Lane Keeping and Coopera- tive Adaptive Cruise Control. An illustration of the simulation can be seen in the Figure 2. Another approach was demonstrated by Chen et al. [8]. They verified the effectiveness of their HiL simu- lation platform by Path Planning and Decision-making Modules Verification, Reinforcement Learning-based Car-following Model Validation, and Autonomous Parking Algorithm. Their scenery was created us- ing OpenStreetMap, a tool for creating terrain and basic building outlines. Various objects like buildings, trees, roads, and other object models were created by 3D modelling software Sketchup. The resulting scenario can be seen in Figure 3. Similarly, Li et al. [5] build upon the open-source OpenStreetMap geographic data to create an elemen- tary road network. Using CityEngine software, they import the attributes of the virtual road network (in- cluding type, width, and the number of lanes) and complete the surrounding environment (see Figure 4). Introduced by Dosovitskiy et al., CARLA (see Fig- 50 vol. 41/2023 U SMART ZONE Figure 3. Simulation for testing Path Planning and Decision-making Modules, Reinforcement Learning-based Car-following Model, and Autonomous Parking [8]. Figure 4. Simulation based on OSM and CityEngine for HiL [5]. Figure 5. CARLA is complex HiL training and test- ing software [9]. ure 5) is complex HiL training and testing software [9]. “The environment is composed of 3D models of static objects such as buildings, vegetation, traffic signs, and infrastructure, as well as dynamic objects such as vehi- cles and pedestrians. All models are carefully designed to reconcile visual quality and rendering speed: we use low-weight geometric models and textures but main- tain visual realism by carefully crafting the materials and making use of variable level of detail.” [9] They used these assets to create two urban environments, initially by modelling roads and sidewalks, then by manually placing buildings, vegetation, terrain, and other elements of transport infrastructure. The two city models have 2.9 and 1.4 km roads where the test vehicle can move. This system is modular and open- source and is used by many researchers for their HiL simulations. Gomes et al. [12] took a different approach. In addition to autonomous driving testing, they also focused on the Quality of Experience of their users. For this reason, they aimed at a higher realism of the simulated environment, especially from the visual side. Their method used photogrammetry for all objects in the simulated environment, which you can see in Figure 6. Continuously more authors use photogrammetry to create photorealistic objects, for instance, Shah et al. [3] uses their AirSim implemented in the Unreal engine (see Figure 7 – left) with its access to hundreds of pre-created models. Guerra et al. [6] took a similar approach with their FlightGoggles. They created their own 3D objects using photogrammetry (see Figure 7 – right). Both environments are generally used for de- velopment and testing of autonomous aerial vehicles. 3. Method We decided to advance the current HiL simulation en- vironment by creating a large-scale and highly realistic model of the Severní Terasa district in Ústí nad Labem, Czech Republic. That is a nearly 7.5 km of public roads with a total area of approximately 1.4 km2. This model contains a great variety of elements, including directionally divided and directionally undivided sec- tions, a roundabout, a traffic light-controlled intersec- tion, a dense public transport network, and frequent exits for integrated emergency vehicles (see Figure 8). Thus, it is ideal for running complex simulations. The actual development of the model is divided into four successive steps. These are (i) elevation model, (ii) photogrammetric model, (iii) laser scanning, (iv) manual modelling. 51 O. Piksa, A. Orlický, M. Scháno Acta Polytechnica CTU Proceedings Figure 6. Scene created only by photogrammetry [12]. Figure 7. A snapshot from AirSim (left) and FlightGoggles (right) [3, 6]. Figure 8. Extent of the reconstructed section in Ústí nad Labem, Czech Republic. Figure 9. Example of terrain rendering by OSM. 52 vol. 41/2023 U SMART ZONE Figure 10. Simple model crafted on low resolution OSM elevation map. 3.1. Elevation model The construction of the polygon commenced with an altitude map of the terrain. That defined the core element upon which more specific models were subse- quently applied. Its visual quality primarily influences the quality of experience of human participants, as it also serves as a visual backdrop providing the model authenticity. At short distances, it is superseded by a more detailed model based on laser scanning or photogrammetry, described below. We compared sev- eral different sources of an altitude map. The first was OpenStreetMap (OSM). OSM (see Figure 9) is preferable in cases with lower standards for an ac- curate description of the relationship between real and virtual environments, as they have a relatively low resolution of elevation points for the vehicle or pedestrian simulation. Beyond that, they offer an ad- ditional option, for instance, the automatic generation of buildings. Nevertheless, the buildings and terrain generated are highly inaccurate, in particular, terrain lacks the subtle details (unevenness, etc.) found in an actual environment (see Figure 10). In our country, a more accurate altimetry called ZABAGED® (short for “Basic Base of Geographic Data”) is available. “Digital Terrain Model of the Czech Republic of the 5th generation (DMR 5G) rep- resents the image of natural or man-modelled terrain in digital form as heights of discrete points in a tri- angulated irregular network (TIN) with a total mean error of the height 0.18m in an open terrain and 0.3m in a forested terrain.” [13] The model was created using airborne laser scanning acquired data (see Fig- ure 11). The availability of such an accurate elevation model makes the surrounding landscape look much Figure 11. View of ZABAGED® elevation with basic texture (up) and without texture (down). Figure 12. View of ZABAGED® elevation without smoothing (up) and with smoothing (down). more authentic and makes it easier to situate objects in it. We further cultivate the visual aspect of the view through low-polygon model smoothing functions (e.g., Auto Smooth in Blender) (see Figure 12). 3.2. Photogrammetric model The above elevation model delivers realistic and high- visual quality before a close-up. Then, it is insuf- ficient from the driver’s perspective. In such cases, photogrammetry can be beneficial as it provides, when used correctly, a relatively accurate point cloud. Con- sequently, accurate surface geometry can be generated, resulting in a high-quality and realistic simulation. And also, this step yields a high-quality texture that further enhances the realism of the overall environ- 53 O. Piksa, A. Orlický, M. Scháno Acta Polytechnica CTU Proceedings Figure 13. The measuring vehicle equipped with cameras for photogrammetry. ment. Photogrammetry is a relatively simple and quickly deployable method for creating 3D models, but its use for longer sections still entails significant time effort for data collection. We are trying to solve this partly by placing the measuring equipment on a vehicle, which we then drive through the recon- structed area. The telescopic-modular design, shown in Figure 13, allows us to adjust the sensor positions to the nature of the site measured. It is adjustable in height between 1.5–3.0metres above the vehicle in any position, even when exceeding the vehicle out- lines. That makes data collection quick and easy. This approach makes it possible to overcome the dis- advantages of conventional fixed measuring devices (see Figure 14). However, this method brings several challenges for acquiring sufficiently clear images. Those range from dynamic shocks to the need for a synchronized shutter speed based on the vehicle speed. We use commer- cially available GoPro Hero 9 cameras to maintain this method cost-efficient. The difference between low and high quality images can be seen in Figure 15. Follow-up experiments showed that it is possible to achieve sufficiently high-quality image while driving. Figure 14. The disadvantage of a fixed scanner (in this case, a laser scanner) is the impossibility of col- lecting data on a site that is not directly visible to the scanner and Example of a model reconstructed from the passage of a measuring vehicle (down). Figure 15. Image of the attainable quality of the photogrammetric model from a detailed environment shot with a static camera (up) and from a driving measuring vehicle (down). 3.3. Laser scan The laser scan functions similarly within the frame- work of the constructed model as photogrammetry. Yet, it is a significantly more expensive technology employing, in our case, a high-speed high-performance dual scanner mobile mapping system Riegl VMX-2HA (Result shown in Figure 16). For this reason, we use the low-cost photogrammetry as an alternative. Con- sequently, we compare photogrammetric results with 54 vol. 41/2023 U SMART ZONE Figure 16. Wired model from high-speed high-performance dual scanner mobile mapping system Riegl VMX-2HA. Figure 17. Combination of multiple textures (plaster + graffiti) before (up) and after (down) application of displacement (focus on graffiti). result obtained via laser scanning. The key difference to be observed will be the quality of the geometric shape of the drivable area to allow the simulation of the realistic behaviour of the vehicle. The compari- son will be performed using the professional vehicle physics simulation software CarSim. 3.4. Manual modelling Many objects had to be modelled manually. There are many tools for simplifying complex geometric shapes obtained by photogrammetry or laser scanning (e.g., decimation, the process of discarding points from the data to improve performance) available. However, we have not come across any that would outperform manual modelling regarding object complexity (mesh LOD) and the resulting visual quality of the model, if a refined texture is used. In addition, many objects in Figure 18. Real stop on modelled section (up) and manually modelled stop with texture from real photos (down). our testing area that cannot be reconstructed without manual modelling (see Figure 17). We use many other tools to substitute surface details commonly lost using simplified object geometry. To illustrate, we apply the Shader effect via Vector Displacement based on the underlying texture (see Figures 17 and 18). This approach achieves simple yet photorealistic objects. For some objects, it is not yet possible to automate the creation of a virtual model with acceptable quality and complexity. An example of such an object can be seen in Figure 19. 4. Summary and future work In this paper, we reviewed the current work on virtual environments for HiL simulation and presented our highly realistic digital model of a complex city environ- ment for HiL simulation. The two main contributions of this model are (i) the highly heterogeneous environment for com- plex simulations, 55 O. Piksa, A. Orlický, M. Scháno Acta Polytechnica CTU Proceedings Figure 19. Complicated model with no option automatic reconstruction. (ii) the possibility of parallel testing in virtual and actual environments to validate the results. Using the real-world environment with all its flaws, including, for example, faulty, worn, or damaged road- way markings or contradictions between traffic signs and road markings, adds depth not previously used in HiL simulations. Thus, the resulting autonomous system derived from using the real-life environment is better equipped to rectify the randomness inher- ent to real road traffic. In the future, we intend to extend our model to include additional areas involv- ing adjacent rural roads and highway segments. For modelling the rural roads, we have selected an area of approximately 8.5 km in length involving cramped villages and parts with limited visibility due to veg- etation. That will allow simulations of emergencies associated with, for example, collisions with wildlife. Also, an approximately 9 km section directly adjacent to the current model area is selected for modelling the highway segments. References [1] A. Gaidon, Q. Wang, Y. Cabon, E. Vig. Virtualworlds as proxy for multi-object tracking analysis. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4340–4349. 2016. https://doi.org/10.1109/CVPR.2016.470 [2] Ş. Y. Gelbal, S. Tamilarasan, M. R. Cantaş, et al. A connected and autonomous vehicle hardware-in-the-loop simulator for developing automated driving algorithms. In 2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp. 3397–3402. 2017. https://doi.org/10.1109/SMC.2017.8123155 [3] S. Shah, D. Dey, C. Lovett, A. Kapoor. Airsim: High-fidelity visual and physical simulation for autonomous vehicles. In M. Hutter, R. Siegwart (eds.), Field and Service Robotics, pp. 621–635. Springer International Publishing, Cham, 2018. https://doi.org/10.1007/978-3-319-67361-5_40 [4] Y. Feng, C. Yu, S. Xu, et al. An augmented reality environment for connected and automated vehicle testing and evaluation. In 2018 IEEE Intelligent Vehicles Symposium (IV), pp. 1549–1554. 2018. https://doi.org/10.1109/IVS.2018.8500545 [5] X. Li, Y. Wang, L. Yan, et al. Paralleleye-cs: A new dataset of synthetic images for testing the visual intelligence of intelligent vehicles. IEEE Transactions on Vehicular Technology 68(10):9619–9631, 2019. https://doi.org/10.1109/TVT.2019.2936227 [6] W. Guerra, E. Tal, V. Murali, et al. Flightgoggles: Photorealistic sensor simulation for perception-driven robotics using photogrammetry and virtual reality. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 6941–6948. 2019. https://doi.org/10.1109/IROS40897.2019.8968116 [7] Y. Kang, H. Yin, C. Berger. Test your self-driving algorithm: An overview of publicly available driving datasets and virtual testing environments. IEEE Transactions on Intelligent Vehicles 4(2):171–185, 2019. https://doi.org/10.1109/TIV.2018.2886678 [8] Y. Chen, S. Chen, T. Zhang, et al. Autonomous vehicle testing and validation platform: Integrated simulation system with hardware in the loop. In 2018 IEEE Intelligent Vehicles Symposium (IV), pp. 949–956. 2018. https://doi.org/10.1109/IVS.2018.8500461 [9] A. Dosovitskiy, G. Ros, F. Codevilla, et al. Carla: An open urban driving simulator, 2017. arXiv:1711.03938 [10] Y. Liu, K. Zhang, Z. Li. Application of digital twin and parallel system in automated driving testing. In 2021 IEEE 1st International Conference on Digital Twins and Parallel Intelligence (DTPI), pp. 123–126. 2021. https://doi.org/10.1109/DTPI52967.2021.9540143 [11] X. Che, C. Li, Z. Zhang. A test method for self-driving vehicle based on mixed reality. In 2021 IEEE International Conference on Smart Internet of Things (SmartIoT), pp. 401–405. 2021. https://doi.org/10.1109/SmartIoT52359.2021.00075 56 https://doi.org/10.1109/CVPR.2016.470 https://doi.org/10.1109/SMC.2017.8123155 https://doi.org/10.1007/978-3-319-67361-5_40 https://doi.org/10.1109/IVS.2018.8500545 https://doi.org/10.1109/TVT.2019.2936227 https://doi.org/10.1109/IROS40897.2019.8968116 https://doi.org/10.1109/TIV.2018.2886678 https://doi.org/10.1109/IVS.2018.8500461 http://arxiv.org/abs/1711.03938 https://doi.org/10.1109/DTPI52967.2021.9540143 https://doi.org/10.1109/SmartIoT52359.2021.00075 vol. 41/2023 U SMART ZONE [12] G. D. Gomes, R. Flynn, N. Murray. A qoe evaluation of an immersive virtual reality autonomous driving experience. In 2020 Twelfth International Conference on Quality of Multimedia Experience (QoMEX), pp. 1–4. 2020. https://doi.org/10.1109/QoMEX48832.2020.9123128 [13] Geoportal ČÚZK – ZABAGED® – altimetry – introduction. [2023-03-06] https://geoportal.cuzk.cz/ (S(vesl5kbxcrvnynvkwudwtb31))/Default.aspx?lng= EN&mode=TextMeta&text=vyskopisZBG&side= vyskopis&head_tab=sekce-02-gp&menu=30. 57 https://doi.org/10.1109/QoMEX48832.2020.9123128 https://geoportal.cuzk.cz/(S(vesl5kbxcrvnynvkwudwtb31))/Default.aspx?lng=EN&mode=TextMeta&text=vyskopisZBG&side=vyskopis&head_tab=sekce-02-gp&menu=30 https://geoportal.cuzk.cz/(S(vesl5kbxcrvnynvkwudwtb31))/Default.aspx?lng=EN&mode=TextMeta&text=vyskopisZBG&side=vyskopis&head_tab=sekce-02-gp&menu=30 https://geoportal.cuzk.cz/(S(vesl5kbxcrvnynvkwudwtb31))/Default.aspx?lng=EN&mode=TextMeta&text=vyskopisZBG&side=vyskopis&head_tab=sekce-02-gp&menu=30 https://geoportal.cuzk.cz/(S(vesl5kbxcrvnynvkwudwtb31))/Default.aspx?lng=EN&mode=TextMeta&text=vyskopisZBG&side=vyskopis&head_tab=sekce-02-gp&menu=30 Acta Polytechnica CTU Proceedings 41:49–57, 2023 1 Introduction 2 Literature review 3 Method 3.1 Elevation model 3.2 Photogrammetric model 3.3 Laser scan 3.4 Manual modelling 4 Summary and future work References