1 American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) ISSN (Print) 2313-4410, ISSN (Online) 2313-4402 © Global Society of Scientific Research and Researchers http://asrjetsjournal.org/ Current Stage of Autonomous Driving Through a Quick Survey for Novice Saman Sarraf* The Institute of Electrical and Electronics Engineers, Senior Member IEEE Email: samansarraf@ieee.org Abstract Today, autonomous driving is considered a branch of artificial intelligence in which various technologies are employed, ranging from computer vision to machine learning-based sensor fusion technologies. This work summarizes the autonomous vehicle advances and also discusses the crucial components required to build such technology. The state-of-the-art architectures of autonomous vehicles compromise several core modules, including sensors, road scene perception, motion planning, core control system, and system management. The research showed that computer vision technologies such as object detection and tracking and localization and mapping techniques, play crucial roles in an advanced autonomous vehicle functional architecture. The current stage of this industry demonstrates the successful prototyping of autonomous vehicles without drivers’ significant interventions. However, the research centers and automobile industries’ ongoing development aim to explore the productization of such highly automated vehicles and seek to improve road scene perception to reduce the number of sensors while enhancing or maintaining the current performance. Keywords: Autonomous Driving; Artificial Intelligence; Machine Learning; Computer Vision. 1. Introduction Autonomous driving is an emerging technology that has been of interest to many companies and researchers over the past decades, and it encompasses a broad area of self-driving cars, drones, trains, and agricultural machines along with military applications [1]. The concept of autonomous driving goes back to the 1920s when American Wonder – a Chandler controlled by radio signals – was demonstrated in New York City [2]. After years, the first autonomous vehicle controlled by itself was developed by Carnegie Melon University in 1986, which was a self-driving van. ------------------------------------------------------------------------ * Corresponding author. American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) (2020) Volume 73, No 1, pp 1-7 2 In 2005, another team at Carnegie Melon University built a self-driving car that could successfully navigate eight miles and won the defense advanced research projects agency (DARPA) grand challenge [2]. Researchers at Google developed an autonomous vehicle in 2010, traveling from Los Angeles and San Francisco and recorded over 140,000 miles of driving over the years. In 2016, numerous automobile companies, including Tesla, BWM, Audio, Mercedes Benz, and some transportation enterprises such as Uber, incorporated self- driving hardware into their products [2]. Other streams of autonomous vehicles, such as drones or trains, could advance faster since their ecosystems were less engaged with human interactions than autonomous cars [3]. The fundamental concept of autonomous vehicles relies on employing artificial intelligence (AI) to automatically control the vehicles based on surrounding information [4]. To build such a complex system including various components shown in Figure 1, different AI branches are utilized which include (a) supervised and unsupervised machine learning techniques for decision-making purposes, (b) computer vision and deep learning technologies for video data analysis, and (c) sensors fusion and processing techniques for vehicle control and safety [4,5]. Also, advances in distributed hardware such as graphical processor units (GPUs) for accelerated processing, big data technologies [6], and network communication (i.e., the fifth generation referring to 5G [7]) play crucial roles in building autonomous vehicles [1]. Figure 1: Autonomous vehicle AI-based components 2. Levels to Autonomy The “autonomous” terminology offers a broad range of definitions, and such an ambiguity led the practitioners in the domain of the autonomous vehicle to categorize the autonomy of driverless vehicles into several levels as follows, implying the level of the automatic controlling system along with drivers’ engagement in a vehicle [8]. Level 0: No automatic and zero controlling driving capabilities are offered at this level except optional and necessary hazardous alarms. Level 1: The automated controlling systems and drivers share vehicles’ control American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) (2020) Volume 73, No 1, pp 1-7 3 while the drivers still have full control. Such vehicles are often provided with advanced driving assistance systems. Level 2: The automated control system offers an optional ability to take full control of the vehicle, but the driver is still fully responsible for conducting the car. Level 3: The autonomous vehicle can take full control without passenger’s supervision; however, drivers must intervene if alarming systems require any manual efforts. Level 4: The automated system takes full control of the vehicle without drivers’ attention or intervention except for unpredicted circumstances where the drivers take back control of the car. Level 5: In this last level, no human intervention is considered. 3. Autonomous Vehicle Functional System Architectures The functional system architectures of autonomous vehicles have advanced over the past decades where the state-of-the-art topology is formed of five major components [9], including (a) sensors, (b) perception and scene understanding, (c) behavior, and motion planning, (d) vehicle control and actuation and (i) system management illustrated in Figure 2. Figure 2: The functional system architecture of a novel autonomous vehicle As seen in the figure, the flow of information starts from sensors to perception systems and continues towards the motion planning component. The processed information by AI-based modules is transferred to the control and stabilization system to adjust steering, acceleration, and brakes in real-time. A system management software algorithm monitors potential system failure, provides backup plans, and ensures a safe operation [9]. The significant subsystems forming an autonomous vehicle functional architecture are discussed [10]. 4. Sensors and Hardware Sensors and hardware are the primary pillars of autonomous driving systems, which are categorized into five groups, including (a) exteroceptive, (b) proprioceptive, (c) communications, (d) actuators, and (e) computational components [11]. Exteroceptive sensors are used for road scene perception purposes, including camera, lidar, ultrasound, or radar, whereas proprioceptive sensors are employed to monitor autonomous vehicles’ operation speed or acceleration [11]. Actuators convert sensors data into electrical signals for digital processing, and American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) (2020) Volume 73, No 1, pp 1-7 4 computational units are designed to store the sensor data [11]. 4.1. Cameras To replace drivers’ visual systems, various types of cameras are required in autonomous vehicles to capture image/video data, including (a) monocular cameras to detect traffic light, (b) omnidirectional cameras providing a panoramic view, (c) event cameras to capture events such as a considerable change in brightness [12]. The cameras often capture video data or static images used for 2D/3D object detection, pedestrian detection, tracking, semantic segmentation, traffic lights, and signs detection by employing advanced computer vision and deep learning techniques [11,12,13,14]. 4.2. RADAR and LIDAR Radio detection and ranging (RADAR) and light detection and ranging (LIDAR) are the sensors that can capture the depth information used to complement the video data captured by cameras [15]. RADAR, LIDAR, and ultrasonic sensors can capture 3D information from surrounding where RADAR emits radio waves whereas LIDAR works with infrared, and the illumination has no impact on the quality of their captured data, unlike the cameras. The data collected using RADAR and LIDAR, along with some cameras’ data, are used for sensor fusion, which can exceed human perception proved by researchers [11,15]. Figure 3 illustrates a typical scene, including various objects whose data captured by different sensors such as cameras, LIDAR, and RADAR. Figure 3: Various components in a typical scene that are recognized by different sensors 5. Localization and Mapping A crucial component in an automated driving system refers to the localization and mapping module in which the position of a vehicle must be accurately and continuously defined, so-called ego-localization [11]. An autonomous vehicle must locate itself in a correct position for safe driving and communicate with the global American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) (2020) Volume 73, No 1, pp 1-7 5 navigation system. Among the localization and mapping techniques, simultaneous localization and mapping system so-called SLAM, global positioning system accompanied with inertial measurement unit referring to (GPS-IMU) and a priori map-based localization techniques have been broadly employed in the state-of-the-art autonomous driving architectures [11]. Another advanced localization technique refers to point cloud matching, a computer vision technology to discover spatial transformation between two cloud points [16]. 6. Perception Road scene perception is the central pillar of autonomous driving systems, encompassing various tasks to extract information from vehicles’ surroundings. Among the sensors used for perceiving scenes, 2D/3D cameras are still the main hardware components collecting image-based data processed by computer vision techniques [17]. The core perception module of autonomous vehicle compromises (a) detection techniques including image-based object detection, semantic segmentation, 3D object detection, (b) tracking techniques such as object tracking, and (c) road and lane detection [11]. Sensor fusion methods aim to simultaneously utilize RADAR, LIDAR, and ultrasound data to measure the depth – a missing part in vision – and provide accurate information about objects and obstacles surrounding vehicles; however, some researchers consider depth- cameras to reduce the dependency of autonomous vehicles on RADAR/LIDAR. One of the significant challenges in the road scene perception is to conduct multimodal data alignment and synchronization to provide frame-level road scene prediction [11]. 7. Performance and Risk Assessment A reliable autonomous vehicle continually evaluates the vehicle’s current risk status and accurately predicts potential risks to drivers, passengers, and surrounding pedestrians [11]. Risk and uncertainty assessment are crucial components of autonomous vehicles, which indicate the level of risks by quantifying the risks using Bayesian and neural network models. Also, surrounding driving behavior assessment and driving style recognition are considered other performance and risk factors assessment tools [11]. 8. Decision-Making A sophisticated post-processing algorithm is often required to stabilize the output of computer vision, deep learning, sensor fusion models, and incorporate the finding from the core assessment module for decision- making [11,18]. Such a module ensures that an autonomous vehicle, its passengers, and surrounding pedestrians are safe, and the risk level is low or manageable with a backup plan. The core decision-making module often includes (a) global planning and (b) local planning to accomplish the objective of a trip, which is to arrive at the destination highly safely without failures [11]. 9. Human-Machine Interaction One of the vital concepts of an autonomous vehicle is to provide a technology in which physical human interactions such as steering is removed [19]. However, a bilateral communication is always required between an autonomous vehicle and its passengers to adjust various parameters about the primary driving tasks (i.e., American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) (2020) Volume 73, No 1, pp 1-7 6 defining the destination) or secondary passengers’ tasks such as adjusting the cabin temperature [19]. The human-machine interaction (HMI) core module is designed using natural language processing (NLP) and speech recognition algorithms to address such requirements. Recently, video-based HMI technologies are implemented in which a multimodal video and audio data are utilized for model development [19]. 10. Conclusion The current autonomous vehicles stage compromises state-of-the-art deep learning, sensor fusion, accelerated hardware, and network technologies incorporated into a system resulting in several autonomous vehicle prototypes considered level-4 autonomy. Soon, scientists and automobile companies plan to enhance road scene perception to address existing issues since the current algorithms sometimes fail to predict the surroundings. Furthermore, they plan to employ improved sensor fusion techniques for better data alignment and vehicle localization. The researchers also seek potential solutions to reduce the hardware components – especially the number of sensors – or onboard data capturing technologies to simultaneously collect multimodal data to reduce the system complexity while offering the same or enhanced performance. The researchers hope the level-4 vehicles to be commercialized shortly; however, a temporary remote-control system might be required for a while to guarantee the safety of vehicles, passengers, and pedestrians. References [1]. J. Wang, J. Liu, and N. Kato, “Networking and communications in autonomous driving: A survey,” IEEE Commun. Surv. Tutorials, vol. 21, no. 2, pp. 1243–1274, 2018. [2]. B. Zito, “2017: The year for autonomous vehicles,” Machine Learnings, 2017. https://machinelearnings.co/2017-the-year-for-autonomous-vehicles-8359fec2d2db. [3]. D. Patil, M. Ansari, D. Tendulkar, R. Bhatlekar, V. N. Pawar, and S. Aswale, “A Survey On Autonomous Military Service Robot,” in 2020 International Conference on Emerging Trends in Information Technology and Engineering (ic-ETITE), 2020, pp. 1–7. [4]. S. Grigorescu, B. Trasnea, T. Cocias, and G. Macesanu, “A survey of deep learning techniques for autonomous driving,” J. F. Robot., vol. 37, no. 3, pp. 362–386, 2020. [5]. S. Sarraf, D. D. Desouza, J. A. E. Anderson, and C. Saverino, “MCADNNet: Recognizing stages of cognitive impairment through efficient convolutional fMRI and MRI neural network topology models,” IEEE Access, vol. 7, pp. 155584–155600, 2019. [6]. S. Sarraf and M. Ostadhashem, “Big data application in functional magnetic resonance imaging using apache spark,” in 2016 Future Technologies Conference (FTC), 2016, pp. 281–284. [7]. S. Sarraf, “5g emerging technology and affected industries: Quick survey,” Am. Sci. Res. J. Eng. Technol. Sci., vol. 55, no. 1, pp. 75–82, 2019. [8]. J. Helgath, P. Braun, A. Pritschet, M. Schubert, P. Böhm, and D. Isemann, “Investigating the effect of different autonomy levels on user acceptance and user experience in self-driving cars with a VR driving simulator,” in International Conference of Design, User Experience, and Usability, 2018, pp. 247–256. [9]. Ö. Ş. Taş, F. Kuhnt, J. M. Zöllner, and C. Stiller, “Functional system architectures towards fully American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) (2020) Volume 73, No 1, pp 1-7 7 automated driving,” in 2016 IEEE Intelligent vehicles symposium (IV), 2016, pp. 304–309. [10]. S. Sarraf and J. Sun, “Functional brain imaging: A comprehensive survey,” arXiv Prepr. arXiv1602.02225, 2016. [11]. E. Yurtsever, J. Lambert, A. Carballo, and K. Takeda, “A survey of autonomous driving: Common practices and emerging technologies,” IEEE Access, vol. 8, pp. 58443–58469, 2020. [12]. G. Beckers et al., “Intelligent autonomous vehicles with an extendable knowledge base and meaningful human control,” in Counterterrorism, Crime Fighting, Forensics, and Surveillance Technologies III, 2019, vol. 11166, p. 111660C. [13]. A. Sarraf, “Binary Image Classification Through an Optimal Topology for Convolutional Neural Networks,” Am. Sci. Res. J. Eng. Technol. Sci., vol. 68, no. 1, pp. 181–192, 2020. [14]. A. Sarraf, A. E. Jalali, and J. Ghaffari, “Recent Applications of Deep Learning Algorithms in Medical Image Analysis,” Am. Sci. Res. J. Eng. Technol. Sci., vol. 72, no. 1, pp. 58–66, 2020. [15]. S. Haag, B. Duraisamy, W. Koch, and J. Dickmann, “Radar and lidar target signatures of various object types and evaluation of extended object tracking methods for autonomous driving applications,” in 2018 21st International Conference on Information Fusion (FUSION), 2018, pp. 1746–1755. [16]. Z. Gojcic, C. Zhou, J. D. Wegner, and A. Wieser, “The perfect match: 3d point cloud matching with smoothed densities,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 5545–5554. [17]. A. Ziebinski, R. Cupek, H. Erdogan, and S. Waechter, “A survey of ADAS technologies for the future perspective of sensor fusion,” in International Conference on Computational Collective Intelligence, 2016, pp. 135–146. [18]. S. Sarraf, C. Saverino, and A. M. Golestani, “A robust and adaptive decision-making algorithm for detecting brain networks using functional mri within the spatial and frequency domain,” in 2016 IEEE- EMBS International Conference on Biomedical and Health Informatics (BHI), 2016, pp. 53–56. [19]. S. Gnatzig, F. Schuller, and M. Lienkamp, “Human-machine interaction as key technology for driverless driving-A trajectory-based shared autonomy control approach,” in 2012 IEEE RO-MAN: The 21st IEEE International Symposium on Robot and Human Interactive Communication, 2012, pp. 913–918.