Microsoft Word - 1071 Article Text, Copyedited.doc Adv Syst Sci Appl 2021; 03; 12-21 Published online at https://ijassa.ipu.ru. Advanced Technologies for Supporting Operational Decision-Making in Civil Aviation Valentin G. Lebedev1*, Elena L. Kulida1 1) V.A. Trapeznikov Institute of Control Sciences, Russian Academy of Sciences, Moscow, Russia E-mail: lebedev@ipu.ru Abstract: In civil aviation, many tasks require the use of modern technologies, in particular artificial intelligence technologies. The features of the crew and dispatcher decision-support tasks that can be solved based on deep neural networks are considered. The features of the intelligent crew interface for improving interaction with the operational decision support system are considered. Optimization of air traffic management by using genetic algorithms is illustrated by the example of two tasks. A special feature of the genetic algorithm for solving the problem of optimizing the sequence and landing times of arriving aircraft is a non-standard crossing operator. A special feature of the genetic algorithm for solving the problem of constructing conflict-free trajectories of a given length at low altitude in conditions of complex terrain is the method of forming the objective function. In conclusion, there is a list of problems that need to be solved for the wider use of artificial intelligence in civil aviation. Keywords: decision support system, air traffic management, artificial intelligence, deep neural networks, genetic algorithms 1. INTRODUCTION One of the main priorities of civil aviation is to ensure flight safety, which is largely determined by the correct and timely actions of pilots and dispatchers, and the serviceable technical condition of aircraft. The development of modern technologies, in particular artificial intelligence technologies, will support the adoption of adequate operational decisions by the crew and dispatcher, especially in critical situations. The complexity of tasks, a large amount of incoming information, short time for decision- making, and responsibility for decisions lead in some cases to a mismatch between human capabilities to the requirements of the process efficient operational management of human- machine systems. In this regard, computer-based operational decision support systems are being increasingly used. The interface between the crew and the system is the most important component of the decision support systems. The goal of creating an intelligent crew interface is to reduce the emotional and sensory burden on the crew and create a human-machine interaction similar to human-to-human communication. Promising technologies to support decision-making by the dispatcher are related to solving the following tasks ‒ • collection, processing, and intelligent analysis of information about the movement of aircraft and its application for air traffic management and organization systems; • increase in throughput by optimizing the structure of the air space, adapting as much as possible to the real flows of aircraft; • optimization of aircraft flight routes, including in the airport area. * Corresponding author: lebedev@ipu.ru ADVANCED TECHNOLOGIES FOR SUPPORTING OPERATIONAL DECISION-MAKING IN CIVIL AVIATION 13 Copyright ©2021 ASSA. Adv. in Systems Science and Appl. (2021) The use of data mining methods is of significant interest for finding difficult-to-formalized patterns in the data on the state of aircraft aggregates and nodes. There are a large number of modern data mining methods for solving the problem of classification of the technical condition of the aircraft. The most well-known of them are neural networks, decision trees, the method of reasoning based on similar cases, the method of support vectors. The listed methods and their various modifications differ from each other according to the principle of forming the final result, but they have a common property – they use information about previously occurring cases (precedents) to search for and formalize patterns. One of the promising areas of application of intelligent technologies is the use in aviation training complexes. Training of individual skills and abilities with the help of such aviation training complexes can be automatically improved based on objective testing data, knowledge verification, work protocols, and then optimized by correlating changes in the training methodology with changes in the learning outcome indicators. Artificial intelligence will allow pilots to have even more realistic simulations, allowing them to observe not only "virtual reality", but also those objects that are difficult to see during a real flight. Aviation training complexes with implemented artificial intelligence generates individual training tasks (training exercises), collect all training data, and records the behavior of the pilot during training. The data collected during training can also be used to improve autopilots. In the modern practice of training civil aviation pilots, the most widespread are complex (Full flight simulator) and procedural (Flight procedures training device) aviation training complexes with a single virtual environment. The use of artificial intelligence tools allows you to practice piloting in a wide range of external conditions. Let's look at some of these areas of use of artificial intelligence methods in civil aviation in more detail. 2. SUPPORT FOR OPERATIONAL DECISION-MAKING BY THE CREW The analysis of aviation accidents shows that the introduction of artificial intelligence methods can be especially useful for the crew in the following situations: • special situations: the approach to critical flight modes and output from them (the output modes at supercritical angles of attack, beyond the altitude and speed characteristics, going beyond the operational limits of alignment, etc.); • special situations relating to the technical status of the onboard systems, equipment, and assemblies (engine failures, hydraulic, fuel, and other systems, fire, etc.); • special situations related to external threats to the safety of flight and the use of airspace (meteorological, geomagnetic phenomena, terrain, risk of collision in the air); • special situations related to the psychophysiological state of the crew (increased psychophysiological load, fatigue, etc.); • special situations of the tactical and special use of the aircraft (unacceptable distance from the runway, violation of the state border, etc.). Hybrid technologies, which combine different approaches, are an integral part of the implementation of decision support systems. Expert systems were the first artificial intelligence approaches for creating decision support systems. For use in aviation, onboard operational-advisory systems were developed to solve specific tactical tasks, including military applications, the purpose of which was to determine a rational way to achieve an operationally assigned goal. During the development of the expert systems, it became clear that one of the most time- consuming tasks is to obtain knowledge from experts and formalize it, that is, to fill in the knowledge base to describe a specific subject area that is the core of the expert systems. In this regard, the task of automatic acquisition of knowledge by the system has come to the fore – the task of machine learning, in the process of solving which deep neural networks and 14 V.G. LEBEDEV, E.L. KULIDA Copyright ©2021 ASSA. Adv. in Systems Science and Appl. (2021) methods of their training has been further developed. Deep optimization methods are being actively developed based on deep neural networks. This group of methods is designed to solve a wide range of problems. 2.1. Automatic object recognition; development of technical vision systems for solving navigation problems Further automation of the decision support systems requires analysis of the state of the external environment based on aerial observation data. Due to the large volume of aerial observation data, it is obvious that artificial intelligence methods and models need to be used to process this information. Currently, training methods for solving problems of pattern recognition and big data analysis have been developed [1]. In the field of image processing and analysis, deep convolution neural networks are used in combination with deep machine learning. Neural network algorithms based on deep convolution neural networks demonstrate good results and can be effectively used, including in solving problems of combining multispectral images, for example, based on a generative adversarial neural network. With the help of the deep neural networks on board the aircraft, image recognition tasks can be solved: monitoring of gas pipelines, power lines, etc. To the problems of image recognition solved by technical vision systems, the problem of recognizing phenomena occurring in space can be added. The task of autonomous navigation of the aircraft based on the images from the technical vision systems is also relevant. In the practical implementation of this task, various difficulties arise associated with image interference caused by weather conditions, errors in the alignment of the technical vision systems sensors, etc., which require the development of algorithms that are invariant to noise and geometric distortions. One of the ways to solve this problem is to use a convolutional neural network that allows you to detect objects even with partial noise. As research shows, the use of a convolutional neural network is quite effective for recognizing characteristic points of the terrain (reference points). The main difficulty of this approach is the selection of reference points and the process of training a neural network to solve such a problem. 2.2. Intelligent interface of the crew The intelligent crew interface should provide: • a visual, integrated, comprehensive, multi-modal representation of information; • improved, synthesized, and combined vision, which is very important especially in landing conditions with poor visibility, as well as in low-altitude flight; • interact in natural language, adaptive to the pilot and the problem being solved, including the dialogue, including by transmitting and receiving commands and unstructured data; the use of artificial intelligence in providing a voice will allow achieving a more rapid and informative when sending messages; • face, voice, and gesture recognition for more accurate identification; • tracking the psychophysical state of the pilot and his actions with the generation of warning signals regarding the detected state, characterizing the level of real human abilities to solve problems and control the aircraft. In addition to the above-mentioned ways of interaction between the crew and the decision support systems, interaction is of interest when, as a result of an in-depth analysis of the state of the aircraft systems, the crew is automatically offered the information in the appropriate form that is most relevant at the current time. The development of the so-called neuro interface is also underway. It is a new promising direction in which a special headset that measures the signals of human brain activity, controls them, and can react in time, for example, when the pilot has begun to fall asleep. Such an interface can be an integral part of the module for monitoring the psychophysical ADVANCED TECHNOLOGIES FOR SUPPORTING OPERATIONAL DECISION-MAKING IN CIVIL AVIATION 15 Copyright ©2021 ASSA. Adv. in Systems Science and Appl. (2021) state of a person. Machine learning models can be trained to distinguish between normal and abnormal patterns about pilots ' eye movements, data from other psychophysiological sensors such as heart rate or blood pressure, and assist pilots in abnormal situations [2]. The top-level architecture of the intelligent interface of the crew is shown in Figure 2.1. Fig. 2.1. The top-level architecture of the intelligent crew interface 3. IMPROVING THE EFFICIENCY OF AIR TRAFFIC MANAGEMENT Promising artificial intelligence methods for improving the efficiency of air traffic management are, in particular, genetic algorithms. The main difference between a genetic algorithm and a random search is the active use of information accumulated during iterations. The main advantage of genetic algorithms is their "creativity", i.e. the ability to obtain unexpected effective solutions that are not generated using other methods. Let's consider the application of genetic algorithms for solving the following two problems: 16 V.G. LEBEDEV, E.L. KULIDA Copyright ©2021 ASSA. Adv. in Systems Science and Appl. (2021) • improving the efficiency of using airport runways by optimizing the sequence and landing times of arriving aircraft; • construction of conflict-free routes of aircraft in the concept of air traffic "free flight" (Free Flight). 3.1. Improving the efficiency of the use of airport runways by optimizing the sequence and landing times of arriving aircraft One of the most difficult tasks in the airport area is the task of optimizing the planning of the sequence and landing times of arriving aircraft, taking into account the attenuation of the resulting vortices and turbulence zones. The mathematical formulation of the problem of forming optimal aircraft queues for landing as a linear or quadratic optimization problem (depending on the selected objective function) is NP-complete [3]. The task of optimizing the sequence and landing times of arriving aircraft is to optimize the global target function for a group of aircraft that are in the airport area to land, P is the number of aircraft waiting to land. Between the aircraft, according to ICAO standards, it is necessary to provide a certain minimum interval to avoid falling into the wake vortex formed from the aircraft going ahead. This interval depends on the types of consecutive aircraft. Known matrix S of size P*P, where – the minimum interval between landing aircraft after the aircraft , , Ci is the type of i-th SU. Also, for each aircraft, a time window is defined , during which the aircraft with the number i can land by its flight characteristics, fuel availability, flight duration, etc. Ei – the earliest possible landing time of the i-th aircraft; xi – the assigned landing time of the i-th aircraft, Li – the latest possible landing time of the i-th aircraft. For each aircraft, the time Ti is also known – the optimal time of arrival of the i-th aircraft under the condition of a free runway, . Depending on the air traffic conditions, the optimized global objective function may depend on many factors, but as a rule, the determining values are the deviation of the real landing times xi from the optimal landing times Ti. For example: • the piecewise-linear objective function is the minimization of the sum of absolute deviations from the optimum time of planting: , where (3.1) • the nonlinear objective function is the minimization of the sum of squares of deviations from the optimal time of planting: , where (3.2) In this task the subject of numerous scientific works, an overview of exact and approximate methods of solving this problem is presented in [3, 4], which deals with the genetic and memetic algorithms, algorithm diffused search and bionic algorithm, ant algorithm, etc. The main obstacle to using the proposed methods in practice in real-time is the unacceptably long counting time. The conducted research shows that heuristic and genetic algorithms are the most promising for solving the problem [5]. A special feature of the genetic algorithm for solving the problem of optimizing the sequence and landing time of arriving aircraft is a non-standard crossing operator. The landing sequence is encoded using the vector , which is a permutation of the aircraft numbers. The standard crossing operator cannot be applied in this case, because as a result of its application, the resulting vectors may not be permutations of aircraft numbers. Figure 3.1 shows this with the example of a single-point crossing. ADVANCED TECHNOLOGIES FOR SUPPORTING OPERATIONAL DECISION-MAKING IN CIVIL AVIATION 17 Copyright ©2021 ASSA. Adv. in Systems Science and Appl. (2021) Fig. 3.1. Standard single-point crossing operator It is proposed to perform the crossing operator of the corresponding vectors of the assigned landing times instead of the crossing operator of the vectors y. Figure 3 shows this by the example of a single-point crossing. Then, the corresponding vectors are reconstructed from the new vectors obtained. Fig. 3.2. Proposed single-point crossing operator In [6], a heuristic algorithm for solving the problem is proposed and the results of the study of the effectiveness of this algorithm are presented. The algorithm is based on using some initial solution, which is then consistently improved during the iterative process. As a result of the research, it was found that the result of applying the algorithm significantly depends on the quality of the selected initial solution. On the other hand, the peculiarity of genetic algorithms is that they do not guarantee an optimal solution, but allow you to quickly get an acceptable solution and then use it as the initial solution for a heuristic algorithm. This hybrid approach proved to be very effective. Table 3.1 shows the results of 20 tests of applying algorithms for a sequence of 17 aircraft for a nonlinear objective function (3.2). The effectiveness of the algorithm is estimated by the value of the minimized objective function for the resulting solution. The values of the objective function are given for the initial solution, the solution obtained using a heuristic algorithm, the solution obtained using a genetic algorithm, the solution obtained using a hybrid algorithm, and for the optimal solution, which for a sequence of 17 aircraft can be obtained in the absence of restrictions on the counting time. Table 3.1 shows that both heuristic and genetic algorithms provide a significant improvement of the initial solution, and the hybrid algorithm allows you to get optimal or fairly close to the optimal solution. For clarity, the results of comparing the algorithms are shown in Figure 3.3. 3.2. Construction of conflict-free routes of aircraft in the concept of air traffic "free flight» A serious problem with the perspective concept of air traffic management "free flight" in the airport area is associated with the limited capabilities of the human dispatcher. The way to overcome this problem is to further improve the automation of air traffic control, in particular, in the automatic synthesis of trajectories and the calculation of speeds along the trajectories to accurately perform the specified time intervals between aircraft during landing. In [7, 8], an algorithm for generating the shortest conflict-free approach trajectories in complex terrain conditions and an algorithm for controlling the speed along the constructed trajectories to maintain the necessary separation rates between aircraft is proposed. 18 V.G. LEBEDEV, E.L. KULIDA Copyright ©2021 ASSA. Adv. in Systems Science and Appl. (2021) Table 3.1. The values of the objective function Test № Initial solution Heuristic algorithm Genetic algorithm Hybrid algorithm Optimal solution 1 704400 307564 310052 293100 293100 2 633057 227376 172175 171687 171687 3 279885 191400 185789 174344 166424 4 286196 141035 144304 141035 141035 5 912663 213026 213026 213026 213026 6 544427 165802 158362 158362 158362 7 1027015 301946 300370 300370 299743 8 261039 74370 74370 74370 74370 9 233146 100882 102166 102166 100882 10 2103394 457870 444770 444770 444770 11 232913 172292 168932 168932 162332 12 162565 150013 150013 150013 150013 13 211737 206756 206756 206756 206756 14 430268 119752 123816 119752 119752 15 323899 243954 230094 230094 230094 16 462540 266424 212784 212784 211584 17 220370 130446 130446 130446 130446 18 1060169 295720 251620 251620 251620 19 345061 153205 135352 134212 134212 20 957920 355969 291844 286465 286465 Fig. 3.3. Comparison of algorithm results ADVANCED TECHNOLOGIES FOR SUPPORTING OPERATIONAL DECISION-MAKING IN CIVIL AVIATION 19 Copyright ©2021 ASSA. Adv. in Systems Science and Appl. (2021) However, the proposed approach does not provide a solution in the case when maintaining the necessary separation standards between the aircraft using speed control on the shortest trajectory is impossible due to speed and acceleration restrictions. In this case, it is necessary to build a trajectory of the desired length, taking into account the difficult terrain, limited maneuverability of the aircraft, restrictions on the trajectory, taking into account the trajectories of other aircraft. The solution to this complex problem is possible with the help of a genetic algorithm, in which the necessary constraints are taken into account in the form of penalties in the optimized objective function [9]. We will explain this by the example of penalties for deviation of the trajectory length from a given target length D, for conflicts of the trajectory with the terrain, and sharp angles between successive segments of a piecewise linear trajectory. Each solution in the genetic algorithm has the form: . This is a set of n integers, which are the numbers of points in the maneuvering area. The coordinates of the point e_i are denoted by . Then: (3.3) Deviation from the specified target length (3.4) Based on the elevation matrix H, a set of horizontal and vertical segments is constructed, so that the trajectory that does not intersect these segments does not have conflicts with the terrain. Each segment of a piecewise linear trajectory is checked for intersection with all segments from the set N1 denotes the number of segments of a piecewise linear trajectory that intersect with segments from the set The total number of acute angles between successive segments of a piecewise linear trajectory is denoted by N2: (3.5) Then the objective function has the form (3.6) where F1 is the penalty for crossing the terrain, F2 is the penalty for an acute angle. The penalties are chosen large compared to the length of the trajectory, for example, , in this case, the genetic algorithm generates such chromosomes that the corresponding trajectories do not intersect with the terrain and do not have sharp corners. Figure 3.4 shows the results of the algorithm for constructing a trajectory with a given length of 250,000 meters in the mountainous surroundings of Yelizovo airport at an altitude of 1500 m. Areas of terrain that need to be flown at the specified altitude due to the risk of collision with the terrain are shaded in the drawing. The constructed trajectory has a length of 250134 meters, has no sharp corners between consecutive segments, and passes through safe areas of the terrain. 1, 1, 1, 1 2 1 1, если 0, , где 0 в противном случае. n i i i i i i i i i d d d N n n - + - + = + - ³ìï= = í ïî å 20 V.G. LEBEDEV, E.L. KULIDA Copyright ©2021 ASSA. Adv. in Systems Science and Appl. (2021) Fig. 3.4. Comparison of algorithm results 4. CONCLUSION In addition to the aforementioned areas of application, the use of AI techniques in civil aviation includes detection and prevention of collisions between aircraft and unmanned aerial systems; automation and autonomy of management; cybersecurity; regulation of traffic flow (simulation of demand for air travel, the facial recognition at airports, baggage and carry-on baggage), and others [10]. The prospects for using artificial intelligence capabilities in civil aviation are extremely diverse, but at present, they are still underutilized [11]. This is due to some problems that need to be solved for the wider use of artificial intelligence: 1. It is important to find the right balance between the use of AI technologies and human skills in ensuring flight safety in terms of piloting, control, and maintenance. 2. It is necessary to take into account the person in the system of support for operational decision-making. A person must be confident in the solutions offered by the AI system, trust them, while the methods of obtaining solutions using artificial intelligence are difficult to understand and explain. 3. It requires analysis of a large amount of source data obtained from different sources, and a large number of calculations to obtain the result. 4. It is necessary to take into account the problem of vulnerability of neural networks, which can lead to serious errors. ADVANCED TECHNOLOGIES FOR SUPPORTING OPERATIONAL DECISION-MAKING IN CIVIL AVIATION 21 Copyright ©2021 ASSA. Adv. in Systems Science and Appl. (2021) Despite these difficulties, the results show that it is difficult to overestimate the possibilities of using artificial intelligence methods in civil aviation. In the future, the construction of an aircraft with one pilot is being considered, the second pilot is planned to be replaced by artificial intelligence, and fully autonomous passenger aircraft are being developed (Boeing, Airbus, etc.). REFERENCES 1. Vizilter, Yu. V., Gorbatcevich V. S., Zheltov S. Yu. (2019) Structure-functional analysis and synthesis of deep convolutional neural networks, Computer Optics, 43(5), 886–900. DOI: https://doi.org/10.18287/2412-6179-2019-43-5-886-900 U [in Russian]. 2. Lutnyk, L., Rudi, D., Martin, R., (2020) Towards Pilot-Aware Cockpits, Conference: 1st International Workshop on Eye-Tracking in Aviation (ETAVI 2020), DOI:10.3929/ethz-b-000407661. 3. Veresnikov, G.S., Egorov, N.A., Kulida, E.L., Lebedev, V.G. (2019). Methods for solving of the aircraft landing problem. I. Exact solution methods // Automation and Remote Control, 80(7), 1317–1334. 4. Veresnikov, G.S., Egorov, N.A., Kulida, E.L., Lebedev, V.G. (2019). Methods for Solving of the Aircraft Landing Problem. II. Approximate Solution Methods // Automation and Remote Control, 80(8), 1502-1518. 5. Kulida, E.L. (2020). Analysis of Algorithms for Solving the Aircraft Landing Problem. Proc. of the 13th International Conference "Management of Large-Scale System Development" (MLSD). Moscow: IEEE, https://ieeexplore.ieee.org/document/ 9247839. 6. Kulida, E., Egorov, N., Lebedev, V. (2020). Algorithm for Scheduling a Flight of a Group of Aircraft to Implement an Optimal Landing Plan. Proc. of the 12th International Conference "Management of Large-Scale System Development" (MLSD). Мoscow: IEEE, https://ieeexplore.ieee.org/document/8911115. 7. Bazhenov, S.G., Egorov, N.A., Kulida, E.L., Lebedev, V.G. (2016). Control of Aircraft Trajectory and Speed to Avoid Terrain and Traffic Conflicts during Approach Maneuvering. Automation and Remote Control. 77(10), 1827-1837. 8. Kulida, E.L., Lebedev, V.G., Egorov, N.A. (2018). The heuristic algorithm for planes queue optimization. Proc. of the 11th International Conference «Management of Large-Scale System Development» (MLSD). Мoscow: IEEE, https://ieeexplore.ieee.org/document/8551908. 9. Kulida, E.L., Lebedev, V.G. (2017). Genetic Algorithm for Generating Trajectories of Specified Length for the Onboard Flight Path Safety System. Procedia Computer Science, 112, 1015-1022. 10. Maheshwari, A., Davendralingam, N., DeLaurentis, D.A. (2018). A Comparative Study of Machine Learning Techniques for Aviation Applications. Aviation Technology, Integration, and Operations Conference, DOI: 10.2514/6.2018-3980. 11. Kashyap, R. (2019). Artificial Intelligence Systems in Aviation, In book: Cases on Modern Computer Systems in Aviation, DOI: 10.4018/978-1-5225-7588-7.ch001.