Aeronautic pilot training and augmented reality ACTA IMEKO ISSN: 2221-870X September 2021, Volume 10, Number 3, 66 - 71 ACTA IMEKO | www.imeko.org September 2021 | Volume 10 | Number 3 | 66 Aeronautic pilot training and augmented reality Simone Keller Füchter1, Mário Sergio Schlichting1, George Salazar2 1 University of Estácio de Santa Catarina - Pesquisa Produtividade Program, Av. Leoberto Leal, 431, São José, Santa Catarina, Brazil 2 NASA, National Aeronautics and Space Administration - Johnson Space Center, 2101 NASA Parkway Houston, 77058, Texas USA Section: RESEARCH PAPER Keywords: Augmented reality; virtual training; flight panel; pilot training Citation: Simone Keller Füchter, Mário Sergio Schlichting, George Salazar, Aeronautic pilot training and augmented reality, Acta IMEKO, vol. 10, no. 3, article 11, September 2021, identifier: IMEKO-ACTA-10 (2021)-03-11 Section Editor: Bálint Kiss, Budapest University of Technology and Economics, Hungary Received January 15, 2021; In final form September 14, 2021; Published September 2021 Copyright: This is an open-access article distributed under the terms of the Creative Commons Attribution 3.0 License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Corresponding author: Simone Keller Füchter, e-mail: simonekf.2011@gmail.com 1. INTRODUCTION Various different materials are used for pilot training on the equipment used in various types of aircraft. Traditionally, the aircraft manual has been an important source of information on each part of the aircraft’s structure, as well as its mechanical, electronic and digital components. While the operating manual remains the main learning resource, a number of multimedia resources have been added for better visualisation of certain tasks and commands during the panel checklist. The array of available printed books can also be a good knowledge resource for students aspiring to be private pilots, also known as PPs. This pilot category involves certification that allows the individual to fly an airplane and carry baggage and passengers without being paid or hired.[1] Videos, photographs, printed posters, flight simulators and virtual reality glasses can also be used to help learn [2] and memorise the components of the panels. Each resource has its advantages and disadvantages. For example, the simulator is fairly complete and close to reality but is not available 24 hours a day for everyone, while its cost per hour of use is relatively high since it involves sophisticated equipment. Meanwhile, while videos present a great deal of detail, they do not involve interaction, and in an effort to find the information one is looking for, it is sometimes necessary to watch a long video to be able to visualise a specific procedure. Software such as Microsoft's Flight Simulator [3] and X-Plane [4], among others, have a wide base of information and procedures as well as high- quality graphics. However, the purpose of the present work is to create a prototype app for smartphones that is easy to use and is specifically focused on familiarising the pilot with the panel and the respective checklists. Thus, the learning and memorisation of a new aircraft, as well as the attendant training, can be faster, bringing great savings and, above all, enhanced flight safety. Checklists are a critical part of preparing for a flight. Apps such as that presented in this paper can be used anywhere without the need for computers, screens, joysticks, or any other hardware; in fact, all that is required is a smartphone. The app can be used, for example, before the pilot goes into a flight simulator. Indeed, the app is not intended for use as a kind of flight simulator, but a specific and cheap tool to help pilots or students to memorise the flight panel components and study them continuously, be it at home, in open spaces or even in rooms without an internet ABSTRACT A pre-flight checklist requires in-depth technical knowledge of the aircraft, including its dashboard, avionics, instruments, functions and cabin layout. To obtain up-to-date certification, students training to be a pilot or advanced pilot must be completely familiar with each instrument and its position on the flight panel. Every second spent searching for the location of an instrument, switch or indicator can waste time, resulting in a poor start-up procedure and possibly a safety hazard. The objective of this research was to obtain preliminary data to determine whether the use of augmented reality as a human interface for training can help pilots improve their skills and learn new flight panel layouts of different aircraft. The methodology used was the human-centred design method, which is a multidisciplinary process that involves various actors who collaborate in terms of design skills, which includes individuals such as flight instructors, students, and pilots. A mobile/tablet application prototype was created with sufficient detail on the flight panel of a Cessna150, an aircraft used in training flights at the Aero Club of Santa Catarina. The tests were applied in Brazil and the results indicated good response and acceptance from the users. mailto:simonekf.2011@gmail.com ACTA IMEKO | www.imeko.org September 2021 | Volume 10 | Number 3 | 67 connection. Indeed, the main objective of this work was to attempt to represent, as realistically as possible, the details of the flight panel without the need of computers and in a cost-effective and efficient way using readily available technology. 2. METHODOLOGY The methodology used for this research with augmented reality (AR) training is the human-centred design (HCD) method. This method is a multidisciplinary process that involves various stakeholders who collaborate in terms of design skills, including the individuals who specifically pertain to the process [5], such as flight instructors, pilots and students. Interactivity is a key point in this field and comes in the form of continuous testing and evaluation of the AR system and the related concepts. The HCD method is a key component of the human-systems integration (HSI) process, which is defined as an interdisciplinary technical and management process that integrates human considerations within and across all system elements [6]. In this context, ISO 13407:1999, part of ICS 13.180 on ergonomics, has been withdrawn and revised in terms of ISO 9241-210:2019 Ergonomics of human-system interaction-Part 210: Human- centred design for interactive systems. [7] The HSI domains define (a) how human capabilities or limitations impact the hardware and software and, conversely, (b) how the system’s software, hardware, and environment influence human performance.[8] Meanwhile, there are four principles of HCD [9]: • Function allocation between user and technology • Design iteration • Multidisciplinary design • Active involvement of users and a clear understanding of user and task requirements. These principles are crucial for the concept of this research since they are focused on the ergonomics, comfort and acceptance of the pilots and students. For this study, a prototype mobile app related to AR was built based on the Cessna150 aircraft panel, which is used for the training of new PPs in a number of flying clubs in Brazil, specifically here, the Aero Club of Santa Catarina. This prototype provides the pilot with the opportunity to familiarise themselves with the panel of a new plane that he/she may fly, accessing it on their smartphone anytime, anywhere, and viewing the panel in a highly realistic way. Furthermore, perhaps more importantly, the pilot can use the main checklist procedures in 3D view, which enables a better understanding of certain movements of push/pull, rotate, and other movements. 3. AUGMENTED REALITY The technology pertaining to AR involves overlaying virtual components in a real-world environment with users viewing it using a specific device. For the most part, virtual objects are added to the real world in real-time during the user experience [10]. To connect with this technology, digital devices such as smartphones, tablets and virtual reality glasses (e.g., Microsoft Hololens) [11] can be used. The difference between AR and virtual reality (VR) is that the latter creates a completely synthetic and artificial world in which the user is completely immersed, while in the former, the user can observe a real environment with virtual objects overlaid. With AR, the pilot can observe a virtual flight panel in front of them and, using a smartphone, can visualise and study each component before moving on to a real physical aircraft. The combination of AR and smartphone provides a useful tool for training, with the professional able to practice in a manufacturing environment [9], a garage or a hangar. When used correctly, the AR technology can present a real environment in terms of creating virtual objects that mimic real-time applications [12]. However, the future of AR depends on huge innovations in all fields [13], and this paper presents a new way of learning and training aircraft-related checklist panels. 3.1. Augmented reality as a training tool This technology is used in various industries [14], military environments, and schools and is highly useful in the health field, making it possible to visualise the inside of a human body. Furthermore, this type of digital interface is suitable for the new generations comfortable with the devices used for VR and AR [15]. When combined with gamification, educational technologies can be improved, largely because the new generations are open to experimenting with new virtual competencies and stimuli and are highly motivated to win [16]. The same idea can be applied to AR, that is, using challenges similar to videogames to promote learning and pleasure, as explained in terms of the dopamine cycle, which pertains to challenges, achievement and pleasure [17]. A recent study conducted on a multi-national European sample of pilots regarding the use of AR and gamification [11] demonstrated that 72.25 % of the women pilots and 56.25 % of the male pilots considered it satisfactory in terms of successfully finishing a task, while, overall, 70.74 % of the pilots regarded the feedback received for corrective actions as satisfactory. This demonstrates that interactivity is important for users. For this reason, the third step of the proof of concept presented in this paper was created to clarify the number of possibilities this prototype can offer. 3.2. Augmented reality and aviation According to the Aviation Instructors Handbook, [18] VR is already part of pilot training, while AR is not mentioned. Therefore, this field requires more research and more applications to find new ways to use this tool, such as external inspection, maintenance, procedures using the flight panel and various other aspects [19], [20], [21]. 4. THE PROTOTYPE In this paper, a life-size prototype was created and tested by six users made up of flight instructors, pilots and students, who applied the visualisation of a virtual flight panel employing AR to their smartphones or tablets. Bringing together the information from the manual, the experience of the instructors and pilots who were studying how to use different aircrafts, and the proofs of concept, various evaluations were created, as shown in Figure 1. A full description follows. 4.1. Preliminary proof of concept The prototype was built in Unity 3D, a game development platform [22], using Vuforia [23] as the AR technology. Here, it was used as a printed marker in banner size (A0), as shown in Figure 2. ACTA IMEKO | www.imeko.org September 2021 | Volume 10 | Number 3 | 68 After the first test, a small ‘pocket’ marker was printed, as shown in Figure 3. The app was developed after several meetings with the instructors and several practice flights with the students. The assessment was made based on how pilots perform a checklist before departure in the aircraft. A sketch was shown with the first placement of the instruments and buttons. The size of the panel was presented, and the quality of the generated 3D image was tested. The result exhibited good aesthetic quality and high definition, but still presented little interactivity. After the first phase, an enhancement was applied where various animations were created and placed in the first menu item, the idea being to present each item in the manual according to the order in which it appears from left to right, following the checklist found in the manual. For each instrument or button displayed, a green highlight appears, as shown in Figure 4. This step was not planned with this type of indication, but with the feedback given in the meetings and discussions, we determined that the first step in the study of the checklist was not to start with the checklist steps. In fact, it was more interesting to present all the instruments and buttons first before moving on to the checklist, using various animations of objects and instruments. The prototype, which ended up acquiring the unexpected new feature described above, now featured the item in the menu that makes it possible to understand not only the names and functions of the buttons but also the kind of movement it performs. If this movement is a turn, it is evidenced in terms of three dimensions: whether it is, for example, 45 ⁰ or 90 ⁰, whether it is a push/pull-type button, or whether it is an on/off switch. As such, it is possible to closely visualise the movement of the mixture control, which consists of a button and a lever that must be pressed at the same time. Therefore, in this second proof of concept, the goal was to demonstrate the panel in a better way than the pictures in the aircraft manual, where it is difficult to clearly observe the controls’ design, format and movements, as shown in Figure 5. In short, an aircraft manual presents 2D images, making it difficult to visualise the array of controls and understand whether each is a rotate, push/pull or press button. Thus, the goal was to demonstrate the panel in high definition and to include animation and interaction, as shown in Figure 6. This app is very specific to a checklist approach and the panel components, which makes it different to other applications such as X-plane. The focus of our app includes a didactical Figure 1. Proof of Concept. Figure 2. A life-size panel that can be viewed at home using a smartphone and a banner printed marker. Figure 3. A small-size ‘pocket’ panel that can be visualised anywhere, anytime in a ubiquitous approach. Figure 4. Each instrument and controls are presented to the user. Figure 5. Typical images presented in aircraft manuals, where full visualisation can be difficult (Cessna 150 Aircraft Manual). ACTA IMEKO | www.imeko.org September 2021 | Volume 10 | Number 3 | 69 explanation for each instrument. The Cessna 150 starting engine checklist [24] was used here, as shown in Figure 7. 4.2. Enhanced proof of concept At this point, greater interaction with the user was pursued, with the same checklist presented and the user asked to perform the movements of the lower control panel. It will only be possible to start the aircraft (start the engine) in this AR simulation if all the buttons have been executed with the correct movement and in the correct order. At the end of this step, the engine is started, and the audio will confirm the complete verification of the checklist in addition to a message. Following discussions with all those involved, it was understood that these features would bring gamification to the app, as the pilot feels challenged ‘to hit’ the movements and finish the checklist perfectly, and as a reward, the engine will start. This stage is still under development, but its concept and technologies have already been finalised. Flight instructors, pilots and students were involved in this experiment. 5. THE EXPERIENCE The prototype was tested by students, pilots and instructors. The experience involved the following steps: • Explaining the objectives of the study; • Obtaining the participants’ consent; • Reading the orientation on how to use the app; • Using the app in front of a banner as a marker for the AR. The user can be standing or seated and can move in front of the panel; • Using the app with a banner marker in A0 size and with a little marker printed on half an A4 page. In the first case, the user can see the AR panel in full scale (life- size). In the second case, the user can move the little marker using their hands and manipulate the panel position so that it can be visualised comfortably; • Filling out a questionnaire for reflection and sharing points of view The tests were performed with instructors, as can be seen in Figure 8, while the experience also involved students preparing to gain their PP license, as shown in Figure 9. 6. RESULTS The results demonstrated that the virtual 3D model is highly realistic and will prove useful for the pilot through the feature of simulating failures in the instruments in order to check whether the pilot had paid attention to the flight indicators or even whether the aircraft has deficiencies in the human interface design, which should be corrected or controlled during the flight. The prototype features various animated controls and items that promote interactive and complex tasks in different situations. The app can help the pilot to be more confident, faster and more secure when flying. Meanwhile, ensuring that less time is spent on the checklist in a real aircraft or a flight simulator will reduce the costs of the process and increase the safety with AR training. Furthermore, the use of AR improves the pilot’s situational awareness (S-A) [25] in perceiving, comprehending and projecting future actions in scenarios in which S-A and the Figure 6. In an AR app, it is possible to identify that a specific control is not a button but a lever. The animation clearly shows how to execute the procedure. Figure 7. The checklist items used in this experience (Cessna150 Aircraft Manual). Figure 8. Flight instructor testing the AR app. Figure 9. Student testing the AR app. ACTA IMEKO | www.imeko.org September 2021 | Volume 10 | Number 3 | 70 system’s mental models are important for minimising human error. Table 1 presents example information and comments that allowed us to conclude how useful this application can be for pilots and students, and even for companies that need to improve their training resources. 7. CONCLUSION This was merely a preliminary study, and more students, pilots and instructors will be invited to participate in future tests of the system. In addition, flight engineers and physicians will also become involved so as to increase the diversity in the evaluation of the system as part of the HCD method. It will be important to ensure that this application goes beyond the data and information that can be found in aircraft manuals and the existing procedures. It will also be crucial to improve the app such that it can capture and share the knowledge and background of qualified pilots, instructors and all stakeholders who may have novice pilots. This will help promote a complete experience, improve security, ensure better quality flights and good tolls, and will present mutually beneficial teaching methods, transferring the knowledge and experience of flying. Unquestionably, this study can be useful for the aircraft industries and for professionals such as those operating in the medical industry, where medical practitioners can receive more information to support the human cognitive and health fields in terms of, for example, medical operation procedures. Essentially, the AR method is independent of the field of application. ACKNOWLEDGEMENTS S. K. Füchter thanks the Aero Club of Santa Catarina, the instructors, Claudia Thofehrn and Thiago Bortholotti, the pilots and the students. We also thank Dr Orly Pedroso, physician specialist in aerospace medicine, development in health, astronaut, and increased VR in cosmic space. He is accredited by the National Civil Aviation Agency (ANAC), Brazil; Postgraduate in the Course for Aerospatiale Medical Examiners, and NASA, Johnson Space Center, Houston, Texas, USA: Physiology and Aerospatiale Medicine Course. S. K. Füchter also thanks the University of Estácio de Santa Catarina Center and the Programa Pesquisa Produtividade (PPP). REFERENCES [1] I. N. Gleim and G. W. Gleim, Private Pilot FAA Knowledge Test (1st ed.), University Station, 2011, pp 2. [2] J. Tustain, Tudo Sobre Realidade Virtual & Fotografia 3600. São Paulo Editora Senac, 2019. [3] Microsoft. Microsoft Flight Simulator. Xbox. Online [Accessed 18 September 2020] http://xbox.com/games/microsoft-flight-simulator [4] Laminar Research. X-Plane 11. X-Plane. Online [Accessed 18 September 2020] https://www.x-plane.com [5] International Standards Organisation (ISO), ISO 13407. Human- Centered Design Processes for Interactive Systems. Online [Accessed 15 September 2020] Fehler! Linkreferenz ungültig.https://www.iso.org/obp/ui/#iso:std:iso:13407:en [6] J. S. Martinez, N. Schoenstein, G. Salazar, T. M. Swarmer, H. Silva, N. Russi-Vigoya, A. Baturoni-Cortez, J. Love, D. Wong, R. Walker, Implementation of human-system integration workshop at NASA for human spaceflight, 70th International Astronautical Congress IAC Washington D. C., USA, 21-25 October 2019, 11 pp. Online [Accessed 20 Sept. 2020] https://ntrs.nasa.gov/api/citations/20190032353/downloads/2 0190032353.pdf Table 1. Users comments. Profile/Function Comments Pros Cons Flight instructor/ coordinator ‘The app has great potential, not only for new students studying to get their private pilot license, but also for experienced pilots when they want to upgrade and get their jet license, for example’. ‘Easy, it can be replicated to other airplanes, high definition’. ‘Check list could have all items, it could be more complete’. Flight instructors ‘Cool. We can see in detail’. ‘It is a good way to learn about the instruments and controls. Good to memorise’. ‘Cool. We can see in detail’. ‘It is a good way to learn about the instruments and controls. Good to memorise’. ‘The prototype is just for Android. Various instructors use iPhone’. Students ‘There are illustrations in the manuals that do not allow us to understand the real movement that is made when activating a certain switch, lever, button, or selector. With AR, it is possible to visualise things as if the panel were really in front of us, and the items are easily seen’. ‘It would have been great to have had an app like this one when I was preparing myself to be a pilot and upgrading my licenses and airplanes’. ‘Functional, detailed, easy’. ‘It could include more procedures’. Students ‘I found it very realistic and I liked the part where I could have it at home and use the panel whenever I wanted’. ‘Super. I can study at home and see the panel in detail. When I use this airplane ( I will be a new student) it will be familiar to me’. ‘Nowadays, I´m learning to fly in an Aeroboeiro aircraft, I would like to have one of these for this airplane’. ‘Cool, portable, useful in training’. ‘The prototype is just for Android. Various students use iPhone; the screen is too small to click on it’. Aerospace Physician ‘Brilliant Project’ ‘Useful and safe in training’. http://xbox.com/games/microsoft-flight-simulator https://www.x-plane.com/ https://www.iso.org/obp/ui/#iso:std:iso:13407:en https://ntrs.nasa.gov/api/citations/20190032353/downloads/20190032353.pdf https://ntrs.nasa.gov/api/citations/20190032353/downloads/20190032353.pdf ACTA IMEKO | www.imeko.org September 2021 | Volume 10 | Number 3 | 71 [7] ISO International Standalisation Organisation, Documentation. Online [Accessed 25 September 2020] https://www.iso.org/standard/52075.html [8] NASA/SP–2015-3709 Human Systems Integration (HSI) Practitioner’s Guide. National Aeronautics and Space Administration Lyndon B. Johnson Space Center Houston, Texas. 2015. Online [Accessed 18 December 2020] https://ntrs.nasa.gov/api/citations/20150022283/downloads/2 0150022283.pdf [9] NASA/TP-2014-218556. Human Integration Design Processes (HIDP). National Aeronautics and Space Administration International Space Station Program Johnson Space Center Houston, Texas. USA, 2014. Online [Accessed 18 December 2020] https://www.nasa.gov/sites/default/files/atoms/files/human_i ntegration_design_processes.pdf [10] P. Milgram, F. Kishino, A taxonomy of mixed reality visual displays. IEICE Trans. Inform. Syst. 77 (1994), pp. 1321-1329. [11] H. Schaffernak, Potential augmented reality application areas for pilot education: an exploratory study. Educ. Sci. 2020, 10, p. 86. DOI: 10.3390/educsci10040086 [12] C. Kirner, and R. Tori, Introdução à realidade virtual, realidade misturada e hiper-realidade. Realidade Virtual: Conceitos, Tecnologia e Tendências. 1 ed. Sã Paulo, SENAC, 2004, v. 1. [13] P. Cipresso, I. A. Giglioli, M. A. Raya, G. Riva, The past, present, and future of virtual and augmented reality research. Front Psychol (2018). DOI: 10.3389/fpsyg.2018.02086 [14] S. Füchter, T. Pham, A. Perecin, L. Ramos, A. K. Füchter, M. Schlichting, O uso do game como ferramenta de educação e sensibilisação sobre a reciclagem de lixo. Revista Educação e Cultura Contemporânea 13(31) (2016), pp. 56-81. DOI: 10.5935/2238-1279.20160022 [15] L. Brown, The next generation classroom: transforming aviation training with augmented reality. In: Proceedings of the National Training Aircraft Symposium (NTAS), Embry-Riddle Aeronautical University, Daytona Beach, FL, USA, 14–16 August 2017. Online [Accessed 8 Jan. 2021] https://commons.erau.edu/ntas/2017/presentations/40/?utms ource=commons.erau.edu%2Fntas%2F2017%2Fpresentations% 2F40utmmedium=PDFutmcampaign=PDFCoverPages [16] S. K. Füchter, M. S. Schlichting, Utilisação da realidade virtual voltada para o treinamento industrial. Revista Científica Multidisciplinar Núcleo do Conhecimento. Ano 03 10(07) (2019), pp. 113-120. [17] G. Zichermann, J. Linder, The Gamification Revolution: How Leaders Leverage Game Mechanics to Crush the Competition. New York: McGraw-Hill, 2013. [18] Aviation Instructor’s Handbook: 2008, Department of Transportation, Federal Aviation Administration, Flight Standards Service: Oklahoma City, OK, USA, 2008. [19] S. K. Fuchter, Augmented Reality in Aviation for Pilots and Technicians. October 2018 XR Community of Practice. Johnson Space Center – NASA. Houston, 2018. [20] S. K. Fuchter, Pre-flight Inspection and Briefing for Training Aircraft Pilots using Augmented Reality. IEEE Galveston Bay Section. Joint Meeting of YP Group and New I&S Section Chapter. University of Houston Clear Lake, Houston, TX, USA, 2018. [21] S. K. Fuchter. Augmented reality as a tool for technical training. Galveston Bay Section Meeting: The Institute of Electrical and Electronics Engineers (IEEE). Gilruth Center NASA-JSC, 17 August 2017. Online [Accessed 18 December 2020] https://site.ieee.org/gb/files/2017/06/IEEE-GBS-081717.pdf [22] UNITY. Documentation. Online [Accessed 18 September 2020] https://learn.unity.com/?_ga=2.143231477.1523753934.163121 9945-1791299178.1627660886 [23] PTC. Vuforia: Market-Leading Enterprise AR. Online [Accessed 22 September 2020] https://www.ptc.com/en/products/vuforia [24] Cessna Pilot’s Operating Handbook. 150 Commuter Cessna Model 150M, 1977. [25] M. R. Endsley, Toward a theory of situation awareness in dynamic systems. Human Factors Journal 37(1) (1995), pp. 32-64. https://www.iso.org/standard/52075.html https://ntrs.nasa.gov/api/citations/20150022283/downloads/20150022283.pdf https://ntrs.nasa.gov/api/citations/20150022283/downloads/20150022283.pdf https://www.nasa.gov/sites/default/files/atoms/files/human_integration_design_processes.pdf https://www.nasa.gov/sites/default/files/atoms/files/human_integration_design_processes.pdf https://doi.org/10.3390/educsci10040086 https://doi.org/10.3389/fpsyg.2018.02086 http://dx.doi.org/10.5935/2238-1279.20160022 https://commons.erau.edu/ntas/2017/presentations/40/?utmsource=commons.erau.edu%2Fntas%2F2017%2Fpresentations%2F40utmmedium=PDFutmcampaign=PDFCoverPages https://commons.erau.edu/ntas/2017/presentations/40/?utmsource=commons.erau.edu%2Fntas%2F2017%2Fpresentations%2F40utmmedium=PDFutmcampaign=PDFCoverPages https://commons.erau.edu/ntas/2017/presentations/40/?utmsource=commons.erau.edu%2Fntas%2F2017%2Fpresentations%2F40utmmedium=PDFutmcampaign=PDFCoverPages https://site.ieee.org/gb/files/2017/06/IEEE-GBS-081717.pdf https://learn.unity.com/?_ga=2.143231477.1523753934.1631219945-1791299178.1627660886 https://learn.unity.com/?_ga=2.143231477.1523753934.1631219945-1791299178.1627660886 https://www.ptc.com/en/products/vuforia