10856 FACTA UNIVERSITATIS Series: Mechanical Engineering https://doi.org/10.22190/FUME220714047S © 2020 by University of Niš, Serbia | Creative Commons License: CC BY-NC-ND Original scientific paper A STUDY OF THE USE OF MIXED REALITY FOR CAPTURING HUMAN OBSERVATION AND INFERENCES IN PRODUCTION ENVIRONMENTS Milos Stojkovic1, Rajko Turudija1, Milan Trifunovic1, Marko Pavlovic2, Ivan Jovanovic3, Nenad Uzelac4, Vladeta Milenkovic5 1University of Nis, Faculty of Mechanical Engineering, Nis, Serbia 2Affine Technologies, Nis, Serbia 3KUVAG Group, USA 4G&W Electric, Bolingbrook, USА 5Netico-Group, Nis, Serbia Abstract. Augmented and mixed reality is already considered as needful technology of the modern production systems. It is primarily employed to virtualize proper digital content, mainly related to 3D objects, into the human visual field allowing people to visualize and understand complex spatial shapes, their mutual relations, and positioning. Yet, the huge potential of the technology is waiting to be revealed in its usage for collecting and recording human observations and inferences about the context of the production environment. Its bi-directional interface makes it the most direct and the most efficient knowledge capturing means to date. The paper presents the challenges and benefits that come from the usage of a conceptual interface of an mixed reality application that is designed to collect data, semantics and knowledge about the production context directly from the man-in-process. As a production environment for the development, implementation, and testing of mixed reality applications for this purpose, various processes for the assembly and maintenance of medium-voltage equipment were used. Key words: Augmented Reality, Mixed Reality, Knowledge capturing, Industry 4.0, Human-to-Machine Communication Received: July 14, 2022 / Accepted December 05, 2022 Corresponding author: Miloš Stojković University of Niš, Faculty of Mechanical Engineering, Aleksandra Medvedeva 14, 18000 Niš, Serbia E-mail: milos.stojkovic@masfak.ni.ac.rs 2 M. STOJKOVIC, R. TURUDIJA, M. TRIFUNOVIC, ET AL. 1. INTRODUCTION Augmented Reality (AR) and Mixed Reality (MR) earned the reputation of highly beneficial technology for the production systems of the Industry 4.0 environment (I4E) standards [1]. The main benefit it provides to the users in the production environment is a fascinating virtualization of digital content into the human visual and haptic field (VHF) (Fig. 1). By mixing the virtualized digital content with the real-world objects, a kind of mixed reality is being created, which provides a much more detailed visual, sonic and potentially tactile experience to the operator wearing AR/MR gear. Such functionality is ideal for a series of applications in a production environment - from visualization of training instructions [2, 3], through visualization of production operating guidelines [4], to effective visualization of measured magnitudes collected from the sensors of equipment and machines [5]. Fig. 1 The space of visual and haptic field of an operator – VHF Digital content being displayed in VHF can be data, i.e., alphanumeric or graphical content or 3D digital objects (Fig. 2). Usually, data are presented on virtual panels projected in the operator visual field covering a corresponding part of real space. In most cases these data are related to the context of VHF of a human equipped with AR/MR gear. When a 3D digital object is being displayed in VHF, mostly it is expected from an AR-app to "overlap" the real object with it (Fig. 3 and Fig. 4). In some specific applications (e.g., assembly), AR-apps can be designed to merely highlight the boundaries of an object to make it easier for a person wearing an AR/MR gear to spot the corresponding detail on a real object, such as highlighting a hard-to-see screw that should be tightened on a real assembly. Of course, AR/MR apps can virtualize a 3D digital object with the data that are related to it in VHF (Fig. 3). By developing the AR/MR solution for a series of applications like production, assembly, diagnostics, and maintenance of medium- and high-voltage equipment (Figs. 2, 3, 4 and 5), it has been determined that AR/MR could be used not only to virtualize digital content into VHF, but also to collect information from the VHF by a human. Even though this direction of information flow is not the primary application of AR/MR, it could provide a lot of benefits to the whole I4E production system. A Study of the Use of Mixed Reality for Capturing Human Observation and Inferences in Production ... 3 Fig. 2 Digital content (alpha-numerical and graphical data and 3D objects) in VHF. (MR- app for assembling the high-voltage equipment. MS HoloLens was used as MR-app running hardware.) Fig. 3 The virtualized high-voltage switchgear with the magnitudes that are being measured by its sensors. (AR-app for diagnosing the high-voltage equipment status in the field. iPhone and Android phones were used as AR-app running hardware.) Machine-to-machine (M2M) communication, which is focal in I4E production systems [6, 7], however, cannot autonomously recognize semantics of the current production context features without big datasets [8, 9]. Semantic categorization of complex datasets 4 M. STOJKOVIC, R. TURUDIJA, M. TRIFUNOVIC, ET AL. gathered from different sources in a production environment as well as the identification of causal-consequential relations between these data and corresponding phenomena and events that might be anticipated by their profound semantic interpretation seems to still be impossible without human intervention. This insufficiency sparked an idea to employ AR/MR technology as a tool for "capturing" the semantics of the process features and context. Of course, a human is the one who can and should do the semantic interpretation and categorization of these data and relate them to the production context that is visible in the VHF (through AR/MR gear). This way, it is possible to facilitate and speed up an I4E information system (which is usually expected to be enriched by artificial intelligence algorithms such as machine learning) to learn about the production process. Finally, a human's observations and inferences about the current production context that are being fed to an I4E information system through AR/MR gear and an AR/MR app are very useful information for production planners and managers. 1.1 Technology challenges The application of AR/MR technology is usually followed by two classes of challenges. The first is related to precise "overlapping" of a digital virtual object with its real counterpart (object) in space, that is, in VHF of an operator wearing AR/MR gear (Fig. 4). Fig. 4 Coinciding the 3D digital model of high-voltage cable termination with its real counterpart. (AR-app for remote assistance in handling the issues with high voltage cable termination. iPad Pro™ was used as AR-app running hardware.) The second is related to the methods of capturing and recording human observations, actions, and reactions about the mixed reality context, which exists in an operator’s VH field. A Study of the Use of Mixed Reality for Capturing Human Observation and Inferences in Production ... 5 The precise "overlapping" (of a virtual object with its real counterpart) is a specific challenge inherent to AR/MR primarily due to the following two reasons. Firstly, because of the inconsistency of optical aberrations between the visual fields of a human and an AR/MR gear optical system. This complicates the transformation of a virtual object projection in the visual field of an operator significantly and results in inadequate matching of the perspective of an operator and AR/MR gear causing the distortion of virtual objects. Secondly, because of impetuous head movements of an operator who wears AR gear, that is, due to fast changing gaze orientation. It is still difficult for an AR/MR app to follow these rapid movements smoothly, which results in the so-called drifting of the projection of a virtual object. Sufficiently precise "overlapping" is important especially for the operations where precise marking of a target location is essential for successful or valid performance of the operation. A typical example is guiding the welder to place two parts correctly before welding, such as positioning a stud onto a proper dent at a steel plate. (Fig. 5) Fig. 5 Guiding the welder towards proper dents in the sheet metal plate. (MR-app hand- welding of studs onto tank sheet plates. MS HoloLens was used as MR-app running hardware.) The challenges related to capturing and recording human observations, actions, and reactions about the mixed-reality content could be considered within two aspects as well. 1. The first aspect is related to the technology of noticing and recognizing appropriate human gestures, voice commands and comments. The technology which is already applied in most of current AR/MR wearables has been proven capable of capturing the hand and fingers gestures and recording human voice [10, 11, 12]. 2. The second aspect is about the challenge of embedding the meaning (semantics) of a captured and recorded human observation. The main benefit that is expected from this capability is to facilitate acquisition of so-called tacit or not-formalized knowledge regarding the mixed-reality content and its context. In the case of a production process, tacit knowledge is related to the process or an operation context, which is usually gained through experience. It is to be expected that the formalization of this kind of knowledge or even just assigning (linking or associating) the right 6 M. STOJKOVIC, R. TURUDIJA, M. TRIFUNOVIC, ET AL. semantic categorization to the MR context and its constituents could improve the performance of AI algorithms engaged within an I4E information system. Modeling the semantics of concepts, data, objects, and the whole context is a well-known research challenge that researchers have been trying to solve for a number of years [13, 14]. However, the use of AR/MR gear (i.e., an AR/MR app) seems to be very convenient for assigning semantics to the objects, data, and features of the MR context since human perception can be directly associated with any of the MR context constituents in a way that was not possible before AR/MR. And yet, it appears that this application of AR/MR has not attracted a lot of research interest until now. Fig. 6 Classification of main challenges for AR/MR application in a production environment 2. RELATED WORK A comprehensive review about the usage of AR in the manufacturing industry was done by Eleonora Bottani and Giuseppe Vignali [15], who analyzed and reviewed the scientific literature spanning from 2006 to 2017. The survey only included studies written in the English language and published in peer-reviewed international journals (174 papers were included in the survey). The review showed that the interest in AR in the manufacturing industry grew over the years and peaked in 2016, since the number of papers was many times higher than in the years prior to 2013. The most popular areas of AR applications were assembly and maintenance, but closely followed by training/learning, with production design, safety, remote assistance, and robotics being less popular areas of research. We should not neglect the possibilities of applying AR for effective visualization and simulation of engineering analyzes of body deformations in real time [16, 17]. Francesco De Pace et al. [1] also provided an overview of AR application but in the industry domain, in which five major areas of AR application were depicted: Human-Robot Collaboration, maintenance/assembly/repair, training, inspection and building monitoring. Even though all these papers recognize the industry benefits A Study of the Use of Mixed Reality for Capturing Human Observation and Inferences in Production ... 7 stemming from implementing AR technology, and even though all roughly agree on the main areas of AR use in the industry domain (maintenance, assembly, training/education, inspection, etc.), the potential for capturing human observations through AR was not mentioned. This is also the case with Feng Zhou et al. [18] and Kangsoo Kim et al. [19] review papers on the trends from the International Symposium on Mixed and Augmented Reality (ISMAR) from 1998 to 2007, and 2008 to 2017, respectively. The most obvious benefit of AR use in maintenance and assembly tasks is the replacement of maintenance guides and assembly instructions. These guides/instructions are often in paper form, which makes them quite difficult for the operator to use, as they need to shift their attention from the guide/instruction to the task at hand. Therefore, overlaying the needed information at the right time through AR can make the operator’s job easier and faster. This was showcased in the studies by Alessandro Ceruti et al. [20] and Dimitris Mourtzis et al. [4]. In Alessandro Ceruti et al. [20], an AR application was used for aeronautics maintenance tasks. The case study that was carried out showed that just finding a needed maintenance task through the developed AR application was 27% faster than with the paper manual. Dimitris Mourtzis et al. [4] developed an AR application for visualizing CAM instructions for a bending machine, instructions for the machine setup and safety instructions. Riccardo Masoni et al. [21] presented an example of a solution for remote maintenance where AR can be of great help. Their AR tool uses symbols, free-hand sketches, and text to convey a required instruction from the remote expert to the local operator. Roberto Pierdicca et al. [22] developed and tested an AR application for training-on-the-job purposes for assembly tasks. Main drawbacks of this AR application, as well as the majority of AR applications, are hardware insufficient processing power and poor ergonomics. Nevertheless, combining AR with practice books that are traditionally used for learning in academic studies can be implemented relatively easy, and can yield great results. Saša Ćuković et al. [23] showcased this through an AR integrated CAD practice book, which helped students with understanding shapes, manufacturing features, engineering drawings, etc., by superimposing virtual objects over specific illustrations in the practice book (every illustration has a square marker associated with some 3D CAD model). Ivo Mal´y et al. [24] used AR technology for human-robot collaboration, and the paper focuses on experiments where the users are moving around а real or virtual object that is being augmented by the application. Different visualization techniques were used and tested, as well as two different devices – AR glasses and a hand-held device. Ashish Doshi et al. [25] showed that not only hand- held devices (phones, tablets, etc.) and vizors can be used, but also that projector-based AR systems can be of great help in industrial environments. A 52% reduction of standard deviations was calculated for spot-welding tasks with an AR system, compared to the same tasks done without an AR system. All of these applications take a relatively long time to build, so some research was also focused on improving the efficiency of creating VR applications (that could potentially also work with creating AR applications). Flotynski et al. [26] used semantic modeling techniques and developed a new method of building and managing VR training scenarios. Gorski [27] suggested using general rules of available knowledge engineering methodologies for building VR applications. Going a step further than merely using AR to replace maintenance manuals, send expert instructions, visualize data or “overlap” the virtual onto the real world, AR systems can also be combined with machine sensors, machine learning technology, and other 8 M. STOJKOVIC, R. TURUDIJA, M. TRIFUNOVIC, ET AL. pillars of I4.0, to get great results. Iñigo Fernández del Amo et al. [28] presented a framework for AR integration in whole maintenance systems, which means not only using AR for operator support, but also for analyzing their performance and recommending improvements. For this, AR applications need to be connected to sensors of the machines that are being maintained, because gathered sensor data can help an operator diagnose problems, but also assess if operators performed the diagnosis correctly. In addition, for performance valuation, gesture tracking was proposed. Dimitris Mourtzis et al. [29] developed a framework that enhances shop floor maintenance scheduling using AR. The AR application displays the status of a machine (remaining time between failures) gathered by a monitoring tool, and if needed, allows the operator to immediately call a remote expert or schedule the maintenance tasks for later, through the shop floor scheduling tool. Fabio Bruno et al. [30] proposed the use of AR for detecting and annotating design changes made during work, in addition to formalizing and automatic collection and transferring of data to the designers, so no information is lost. The proposed AR tool can also be used for assembly activities, as it is capable to overlay virtual objects on real ones. Gang Zhao et al. [31] used deep learning and knowledge modeling in combination with a mobile AR application to enhance the outdoor learning experience. Essentially, the camera from a mobile device is used for obtaining a video stream, the category of the learning scene is than recognized, which is crucial for obtaining the virtual learning resource associated with the scene. Then textual and audio instructions, pictures and 3D models are superimposed on the scene to give the user a good outdoor learning experience. If looking at a wider scope, in terms of capturing tacit knowledge, several papers come up. Teegan L Johnson et al. [32] emphasize why capturing tacit knowledge of industry operators is essential. This paper presented case studies in two manufacturing facilities for inspection tasks. The conclusion was that the use of tacit knowledge in some operations could range from 15%, up to 50%, indicating that losing of this knowledge could cause substantial decrease of performance in terms of productivity quality and safety. Mikhail Fominykh et al. [33] reviewed of AR and VR technologies used in, as they say, capturing “human experience”. However, the research primarily points out that AR and VR can be used for the purpose of recording physical places and human actions, which, later, can be experienced by someone elsewhere replaying the recorded content. Adam Dudek et al. [34] presented a model of tacit knowledge transformation that can be used in a manufacturing company. The knowledge transfer is being done through automatic speech recognition and natural language processing by transformation of operator’s observations given by the voice into digitally recorded comments. Jason Hashimoto and Hyoung-June Park [35] used smart device augmented reality (SDAR) for exploring alternative fenestration design. By applying users’ tacit knowledge and intuition, they generated different virtual fenestration designs by adding/deleting openings, positioning them in different locations in a real space and modifying their size, and SDAR helped in visualizing the effects of the design on interior day lighting on surfaces (walls, floors, and objects inside the room). All the papers reviewed above suggest that the use of AR/MR technology for capturing human observation has been, to the best of the authors’ knowledge, poorly explored. A Study of the Use of Mixed Reality for Capturing Human Observation and Inferences in Production ... 9 3. THE CONCEPTUAL SOLUTIONS 3.1 Material, methods and means The feasibility of applying AR technology has been explored through several projects (2017-2019) for the client involved in production and maintenance of high- and medium- voltage equipment such as switchgears, reclosers, circuit brakers, cable couplings and terminations. As a result of the research, a few conceptual solutions of AR/MR-apps were developed and explored for various applications and AR/MR gear. Specifically, the research was dedicated to the application cases given/represented in Table 1. For all applications (Table 1), 3D digital objects were virtualized in VHF and overlapped with its real counterpart. Also, for all cases, it is possible to cancel the strict overlapping of the digital and real object and place the digital one in the most convenient area of VHF. This option has been proven as very ergonomic to the operator. The testing of the usage of the MR visor showed that permanently active overlapping reduces the visual cognitive performance of operators and makes it very difficult for them to notice all the necessary details on a real object while it is overlapped by its digital counterpart. Table 1 AR application cases that were used for the research and experiments conducting Application case AR/MR gear 1 Diagnosing the equipment status in the field, which included checking the functionality of the high-voltage switchgear, gathering and visualizing the data about measured magnitudes of interest such as voltage, current, power, etc. Phones (iOS, Android) 2 Training the maintenance operators to handle equipment and perform maintenance operations (dry run simulations) Tablet (iPad Pro), Vizor (MS HoloLens) 3 Remote surveillance and assistance in equipment handling and performing the maintenance operations in the field Phones (iOS, Android), Tablet (iPad Pro), Vizor (MS HoloLens) 4 Assembling equipment operations in the shopfloor Vizor (MS HoloLens) 5 Performing manufacturing (hand-welding) operations in the shopfloor Vizor (MS HoloLens) 3.2 Data stream-oriented interface features There are two data/information/knowledge streaming directions - the first is from the information system towards the human, and the second one is from the human-in-process towards the information system (Fig. 7). In accordance with it, the AR/MR app interface features have been designed in a way to enable data presenting and data collecting. 10 M. STOJKOVIC, R. TURUDIJA, M. TRIFUNOVIC, ET AL. Fig. 7 Two principal directions of data (information and knowledge) stream while using AR/MR 3.2.1 System to Human data stream (S2H) Concerning the digital content that is streamed to the operator from the process via Cloud, aside from the virtualization of 3D digital objects in their VHF, the AR/MR app feeds the operator with the data relevant to the current process context and equipment which is in the operator's VHF. The data are gathered from the sensing equipment and processed in the cloud before being forwarded to the operator. This way, the data become meaningful and valuable information for the operator since they are related to the (process and) ongoing operation context. Additionally, this data stream feeds the operator with the current operation execution time, comparing it with normed (standard) operating times (unit, preparatory-final, main, and auxiliary time). At the same time, by tracing the execution of operations in real time it is easy for the information system to interpolate operating times and provide the anticipation of the realization of operations to the operators and production management staff. 3.2.2 Human to System data stream (H2S) The main contribution stemmed from the research is the original interface features built to allow operators to feed the system with their observations and inferences. In terms of collecting data from the operator about the process, the AR/MR interface was established predominantly through a series of prepared questionnaires to which the operator should provide appropriate responses, as in the case of a quality assurance check list (Fig. 8 - The image originates from the AR/MR app developed by the authors of the paper for G&W Electric for the case of AR/MR-aided assembling the high-voltage cable joint. MS HoloLens was used as AR/MR-app running hardware.). A Study of the Use of Mixed Reality for Capturing Human Observation and Inferences in Production ... 11 Fig. 8 Prepared questionnaire for the quality assurance checklist (The sequence where QA requires from an operator to take a photo as an argument that the operation is done correctly) The operator’s responses to the predefined questions enable the production management system to monitor process-related details in real time, immediately identify challenges and failures, and anticipate their consequences in terms of time, quality, productivity, safety and costs. At the same time, passing through predefined questionaries on the status of an operation completion and process interventions, the operator generates the relevant information about the process to other operators nearby. Hence, AR/MR gear may be considered as an ideal platform for notification about the ongoing process status, integrating the Kanban and Andon notification systems in the VHF in an elegant and ergonomic manner. Moreover, through the AR/MR app, notifications are selected regarding their relevancy for each specific process participant. Besides a series of prepared questionnaires about (the part of) the process context, which are perceptible in the operator's VHF, a specific interface toolset that was built allows the operator to initiate an observation note, which can be corroborated by a video, photo or audio recording. What is even more important, this feature enables the operator to associate his observation to the object in VHF to which this observation refers (Fig. 9). Thus, in this way, the operator participates in the creation of additional semantic content that is related to the process' context. In terms of a perpetual effort to continuously improve the process (Kaizen), self-initiated context-related observations, that the AR/MR app enables, are very valuable since this content is immediately visible and accessible to production managers and can be further analyzed, reviewed, and acted upon to improve the process. 12 M. STOJKOVIC, R. TURUDIJA, M. TRIFUNOVIC, ET AL. Fig. 9 Operator’s observation based on captured data from the sensing equipment can be formalized into an expert rule for reaction (MR-app for the case of assembling three phase recloser. MS HoloLens was used as MR-app running hardware.) 3.3 Measuring the benefits and shortcoming As it was mentioned before, one of the greatest imperatives in I4E is to capture large data about an ongoing manufacturing process. The large data set enables an AI toolset, which is expectedly embedded and is running in the Cloud computing service, to try to identify and semantically categorize data patterns that can empower the capacity of the information system to anticipate process realization and possible challenges. However, in real industry practice, it is usually not so easy for the AI toolset to learn causal- consequential relations between collected data and the corresponding process context. The possibility to instruct the AI toolset to interpret data patterns properly by an operator (human) in the process can shorten the learning process remarkably. For the simplest data pattern (built by 3 to 4 data sources), the AI toolset needs minimum of 102 occurrences to learn (be trained), i.e., to recognize the “possible” causal-consequential relation by itself. With an operator who uses an AR/MR app to introduce the AI toolset to a new data pattern that is associated to a specific process context, the AI toolset needs just one (first) or a few occurrences. 3.3.1 Process improvement indicators The impact of the usage of AR/MR for capturing human observations and inferences about the process is measured by the following indicators: 1. Speed of disseminating important information about the process (SoD) between the operators and process management staff. This measure indicates how fast the relevant information reaches the process supervisors in-charge as well as the operators who should be informed (e.g., who operate preceding or succeeding sequences). The typical important information about the process that requires to be disseminated as soon as possible is the identified challenge/issue within the operation (regarding assembly, handling the equipment, maintenance, manufacturing). In comparison to the conventional information flow, which usually involves issue reporting (creating a A Study of the Use of Mixed Reality for Capturing Human Observation and Inferences in Production ... 13 digital or paper file) and later reading/analysing these reports, AR/MR-aided reporting through prepared questionnaires makes dissemination immeasurable faster. The information reaches the ones who should be notified immediately instead of at least dozens of minutes or even several hours later, which is common in conventional reporting. One should also emphasize the high relevancy of the information for the designated process participant in the case of AR/MR-aided reporting. In conventional report files, as it was identified through the interviews with operators in the production and assembling processes (12) and management staff (4), usually only 20- 25% of the content concerns the designated participant, since the report content is aimed at several (4-5) participants in the process. The direct consequence of the improved speed of dissemination of important information about the process is the improved reactivity which is reflected in focusing on overcoming the process issue and finding a solution faster. The reactivity of those who are in-charge to react is incented additionally by scoring the reaction (by the time and effect of a solution that they built). The indirect consequences are improved quality that comes from reducing the repetition of operating mistakes (reduction of rework), improved productivity that comes from the developed/upgraded and formalized solutions and improved safety. 2. Speed of learning the process (SoL). The learning is related to both human participants (operators and process management staff) and the AI toolset. Regarding the AI toolset, this measure indicates how fast the data processing algorithms, which are usually a kind of machine learning algorithms, will be able to correctly recognize and semantically categorize the data patterns from the process. The smaller the number of data-pattern cases needed for learning, the greater the speed of learning. It should also be mentioned that the time and expertise needed for preparing the data pattern for training should be taken into account as well. There are three approaches to training an artificial intelligence system. The first is supervised machine learning which requires at least one knowledge engineer to design predefined training data patters and expected outcomes. The second is to try to make the system learn by itself autonomously, passing through a two-step learning procedure. In the first step, the system should notice and extract the relevant features of real data patterns without the help of a knowledge engineer. Then, in the second step, the system is expected to build the algorithm for semantic interpretation and categorization of real data patterns, again autonomously. The third approach suggests employing an AR/MR- app to allow an operator to directly interpret and categorize the semantics of real process data patterns as they appear. The third approach does not need time and effort of a knowledge engineer to prepare training data patterns, but it still requires a kind of self-initiated engagement of an operator to “build the case” by indicating distinctive features of the process context and outcome, which come or can come out of this context. The research involved the first and the third approach because it was assumed that the second approach could not be sufficiently efficient (comparable with the first and the third). Additionally, due to safety reasons and inference traceability poorness, the second approach could not be applied, that is, it was not possible to leave the artificial intelligence system to build inferences completely on its own. 14 M. STOJKOVIC, R. TURUDIJA, M. TRIFUNOVIC, ET AL. 3.3.2 Measuring and results Testing the use of AR/MR and its effect on improving operational performance was carried out through a series of experiments in several sectors already denoted in table 1. To measure the “speed of disseminating the important information about the process” (SoD) objectively, it is important to identify four discrete activities that are involved in the process of dissemination itself (Fig. 10). Fig. 10 Dissemination route of the operator’s observations through the system The first activity is about generating the operator's observation. Within this activity the AR/MR app interface plays a crucial role because the efficiency of generating the observation is related to the intuitiveness of the interface. However, the solely intelligent sub-process of this activity is about the categorization regarding the importance of an occurrence that will be noticed, and it has to be done by the operator in the process through the AR/MR app. The second activity is about sorting out the observations, i.e., identifying the target person(s) to whom an operator's observation should be disseminated (the one or more who supervise and coordinate that part of the process). The cloud computing service does this activity in accordance with the predefined observation's tags that are generated by the AR/MR app (and the operator) regarding the process context. The third activity is about presenting the operator’s observation to the supervisors (who are called to react). In the case of experiments, that was done by using a pop up notice through the already existing process supervision application (sending it to the supervisors via computer or phone). The fourth step is about inducing reaction from the “person-in- charge”. This step depends on the incentive procedure that is adopted by the company. In the case of experiments, this was the “descending reaction value through time” procedure which induces supervisors to react as soon as they are able in order to earn high values for the reactivity indicator at the end of the control period. In measuring the impact of AR on SoD, only the first activity is relevant. The second is done autonomously and instantly by A Study of the Use of Mixed Reality for Capturing Human Observation and Inferences in Production ... 15 the computer, and the third and fourth are the activities that (can) also exist without using AR/MR. So, the time lapses from generating the observation to noticing it by the process supervisor or coordinator was measured and compared (the conventional approach of typing the file versus the approach featured in the AR/MR usage). The measurements are given in table 2. Table 2 Features and results of the conducted tests # Process AR gear Test conditions NO NT SoD1 (collateral, indirect indicators) Q1 P1 P2 Conv AR Con AR/MR Con AR/MR Con AR/MR 1 Diagnosing the equipment status Phone Lab. env. 4 >30 n/a n/a 1* 0.5 1 0.25 n/a n/a Phone Real. env. 6 >100 n/a n/a 1 0.5 1 0.25 n/a n/a 2 Training handling and maintenance op. Vizor Lab. env. 6 >50 n/a n/a 1 0.04 1 0.55 n/a n/a Tablet Real. env. 4 >20 n/a n/a 1 0.05 1 0.7 n/a n/a 3 Remote assistance in equip. handling and maintenance in the field Vizor Lab. env. 3 >50 1 0.2 1 0.4 1 0.75 1 0.67 Phone Real. env. 4 >100 1 0.1 1 0.5 1 0.67 1 0.8 Tablet Real. env. 4 >50 1 0.12 1 0.5 1 0.6 1 0.75 4 Assembling the recloser Vizor Lab. env. 4 >50 1 0.12 1 0.4 1 0.5 1 0.64 5 Hand-welding studs Vizor Lab. env. 2 >30 1 0.2 1 0.85 1 0.75 1 0.9 * All values are normalized to avoid publication of real data. For example, if for some activity conventional approach requires 10 repetitions it is normalized as 1, and if the same activity done with AR/MR requires 2 repetitions it is normalized as 0.2. The legend of table 2 is given below: - NO: Number of operators participating in tests. - NT: Number of conducted tests. - SoD1: Speed of disseminating - Time to generate and deliver an observation for a test case by the conventional channel (typing/filling the QA file or similar) and the AR app. Collateral benefits that demonstrate the effect of applying AR to disseminate observations of the operator in the process can be seen indirectly through the following operation performance indicators. - Q1: 1st quality performance indicator - Number of rework operations within test cases that were counted when operators’ observations were recorded and disseminated through the conventional channel (typing/filling the QA file) and through the AR app. - P1: 1st productivity performance indicator - Operation preparatory time within test cases measured when operators’ observations are recorded and disseminated through the conventional channel and through the AR app. - P2: 2nd productivity performance indicator – Operation performing delay caused by not getting the info about the process disturbance promptly. 16 M. STOJKOVIC, R. TURUDIJA, M. TRIFUNOVIC, ET AL. Measurements presented in the paper were done mainly in a laboratory environment, but some were done in a real ambient. Laboratory measurement involves a predefined scenario regarding test operation and disturbance within a controlled ambient (in special workshop test areas). Measuring the speed of learning (SOL) was done in the laboratory environment only, and for the cases of 1) recloser assembly and 2) remote assistance in equipment handling and maintenance in the field. The test scenario comprehended feeding the AI toolset with five data inputs, three analogue and two digital signals, whose combination values builds a specific process context, i.e., its features that, if they are semantically interpreted correctly, anticipate potential issues regarding quality and productivity that should be expected to appear. As an alternative solution to this learning procedure, a series of micro-services running within the AR app was designed to allow the operator to formalize knowledge about the process gained through the experience into expert rules which can also anticipate the same potential issues. The speed of learning about the process was measured by the effort invested to enable the AI toolset to semantically interpret the process context features correctly. These two possible routes of teaching the information system about the process are shown in Fig. 11 – the example is about recloser assembly. It is about recloser assembly process. Fig. 11 Two possible routes for teaching the system about the process There are three analog inputs: A1: Plunger body displacement, A2: Handle angular position, A3: Locking spring compression, and two digital signals: D1: Status of recloser mechanism position (Opened/Closed), D2: Vacuum interrupter contact (On/Off). A certain set of values of these inputs indicates possible issues to be expected: A Study of the Use of Mixed Reality for Capturing Human Observation and Inferences in Production ... 17 1. Regarding quality – Premature failure of Vacuum interrupter should be expected due to fatigue since the assembly is overtightened. 2. Regarding productivity – Prolonged installation and adjusting of the recloser in the field should be expected since the assembly is adjusted improperly. 3. Regarding productivity – Production delay should be expected due to the necessity to substitute the middle subassemblies since they are arriving in the final assembly operation improperly adjusted earlier in the process. To make the AI toolset capable of anticipating issues based on an input data set, a knowledge engineer needs to feed the AI toolset with numerous samples of the input data set. It becomes even harder for learning if it is necessary to learn the AI toolset to consider what value appears first and what later because the timeline of values appearing can improve the distinction of the process context and anticipation of potential issues that can be caused (for this research such cases were not included). Unlike a knowledge engineer, who must first learn the process context and its features by himself, and then prepare a suitable set of samples for teaching the AI toolset, a process operator who uses MR gear performs implicit semantic categorization and interpretation of process context features in real time, creating the appropriate expert rules (potential reaction). Table 3 shows the investment in learning of the AI toolset to be able to recognize the context of the process (based on data patterns) that indicates the forthcoming occurrence. The displayed values are obtained through testing in the lab environment. Table 3 Efficiency of learning AI toolset about the process – comparison between two learning routes: MR aided learning and conventional ML approach. Number of data pattern cases (based on 3 analogue and 2 digital data sources) Forthcoming occurrence With AR/MR Conv. ML approach Quality decline or an accident <10 >102 Operation productivity decline <10 >102 Operation performing delay <10 >102 It is important to also notice that the results showed in the table do not consider time and costs investment for training the AI toolset by engaging a proper knowledge engineer (which, inevitably, steals the time from the operators in the process and makes even more waste). Finally, it should not be neglected that engaging a knowledge engineer to teach the system about the process is a one-time job, and the AR/MR-aided learning is a tool for permanent learning. It is questionable whether it is possible to predict all the combinations of data patterns which can appear in real practice at once. One can argue that the main advantage of AR/MR -aided process learning comes from the ability to narrow the context envelope (VHF) and exclude practically impossible data- pattern cases in a few steps. Additionally, operators with AR/MR gear impose their inference, that is, semantic categorization of the current process context, generating a kind of expert rule in the form of an if-then logical relation between an input data pattern and consequences (Fig 9). In that way, the operator formalizes their tacit knowledge regarding the current process context and embeds it into the information system. 18 M. STOJKOVIC, R. TURUDIJA, M. TRIFUNOVIC, ET AL. From the perspective of human participants, especially process management staff, an AR/MR-aided bottom-up tacit knowledge stream speeds up the learning about the process, also. This data flow helps with getting more profound knowledge about the process features directly from the shopfloor operators, which is usually missing. However, for this research, we did not measure neither compare the learning speed between operators when the AR/MR -app was used and when it was not. An indirect consequence of the AI-toolset (learned) ability to categorize the process context accurately (relevantly) based on data patterns is its functionality to anticipate forthcoming occurrences fully autonomously and to react preventively, at least by sending a warning message to the participants in charge. Finally, these reactions of the AI toolset force the participants in the process to focus on finding the corresponding solutions that should prevent a decline in quality, productivity or extending deadlines for the given cases. As the collateral gains derived from data collecting by the AR/MR -app are: 1. Reduction of quasi-stochastic disturbances that cause process/operation performing delays. Actually, these disturbances are not random, but the information system, operators and supervisors do not know about their origins, so they are classified as quasi-stochastic. The AR/MR -app helps the system to increase the awareness of the origin of these disturbances. For example, the experiments done with assembly tasks showed that the number of quasi-stochastic disruptions that led to a process performing delay was reduced from 22 to 14 over 96 working hours (Refer to table 3, the column denoted by productivity indicator P2). 2. Eliminating the time needed for tedious reporting and getting more time for performing the primary job. The time saved by using the AR/MR-app instead of conventional reporting is measured between 40 to 70 %, depending on the application case. For example, reporting by typing/filling the QA file in an assembly operation cell usually requires 20 to 45 min for the operator per shift while the same job – generating the QA file within the AR/MR application interface takes 10 to 15 min (additionally it can involve voice comments, and photos are being made and attached faster). In terms of collecting human observations and inferences that have an impact on speeding up the AI toolset learning about the process, the main observed shortcoming was the possibility of wrong indoctrination of the AI toolset. Wrong inferences and observations that an operator can add to the system by associating them to the particular process context can mislead the system about the process, result in the wrong process anticipation and consequently in wrong reacting proposals. It is currently possible to overcome this shortcoming by adding (at least one) observation review step, which should be necessary in adding an observation/inference to the system. 4. CONCLUSION AND FUTURE WORK Through exploring the possibilities of applying the AR/MR app for collecting data and tacit knowledge about the (production) process context from a human-in-process, a few original conceptual interface solutions were developed for several different applications and different hardware platforms (phones, tablet and vizor – MS HoloLens) and different A Study of the Use of Mixed Reality for Capturing Human Observation and Inferences in Production ... 19 operative systems (Android, iOS, Windows). Concerning the human-to-system data flow, which was in the focus of this research, AR/MR applications were equipped with two conceptually different communication forms: prepared questionnaires and self-initiated observations, semantic interpretation and categorization of the process context features. The results of a series of experiments proved that the AR/MR application is a very powerful tool for capturing data, information, and knowledge about the ongoing process context that cannot be built just on data gathered by sensors. The main benefit that comes from using the specific interface features of AR/MR app developed for collecting human observations and semantic interpretations of process context features is fast and profound learning about the process, both by the information system and the human participants in the process. As an indirect benefit of this feature, a production environment becomes significantly reactive (agile) to prevent any decline in terms of operating productivity and quality or any accident in terms of safety. Having on mind the results of this research, we can firmly argue that AR/MR technology provides humans with quite a new role in a modern production environment, as some kind of an intelligence agent in the production process. In the near future, we should expect that AR/MR would figure as a key technology in facilitating the human-supervised learning of artificial intelligence systems. When it comes to future research, we would like to focus on using AR/MR for embedding semantics of production process context features. We believe that AR/MR can be used very efficiently for building or upgrading a semantic network by direct linking the digital content virtualized in VHF and "intelligent" semantic interpretations regarding that content made by a human participant in the production process. By mapping the semantic network with data and objects in VHF it would be possible to code experiential knowledge into formalisms, that is, into strongly structured knowledge about the process, which can then be searched and reused, for instance by using analogy-based reasoning [36, 37]. Acknowledgement: This research was financially supported by the G&W Electric, USA, Netico Solutions d.o.o, Serbia. and Ministry of Education, Science and Technological Development of the Republic of Serbia (Contract No. 451-03-68/2022-14/ 200109) REFERENCES 1. Pace, F.D., Manuri, F., Sanna, A., 2018, Augmented Reality in Industry 4.0., American Journal of Computer Science and Information Technology, 6(1), doi: 10.21767/2349-3917.100017 2. Sorko, S.R., Brunnhofer, M., 2019, Potential of Augmented Reality in Training, Procedia Manufacturing, 31, pp. 85-90. 3. Gorski, F., Grajewski, D., Bun, P., Zawadzki, P., 2021, Study of Interaction Methods in Virtual Electrician Training, IEEE Access, 9, pp. 118242-118252. 4. Mourtzis, D., Zogopoulos, V., Katagis, I., Lagios, P., 2018, Augmented Reality based Visualization of CAM Instructions towards Industry 4.0 paradigm: a CNC Bending Machine case study, Proceedings of the 28th CIRP Design Conference, 70, pp. 368-373. 5. Mourtzis, D., Siatras, V., Zogopoulos, V., 2020, Augmented Reality Visualization of Production Scheduling and Monitoring, Procedia CIRP, 88, pp. 151-156. 6. Mumtaz, S., Alsohaily, A., Pang, Z., Rayes, A., Tsang, K.F., Rodriguez, J., 2017, Massive Internet of Things for industrial applications: Addressing wireless IIoT connectivity challenges and ecosystem fragmentation, IEEE Industrial Electronics Magazine, 11(1), pp. 28-33. 20 M. STOJKOVIC, R. TURUDIJA, M. TRIFUNOVIC, ET AL. 7. Meng, Z., Wu, Z., Gray, J., 2017, A Collaboration-Oriented M2M Messaging Mechanism for the Collaborative Automation between Machines in Future Industrial Networks, Sensors, 17(11), 2694. 8. Berges, I., Ramírez-Durán, V.J., Illarramendi, I., 2021, A Semantic Approach for Big Data Exploration in Industry 4.0, Big Data Research, 25, 100222. 9. Ceravolo, P., Azzini, A., Angelini, M., Catarci, T., Cudré-Mauroux, P., Damiani, E., Mazak, A., Van Keulen, M., Jarrar, M., Santucci, G., Sattler, K.U., Scannapieco, M., Wimmer, M., Wrembel, R., Zaraket, F., 2018, Big Data Semantics, Journal on Data Semantics, 7, pp. 65-85. 10. Jain, V., Aggarwal, S., Mehta, S., Hebbalaguppe, R., 2019, Synthetic video generation for robust hand gesture recognition in augmented reality applications, ICCV 2019 Workshop: The 5th International Workshop on Observing And Understanding Hands In Action, arXiv preprint arXiv: 1911.01320. 11. Kolsch, M., Bane, R., Hollerer, T., Turk, M., 2006, Multimodal interaction with a wearable augmented reality system, IEEE Computer Graphics and Applications, 26(3), pp. 62-71. 12. Aouam, D., Benbelkacem, S., Zenati, N., Zakaria, S., Meftah, Z., 2018, Voice-based Augmented Reality Interactive System for Car’s Components Assembly, Proceedings of the 3rd International Conference on Pattern Analysis and Intelligent Systems (PAIS), Tebessa, Algeria, pp. 1-5. 13. Bertoli, P., Corcoglioniti, F., Francescomarino, C.D., Dragoni, M., Ghidini, C., Pistore, M., 2022, Semantic modeling and analysis of complex data-aware processes and their executions, Expert Systems with Applications, 198, 116702. 14. Paulus, A., Burgdorf, A., Pomp, A., Meisen, T., 2021, Recent Advances and Future Challenges of Semantic Modeling, Proceedings of the IEEE 15th International Conference on Semantic Computing (ICSC), Laguna Hills, CA, USA, pp. 70-75. 15. Bottani, E., Vignali, G., 2019, Augmented reality technology in the manufacturing industry: A review of the last decade, IISE Transaction, 51(3), pp. 284-310. 16. Marinkovic, D., Zehn, M., 2019, Survey of finite element method-based real-time simulations, Applied Sciences, 9(14), 2775. 17. Marinkovic, D., Zehn, M., Rama, G., 2018, Towards real-time simulation of deformable structures by means of co-rotational finite element formulation, Meccanica, 53(11-12), pp. 3123-3136. 18. Zhou, F., Duh, H.B.L., Billinghurst, M., 2008, Trends in augmented reality tracking, interaction and display: A review of ten years of ISMAR, Proceedings of the 7th IEEE/ACM International Symposium on Mixed and Augmented Reality, pp. 193-202. 19. Kim, K., Billinghurst, M., Bruder, G., Duh, H.B.L., Welch, G. F., 2018, Revisiting Trends in Augmented Reality Research: A Review of the 2nd Decade of ISMAR (2008–2017), IEEE Transactions on Visualization and Computer Graphics, 24(11), pp. 2947-2962. 20. Ceruti, A., Marzocca, P., Liverani, A., Bil, C., 2019, Maintenance in aeronautics in an Industry 4.0 context: The role of Augmented Reality and Additive Manufacturing, Journal of Computational Design and Engineering, 6(4), pp. 516-526. 21. Masoni, R., Ferrise, F., Bordegoni, M., Gattullo, M., Uva, A.E., Fiorentino, M., Carrabba, E., Di Donato, M., 2017, Supporting Remote Maintenance in Industry 4.0 through Augmented Reality, Procedia Manufacturing, 11, pp. 1296-1302. 22. Pierdicca, R., Frontoni, E., Pollini, R., Trani, M., Verdini, L., 2017, The Use of Augmented Reality Glasses for the Application in Industry 4.0, Proceedings of the International Conference on Augmented Reality, Virtual Reality and Computer Graphics, 10324, pp. 389-401. 23. Cukovic, S., Devedzic, G., Ghionea, I., Fiorentino, M., Subburaj, K., 2016, Engineering design education for industry 4.0: Implementation of Augmented Reality concept in teaching CAD courses, Proceedings of the Augmented Reality for Technical Entrepreneurs International Conference – [ARTE2016], Bucharest, Romania, pp. 11-16. 24. Maly, I., Sedlacek, D., Leitao, P., 2016, Augmented reality experiments with industrial robot in industry 4.0 environment, Proceedings of the IEEE 14th International Conference on Industrial Informatics (INDIN), Poitiers, France, pp. 176-181. 25. Doshi, A., Smith, R.T., Thomas, B.H., Bouras, C., 2017, Use of projector based augmented reality to improve manual spot-welding precision and accuracy for automotive manufacturing, The International Journal of Advanced Manufacturing Technology, 89(5-8), pp. 1279-1293. 26. Flotynski, J., Sobocinski, P., Strykowski, S., Strugala, D., Bun, P., Gorski, F., Walezak, K., 2021, Semantic Representation of Domain Knowledge for Professional VR Training, Proceedings of the 24th International Conference on Business Information Systems, 1, pp. 139-150. 27. Gorski, F., 2017, Building Virtual Reality Applications for Engineering with Knowledge-based Approach, Management and Production Engineering Review, 8(4), pp. 64-73. A Study of the Use of Mixed Reality for Capturing Human Observation and Inferences in Production ... 21 28. Fernández del Amo, I., Erkoyuncu, J.A., Roy, R., Wilding, S., 2018, Augmented Reality in Maintenance: An information-centred design framework, Procedia Manufacturing, 19, pp. 148-155. 29. Mourtzis, D., Vlachou, E., Zogopoulos, V., Fotini, X., 2017, Integrated Production and Maintenance Scheduling Through Machine Monitoring and Augmented Reality: An Industry 4.0 Approach, Proceedings of the IFIP International Conference on Advances in Production Management Systems, Hamburg, Germany, 513, pp. 354-362. 30. Bruno, F., Barbieri, L., Marino, E., Muzzupappa, M., D’Oriano, L., Colacino, B., 2019, An augmented reality tool to detect and annotate design variations in an Industry 4.0 approach, The International Journal of Advanced Manufacturing Technology, 105(1-4), pp. 875-887. 31. Zhao, G., Liu, S., Zhu, W.J., Qi, Y.H., 2021, A Lightweight Mobile Outdoor Augmented Reality Method Using Deep Learning and Knowledge Modeling for Scene Perception to Improve Learning Experience, International Journal of Human–Computer Interaction, 37(9), pp. 884-901. 32. Johnson, T.L., Fletcher, S.R., Baker, W., Charles, R.L., 2019, How and why we need to capture tacit knowledge in manufacturing: Case studies of visual inspection, Applied Ergonomics, 74, pp. 1-9. 33. Fominykh, M., Wild, F., Smith, C., Alvarez, V., Morozov, M., 2015, An Overview of Capturing Live Experience with Virtual and Augmented Reality, Proceedings of the 11th International Conference on Intelligent Environments, 19, pp. 298-305. 34. Dudek A., Patalas-Maliszewska, J., 2016, A Model of a Tacit Knowledge Transformation for the Service Department in a Manufacturing Company: A Case Study, Foundations of Management, 8(1), pp. 175-188 35. Hashimoto, J., Park, H.J., 2020, Capturing tacit knowledge with smart device augmented reality (SDAR), Proceedings of the ECAADE 2020, Berlin, Germany, pp. 165-172. 36. Stojkovic, M., Trifunovic, M., Misic, D., Manic, M., 2015, Towards Analogy-Based Reasoning in Semantic Network, Computer Science and Information Systems, 12(3), pp. 979-1008. 37. Trifunovic, M., Stojkovic, M., Misic, D., Trajanovic, M., Manic, M., 2015, Recognizing Topological Analogy in Semantic Network, International Journal of Artificial Intelligence Tools, 24(3), pp. 1550006-1 - 1550006-25.