Frontiers in Computing and Intelligent Systems ISSN: 2832-6024 | Vol. 8, No. 2, 2024 13 Semi Solid MR Coal Mine Tunneling Machine Operation Training System Xu Yang 1, Mingquan Zhou 2, Chao Chen 2, *, Weiwei Zhong 2, Fen Yue 2 1 Mining Safety College, North China Institute of Science and Technology, Langfang, Hebei, China 2 School of Computer Science, North China Institute of Science and Technology, Langfang, Hebei, China * Corresponding Author: Chao Chen Abstract: At present, the coal mining industry is increasingly focusing on improving safety and operational efficiency, which has led to an increasing demand for training of coal mine excavation equipment operators, and the corresponding training costs have also increased. To alleviate these pressures, researching a training program for cantilever tunneling machines based on digital twin technology has become an innovative initiative. This article constructs a virtual environment for underground tunnels by combining mixed reality technology with physical entities. The operator commands the activities of the virtual tunneling machine through actual control of the equipment, and the system evaluates the operation process by tracking and analyzing the operator's posture, ultimately generating detailed evaluation scores. This innovative training model not only greatly enriches the user training experience, but also provides comprehensive data support, which helps to improve the skills and work efficiency of operators, and meets the new requirements and development trends of coal mine safety production. Keywords: Mixed Reality Technology; Tunnel Boring Machine Training; Attitude Recognition. 1. Introduction The core objective of the "Regulations on Coal mine Safety Production", scheduled to be implemented on May 1, 2024, is to improve safety measures in the coal mine production process, and is committed to preventing and reducing the occurrence of safety accidents, so as to ensure that the safety of people's lives and property is effectively guaranteed. The regulatory responsibilities at all levels from the local level to the leadership have been clarified, and the regulatory authorities have been standardized, requiring them to regularly inspect coal mines and take appropriate measures to promote the development of safety work in the coal mine industry to a higher level[1]. In order to actively respond to national policies, improve coal mine safety awareness, promote coal mine production efficiency, and realize efficient boring machine operation, it is particularly important to improve the technology and equipment of coal mine roadway excavation. At present, the coal mine tunnel driving technology includes the cantilever tunnel driving technology, the continuous mining machine driving technology and the anchor driving technology. The development of these technologies has a direct impact on the stability, safety and production efficiency of coal mining[2]. It is of great significance for improving the safety and efficiency of coal mine production to reduce the fault incidence of cantilever boring machine, master its operating state comprehensively and study the current situation and development trend of roadway driving technology and equipment[3]. Through in-depth research and application of advanced digital twin technology, comprehensive monitoring and real- time control of the running state of the roadheader can be realized, and the operation efficiency and safety of the roadheader can be further improved [4, 5]. For this reason, many scholars began to focus on the application of virtual reality technology in the field of engineering, and the combination of tunnel boring machine to carry out research.For example, Yu Yang has studied the remote control system of roadheader based on virtual reality technology, with the purpose of improving the efficiency and safety of roadheader operation through high-precision simulation model and remote control technology [6]. Zhang Qianwen proposed a remote virtual automatic cutting control system for roadheader, which uses virtual reality technology to realize unmanned control of the roadheader, and significantly improves the cutting accuracy and operation safety [7]. Wu Chun developed the tunnel boring machine virtual simulation system and applied it to practical teaching, which effectively improved the professional skills and practical ability of underground engineering students by simulating the real construction environment and operation process. The use of simulation technology not only helps to improve the level of coal mine safety, but also provides a new path and opportunity for the intelligent and digital development of the coal mine industry[8]. Based on the digital twin technology, this paper studies the training scheme of the cantilever TBM which combines mixed reality and physical entity. By using 3DMax and Unity software, the cantilever boring machine was modeled and the corresponding underground roadway environment was built. Operators use devices such as matrix keyboards and joysticks to control the movement of a virtual boring machine in a mixed reality scenario. At the same time, the system can monitor whether the operator's attitude meets the standard and record the operation score. After the training, users can view the training data and historical records on the Web interface, which improves the training experience of users and provides comprehensive data support for the training process. This research result not only promotes the innovation and upgrading of TBM technology, but also provides new ideas for education and training in the digital era, and promotes the improvement of the quality and efficiency of talent training. 14 2. System Scheme 2.1. Overall Design The overall design of the semi-solid MR Coal mining machine operation training system based on mixed reality technology is shown in Figure 1. The user first wears the MR Device (Hololens2.0), and then connects and activates the required device as required. By controlling the relevant equipment, such as pressing the matrix keyboard, flight throttle arc base and other hardware, the data is transmitted to the server, and the server sends the data to the MR Device and modifies the corresponding scene information. During the operation, the camera monitors the operator's body posture in real time, records the score according to the qualification of the operating posture and stores it in the database. After the training, users can access the database through the Web interface to view the relevant data of this training for training details. This system makes full use of advanced technology, enhances the user's training experience, and provides comprehensive data support for the training process. Figure 1. Overall design 2.2. System Framework Design As shown in Figure 2, the highly accurate mapping of the TBM is realized through equiscale modeling of TBM and mine roadway in the virtual scene. In the virtual reality scene, the system integrates the scene object and the geometric object of three-dimensional space to achieve seamless interactive experience between the user and the virtual environment. The user simulates the various functions of the TBM model by operating the matrix keyboard, and at the same time controls the movement direction of the boom and the advance and retreat of the TBM by the flight throttle arc seat. When the operator controls the physical device, the system presents the corresponding effect in the virtual scene according to the incoming data of the device, and evaluates and gives a score according to the operation behavior of the virtual space. Figure 2. System framework In addition, the system uses the camera connected to the PC to recognize the body posture of the operator in the process of using the device, and scores according to the scoring criteria. These scores are stored in the cloud server according to the corresponding location. The data received by the server is stored in the MySQL database. In order to realize the real- time viewing and analysis of the training data, the client establishes a connection with the Web server through Http communication, calls the incoming data in the MySQL database for statistics, and records the statistical results in the database for subsequent viewing and analysis. 15 3. Physical Equipment 3.1. Flight Throttle Arc Seat Figure 3. Flight throttle arc seat The flight throttle arc seat is a hand-held controller, which is mainly used for the control of the TBM in the virtual scene, including the fuselage, TBM cutting and other parts, so as to obtain the real operation feeling and achieve more efficient training effect. In addition, the operator grips and moves the device to stimulate muscle and nerve responses in the human body, simulating a real operating experience. 3.2. Homemade Matrix Keyboard Figure 4. Homemade matrix keyboard Matrix keyboard is a common input device in embedded system, which realizes user input data through matrix arrangement and row circuit connection of keys. Compared with the traditional switch keys that directly control IO ports, the matrix keyboard has flexibility and scalability, can realize key coding through the chip, and effectively use the MCU resources, and is usually used in the control system. In this system, the matrix keyboard is used to control the operation of the boring machine button, such as control spray, crab claw, cutting head start, power supply, emergency stop, etc. The use of matrix keyboard as input device significantly improves the ease of user operation. 4. The Realization Process of Attitude Recognition LSTM is a common deep learning algorithm that can be used to model and predict sequence data. LSTM long and short term memory neural network is adopted in deep model learning. It is a variant network structure of recurrent neural network RNN architecture, and has three activation functions, namely "forgetting gate", "input gate" and "output gate". The three structures can carry out cell editing on the cell state that runs through it, so that the final training target of LSTM retains useful information. It can solve the problem of disappearing gradient better. The neuronal structure of LSTM is shown in Figure 5: Figure 5. Neuron structure of LSTM Opencv and Mediapipe were used to identify and analyze the actual actions of the students, and compared with the results predicted by the model to obtain the corresponding scores and evaluation results. According to the feedback and evaluation results of the students, the model is optimized and improved to improve the accuracy and stability of the model. The specific steps of using deep learning to realize posture recognition are shown in Figure 6: collect the postures and movement data of students in the operation of the boring machine, including key point coordinates, movement types and other information. The collected data is preprocessed, including data cleaning, standardization and other operations, in order to facilitate subsequent modeling and prediction. LSTM algorithm is used to model and train the pre-processed data to realize the prediction and analysis of the trainees' actions. Mediapipe can convert the identified human node into 30 frames of video human movement data, obtain the human movement data set, and predict whether to operate in time through deep learning algorithms. 5. Virtual Reality Scene Construction The system uses digital twin technology to reproduce the diverse scenarios of the boring machine operation, covering various geological conditions and changing operating environments. Operators can conduct simulation drills of TBM control in the simulation environment, including key operation processes such as tunneling, steering, and ore collection, effectively cultivating and improving the proficiency of actual TBM operation skills. The Internet of Things technology is used to integrate the control of the real boring machine with the dynamic synchronization of the virtual environment, which greatly enhances the authenticity of the operating experience. Operators can use physical equipment to control, to achieve accurate operation of the boring machine in virtual space. In summary, the "Semi-physical MR Mine TBM Operation Training System" provides TBM operators with an efficient, safe and comprehensive operational training platform with the help of digital twin technology. Through simulation training to improve the operational skills of operators in actual operations and the ability to deal with complex situations. 16 Figure 6. Implementation steps of attitude recognition Figure 7. Virtual scenario 6. Assessment Module In order to optimize the training results of the "semi- physical MR Coal Mining Machine Operation Training system", the system monitors and records the operator's behavior in the virtual environment in real time, and compares the collected data with the standard operation process, so as to accurately quantify the operator's technical proficiency and other related performance indicators. By invoking pose recognition to analyze the body posture and movement of the operator, the core evaluation factors such as operation accuracy, error recognition and reaction time are effectively evaluated by performing a fine comparison against the preset standards. In addition, each link of the operation process is given appropriate weight, combined with the score evaluation method, to ensure the comprehensiveness and accuracy of the evaluation results. In the training process, the trainees will face a variety of situations, including the control of the TBM, the adjustment of the TBM cantilever swing, the direction of the fuselage and the normative inspection of the body posture. When students perform well in the training and successfully complete the task, they will receive positive points to reward their skills improvement. Accordingly, if the operation is wrong, the system will record in the form of a negative score, and provide feedback and improvement suggestions. At the end of the training, the student's overall performance is calculated by the system into a total score and stored in the database. Through the Web interface, participants can easily view and analyze the specific evaluation of each training, including operational strengths and areas for improvement. Ensure the transparency of the assessment and the reliability of the results. The whole training and assessment process is designed so that the system is not only a simple training tool, but also a platform for continuous iteration and optimization. Through continuous absorption of trainees' operational data and analysis of feedback, the teaching content and methods are adjusted, and the training is committed to maximizing the effect and efficiency. The evaluation mechanism not only provides solid data support for the trainers, but also effectively promotes the continuous optimization of the training content and humanized teaching methods, thus improving the efficiency of the training operation of the boring machine. 7. Conclusion The semi-solid MR Coal mine boring machine operation training system shows significant advantages in many aspects. First, the system uses virtual simulation technology to save costs and improve skills. Evaluate student operations through real-time monitoring and scoring mechanisms, generate an overall score at the end of the training and view it via a Web interface, providing personalized learning recommendations. The data analysis capability of the system provides a comprehensive and objective evaluation of the training effect, thus promoting the optimization of operational skills and training content. Secondly, its excellent security design greatly reduces the risks faced by students during the learning process. Students are free to make mistakes and experiment in a safe simulation environment, and through first-hand experience of the serious consequences that can be caused by wrong operations, they 17 are deeply aware of the risk of control errors without fear of actual damage to people, the environment or equipment. Most importantly, it achieves a high degree of integration between the system and the operator, thus ensuring the safety of the operation while improving the efficiency of training. Acknowledgments Fund project: supported by “the Fundamental Research Funds for the Central Universities” (3142023069). References [1] Promulgation of the Regulations on Coal Mine Safety Production [J]. China Coal Industry,2024(03):34. [2] Zhao Zhongqiang. Coal mine roadway construction present situation and development trend of technology and equipment [J]. Energy and energy conservation, 2024, (01) : 200-203. The DOI: 10.16643 / j. carol carroll nki. 14-1360 / td. 2024.01.012. [3] HOU Lei. Design and Application of Intelligent Monitoring System of Coal Mine cantilever boring Machine [J]. Jiangxi Coal Science and Technology,2024, (01):181-183. [4] Cao Yu. The application of digital twin control system in coal mine machine [J]. Journal of coal mine machinery, 2024, (01) : 179-181. The DOI: 10.13436 / j. m KJX. 202401055. [5] Zhang Xuhui, Yang Wenjuan, Xue Xusheng, et al. Remote intelligent coal mine tunneling face challenge and research progress [J]. Journal of coal, 2022,47 (01) : 579-597. The DOI: 10.13225 / j. carol carroll nki JCCS. Yg21.1941. [6] Yu Yang, Wu Lei, Liang Huaqiang. Virtual simulation system of tunnel boring machine design and training mode study [J]. Journal of experimental technology and management, 2018, 35 (11): 141-143 + 148. DOI: 10.16791 / j.carol carroll nki SJG. 2018.11.032. [7] Zhang Qianwen. Remote virtual machine automatic cutting control system applied research [J]. Mechanical management development, 2023, 38 (01) : 206-207 + 210. DOI: 10.16525 / j. carol carroll nki cn14-1134 / th. 2023.01.082. [8] WU Chun. Research on Remote Control System of boring machine based on Virtual Reality Technology [J]. Mechanical management development, 2019 (11): 229-230 + 235. DOI: 10.16525 / j. carol carroll nki cn14-1134 / th. 2019.11.096.