DOI: 10.3303/CET2290125 Paper Received: 3 January 2022; Revised: 20 March 2022; Accepted: 29 April 2022 Please cite this article as: Yang S., Demichela M., Geng J., Tao H., 2022, Analysis of human and organization factors related accident reports based on natural language processing, Chemical Engineering Transactions, 90, 745-750 DOI:10.3303/CET2290125 CHEMICAL ENGINEERING TRANSACTIONS VOL. 90, 2022 A publication of The Italian Association of Chemical Engineering Online at www.cetjournal.it Guest Editors: Aleš Bernatík, Bruno Fabiano Copyright © 2022, AIDIC Servizi S.r.l. ISBN 978-88-95608-88-4; ISSN 2283-9216 Analysis of Human and Organizational Factors Related Accident Reports Based on Natural Language Processing Shuo Yanga*, Micaela Demichelaa, Jie Gengb, Hongjie Taob a Department of Science Application and Technology, Politecnico di Torino, Torino, Italy b Zhejiang University of Finance and Economics, Hangzhou, China shuo.yang@polito.it Lacking data has always been a challenging problem for risk analysts on human and organizational factors (HOFs) since the theme comes to birth. Accident reports are an essential source of HOFs information, but they are often in the form of unstructured text, making it challenging to apply the number statistic method directly. The traditional manual coding of accident records could introduce uncertainties and inefficiencies, especially when a large number of records is available. Thanks to the development of the natural language processing (NLP) technique, some analysts have attempted to mine the text of accident reports (Single et al., 2020). A similar approach was adopted to highlight HOFs contributing to the accidents. The NLP and HOFs categories have then been introduced to obtain the critical structure of HOFs related accidents. Furthermore, the approach of text similarities calculation is applied to support the relationship analysis of performance influencing factors (PIF) based on the mining of data of the EU Major Accident Reporting System’s (eMARS). In general terms, a framework is proposed to efficiently exploit the information contained in accident records to assess the HOFs elements better to be included in process risk assessment. 1. Introduction HOFs are essential contributors and often the root cause of some accidents. Since the 1980s, nearly 50 Human Reliability Analysis (HRA) methods have been developed (Xing et al., 2021). Among them, Many HRA methods identified HOFs, such as Technique for Human Error Rate Prediction (THERP) (Swain & Guttmann, 1983), Human Cognitive Reliability Correlation (HCR) (Hannaman & Spurgin,1984), Success Likelihood Index Methodology (SLIM) (Embrey et al., 1984), Cognitive Reliability and Error Analysis Method (CREAM) (Hollnagel, 1998), the Standardized Plant Analysis Risk Human Reliability Analysis (SPAR-H) (Gertman et al., 2005.), Human Error Assessment and Reduction Technique (HEART) (Williams, 1988), although using different descriptions, like Performance Influencing Factors (PIFs), Performance Shaping Factors (PSFs), Common Performance Conditions (CPCs), and so on. But the long-standing difficulty is a lack of data to validate those PIFs, PSFs, or CPCs. Learning from the occurred events may be a possible way. Machine learning methods have been already adopted to analyze accident databases (Comberti et al., 2015). Comberti et al., 2018 grouped and visualised data in a readable way. Baldissone et al.(2019) used accident data to develop an Accident Precursors Management System, and Comberti et al. (2015) proposed clustering methods. It was recognised that, although simulation technology can generate a large amount of data now, accident reports that record real accident scenarios are still essential sources. However, with unstructured texts in the accident reports, the required information is quite challenging to be obtained. Traditional manual coding of accident records could bring uncertainties and inefficiencies, especially when many records are available. NLP technique provides an attempt to mine the text of accident reports. Kanza Noor Syeda et al. (2011) applied stemming, lemmatization and Part of Speech (POS) tagging to exploit the railway incident reports. A custom tag-based pattern recognition technique extracted general risk information from eMARS (Single et al., 2020). In this work, the HOFs influences on the accidents regarding the eMARS database are highlighted. The research questions of this study are: 1) What is the core structure of a HOFs accident scenario? 745 2) How to build a model that can intelligently extract the core information of an accident scenario from accident reports? 3) How is this model performed when applying to the HOFs related accident reports? 2. “4W” information structure and the extract method This research approach analyses the raw text of the HOFs related accidents using NLP techniques. First, the raw accidents text from eMARS is filtered with “cause of the accident”. Only the HOFs related cases have been collected. Meanwhile, this research has not considered the malicious and no/too simple cause description cases. The basic statistic description of the database is shown in Table 1. Secondly, the “4W” information structure framework has been built to support the analysis, as shown in Figure 1. The core HOFs relevant information, including the working periods when the accident occurred, the equipment (location), the actor, and the HOFs, have been involved. Table 1: Basic statistic description of the database Total Human error cases Organizational factor cases Redundant cases No/too simple cause description cases Other cases Final HOFs cases 1128 209 464 142 42 639 489 Figure 1: “4 W” information structure of accident scenario Then, the keywords have been extracted using the SpaCy package (Honnibal & Montani, 2017) under the Python programming platform. For the pre-processing part, the raw texts are tokenized in spaCy’s built-in pipelines become ‘docs’, Tokenization is the process of breaking text into sentences and words, e.g., the sentence ”A leakage in a pipeline caused the release of chlorine.” The ‘doc’ result, as Table 2 shows, then the tokens in ‘doc’ can be selected, transformed, and analyzed. For the keywords extraction part, the Named-entity Recognition (NER) process is employed, including fine-tuning pre-trained model, spaCy has many built-in models, a few of the top layers of a frozen model base, and jointly train both the newly-added classifier layers and the last layers of the base model. Further, those keywords supported the analysis. Table 2: SpaCy Tokenized doc TEXT LEMMA POS TAG DEP SHAPE ALPHA STOP A a DET DT det X TRUE TRUE leakage leakage NOUN NN nsubj xxxx TRUE FALSE in in ADP IN prep xx TRUE TRUE a a DET DT det x TRUE TRUE pipeline pipeline NOUN NN pobj xxxx TRUE FALSE caused cause VERB VBD ROOT xxxx TRUE FALSE the the DET DT det xxx TRUE TRUE release release NOUN NN dobj xxxx TRUE FALSE of of ADP IN prep xx TRUE TRUE chlorine chlorine NOUN NN pobj xxxx TRUE FALSE . . PUNCT . punct . FALSE FALSE 746 2.1 “When” information structure and extract method The working periods are divided into two categories “operational periods” and “maintenance periods”, with maintenance periods including shutting downtime, cleaning/purging time, repair time, solder/welding time, and isolation time”. To identify all the maintenance periods, like “during, before, and after” maintenance periods, the inclusive match patterns are built, using token patterns to be found in the accident description with rules as: {'LEMMA':{'IN':['maintenance','clean','repair','shut','weld','solder','hot']}},{'LOWER':{'IN':['routine','out']},'OP':'?'}, {'LOWER':{'IN':['work','operation']},'OP':'?'} 2.2 “Where” information structure and extract method The equipment (location) terms are not limited to a certain word list, so it is not an excellent solution to use a rule-based method to extract the information. Using the prodigy package, this research employs the custom model to mix the rule-based and statistic model. First, use “pipe, tank, pump” as seeds to generate a worklist pattern, then use the “Cause of accident” text of the eMARs database to teach and correct the model. Finally, train the NER model combined with the pre-trained model in SpaCy, using training data of 514 entities and evaluation data of 236 entities (30% split). The precision, recall, and F score are employed to evaluate our NER model; the performance evaluation of the model shows good results, with the entities precision 93.00, recall 93.94, and f score 93.47. 2.3 “Who” information structure and extract method The actors are divided into employed operators and contract operators, extracted directly from the dataset. 2.4 “Why” information structure and extract method This section is the one that is related to the PIFs; in particular, this research selects the set of PIFs developed by Katrina M. Groth and Ali Mosleh (Groth & Mosleh,2012), the adapted PIFs are shown in Table 3. Table 3: Performance Influencing factors adapted Organization-based Team-based Person-based Situation-based Machine- based Training Communication Attention External environment HSI Corrective action Direct supervision Physical & psychological abilities Task load Safety culture Team coordination Knowledge/experience Time load Staffing Team cohesion Skills Task complexity Scheduling Role awareness Bias Stress procedures Familiarity with situation Perceived situation Workplace adequacy Morale/motivation/attitude Perceived decision procedures tools information Based on this set of PIFs, extend the management activities to outsourcing management, permit management manage of change, process analysis, and risk analysis. A set of terms category is developed as shown in Table 4-8. These show the Tag of the PIFs to be used to recognize them in the accident description, as a single word( single pattern tag), double words(double pattern tag), and triple words(triple pattern tag). They were defined for the different categories of PIFs. Table 4: Organization-based factors terms Single Pattern Tag Double Pattern Tag Triple Pattern Tag text tag text Tag text tag training TRA corrective action COR ACT manage of change MANAG OF CHAN culture CUL workplace adequacy WOP ADEQ staffing STA outsourcing management OUT MANAG scheduling SCHE permit management PERM MANAG procedures PROD process analysis PROS ANS tools TOOL risk analysis RIS ANS information INFO design DESN 747 Table 5: Team-based factors terms Table 7: Situation/Stress-based factors terms Single Pattern Tag Double Pattern Tag text tag text tag environment ENV condition events CON EVE stress STRE task load TS LOD time load TI LOD task complexity TS COMP perceived situation PERC SITU perceived decision PERC DECI 3. Result The application of the proposed method leads to an automatically populated core accident database information. In the following, the extracted keywords about ”4W” are analyzed. 3.1 Extracted “When” information Nearly three times HOFs related cases occurred during maintenance compared to other cases. About 20% of the HOFs case happened during the maintenance periods, while only 7% of other cases happened during the maintenance periods, as shown in Figure 3. This result validates human errors often occur during maintenance operations; moreover, many accidents happen because of inadequate procedures and instructions about the maintenance work. It is reasonable that the decision-maker should pay more attention to the maintenance working periods when enforcing risk-reducing policies and allocating safety resources. Figure 3: The accident involved working periods 3.2 Extracted “Where” information Single et al. (2020) conducted a rule-based NLP match analysis of the main categories of the eMARS database: from 889 cases, 77 locations were identified, thus only for the 8.7% of cases. 1128 cases are analyzed in this research; 1728 pieces of equipment are identified since, in some cases, more than one piece of equipment was involved in the accident. In Figure 4, the results of identified equipment(locations) frequencies are compared. Other cases HOFs cases maintenance periods 26 96 operational periods 613 393 0 200 400 600 800 Single Pattern Tag Double Pattern Tag text tag text tag communication COM role awareness ROL AWAR coordination COO cohesion COH supervision SUP Table 6: Person-based factors terms Single Pattern Tag Double Pattern Tag text tag text tag attention ATTE sensory limits SENS LIM alertness ALE fatigue FATI impairment IMPA knowledge KNOW experience EXPE skills SKIL bias BIA morale MOR motivation MOTI attitude ATTI familiarity FAM Table 8: Machine based factors terms Triple Pattern Tag text tag human machine interface H M I human system interface H S I 748 Figure 4: Identified equipment(locations) frequencies comparison More than half of the HOFs cases happened around a reactor, compared to other cases. More than one-third of the HOFs cases happened around a valve compared to other cases. Followed by tank and pipe locations, as shown in Figure 5. Figure 5: Identified equipment(locations) frequencies 3.3 Extracted “Who” information 44 cases involved contract operators. In 37 cases, contract operators carried out maintenance work, including hot work, cleaning, repair, and replacement work. In 5 cases they carried out transport services. 3.4 Extracted “Why” information About the PIFs information part, after extracting the HOFs information, 48 cases have three factors identified, 91 cases have two factors identified, 146 cases have one factor identified, and 167 cases have no factors identified. More than 2/3 of the identified factors are organizational factors. The distribution of each PIFs frequency on the total amount of accident-related HOFs is shown in Figure 6. The frequency weight of organizational-based factors of “procedures” is 24%, “maintenance” is 22%, “design” is 18%, and “training” is 10%. Figure 6: Identified HOFs distribution 749 4. Discussion and Conclusions A new framework has been proposed for analyzing the information of HOFs related accident reports. The main challenge in analyzing the HOFs related accident scenarios is automatically extracting the core information. The NLP method has been used in this study. The NER model has been trained to identify different equipment (locations) involved. This process needs human effort to do annotations work. Another challenge of this framework is that some HOFs relevant cases are not described using the keywords scheme we employed. It will be hard to extract that part of the information. Therefore, this framework is more suitable to analyze the organizational factors. Future work can focus on the automatic approach to extract operator-based factors information. Future studies will be devoted to the analysis of specific accident domains. Acknowledgements The work is supported by China Scholarship Council. References Baldissone, G., Comberti, L., Bosca, S., Murè, S.; The analysis and management of unsafe acts and unsafe conditions. Data collection and analysis. (2019) Safety Science, 119, pp. 240-251. Comberti, L., Demichela, M., Baldissone, G.; A combined approach for the analysis of large occupational accident databases to support accident-prevention decision making (2018) Safety Science, 106, pp. 191- 202. Comberti, L., Baldissone, G., Demichela, M.; Workplace accidents analysis with a coupled clustering methods: S.O.M. and K-means algorithms (2015) Chemical Engineering Transactions, 43, pp. 1261-1266. Comberti, L., Baldissone, G., Bosca, S., Demichela, M., Murè, S., Petruni, A., Djapan, M., Cencetti, S.; Comparison of two methodologies for occupational accidents pre-cursors data collection (2015) Safety and Reliability of Complex Engineered Systems - Proceedings of the 25th European Safety and Reliability Conference, ESREL 2015, pp. 3237-3244. EMARS (Major Accident Reporting System), Major Accident Hazards Bureau, European Commission, https://emars.jrc.ec.europa.eu/?id=4, accessed 14.10.2021. Hannaman, G. W., & Spurgin, A. J. (1984). Systematic Human Action Reliability Procedure (SHARP) Interim report (p. 141). Hollnagel, E. (1998). Cognitive reliability and error analysis method: CREAM (1st ed). Elsevier. Honnibal, M., & Montani, I. (2017). spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. Jing Xing, James Chang, Jonathan DeJesus. (2021). NUREG-2198, "The General Methodology of an Integrated Human Event Analysis System (IDHEAS-G)" Single, J. I., Schmidt, J., Denecke, J. (2020). Knowledge acquisition from chemical accident databases using an ontology-based method and natural language processing. Safety Science, 129, 104747. Syeda, K.N., Shirazi, S.N., Naqvi, S., & Parkinson, H.J. (2017). Exploiting Natural Language Processing for Analysing Railway Incident Reports. Katrina M. Groth, Ali Mosleh,A data-informed PIF hierarchy for model-based Human Reliability Analysis,Reliability Engineering & System Safety,Volume 108,2012,Pages 154-174,ISSN 0951- 8320,https://doi.org/10.1016/j.ress.2012.08.006. Williams, J. C. (1988). A data-based method for assessing and reducing human error to improve operational performance. Conference Record for 1988 IEEE Fourth Conference on Human Factors and Power Plants, 436–450. https://doi.org/10.1109/HFPP.1988.27540 Swain, A. D., & Guttmann, H. E. (1983). Handbook of human-reliability analysis with emphasis on nuclear power plant applications. Final report (NUREG/CR-1278, SAND-80-0200, 5752058; p. NUREG/CR-1278, SAND- 80-0200, 5752058). https://doi.org/10.2172/5752058 750 lp-2022-abstract-228.pdf Analysis of Human and Organizational Factors Related Accident Reports Based on Natural Language Processing