CET vol 99 DOI: 10.3303/CET2399074 Paper Received: 17 December 2022; Revised: 27 March 2023; Accepted: 25 April 2023 Please cite this article as: Pera F., Murino T., Madonna M., Di Nardo M., Bizzarro R., 2023, A Conceptual Framework to Defining Leading Indicators to Measure Safety Management System Performance, Chemical Engineering Transactions, 99, 439-444 DOI:10.3303/CET2399074 CHEMICAL ENGINEERING TRANSACTIONS VOL. 99, 2023 A publication of The Italian Association of Chemical Engineering Online at www.cetjournal.it Guest Editors: Sauro Pierucci, Flavio Manenti Copyright © 2023, AIDIC Servizi S.r.l. ISBN 978-88-95608-98-3; ISSN 2283-9216 A Conceptual Framework to Defining Leading Indicators to Measure Safety Management System Performance Fabio Peraa, Teresa Murinoc, Marianna Madonnab*, Mario Di Nardoc, Roberta Bizzarroc aINAIL, Dipartimento Innovazione Tecnologica e Sicurezza degli Impianti, Prodotti e Insediamenti Antropici, Via Roberto Ferruzzi, 38, 00143 Roma, Italy. bINAIL, Unità Operativa Territoriale di Certificazione Verifica e Ricerca di Napoli, Via Nuova Poggioreale, 80143 Napoli, Italy. c Università degli Studi di Napoli Federico II - Dipartimento di Ingegneria dei Materiali e della Produzione, Piazzale Tecchio, 80, 80125, Napoli, Italy. m.madonna@inail.it The measurement of health and safety management performance in organisations is a key step, as well as in any management process, as it provides information to identify the critical areas that require actions to achieve continuous improvement. Within the safety management system (SMS), the performance measurement is therefore associated with the check step in the plan-do-check-act (PDCA) cycle. However, it is also in the planning phase that we need to think about what we will have to measure in order to be able to make decisions in key areas. Unfortunately, based on existing literature and standardisation, there is no unique checklist for measuring the performance of safety management systems. Although theorists and researchers have not yet found standard indicators for measuring the SMS performance, they seem to agree in classifying performance indicators into lagging and leading indicators. While lagging indicators measure the frequency of injuries, illnesses and fatalities that occurred in the past, leading indicators provide information about safety and health activities by enabling organisations to implement preventive measures. In this paper, the authors have proposed a conceptual framework for defining key indicators to measure safety performance. The objective is to design a structured path to identify indicators, among those existing in the literature and those widely used by experts, for each element of the SMS. The steps for using these indicators are also traced. 1. Introduction The Safety Management System (SMS) is an integral part of the wider management system of an organisation and has been implemented since the mid-1980s according to international standards such as the ILO-OSH guidelines (2001), OHSAS 18001 (2007) and the recent ISO 45001 (2018). Safety management models, even if they are non-mandatory, aim at the reduction of accidents at work. However, although occupational health and safety management systems (OHS MSs) have been widely adopted over the years, the number of occupational accidents and diseases is still high. The performance of any management system is measured using indicators. Although the standards on the implementation of management systems refer to the use of indicators to measure performance objectives, there is no exhaustive and standardized list of these indicators to date. For this reason, in recent years, researchers and management systems experts have conducted several studies on how to structure an effective system performance measurement. Even if a framework for measuring the performance of safety management systems has not yet been found, the theorists and the researchers seem to agree on classifying performance indicators into leading and lagging indicators. While lagging indicators measure the frequency of injuries, illnesses and fatalities that occurred in the past, leading indicators provide information about safety and health activities by enabling organizations to implement preventive measures. According to many authors, as the lagging indicators are based on accidental events that have already occurred, they do not make preventive interventions possible, but they allow to intervene only with corrective actions. 439 Furthermore, it is widely acknowledged that, due to their retrospective nature, lagging indicators are insufficient to indicate the current level of performance and, as a result, cannot predict or enhance future performance. In addition, if it is also considered that the high impact accidents are rare and difficult to predict, the number of accidents occurred cannot be taken as unique indicators of the effectiveness of a safety program of an organization or its overall level of safety. (Carra et al, 2022). Especially for those organizations approaching “zero harm”, it becomes difficult to ensure safety by monitoring lagging indicators as no precursors of circumstances are detected (ICMM, 2012). On the other hand, the leading measure the direct and indirect precursors to harm, and give early warning before an undesired outcome occurs. Leading indicators allow timely and effective intervention in the event of possible non-compliance with the management system even before the negative consequences of such situations, such as accidents at work, occur (Podgorski, 2015). With this in mind, leading indicators appear more useful for driving change by providing early warning signs of potential failures that might turn into accidents and injuries. 2. Materials and method In the literature on the safety management performance, many authors have developed indicators to measure performance and its impact on incident rate or other safety outcomes. Most of these indicators referred to specific sectors, such as systems for preventing major industrial accidents in processing industries, road and air transport safety management systems, etc. In this paper, the authors have proposed a conceptual framework for tracing the path to identify indicators for each safety management system element. Figure 1 shows the steps that all types of organizations could use as a tool to build an effective system performance measurement, even if its implementation requires expert people in management systems and an exhaustive state of the art. The proposed framework consists of three phases structured in steps that lead to the definition of i-indicators for each j-element of the management system. The challenge in the existing literature is to find an appropriate number of indicators. On the one hand, a high number of indicators would lead to redundancy due to the interrelationships between indicators and an excessive expenditure of economic resources for their management (data collection, analysis and decision-making). On the other hand, a small number of indicators could lead to not considering the early warnings that would derive from the omitted indicators. In this preliminary study, the aim is not to identify all the indicators but to build a structured path to consider all the main elements of a management system, which should be monitored in order to allow management to implement preventive measures. Figure 1: Framework for defining leading indicators 440 2.1 Phase I - Identification of SMS areas and elements To date, there is no unique model or standard for the implementation of an occupational health and safety management system, but the existing ones are all based on the Deming cycle in order to achieve continuous improvement, which is based on the phases of the plan, do, check and act (PDCA cycle). To underline the importance and integration of the tool, it is possible to highlight how much the certification standards of the latest generation management systems are broken down by it into: a) Plan: establish the objectives and processes for occupational health and safety following the organization’s policy. b) Do: implement processes as planned. c) Check: monitor and measure activities and processes according to the occupational health and safety policy and objectives. d) Act: make improvement actions based on the results of the check phase. Therefore, as a first step, the authors decided to trace the main areas of a management system to the four phases of the Deming cycle. In particular, it was decided to refer to the ISO 45001 (2018) standard, as it represents the most recently developed state of the art and integrates well with the other ISO management systems because they are based on the same high-level structure (HLS). Furthermore, ISO 45001, like the other ISO management systems, also incorporates risk management principles (Pera et al., 2020) (Table 1). Table 1: k-SMS areas based on PDCA cycle phase of the Deming cycle corresponding k- SMS area Plan Do Check Act Leadership and worker participation Planning Support Operation Performance evaluation Improvement The second step of the first phase consists in identifying the main elements of the management systems for each k-SMS area. Again, ISO 45001 has continued to be used as a reference for the nomenclature used to identify the j-SMS elements, as shown in Table 2. Table 2: j-SMS elements based on ISO 45001 k - SMS area corresponding j - SMS element Leadership and worker participation Leadership and commitment OH&S policy Organisational roles, responsibilities and authorities Consultation and participation of workers Planning Support Operation Performance evaluation Improvement Actions to address risks and opportunities OH&S objectives and planning to achieve them Resources Competence Awareness Communication Documented information Operational planning and control Emergency preparedness and response Monitoring measurement, analysis and performance evaluation Internal audit Management review Incident, nonconformity and corrective action Continual improvement 441 2.2 Phase II - Selecting leading indicators In order to select key performance indicators (KPIs) to measure individual elements of the SMS, a literature- based review of management system applications should be conducted. There are numerous publications in the literature concerning the measurement of the performance of safety management systems. Many of these have come to define different indicators, both lagging and leading. In addition, many organizations have also developed, based on experience, their own set of indicators to measure the effectiveness of their system. For this reason, the authors suggest that experts or researchers carry out the bibliographic analysis necessary to provide a range of indicators to measure the main SMS elements. For only example, an extract of the bibliographic research conducted by the authors is reported, but not entirely presented in the present work as it is a research output to be published soon. The literature review could carry out the analysis of the Scopus database through sets of keywords that allow identifying relevant documents for the study goal. A set of keywords that could be used to acquire the literature related to safety management systems and their performance, are “Safety”, “Key Performance Indicators”, “Occupational Health and Safety” “Leading Indicators” and “Lagging Indicators”. All topic areas and documents such as “article”, “conference paper”, “review”, “book” and “book chapter” could be included. The period of the last fifteen years (2007-2022) may be considered, as the interest in OSH began to grow significantly starting from 2007. According to these criteria, the resulting documents were 97, from which it is possible to evaluate the state of the art. Furthermore, to ensure the alignment of the field of application, a further screening could be carried out to eliminate any redundancies by examining the title and the abstract, which led to a further selection of documents based on researchers’ experience. So, the final documents might be selected by critically evaluating the full text to assess its relevance to the study's objective. The effort is to select at least one indicator from the literature for each SMS element identified in table 2. The experience of the person conducting the analysis must be such as to eliminate redundant indicators or those which have relationships with others such as to be deducible from them. The further selection, that is requested to be carried out in the second step, depends on the authors’ choice to identify only the leading indicators, based on considerations made in the introduction concerning the predictive nature of these indicators. This choice leads to the construction of a system of indicators that becomes a prevention tool, as it can give early warming, allowing actions to be implemented before accidents occur. In order to select indicators that well suited to all types of organization, it is appropriate to identify those which, according to Combon et al. (2005) are called structural performance indicators rather than operational performance ones. Structural performance indicators get a picture of what the system consists of rather than how a given system operates at the shop-floor level. The third step of the second phase provides for a further selection of leading indicators, selected in the previous steps, according to SMART criteria proposed by Shahin et al. (2007). These criteria for evaluating and selecting KPIs are frequently recommended in the literature. The acronym SMART stands for: Specific, Measurable, Achievable, Realistic and Time-sensitive. The indicator should be specific to precisely measure the achievement of specific goals and be understandable to all users. In order to achieve objectives, each goal should be measurable. This measurement should be accurate and repeatable. Setting goals should be reasonable and attainable, so the values of the indicator should be achievable under given conditions and within the foreseeable period. A goal should be attainable but not realistic, so the indicator should be realistic for the functioning of an organisation. Finally, an indicator is time sensitive if it should be achieved in a given period, and allows the analyst to monitor progress along the path to reach the goal. 2.3 Phase III - Using leading indicators Phase II concluded with the definition of a set of indicators for each j-SMS element, according to the criteria established in that phase. In phase III, however, the steps to be followed in order to effectively use the selected indicators are explained. For this purpose, the approach reported in an OSHA document (2019) based on the Deming cycle was followed (Figure 2). The cyclic nature of the steps allows for reassessing the chosen indicators if the target objective is not achieved. In the first step, setting a goal for each leading i-indicator chosen to measure an SMS element is necessary. Next, it is necessary to specify whether mandatory lower bounds exist for a given indicator or, alternatively, to provide an evaluation criterion for the single indicator to provide an evaluation scale, qualitative or quantitative, which will allow the performance to be measured. In the second step, the indicator begins to be used by collecting data and is periodically measured for comparison with the target objective over time. After monitoring the i-indicator for a while, it could be possible to evaluate progress toward the target goal. However, many indicators need more time to assess their effectiveness. It has been established that specific leading indicators tend to be more predictive of future performance when deployed over time and often assessed, as opposed to when applied just once (Alruqi and 442 Hallowell, 2019). In addition, the evaluation of the leading indicator allows learning what to improve in the system and, if necessary, changing the indicator. Figure 2 – Steps to effective use leading indicators 3. Discussion The aim of developing a conceptual framework for defining leading indicators is to design a proactive safety management approach. The advantage of the proposed framework is to provide a structured path to guide step by step the management in identifying leading indicators for each main SMS element and cyclical steps in the use of indicators based on continuous improvement leading to reassessing indicator and their target goal. Furthermore, the cyclical nature of the steps in using indicators also allows learning from the measurement result. In this way, organizations can identify and correct critical issues before they turn into accidents. The framework seems easily usable in any type of organization regardless of its size and suitable for monitoring performance according to continuous improvement. However, the framework is a qualitative approach and does not guarantee the identification of an adequate number of indicators to monitor the elements of the SMS. The identification step is delegated to the experience of management systems analysts and their ability to carry out a bibliographic analysis on the leading indicators developed in the research. This represents a strong weakness even if it is the same for all the models proposed for performance measurement. Standardization is also trying to overcome this issue by developing guidelines on performance evaluation. Furthermore, despite having taken ISO 45001 as a management system reference, the main elements that constitute it were identified but not the sub-elements. Consequently, the analyst implementing the framework will be led to not defining indicators for the sub-elements. This choice is based on the authors’ decision to consider only the structural indicators instead of the operational performance indicators. 4. Conclusions The growing interest in a proactive approach to safety by using leading indicators has led researchers to develop models for measuring the SMS performance. However, in the literature, no proposed method provides an exhaustive list of indicators, both lagging and leading, that could be used by all types of organizations and regardless of their size. To date, there is no single system for measuring the performance of safety management systems, even though researchers have proposed various models to make them fully effective. Standardization is also moving in this direction, and the authors are also monitoring its developments. In this context, the proposed conceptual framework aims to drive management to define leading indicators for each main SMS element and effectively use them. Furthermore, attention has been paid to leading indicators due to their predictive nature, which allows management to respond quickly to early warnings detected in all SMS elements. This choice is also preparatory to the research that the authors are developing on the ability to measure organizational resilience through the SMS leading indicators (Pera et al., 2020). Based on the organizational resilience concept formulated by Hollnagel et al. (2006) define, it as "intrinsic ability of an organization (or system) to adjust its functioning prior to or following changes and disturbances to continue 443 working in the face of continuous stresses or major mishaps”. For this purpose, the SMS leading indicators are suitable for detecting those early warning signals on possible malfunctions of safety systems. In fact, these indicators allow monitoring the normal operation of the system and detecting its changes. References Alruqi, W.M., Hallowell, M.R., 2019, Critical success factors for construction safety: review and meta-analysis of safety leading indicators, J. Constr. Eng. Manag. 145. BSI (British Standards Institution), 2007, Occupational Health and Safety Management Systems – Specification (BS OHSAS 18001:2007), Occupational Health and Safety Assessment Series, BSI, London, UK. Carra, S., Monica, L., Di Girolamo, C., & Salerno, A, 2022, Accidents and Near-Misses in the History of a High- risk Chemical Installation: Analysis of the Human Component, Chemical Engineering Transactions, 91, 133- 138. Cambon, J., Guarnieri, F., & Groeneweg, J., 2006, Towards a new tool for measuring Safety Management Systems performance, in Proceedings of the Second Resilience Engineering Symposium. Mines Paris, Less Presses, Antibes–Juan-les-Pins, France (pp. 53-62). Hollnagel, E., Woods, D.E., Leveson, N., 2006, Resilience Engineering: Concepts and precepts, Ashgate Publishing Limited, Hampshire, UK. ICMM (International Council on Mining & Metals), 2012, Overview of leading indicators for occupational health and safety in mining. ILO (International Labour Office), 2001, Guidelines on Occupational Safety and Health Management Systems, (ILO-OSH, 2001), Geneva, Switzerland. ISO 45001:2018 - Occupational health and safety management systems - Requirements with guidance for use. OSHA. June 2019. Using Leading Indicators to Improve Safety and Health Outcomes. www.osha.gov/leadingindicators. Pera, F., Madonna, M., Del Prete, 2020, Enhancing the organisational resilience through the safety management system, 30th European Safety and Reliability Conference, ESREL 2020 and 15th Probabilistic Safety Assessment and Management Conference, PSAM 2020, pp. 3806–3812. Podgórski, D., 2015, Measuring operational performance of OSH management system – A demonstration of AHP-based selection of leading key performance indicators. Safety science, 73, 146-166. Shahin, A., Mahbod, M.A., 2007, Prioritisation of key performance indicators. An integration of analytical hierarchy process and goal setting, J. Prod. Perform. Manage, 56 (3), 226–240. 444 446pera.pdf A Conceptual Framework to Defining Leading Indicators to Measure Safety Management System Performance