179 journal homepage: www.fia.usv.ro/fiajournal Journal of Faculty of Food Engineering, Ştefan cel Mare University of Suceava, Romania Volume XIII, Issue 3 – 2014, pag. 179 – 187 EVALUATING AN ORGANISATION’S MANAGEMENT PERFORMANCE THROUGH THE CERTIFICATION AUDIT *Lucica BRUDIU1, Mariana-Liliana PǍCALǍ2 , Constantin MILITARU3 1 S.C. VIO-MARPREST S.R.L., Department of Quality Management, 900581 Constanta, Romania; University Politehnica of Bucharest, Faculty of Engineering and Management of Technological Systems, Bucharest, Romania; brudiu.lucica@yahoo.com 2 Lucian Blaga University of Sibiu, Department of Agricultural Science and Food Industry, 7-9 Dr. I. Ratiu St., 550012 Sibiu, Romania, Tel.: +407-018-5438; fax: +4-026-921-2558; mariana.pacala@yahoo.com 3University Politehnica of Bucharest, Faculty of Engineering and Management of Technological Systems, Bucharest, Romania; prof_militaru@yahoo.com *Corresponding author Received July 30th 2014, accepted August 29th 2014 Abstract: The article proposes a strategy which reflects in the best way possible the evaluation of the integrated management system subjected to certification. For this purpose, 17 indicators of performance for the supply process of an organization have been defined and analysed so that the management performance of senior management can be improved during a certification cycle. The structure of the proposed strategy was tested in terms of applicability for the process of supply of an organization with an integrated management system. The Case Study "Certification audit and evaluation of performance indicators for the supply process" has showed a decrease of non-compliant indicators and an even greater decrease of indicators that require analysis and what strategy to evaluate the performance indicators for the audited integrated management system is usable. The certification audit emphasizes the importance of integrated management system which is enlarged or minimized depending on how management addresses issues of responsiveness, adaptability and compliance, as well as its ability to pursue strategic objectives. Keywords: integrated management, performance indicator, supply process 1. Introduction A key objective of any organization is getting performance. Implementing a system of performance indicators helps to assess the strategic objectives of the organization, so that at any moment one could be able to determine wheter it works the way it was established or not. A detailed knowledge of the state of the organization is not possible without a “system of performance indicators” [1] to inform management about the results obtained in all key activities and processes of the organization . Implementation of the integrated management system in an organization begins by setting its specific performance indicators. By means of the integrated management system documents and records are generated by employees in order to give values of performance indicators. Strategic objectives should be checked and measured objectively, otherwise the strategy remains a simple act. Measuring the achievement of “strategic objectives” is done by evaluating the performance indicators [2]. The process of identification and definition of performance indicators should be established with the involvement of both “top management and the staff responsible” [3] for carrying out the processes in the organization. Food and Environment Safety - Journal of Faculty of Food Engineering, Ştefan cel Mare University – Suceava Volume XIII, Issue 3 – 2014 Lucica BRUDIU, Mariana-Liliana PǍCALǍ, Constantin MILITARU, Evaluating an organisation’s management performance through the certification audit, Food and Environment Safety, Volume XIII, Issue 3 – 2014, pag. 179 – 187 180 For each performance indicator the optimal value is defined and the period of time required to reach the indicator is established (in months, years, etc.). Performance indicators are chosen so as to meet the strategic objective meaning that the optimal value reflects the strategic objective and the intermediate values to correspond to the achieved progress. Identification, definition and implementation of the performance indicators should take into account their utility, namely they will not consider indicators whose calculation / evaluation requires offering too many resources (systems, work hours, spending surveys etc.) compared to the benefits For example, the classic performance indicators can be calculated based on the data stored in computer systems, such as financial data, production, quality statistics, etc. These are indicators that can be measured most frequently and are most easily determined. Another type of performance indicators has the base of internal assessments or third party assesements. Their advantage is that they can not be influenced, but have the disadvantage of being available only at longer intervals, and getting them can often cause high costs. Indicators based on management's assessment of the organization on the functioning of the integrated management system can be used very well when they are comparable. It is clear that performance indicators need to be calculated in the same manner and on the same data in different processes of the organization. Otherwise they lose consistency, the ability to reflect actual progress and ensure effective comparability. At the effective implementation of the integrated management the indicators will meet the following criteria: • indicator definition - has a real basis ;is not purely theoretical; • indicator implementation costs - benefits of measurement and monitoring outweigh the calculation of costs; • acceptance and responsabilisation - are appropriated and taken out by the responsible managers to remain not only a bureaucratic exercise; • the capacity of the indicator to be defined, the indicator is a known name, a defined evaluation formula and used in the same way across the organization; • the evaluation/calculation frequency is defined. The certification of the integrated management system through certification auditing by a certification organism is also meant to evaluate performance indicators indentified by observing the measuring technique and monitoring the processes according to the referential requirements. The certification audit can find:  Conformity – when a standard requirement is met;  Non-conformity - when a standard requirement is not met. The top management analysis of audit findings made after the certification audit allows the calculation of performance indicators’ values and the evaluation of the organisation status on meeting and improving organisational performance. Without limiting the identification and use of performance indicators, we shall explain them for the supplying process from the integrated system management consisting of the quality management system according to standard SR EN ISO 9001: 2008[4], the food safety management system according to standard SR EN ISO 22000:2005[5], the enviroment management system according to standard SR EN ISO 14001:2005 [6] and occupational health and safety management system according to standard SR OHSAS 18001: 2008[7]. The purpose of this article is to valorize the important role of the certification audit and Food and Environment Safety - Journal of Faculty of Food Engineering, Ştefan cel Mare University – Suceava Volume XIII, Issue 3 – 2014 Lucica BRUDIU, Mariana-Liliana PǍCALǍ, Constantin MILITARU, Evaluating an organisation’s management performance through the certification audit, Food and Environment Safety, Volume XIII, Issue 3 – 2014, pag. 179 – 187 181 of the “audits for monitoring the integrated management system” 8] in the overall vision of the organisation and processes, by identifying improvement opportunities, which leads to optimizing the performance indicators of the supply process. 2. Matherial and methods The case study consisted in testing and determining the degree of application of the evaluation strategy within an organisation in which the integrated management system for the supply process was implemented, functional and subjected to the certification process. This study has used for research the qualitative method: conceptual research and case study. As research tools we used the qualitative analysis of documents and observation as well as audit reports as support. The organisation involved in this study was chosen based on several factors:  it should have a documented and functional integrated management system;  it should have undergone a certification cycle for the integrated management system (quality, environment, food safety and occupational health and safety);  its field of activity should be the manufacturing of food products;  it should be a large organsation. In order to develop the model for evaluation and performance analysis, the following data of the organisation and input-data of the integrated management system and research have been identified, as shown in the table 1. The study was carried out for the supply process within the organisation in which the integrated management system is implemented and functional. Table 1 Values of Data-Input Total no. of suppliers 90 Value of supply costs (thousands lei) 800 No./value of products returned to supplier (thousands lei) 10 No. of orders not completed in due time 100 No. of suppliers with supply agreements 52 Requirement 6.2. of referential standard “Management System Integration Standard” [9] that is audited and connected to each integrated standard, appears in Figure 1 Figure 1. Common requirements of standard for the Supply Process 3. Results and discussion The calculation formula and optimal values have been determined for the performance indicators defined in this paper (Table 2). The calculation formula is the percentage ratio between the determining measurements of each performance indicator. (1)  %100 b a I x where : IX – value of performance indicator, [%]; a – measured value at a certain moment, different from the initial moment; b – measured value at the initial moment. Food and Environment Safety - Journal of Faculty of Food Engineering, Ştefan cel Mare University – Suceava Volume XIII, Issue 3 – 2014 Lucica BRUDIU, Mariana-Liliana PǍCALǍ, Constantin MILITARU, Evaluating an organisation’s management performance through the certification audit, Food and Environment Safety, Volume XIII, Issue 3 – 2014, pag. 179 – 187 182 Table 2. Definition and Optimal Values of KPI for the Supply Process KPI DESIGNATION KPI OPTIMAL LEVEL VARIABLES KPI 129 Degree assessment suppliers 100% a No. evaluate suppliers b Total no. of suppliers KPI 130 Rate of steadily no of suppliers 90% a No.reevaluate suppliers maintained that accept b No.suppliers reassessed KPI 131 Degree of acceptance suppliers 100% a No. suppliers accept b No. evaluate suppliers KPI 132 Degree of uniqueness suppliers 5% a No. single supplier b Total no. of suppliers KPI 133 Degree of cost supply 20% a Value of supply costs (thousands lei) b Total cost value KPI 134 Utilization degree 1 100% a The amount of raw materials used per month b The amount of monthly stock KPI 135 Degree partial according to supply 0% a No. of noncompliance reports done for the supplied products noticed as non copliant at reception b Total no. of non-conformities KPI 136 Degree of compliance with food risk 0% a No. of noncompliance reports regarding food safety done for the supplied products noticed as non copliant at reception b Total no. of non-conformities KPI 137 Partial degree of compliance with food risk 0% a No. of noncompliance reports done for the supplied products noticed as non copliant at reception b No. noncompliance reports on food safety KPI 138 Degree of return to supplier 0% a No./value of products returned to supplier (thousands lei) b No. / Value products supplied KPI 139 Auditing degree of suppliers 100% a No. second party audits conducted at suppliers b No. second party audits conducted at planned suppliers KPI 140 Budget utilization rate of supply 100% a Amount paid on goods supplied b Value established by the Budget KPI 141 Level suppliers supply contracts 80% a No. of suppliers with supply agreements b Total no. of suppliers KPI 142 Degree of compliance of supplied products 100% a No. supply of products with documents attesting b No. total merchandise purchases KPI 143 Assessment degree according to the reception 100% a Requirements (qualified personnel + frequency analysis method of analysis + sample analysis) actually used b Requirements (qualified personnel + frequency analysis method of analysis + sample analysis) set KPI 144 Degree of honor purchase orders 5% a No. unfulfilled orders on time b No. orders issued KPI 145 Share of non- compliant products obtained stocked with hidden flaws 0% a No. / Assets obtained with non-compliant products supplied with hidden flaws b No. / Value of non-conforming products obtained The following non-conformities were noticed for the supply process of the organisation, audited during the certification audit of year I:  Not all suppliers have been evaluated. This way it was possible to calculate the value of KPI 129.  Not all the previously accepted suppliers have been reevaluated. This way it was possible to calculate the value of KPI 130.  Not all the second-party audits planned for the suppliers have been done. This way it was possible to calculate the value of KPI 139. Food and Environment Safety - Journal of Faculty of Food Engineering, Ştefan cel Mare University – Suceava Volume XIII, Issue 3 – 2014 Lucica BRUDIU, Mariana-Liliana PǍCALǍ, Constantin MILITARU, Evaluating an organisation’s management performance through the certification audit, Food and Environment Safety, Volume XIII, Issue 3 – 2014, pag. 179 – 187 183  Not all the supplied products have documents that assure quality. This way it was possible to calculate the value of KPI 142.  Not all the responsible personnel are instructed in the “supplying” procedure. This way it was possible to calculate the value of KPI 143.  Not all the orders to the suppliers have been delivered on time. This way it was possible to calculate the value of KPI 144.  Not all methods of analysis of the products at the reception are likely to identify potential nonconformities. This way it was possible to calculate the value of KPI 145. Following the analysis performed by the top management and relevant positions of the organisation involved in the certification process, based on the certification audit report and the non- conformity reports included in it, it was possible to calculate also the values of the indicators: KPI 131, KPI 132, KPI 133, KPI 134, KPI 135, KPI 136, KPI 137, KPI 138, KPI 141. This way, the values of the performance indicators for KPI 129-KPI 145 were calculated and shown in Table 3. Table 3 Values of KPI of the Certification Cycle (year I, II and III) for the Supply Process KPI VALUES For year I VALUES For year II VALUES For year III a b KPI, % a b KPI, % a b KPI, % KPI 129 78 90 87 92 100 92 110 110 100 KPI 131 69 78 8 84 92 91 96 110 87 KPI 132 6 90 7 6 100 6 6 110 5 KPI 133 800 1600 50 360 1600 23 320 1600 20 KPI 134 2 4 50 2.5 4 63 3.5 4 88 KPI 135 1 12 8 15 0 16 0 KPI 136 0 12 0 1 15 7 0 16 0 KPI 137 0 12 0 1 15 7 1 16 6 KPI 138 0.10 10 1 0.5 100 0.5 0 10 0 KPI 139 0 12 0 6 12 50 10 12 83 KPI 140 3 3.3 91 3.4 3.3 103 3.2 3.3 97 KPI 141 52 90 58 65 100 65 75 110 68 KPI 142 2.3 2.35 98 2.2 2.3 96 2.31 2.4 96 KPI 143 4 5 80 5 5 100 5 5 100 KPI 144 0.1 2.35 4 0.5 2.3 2 3.3 2.4 1 KPI 145 2 10 20 1 10 10 0 10 0 3.1. Evaluation of performance indicators The next step in setting the strategy for the evaluation of management performance based on performance indicators is comparing their value determined during the certification audit to an optimal value. The optimal value of performance indicators was set as based on the organisational history, the analyses performed by the top management on the Food and Environment Safety - Journal of Faculty of Food Engineering, Ştefan cel Mare University – Suceava Volume XIII, Issue 3 – 2014 Lucica BRUDIU, Mariana-Liliana PǍCALǍ, Constantin MILITARU, Evaluating an organisation’s management performance through the certification audit, Food and Environment Safety, Volume XIII, Issue 3 – 2014, pag. 179 – 187 184 evolution of the organisation in terms of organisational and financial strategy and meeting the requirements of the “management system integration standard” [9]. Table 3 shows the optimal values of performance indicators. Three degrees of evaluation were set for performance indicators based on the optimal value:  Degree I - conformity – if the value of the performance indicator is equal to the optimal value.  Degree II – non-conformity – if the value of the performance indicator is lower than the optimal value. In this case there shall be identified and analysed the cause of not reaching the optimal value and there shall be implemented the corrective action, sometimes the preventive action within a non-conformity report written by the responsible for the process considered to be non-conforming when the audit is performed. This non- conformity is not designed to affect the functioning of the integrated management system.  Degree III – requires analysis – if the value of the performance indicator is higher than the optimal value, in the case of indicators involved in evaluating food safety risk, product design, customer satisfaction or financial loss. This requires an analysis by all factors involved in the non-conforming process. Throughout the development of the certification cycle of the integrated management system, consisting in the certification audit and the annual monitoring audits performed by auditors representing the certification organism, their impartiality has helped to determine the degrees of performance indicators. For the case study applied to the supply process within the integrated management system according to the classification of performance indicators into the three degrees, there resulted the situation presented in Table 4 and in Figure 2. Table 4 Classification of KPI in Evaluation Degrees from the Certification Cycle (year I, II and III) for the Supply Process within the Organisation KPI OPTIMAL LEVEL, [%] Year I Interpretation Year II Interpretation Year III Interpretation KPI 129 100 87 NC 92 NC 100 C KPI 130 90 76 NC 88 NC 84 NC KPI 131 100 88 NC 91 NC 87 NC KPI 132 5 7 NA 6 NA 5 C KPI 133 20 50 NA 23 NA 20 C KPI 134 100 50 NC 63 NC 88 NC KPI 135 0 8 NA 0 C 0 C KPI 136 0 0 C 7 NA 0 C KPI 137 0 0 C 7 NA 6 NA KPI 138 0 1 NA 0.5 NA 0 C KPI 139 100 0 NC 50 NC 83 NC KPI 140 100 91 NC 103 NA 97 NC KPI 141 80 58 NC 65 NC 68 NC KPI 142 100 98 NC 96 NC 96 NC KPI 143 100 80 NC 100 C 100 C KPI 144 5 4 NC 2 NC 1 NC KPI 145 0 20 NA 10 NA 0 C (C-conforming, NC-non-conforming, NA-requires analysis) Food and Environment Safety - Journal of Faculty of Food Engineering, Ştefan cel Mare University – Suceava Volume XIII, Issue 3 – 2014 Lucica BRUDIU, Mariana-Liliana PǍCALǍ, Constantin MILITARU, Evaluating an organisation’s management performance through the certification audit, Food and Environment Safety, Volume XIII, Issue 3 – 2014, pag. 179 – 187 185 The results of applying the strategy for evaluating management performance after the monitoring audit of year I, II, respectively year III, of the certification cycle were adapted and included in Table 4. By keeping the same optimal values of the performance indicators, following the analysis performed by the top management, based on the report for the monitoring audit for year I, II, respectively year III, of the certification cycle, there has resulted the interpretation of performance indicators in Table 4 and the graphic representation in Figures 2 show the important role of the certification audit, the monitoring audits of year I, II and year III, by pushing the values of performance indicators towards the optimal value and improving the supplay process after each annual audit. By comparing the KPI values to the optimum value that had only 6 of the optimal values were achieved and it can be concluded that the supply takes place efficiently and effectively in a proportion of 35.3%. Another purpose of the evaluation is to determine the performance management of all the causes that have generated employment performance indicators in levels of "non-compliant" and "needs analysis" in an analysis of the management. Figure 2. KPI values for year I, II and III compared to optimal KPI values Performance indicators were identified and adapted according to the strategy of evaluating management performance applied in the case study, as shown in the diagram in figure 3. 0 20 40 60 80 100 120 KPI 129 KPI 130 KPI 131 KPI 132 KPI 133 KPI 134 KPI 135 KPI 136 KPI 137 KPI 138 KPI 139 KPI 140 KPI 141 KPI 142 KPI 143 KPI 144 KPI 145 OPTIMAL Values year 1 Values year 2 Values year 3 Food and Environment Safety - Journal of Faculty of Food Engineering, Ştefan cel Mare University – Suceava Volume XIII, Issue 3 – 2014 Lucica BRUDIU, Mariana-Liliana PǍCALǍ, Constantin MILITARU, Evaluating an organisation’s management performance through the certification audit, Food and Environment Safety, Volume XIII, Issue 3 – 2014, pag. 179 – 187 186 Figure 3. Identification Diagram for the System of Indicators within Organisations with Certifiable or Certified Integrated management System 4. Conclusion Following the analysis and implemented corrections, corrective and preventive actions for non-conformities identified during the certification audit by auditors of the certification organism, the monitoring audits of year II and III of the certification cycle have shown that some of the performance indicators have reached optimal values and some indicators of the supply process have values that require analysis and correction within the non- conformity reports. By applying the stratgy suggested for improving management performance within an organisation, the audits of the certification cycle have shown a decrease of 20 % of non-conforming indicators and of 80 % of indicators that require analysis at the end of the certification cycle. A highly positive evolution was recorded for indicators KPI 139 - Degree auditing suppliers, KPI 142 - Degree of conformity assessment at reception. The certification of the integrated management system by performed audits supports the strategy for improving management performance which is a key element for developing an organisation. Focusing on the certification of the integrated management system does not imply the need to use costly last generation equipment, but only team work at all organisational levels. The certification audit and the monitoring audits of the integrated management system have an important role in the overall vision of the organisation and processes, by identifying improvement opportunities and optimizing performance indicators. 5. Acknowledgments We would like to thank the management team of S.C. SAFIR S.A. for the permission to do the case study on the organisation and also to acknowledge the certification organism for letting us use the audit reports regarding the enterprise. Define strategy Define KPI Identifiy KPI / monitor KPI Implement KPI Main actions • defining strategic objectives and functions • define how to achieve strategic objectives Main actions • KPI identification • -measured • -can be measured • -need to be measured • define selected KPI • define optimum for selected KPI Main actions • define data collectiong process • define monitoring process -steps • -roles and responsabilities - measurements frequency • - KPI reporting type Main actions • KPI analisation and comparison to optimum • corelation with IMS • optimum adjustment and definition of improvement actions of performance measurement sistem Food and Environment Safety - Journal of Faculty of Food Engineering, Ştefan cel Mare University – Suceava Volume XIII, Issue 3 – 2014 Lucica BRUDIU, Mariana-Liliana PǍCALǍ, Constantin MILITARU, Evaluating an organisation’s management performance through the certification audit, Food and Environment Safety, Volume XIII, Issue 3 – 2014, pag. 179 – 187 187 6. References [1]. KAPLAN S.V. R., NORTON D.P., The Balansed Scorecard, Harvard Business School Press, 1997; [2]. NEELY et al., Designing, implementing and updating performance measurement systems, International Journal of Operations & Production Management Vol. 20 Iss: 7, 1995, 754 - 771; [3]. HĂHĂIANU, L., How to motivate your employees to achieve outstanding performance, Rentrop&Straton, Group publishing and consulting business, Bucharest, 2002 (in Romanian). [4]. SR EN ISO 9001:2008 - Quality management systems. Requirements (in Romanian). [5]. SR EN ISO 22000:2005 - Food safety management systems. Requirements for any organization in the food chain(in Romanian). [6]. SR EN ISO 14001:2005 - Environmental management systems. Requirements with guidance for use (in Romanian). [7]. SR OHSAS 18001:2008 - Management Systems Occupational Health and Safety. Requirements (in Romanian). [8]. KANTER R. M., Borders management, Meteor Press Publishing, Bucharest, 2006 (in Romanian). [9]. BRUDIU, L., PACALA, M.-L., ABDEL- RAHIM Al Ali (2013) - Systems management and integration, Journal of Management Systems - Quality, 14(136), 9-13 (in Romanian).