J Global Clinical Engineering Vol.4 Issue 1: 2021 14 Received February 28, 2020, accepted December 9, 2020, date of publication February 3, 2021 Evaluation and Optimization of CES Performances: Application of the Pareto Principle to KPIs By Alessia Cecchini, Grazia Maria Pia Masselli, Sergio Silvestri Clinical Engineering Service, University Hospital Campus Bio-Medico, Rome, Italy ABSTRACT In recent times the approach to health care has been mostly influenced by the growing quantity of biomedical equipment used in hospitals, which needs the support of the clinical engineering service (CES). This work aims to suggest a methodology to improve the performance of a CES through the application of Pareto principle to the leading Key Performance Indicators (KPIs). The methodology is applied by focusing on using KPIs that represent a quantifiable measure of achieving goals set by an organization. In this study, five KPIs are considered: Uptime, MTTR (mean time to repair), PPM (percentage preventive maintenance completion), MTBF (mean time between failures), and the COSR (cost of service ratio). The first three indicators express the measure of CES efficiency in ensuring regular maintenance. The first step consists of retrieving data related to work orders for 2015-2016 on 6000 installed devices, carried out by manage- ment software. The second step is to get the results by using an environment for numerical calculation and statistical analysis. To identify the main critical issues that may be present, three indicators (Uptime, MTTR and MTBF) are analyzed by applying the Pareto principle (i.e., 20% of the causes produce 80% of the effects). Considering the totality of work orders, it is possible to concentrate on only 20% of them to focus on a small group to understand the correlations between them. Identifying these characteristics means identifying the main critical issues that are present, on which action must be taken, and which affect 80% of the overall behavior. Instead, the COSR and PPM indicators suggest distribution models that focus on the most critical devices. In conclusion, the way to analyze the results is obtained, when possible, by applying Pareto principle. Therefore, a CES will be able to focus on a few causes of poor performance. The achievement of these results could allow the standardization of the method used, enabling it to be applied to any healthcare system. Keywords – CES, KPIs, UPTIME, MTTR, MTBF, COSR, PPM, UCBM. Copyright © 2021. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY): Creative Commons - Attribu- tion 4.0 International - CC BY 4.0. The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms. INTRODUCTION • Key Performance Indicators (KPIs) represent a quantifiable measure of achieving goals set by an organization, both operational and strategic. Generally speaking, companies have different KPIs depending on their priority criteria. KPIs can also be established arbitrarily but, to be useful, they must meet the fol- lowing requirements1: • Quantifiability - KPIs must be presented in the form of numbers. http://www.globalce.org http://globalce.org http://globalce.org https://creativecommons.org/licenses/by/4.0/ https://creativecommons.org/licenses/by/4.0/ 15 J Global Clinical Engineering Vol.4 Issue 1: 2021 Cecchini, Masselli, Silvestri: Evaluation and Optimization of CES Performances: Application of the Pareto Principle to KPIs • Practicality - they integrate well with current busi- ness processes. • Directionality - they help to determine whether companies are improving. • Operations - they can be related to the practical context to measure an effective change. The four leading indicator typologies are1: 1. General indicators - measure the amount of work in the process. 2. Quality indicators - evaluate the quality of the process output based on certain standards. 3. Cost indicators. 4. Service or time indicators - they measure the response time from the start of the process until its conclusion. Thanks to a set of KPIs, it is possible to evaluate the performance of a clinical engineering service (CES). This paper discusses the current status of benchmark indicators within the field of clinical engineering. The paper focuses on the evaluation and optimization of the medical equipment repair and maintenance activities of a CES by applying the Pareto principle to KPIs to focus on main critical activities. STATE OF THE ART According to Cohen et al., KPIs represent the process of comparing business performance levels to identify op- portunities to improve. The results provided an idea of what should be changed and how. However, comparisons have met with limited success due to poor and inconsistent definitions of the parameters measured and lack of qual- ity data available. In the first phase, it was necessary to identify basic indicators that are applied to any healthcare facility. They must be uniquely defined and consequently calculated by the same method so that they can be com- pared. Afterward, it was possible to start to build other indicators that will be different depending on the specific needs and the technical-economic information that ev- eryone possesses. For this reason, the primary purpose of this paper was to detail some of the quantitative per- formance and cost benchmark indicators that have been historically used in medical equipment maintenance and repair and to make recommendations on how the clinical engineering profession can develop good quality, useful and meaningful benchmarks. The general characteristics of a useful benchmark indicator are2: • well defined; • objective; • measurable; • based on current knowledge experience; and • valid. The study by Bassem et al. aims to evaluate CES's performance at the University of Cairo, Egypt, using quantitatively measured parameters to allow comparison and improvement objective.3 In addition to the param- eters proposed by Cohen et al., considered insufficient, Bassem proposed new indicators. Data were collected by 10 hospitals, corresponding to different healthcare orga- nizations.2,3 These data were subsequently analyzed by a software tool, providing a score for each CES. The first step was to decide what exactly to evaluate and monitor. Some of the studies reported by Bassem adopted a survey technique as CES directors were asked to select from a list of proposed performance indicators that could be used for performance measurement benchmarking. Their response revealed three mainstream performance indica- tors. Other indicators had to be added and measured to evaluate the performance of the other services. They used additional essential indicators that should be involved to get an increased accurate evaluation. The results indicated an average gap of 67% between the performance of the CES and the reference they have identified, considered the ideal target.3 According to the Tiwari study, service performance on medical equipment serviced by external suppliers is assessed.4 The performance indicators of CES are first defined according to the needs and benefits required in the specific hospital structure and are then categorized and finally measured as indicated below. 1. The definition of KPIs considering the opinion of experienced staff. 2. The categorization of KPIs into four groups. 3. The measurement of KPIs. Cecchini, Masselli, Silvestri: Evaluation and Optimization of CES Performances: Application of the Pareto Principle to KPIs J Global Clinical Engineering Vol.4 Issue 1: 2021 16 The Tiwari study is an example of an outsourced CES, and the key to success is the measurement of performance to quantify the expected benefits.4 METHODOLOGY This study, conducted by the University Campus Bio- Medico of Rome (UCBM), where there are about 6,000 pieces of biomedical equipment for the year 2016. Before going into the explanation, and then in the calculation of the KPIs, it is vital to understand what type of data have been used. The CES utilizes an equipment management software, in which all the data are collected and related to the inventory number of biomedical equipment: the description, the serial number, the manufacturer, the purchase value, the intervention priority, therefore any information useful to characterize a specific piece of equip- ment. From the mentioned software, further information can be obtained relating to work orders and the schedule for preventive maintenance. In the first case, the CES takes care of entering all the work orders executed. In the second case, the CES takes care of inserting, within a schedule, all the equipment and the corresponding deadlines for preventive maintenance, in order to record the periodicity with which it is required to carry out maintenance. This approach maximizes effectiveness and efficiency in tech- nical management and ensures economic and technical control of maintenance. It has the objective of providing operational and decision-making support for further optimizing the processes related to registry/inventory management. In this way, from this software, it is possible to obtain categorized data, from which it is possible to calculate the KPIs. Starting from these data, the results are obtained through the use of "Matlab," an environment for numerical calculation and statistical analysis, which also includes the programming language. It allows the calculation of the KPIs considered here. The following paragraph will report the explanation, and the subsequent calculation, of the identified and measurable KPIs. The following KPIs are used and calculated: 1. Uptime. This denotes the time the biomedical equip- ment has been working for over one year; downtime is its complementary statistic and denotes the state of a not operational system. This may be due to fail- ure, preventive maintenance, or other causes. The measurement is carried out in absolute values or percentage. Uptime is particularly important for all machines where stability and availability are funda- mental. Through Uptime, efficiency can be deduced: a high Uptime indicates that the equipment is well configured, while a low Uptime could mean instability of the equipment. To get more evidence on the critical issues, this KPI calculation involves the initial use of data from all the equipment from which one or more work orders have taken place. Also, all devices that have not undergone a work order are then considered and has always been functional; a maximum Uptime value will appear. The Uptime calculation, represented by a percentage, is carried out by first calculating the downtime: the work orders corresponding to each in- ventory number of the equipment are considered and, consequently, the duration given by the sum of all the work orders for that specific inventory is calculated. It is then divided by the number of days within a year to indicate, the percentage of the number of days that a specific piece of equipment remained inoperative relative to the total period.4 The formulas used are the following: 2. MTTR (Mean Time To Repair). This denotes the Time to Restore (TTR) expected value, where the TTR is the time interval where the equipment is unavailable due to a failure. The MTTR includes the time for diagnosis, the arrival of the maintenance technician, the arrival of the component(s) to be replaced, and the actual repair. It is a useful parameter for evaluating the effectiveness of the CES in terms of the logistic organization. The calculation of this coefficient involves data from all equipment on which a work order has occurred involv- ing corrective maintenance or functional verification. The MTTR is calculated according to the work orders corresponding to the equipment’s inventory and the duration given by the sum of the times to repair in all the work orders, or that specific item is calculated. This is referred to as TTR and it is then divided by 17 J Global Clinical Engineering Vol.4 Issue 1: 2021 Cecchini, Masselli, Silvestri: Evaluation and Optimization of CES Performances: Application of the Pareto Principle to KPIs the total number of work orders within a year.4 The formula used is: 3. MTBF (Mean Time Between Failures). This term cor- responds to the average time interval between two successive failures (TBF) and indicates the frequency with which a failure can occur. This is mainly a reli- ability parameter used to indicate the probability that equipment operating under certain conditions will retain, after a predetermined time, the functional capacities for which it was built. The calculation of this coefficient involves data from all equipment on which a work order has occurred that involved corrective maintenance or functional verification. The work orders correspond to each inventory number of the equipment and consequently the total time that elapses between the start date of one work order and the start date of another. This would be the TBF. The TBFs from all work orders are added together and then divided by the total number of work orders within one year.4 The inverse parameter, defined as “frequency of failure,” indicates the rate at which technicians must carry out maintenance. The formulas used are the following: 4. COSR (Cost Of Service Ratio). COSR is an economic parameter that represents the sustainability of costs. It is calculated as the ratio between the overall main- tenance cost and the purchase cost, assessed through a percentage measure.4 The calculation of the overall COSR is carried out by proceeding in three phases outlined below. 1. Equipment with maintenance contracts, to which maintenance cost (if any) has been added the cost of the pieces of spare parts. 2. Equipment with only spare parts and without maintenance contracts. 3. Company cost of all the personnel working in the CES. The formula used is the following: 5. PPM (Percentage Preventive Maintenance). This term expresses the overall number of preventive maintenance events or carried out within the deadline, divided by the total planned preventive maintenances within a year expressed as a percentage.4 This calculation is made for each piece of equipment based on the future expiration date of preventive maintenance and the scheduled maintenance frequency. This makes it pos- sible to derive the previous preventive maintenance expiration date, which is compared with the date of the beginning of the maintenance carried out on each piece of equipment, allowing us to understand if the maintenance has been carried out before or after the deadline. The PPM calculation is also necessary in light of the accreditation manual of the Hospitals of the Joint Commission International.5 According to this manual, “all medical equipment and technologies are regularly subjected to inspections, maintenances and calibrations and these activities are documented in the appropriate registers. The staff ensure that all medical equipment and technologies operate at ac- ceptable levels and safely for operators.” When there is a need to manage the maintenance of many pieces of equipment, it is necessary to adopt criteria allow- ing priority. The equipment is not all critical in the same way, so it is necessary to distinguish the critical equipment on which the patient's life depends from the less critical ones for which the priority level is lower. There is, therefore, an issue to solve: if the technician receives two maintenance requests at the same time, the technician must be able to evaluate what the prior- ity request is. To do this, a risk assessment is carried out, which is done with objective criteria and not left to free interpretation. In our case, the criticality analysis is carried out based on the assignment of five scores, respectively relating to five categories of equipment criticality (Equipment Management Program Mayo Clinic6 has taken up this method). According to these criteria, equipment that should be excluded from the Cecchini, Masselli, Silvestri: Evaluation and Optimization of CES Performances: Application of the Pareto Principle to KPIs J Global Clinical Engineering Vol.4 Issue 1: 2021 18 plan can still be included if requested by a certification body or if scheduled by the manufacturer for periodic maintenance or calibrations. The formulas of PPM used are the following: RESULTS AND DISCUSSION Firstly, the first three indicators are analyzed, where possible, by applying the Pareto principle, one of the most used tools in quality management. The Pareto principle, also known as the 80/20 rule, took its name from Vilfredo Pareto, an Italian engineer who, in 1906, observed that the distribution of wealth in his country had an unequal distribution: 20% of the population owned 80% of the wealth.7 By expanding this idea to other areas and con- cepts, an empirical theory has been formulated which is respected in the majority of cases (this means that the distribution referred to it must be a very numerous distribution). The Pareto diagram's construction, based on this principle, shows that 20% of the causes produce 80% of the effects. The advantages that derive from the application of the Pareto principle and diagrams are: • to help to break down big problems into smaller problems and to establish what are the main factors causing them; • to help to focus on the most important causes in order to establish priorities, using the available time more effectively; • to help to link causes with effects; and • to support in evaluating the improvement based on an analysis of the situation before and after the application of the corrective action. To identify critical issues, the data relating to the first three indicators, namely Uptime, MTTR and MTBF, are analyzed by applying the Pareto diagrams, showing that, where applicable, 20% of the causes produce 80% of the effects. Considering all the medical equipment, it is possible to concentrate efforts only on 20% of causes to obtain a significantly better result. However, the analysis of data through the Pareto law is not always possible, but to extend its application, it expanded to 30% of causes. However, where the percentage is more than 30%, its ap- plication is not considered significant. This analysis makes it possible to focus on a small group of medical devices and understand the correlations between them (type of equipment, manufacturer). Identifying these characteristics allows the discovery of the main critical issues present within the health structure and what action is required. This will pertain to 80% of the overall behavior. For the construction of the Pareto diagram, a combina- tion of a bar chart can be considered showing the data in order of decreasing KPIs (Uptime, MTTR, MTBF), and of a cumulative curve, constructed by adding the i-th value to the previous values. This allows for immediate identifica- tion and effect of the relevant elements. On the other hand, the COSR and PPM indicators help analyze the data and suggest distribution models that enable focus on the most critical equipment. The graph of Uptime only considered those devices on which one or more work orders took place, which involved a value of Uptime <100%. It is precisely on these param- eters that the analysis of Pareto is carried out. Focusing will be on the downtime rather than on the Uptime to immediately highlight any critical issues. Overall, we note distribution of data mainly concentrated around 100%, while only a smaller percentage corresponds to a value of Uptime less than 100%, mainly distributed between 20% and 30%. Figure 1 shows the average value calcu- lated, which is very high and aligned well with similar measurements noted in the bibliography. The MTTR graph shows a data distribution mainly concentrated in a range between 0 and 1000 hours (42 days), while only a smaller part, the initial one, shows an increase in the number of hours that goes up to 1800 hours. As shown in Figure 2, this reflects the average value calculated. Since MTBF is the time between failures, the calcula- tion is performed if there are, for each inventory item, at least two failures, therefore two work orders within a year. 19 J Global Clinical Engineering Vol.4 Issue 1: 2021 Cecchini, Masselli, Silvestri: Evaluation and Optimization of CES Performances: Application of the Pareto Principle to KPIs The data analysis focuses on the frequency of faults, and it is carried out using Pareto diagrams. The MTBF graph (Figure 3) shows data distribution mainly concentrated in a range between 0 and 832 hours (35 days), while only a smaller part, the initial one, shows an increase in the number of hours that rises to 1354 hours. The COSR is calculated by paying attention to equip- ment with COSR> 0% to highlight any critical issues more efficiently. A histogram has been constructed (Figure 4) from these devices: the COSR trend has been highlighted through 0.5% intervals and the number of devices is then reported, having the corresponding COSR. The COSR trend resembles a Poisson distribution, in fact the data are distributed bell-shaped around a value belonging to the COSR range between 1.5 and 2%, corresponding to which we have 115 devices in 2015 and 124 devices in the year 2016. The trend over the two years is, in fact, the same. However, the highest histogram bar remains the one with COSR> 10% and will be analyzed later in detail for the analysis of critical issues. As can be seen, the COSR values are quite low at around 1%, but personnel costs must be added to this value, so overall it is around 4%. The data analysis for the PPM is carried out through the use of histograms (Figure 5) that highlight the equipment on which preventive maintenance took place in advance FIGURE 1. Pareto diagram for calculating downtime for equip- ment in 2015. x axis = number of devices; y-axis = percentage of time out of service (Downtime). FIGURE 2. Pareto diagram for calculating MTTR for equipment in 2015. FIGURE 3. Pareto diagram for calculating the frequency of failures for equipment in 2015. FIGURE 4. Histograms for calculating COSR interspersed by 0.5% in 2016. Cecchini, Masselli, Silvestri: Evaluation and Optimization of CES Performances: Application of the Pareto Principle to KPIs J Global Clinical Engineering Vol.4 Issue 1: 2021 20 (negative numerical value of the days) and late (positive numerical value of the days) concerning the scheduled deadline. If the number of days is 0, the preventive main- tenance took place on the same day as the planned date. It should be noted that the maintenance carried out in advance is greater than the delayed since the initial sec- tion of the graph is greater than the final one. If we want to summarize in a single value what has just been shown, the value of the Uptime, MTTR, MTBF, COSR, and PPM is reported expressing the measure according to its mean value and standard deviation. Each KPI is calculated for 2015 and 2016, as follows (Equipment that does not have work orders is also included in the Uptime calculation, resulting in 100% Uptime). As shown earlier, the Tiwari study, which shows numeri- cal calculations and graphs of identified KPIs, assesses the performance of medical equipment serviced by external suppliers; the CES examined in this work, on the other hand, is a predominantly internal CES, so a first difference is immediately apparent. In detail, however, we note that the numerical value of Uptime and COSR obtained from the two studies is comparable; the calculation of MTTR was carried out individually for each month by Tiwari, so it is clear this type of comparison is inconsistent. Also, because some work orders last more than one month; finally, in order to be able to compare the PPM, a clarifica- tion is needed, that is, it is necessary to take into account that, in the UCBM Polyclinic, the system revolves around a risk classification that guides the professionals of the CES to act according to different priorities. It is clear that comparing the two numerical values, they are different, FIGURE 5. Histograms of the number of days in advance and delay concerning the scheduled deadline from the next preven- tive maintenance date. but, taking this into account, it would then have been more significant to take as a reference the value of the calculated PPM for priority equipment, at 93.7%. Comparison with the study by Tiwari et al is shown in Table 1. Starting with the results obtained critical issues are analyzed. In the case of the Uptime, MTTR, and MTBF, this analysis is conducted globally as there is a correlation between them and the level of the numerical calculation performed. They are calculated starting from the durations of the work orders that are carried out during a year. It is therefore important to concentrate on the equipment for which this phenomenon is most evident and this is possible with the analysis carried out using the Pareto diagrams, which are found to be applicable only in those cases where 20% or 30% of the causes have produced 80% of the effects. Therefore, making a detailed analy- sis for each of these KPIs, a global analysis is deduced, identifying the equipment that more frequently falls into 20% or 30% of the causes. Also, in the case of COSR, the critical issues present are analyzed and made possible by focusing on the equipment for which COSR is more than 10%. Finally, the same reasoning is carried out for the PPM, which, regarding preventive maintenance, focuses on the type of intervention priority, such as equipment of priorities I, II, III. For example, it is reported as an average across the fleet of pressure such as therapy units for that particular manufacturer of equipment, considered from the critical- ity analysis of the Uptime, MTTR and MTBF (Table 2). TABLE 1. Comparison with the Study by Tiwari and Tiwari. 21 J Global Clinical Engineering Vol.4 Issue 1: 2021 Cecchini, Masselli, Silvestri: Evaluation and Optimization of CES Performances: Application of the Pareto Principle to KPIs This paper summarizes medical equipment repair and maintenance benchmark indicators that can be used. Therefore, the clinical engineering profession must develop and use indicators that accurately reflect the true costs and quality of medical equipment repair and maintenance. The way to analyze the results obtained is, when pos- sible, using Pareto diagrams. They help to break down the big problems into smaller problems and to determine which are the main factors that cause them; to help to focus on the most important causes and to set priorities, using the time available more effectively help to link the causes with the effects. This methodology makes it possible to have precise information on the critical equipment that will then be replaced or repaired correctly, which will be taken through work experience and information. TABLE 2. Analysis of Critical Issues through the Pareto Diagram for Uptime, MTTR and MTBF: Types of Equipment that Fall Within 20% or 30% of the Causes Performance measurement of clinical engineering departments in hospitals using these indicators will get more accurate and fairer performance evaluation. We will be able to find the real reasons for failure and improve performance. Further analysis may be required to better define creating a standard and substantive performance evaluation benchmarks and solve it. REFERENCES 1. Project Management Europa. Executive Dashboards and KPI: To Know to Decide. Available at: http://www. projectmanagementeuropa.com/cruscotti-direzionali- e-kpi-conoscere-per decidere/ 2. Cohen T, Bakuzoinis C, Friedman SB, and Roa RL. Benchmark indicators for medical equipment re- pair and maintenance. Biomed Instrument Technol 1995;308–21. 3. Ouda BK, Ayman M. An integrated evalutation for the performance of clinical engineering department. IEEE Engineering in Medicine and Biology Society 2014;1:3110–3. 4. Tiwari A and Tiwari A. Performance evaluation of out- sourced medical equipment maintenance service in a tertiary care hospital. Int J Sci Res Pub 2014;4(9):1–9. 5. Joint Commission International. Joint Commission International Accreditation Standards for Hospitals; 2017. Available at: https://www.jointcommissioninter- national.org/-/media/jci/jci-documents/accreditation/ hospital-and-amc/jci-standards-only_6th-ed-hospital.pdf 6. Mayo Clinic. Equipment Management Program Rev 6-08. Rochester: Au. 7. whatissixsigma.net. Pareto Chart and Analysis. Available at: http://www.whatissixsigma.net/ pareto-chart-and-analysis/ http://www.projectmanagementeuropa.com/cruscotti-direzionali-e-kpi-conoscere-per http://www.projectmanagementeuropa.com/cruscotti-direzionali-e-kpi-conoscere-per http://www.projectmanagementeuropa.com/cruscotti-direzionali-e-kpi-conoscere-per https://www.jointcommissioninternational.org/-/media/jci/jci-documents/accreditation/hospital-and-amc/jci-standards-only_6th-ed-hospital.pdf https://www.jointcommissioninternational.org/-/media/jci/jci-documents/accreditation/hospital-and-amc/jci-standards-only_6th-ed-hospital.pdf https://www.jointcommissioninternational.org/-/media/jci/jci-documents/accreditation/hospital-and-amc/jci-standards-only_6th-ed-hospital.pdf http://whatissixsigma.net http://www.whatissixsigma.net/pareto-chart-and-analysis/ http://www.whatissixsigma.net/pareto-chart-and-analysis/