Hrev_master Abstract Key Performance Indicators (KPI) regard key areas of a partic- ular service which needs to be evaluated. Within the health system, they represent specific and measurable elements and are based on standards generally set by the scientific literature. KPI can be used to assess performances in different areas and achievement of goals. To develop and calculate a set of KPI in the Italian Region Friuli Venezia Giulia (1 million inhabitants) as an audit tool for the regional Emergency healthcare system, with a focus on three time- dependent conditions: trauma, acute myocardial infarction, and stroke. To develop the set, a modified Delphi process was applied among Emergency care experts in Friuli Venezia Giulia. Then, the indicators were calculated from anonymous administrative health databases (Emergency Medical System, Emergency Department, hospital discharge, cardiac catheterization laboratory). Databases could be linked with each other at the individual level through a univocal stochastic key. Sensitivity analyses were conducted where different results were expected using different databases. Sixty-one indicators were calculated for the year 2018. Five indicators were summary descriptive measures, 10 were outcome measures, the others were process indicators; 20 were specific on acute myocardial infarction, 12 on trauma, 4 on stroke. Values for some indicators varied depending on the data source. These KPI provided new and interesting information and are used for periodic audit purposes in Friuli Venezia Giulia. Higher quality, completeness and richness of the administrative databases should be promoted to further increase the value of the indicators. Introduction Performance monitoring is a continuous process of data collec- tion and analysis to assess whether a particular health service meets its targets and satisfies the desired standards. The feasibility and usefulness of such process depend on the availability of good quality health information and on the existence of a systematic and consistent data collection system. Key Performance Indicators (KPI) are frequently used to support the performance monitoring of a certain organization, with the final goal of contributing to the improvement of quality and safety. In short, they are performance indicators regarding key areas of the service which needs to be evaluated. They represent specific and measurable elements of the health system and are based on standards set by the scientific liter- ature or experts’ consensus when the scientific evidence is not available or insufficient. KPI do not improve the quality of a sys- tem by themselves, but they can be used to identify areas where the performance is good and those where an improvement is needed and for benchmarking. In addition, they can be used to assess achievement of goals with a linked payment. In Emergency healthcare, services provided are various and include numerous procedures. Thus, any suite of indicators should include several items to reflect in a balanced manner not only the timeliness of care but also quality and health outcomes. A system- atic literature review on the Emergency Department performance measures,1 in agreement with Kelman e Friedman,2 in fact, under- lined the potential danger of focusing on a single time-dependent indicator, which does not necessarily correspond to high levels of quality and may originate disfunctional behaviors, although there is no agreement on the ideal number of indicators to be included in the suite. According to the review, time intervals and patient-relat- ed measures are the most commonly used indicators for the Emergency Department (ED), in the attempt to address the [page 102] [Emergency Care Journal 2020; 16:8910] Emergency Care Journal 2020; volume 16:8910 Correspondence: Francesca Valent, SOC Istituto di Igiene ed Epidemiologia Clinica, Azienda Sanitaria Universitaria Friuli Centrale, Via Colugna 50, 33100 Udine, Italy; Tel.: +39 0432 559158; E-mail: francesca.valent@asufc.sanita.fvg.it Acknowledgments: The authors thank Dr. Mario Calci for coordinating the CREU working group meetings, Dr. Andrea Perkan, Dr. Chiara Rocco, Dr. Giorgio Berlot, Dr. Vincenzo Mione for participation in the CREU working group, and Dr. Michele Chittaro for the discussions on the EMS database. Funding: This work was funded by the Italian Ministry of Health (Ministero della Salute) and co-funded by the participating Regions Lazio, Friuli Venezia Giulia, Piemonte, Emilia-Romagna, Lombardia, Calabria through the “Programma di rete Ricerca Finalizzata Bando 2016 - NET-2016-02364191”. Ethics approval and consent to participate: The manuscript does not contain any individual person's data in any form. Conflict of Interest: the authors declare no potential conflict of interests. Key words: key pefromance indicator; emergency care; acute myocar- dial infarction; trauma; stroke. Received for publication: 22 February 2020. Accepted for publication: 18 May 2020 This work is licensed under a Creative Commons Attribution 4.0 License (by-nc 4.0). ©Copyright: the Author(s), 2020 Licensee PAGEPress, Italy Emergency Care Journal 2020; 16:8910 doi:10.4081/ecj.2020.8910 Emergency Key Performance Indicators in the Italian Region Friuli Venezia Giulia Francesca Valent,1 Antonio Di Chiara,2 Stefano Di Bartolomeo3 1SOC Istituto di Igiene ed Epidemiologia Clinica, Azienda Sanitaria Universitaria Friuli Centrale; 2Direzione centrale salute, politiche sociali e disabilità, Regione autonoma Friuli Venezia Giulia; 3SOC Anestesia e Rianimazione 1, Azienda Sanitaria Universitaria Friuli Centrale, Udine, Italy Non -co mmerc ial us e o nly Institute of Medicine’s six quality domains: safety, effectiveness, patient-centeredness, timeliness, efficiency, fairness.3 In this area, in the past twenty years, various KPI suites, including from about 30 to about 100 indicators each, have been developed in different countries, initially regarding the intra-hospi- tal emergency care, and subsequently also pre-hospital emergency.4-9 In most cases, a preliminary literature review was carried out to set out the knowledge bases and then a Delphi approach with experts knowing the local context and issues was used to select the indicators.10 The selection process may affect the usefulness of the indicators. In an attempt to overcome the arbitrariness often affect- ing the selection of KPI, a checklist has been proposed to critically assess the indicators.11 In fact, a systematic review12 found that most KPI developed for the ED had limited evidence according to the criteria of the Oxford center for evidence-based medicine.13 In particular, it is unclear whether the process indicators, which are the most used and studied, may have an impact on the healthcare outcomes.14 Impact of KPI on patients’ outcomes is a very impor- tant aspect to consider. There are examples of use of KPI with practical implications. For example, the Iranian Ministry of Health used some KPI to assess the performance of the national ED before and after a health reform.15 Analogously, a set of KPI regarding waiting times in the ED has been measured routinely since 2012 in South Africa and is being used to conduct biannual audits, allowing the detection of an improvement.16 Also, recent literature was published on indicators regarding the performance of the Emergency Medicine Communication Centre, a key step in the survival chain for patients who seek pre-hospital aid.17-19 The Italian Region Friuli Venezia Giulia (FVG), located in the North-East of the county, has an area of 7862 Km2, including a Northern mountains area, a central plain area, and a Southern coastal area, and a population of approximately 1,200,000 (Figure 1). In this Region, the emergency health care is regulated by the Regional Health Emergency Plan, issued in 201520and gradually implemented since 2016. The Plan established the requirements of the unified Regional EMS headquarters and call center, the number and types of EMS response units, including one helicopter, the characteristics of the hub and spoke ED, the role of networks for the management of time-dependent conditions (cardiac, stroke, trauma), and the need to develop indicators to monitor performanc- es, in addition to the few generic indicators already available from national outcome evaluation programs.21,22 In 2016, a process for developing a set of indicators had started, under the supervision of the Regional Emergency-Urgency Committee (CREU) of FVG. However, due to changes in the regional top health management and to lack of dedicated resources, the indicators had never been calculated up to 2019. Since April 2019, the FVG Region has been in charge of a work package within a network research program funded by the Italian Ministry of Health and co-funded by 6 Italian Regions on the effectiveness of Audit & Feedback strategies to improve healthcare practice and equity in various clinical and organization- al settings (EASY-NET; http://easy-net.info/). The work package lead by FVG regards the organizational setting of emergency health care, with a specific focus on three time-dependent condi- tions (trauma, Acute Myocardial Infarction (AMI), stroke). Within the EASY-NET project, we decided to take advantage of the work led by CREU and to use the relevant indicators identified in the process carried out in 2016 as an audit tool assessing the perform- ance of the regional Emergency system. The indicators were actu- ally calculated from April to July 2019 and a report containing the results of this KPI-based audit was e-mailed as a feedback to the heads of 76 regional clinical units and sub-units acting in the emer- gency management of trauma, AMI, and stroke in October 2019. In this article, we describe the suite of KPI used as audit tool within the EASY-NET project and discuss issues regarding their validity and potential usefulness. Materials and Methods Development of Emergency KPI for the Friuli Venezia Giulia Region In 2016, to identify a suite of Emergency KPI to be used in FVG, a working group had been appointed by the CREU of FVG. At that time, there was no specific interest in particular emergency areas. The working group members had been chosen by CREU in agreement with the Local Health Agencies of FVG based on field of experience and time availability to participate in periodic meet- ings. The group, finally including two ED physicians, one cardiol- ogist, one anesthesiologist, and one epidemiologist, had designed and applied a modified Delphi process similar to the one described by Wakai et al.,6 Madsen et al.,7 and Murphy et al.8 In short, each working group member conducted a non-systematic review of the scientific literature and of the updated guidelines published by rel- evant scientific societies to identify indicators regarding both pre- hospital and hospital emergency care. The different lists were then put together into a unique one which was discussed to exclude indicators not applicable to the Regional context or unfeasible due to lack of necessary data. A final list of 134 indicators was agreed upon and was used to feed round I of the Delphi process. Delphi panelists (n=49) were the heads of clinical units of anesthesiology and intensive care unit, cardiology, neurology, and Emergency Departments of the FVG Region. After a pre-test of the questionnaire, it was sent by the CREU working group to the institutional e-mail addresses of the panelists. The e-mail contained a weblink to an online survey. The panelists were asked to rate each of the 134 potential KPI Article Figure 1. Location of the Friuli Venezia Giulia Region, Italy. [Emergency Care Journal 2020; 16:8910] [page 103] Non -co mmerc ial us e o nly using a 5-point Likert scale (from1: strongly disagree to 5: strongly agree) by clicking on the corresponding answer. Indicators receiv- ing a score of 4-5 by at least 70% of responding panelists were included in the final list. A short presentation of round I results was given to the CREU assembly, showing the proportion of panelists agreeing to each proposed indicator. New potential indicators pro- posed by panelists in round I were also described and added to the list of indicators to be rated in round II, which also included indi- cators that had not reached agreement in round I. For round II, the same panelists were asked to rerate those indicators plus the addi- tional ones suggested by their colleagues. The phases of the process, which had started on May 10th, 2016, along with dates, participation rates, and summary outcomes are described in as fol- lows: i) Review of scientific literature and guidelines regarding Emergency KPI to identify potential indicators; ii) Assessment of relevance of potential indicator to the local context; iii) Assessment of feasibility of calculation of each potential indi- cator based on availability of information in the administra- tive databases of the Regional Health Information System (RHIS), or on the existence of a plan to promote the informa- tion collection in the RHIS; iv) Initial list of 134 indicators: Type (14 summary descriptive measures, 20 structures, 80 processes, 20 outcomes) and Topic (57 general, 19 traumas, 53 cardiovascular, 5 neurolog- ical); v) Identification of a panel of 49 experts for Delphi process (healthcare professionals from all the geographical areas of the Region and from all the relevant disciplines involved in prehospital and hospital Emergency care); vi) Design of questionnaire for round I of Delphi process, pre-test of the questionnaire on a sample of 4 experts; modification of the questionnaire, and upload of the final version of a web form including the 134 indicators; vii) Round I of Delphi process: a link to the web questionnaire and accompanying letter of presentation was sent to the experts; those willing to participate had to indicate the level of useful- ness of each proposed indicator on a Likert-like scale with a score from 1 (not useful) to 5 (very useful). Comments could be added. New indicators could be proposed. All data were recorded anonymously. Duration: August 2nd to September 20th, 2016; viii) Round I analysis: participation rate 43%. Indicators selected (proportion of participants who assigned a score of 4 or 5 > 70%): 66; ix) Round II of Delphi process: new form was sent to the same 49 subjects of round I including 68 indicators not adopted after round I and 17 new indicators proposed by participants in round I. Duration: October 27th November 11th, 2016; x) Round II analysis: participation rate 20%. Additional indica- tors selected (proportion of participants who assigned a score of 4 or 5 > 70%): 47; xi) Final set of 113 indicators approved by the Regional Emergency-Urgency Committee (CREU): Type (8 summary descriptive measures, 14 structures, 76 processes, 15 out- comes) and Topic (39 general, 15 traumas, 47 cardiovascular (29 on AMI), 7 strokes, 5 neonatal). After the approval of the final pool of desirable indicators by CREU, efforts were made to integrate into the Regional Health Information System (RHIS) all the necessary information which was not yet routinely available. Selection of KPI for the EASY-NET audit In the EASY-NET project, we planned to use for the Emergency system audit only indicators whose calculation was entirely based on administrative databases, because we wanted to be able to calculate them periodically on a yearly basis, with no need to rely upon ad hoc personal data transmission. Of the final pool of 113 desirable indicators according to the process carried out by CREU in 2016, only 90 regarded the conditions of interest for the EASY-NET audit, the others dealing mainly with cardio- vascular conditions other than AMI and neonatal emergency trans- ports. At the time of writing this article, 52 of those KPI were entirely calculable with information contained in the RHIS. No structure indicators are included among those 52, since currently they can only be calculated through information provided separate- ly by each hospital and are not calculable from RHIS databases. The RHIS includes various anonymous health-related databas- es, which can be deterministically linked with each other at the individual level through an anonymous univocal stochastic key. Databases relevant to the calculation of Emergency KPI include the list of potential healthcare beneficiaries and vital status, the Emergency Medical System (EMS) database, the ED database, the hospital discharge records database with discharge diagnoses and procedures, the hemodynamics database from the cardiac catheter- ization laboratory. These databases cover all the regional hospitals (hospital discharge records, ED and hemodynamics data) and the whole regional area (EMS). All the variables in the hospital dis- charge records and ED data are virtually complete, whereas in the hemodynamics database missing data in the variables of interest affect less than 10% of observations. In the EMS database, tele- phone call, alert and transport data are registered mostly automat- ically and are virtually complete; on the other hand, patient record is paper-based and, due to lack of human resources, it is not always computerized. Thus, patient data may be incomplete. In addition, there may be no demographic information if the patient is not iden- tified by EMS personnel. The 52 indicators calculable with RHIS data were included in the pool of KPI for the EASY-NET audit. Nine additional KPI were included in the EASY-NET pool: i) 4 indicators aimed at monitoring the performances of the newborn unique Regional EMS call center, similar to those proposed by Van Buren et al.;17 ii) 3 indicators to assess the use of the EMS dispatch, which was started after the end of the above-described Delphi process; iii) 2 KPI suggested by experts after the end of the Delphi process, con- sidered useful for audit purposes: patients who leave the ED with- out being seen or during treatment23 and patients contemporane- ously waiting to be seen by a doctor as a possible measure of an ED’s workload. The list of the 61 EASY-NET KPI is shown in Appendix. Although we also conducted analyses stratified by geographic area or health center, as appropriate to better describe the hub and spokes network, in this article we present only data for the overall Friuli Venezia Giulia Region. For some indicators, where the use of different RHIS databases as sources of information returned substantially different esti- mates, sensitivity analyses were conducted and values obtained using different sources of data or criteria are presented. The detailed algorithms and sources of data used to calculate each indicator are presented as Supplementary File (in Italian). All the analyses are referred to the year 2018, the first entire year with EMS and cardiac catheterization laboratory data after the Health Emergency Plan was issued. All the analyses were conducted using SAS v9.4 (SAS Institute Inc., Cary, NC. USA). Article [page 104] [Emergency Care Journal 2020; 16:8910] Non -co mmerc ial us e o nly Ethical considerations The authors assert that all procedures contributing to this work comply with the ethical standards of the relevant national and insti- tutional committees on human experimentation and with the Helsinki Declaration of 1975, as revised in 2008. Since this analy- sis was based on anonymous administrative data, patient informed consent and Ethical Committee approval were not required in Italy. Patient and public involvement Patients or the public were not involved in the design, or con- duct, or reporting, or dissemination plans of our research. Results Appendix and Figure 2 show the values of the 61 emergency KPI calculated for year 2018 for the Friuli Venezia Giulia Region. Five indicators are summary descriptive measures, 10 KPI meas- ure outcomes, all the others are process indicators. Twenty KPI regard specifically AMI, 12 are on major trauma, 4 on stroke. For 12 indicators, different data sources or different criteria (see Supplementary File) produced substantially different results and more than one value is reported. Discussion This article describes the first large set of KPI used for moni- toring the performances of the emergency health care system in the 1,200,000-inhabitant Region Friuli Venezia Giulia. Their calcula- tion is entirely based on electronic, mostly administrative, databas- es integrated in the Regional Health Information System, which offer a number of advantages. In fact, the use of these data allows to conduct timely audits of the emergency care system in a rela- tively fast way: all the information is available with a relatively short delay (months) and, once the calculation algorithms have been defined, the data can be processed quickly. This offers further advantages, such as objectivity of information, uniformity of crite- ria across the regional geographic area, and reproducibility of cal- culations in time. In addition, the RHIS of FVG allows the imple- mentation of indicators that encompass bit pre-hospital and in-hos- pital emergency care. Nonetheless, the use of RHIS has also some limitations. Some limitations depend on the type of information contained. In fact, no structural indicators could be calculated using this source of data. The structural indicators identified in the Delphi process will need active, periodical interviews with all the hospitals in the Region to collect the necessary information (e.g., the number of doctors or nurses employed in the ED of each hospital, or the number of nurs- es in each shift in each of the regional Intensive Care Units). Other limitations depend on the quality and completeness of the available databases. Among the KPI presented in this article, there were sub- stantial discrepancies between the values obtained different sources of data. For example, we observed a more than 3-fold variation in the values of indicators regarding major trauma when different data- bases and criteria were used: we estimated 80 annual major trau- mas per 100,000 inhabitants considering patients presenting to the ED and admitted to ICU, however the rate decreased to 46 when only the subset of patients with EMS medical data were counted, and increased to 250 when we based the calculation on EMS trans- ports only (alert with red code, the highest priority, and yellow/red return codes). These estimates are too distant from one another to allow a comparison with other international rates, such as the one from British Columbia, where a trauma (Injury Severity Score (ISS) ≥9) admission rate per 100,000 population was 113.75 in 2016.24 Nonetheless, since all 3 algorithms have been defined, trends using all 3 methods can be monitored in FVG. In this Region, there is no trauma center and no trauma score is routinely registered in all the relevant databases of the RHIS. No trauma score is reported in the hospital discharge record or is routinely cal- culated at the ED. The Revised Trauma Score (RTS) is calculated by EMS personnel and reported on the EMS patient record, how- ever, unlike the EMS call, alert and transport data whose registra- tion is mostly automatic and virtually complete, the EMS patient record is paper-based and, due to lack of human resources, not always computerized. For example, in 2018, of 89,286 patients who were transported to the ED by an EMS ambulance or helicop- ter according to the ED records, only for 58,132 (65%) the corre- sponding EMS patient record was found in the RHIS of FVG. Thus, for example, we must be aware that at this moment the pro- portion of STEMI patients transported to hub and spoke hospitals by the EMS is likely underestimated. This is certainly one result of the audit: there are evident gaps in EMS data registration and they need to be filled, otherwise the information that is being collected cannot be used effectively to monitor the system performances. An alternative strategy to improve completeness of data registration would be to have rele- vant variables (e.g., a trauma score) included in the hospital dis- charge database. This would be a major challenge, since the hospi- tal discharge data set is defined at a national level and is common to all the Italian hospitals. However, as additional variables such as systolic blood pressure, the ejection fraction, or serum creatinine have been included in the new Italian hospital discharge data set since 2017 (after the Ministerial Decree 261 of 2016), further clin- ical data could be required in future versions. Another example of variation in an indicator value depending on the source of data regards PCI in patients with STEMI: hospital discharge records identified approximately 20-80% more PCI in patients admitted with a STEMI than did the hemodynamics data- base from the cardiac catheterization laboratory, suggesting either that the latter database may be incomplete or that PCI procedures found only in the hospital discharge data are not primary PCI. The Article Figure 2. Number of contemporary busy EMS call-center by hour (indicator id 53). [Emergency Care Journal 2020; 16:8910] [page 105] Non -co mmerc ial us e o nly [page 106] [Emergency Care Journal 2020; 16:8910] second hypothesis is supported by the fact that if we consider also non-primary PCI from the cardiac catheterization laboratory data- base, the proportion of overall STEMI treated with PCI increases from 48% to 60%, which approaches the 67% of cases resulting from hospital discharge data. In addition, if we restrict PCI from hospital discharge data to procedures performed within one day from admission (more likely to be primary), the proportion of overall STEMI treated with PCI is 55%, a value which is closer to the one obtained from hemodynamics data. One piece of informa- tion which would be extremely useful in interpreting data on pri- mary PCI would be the time of symptoms onset. Although this important information is often collected by EMS personnel, it is rarely recorded in the electronic database, so we could not analyze it. No database is exempt from quality issues. For example, in the ED, despite completeness of registration, the discharge diagnoses may be imprecise, both because some elements needed for an exact diagnosis may not be available at that point, and because the ED doctor may not devote enough time and attention to the coding step. To explore the issue of ED diagnoses, those recorded as ICD- 9 codes will be checked against free text fields compiled by the ED doctor during or at the end of the visit, which are a useful resource for enriching the ED information. Hospital discharge diagnoses may also be miscoded, as shown in this Region for some particular conditions.25 For the identifica- tion of AMI in the KPIs, however, we decided to use the hospital discharge records because we had already verified that they are accurate: using troponin concentration values as the gold standard, hospital discharge main diagnosis code has very high positive pre- dictive values, although sensitivity may be suboptimal.26 Despite data quality issues, the KPI provide interesting infor- mation. For example, although different numbers of major traumas were estimated using different data sources, the distribution of time on scene was similar regardless of the source of data: in FVG, EMS remains on the scene with half trauma patients for 19-20 min- utes or more. These indicators calculated in FVG are useful not only for monitoring performances in the Region, but also allow a comparison with other national and international systems ruled by the same concepts as ours (i.e., stay and play for trauma care), which is fundamental for benchmarking. The set of indicators developed in FVG should be improved not only promoting higher quality and completeness of the existing data, but also enriching the RHIS with additional databases so that more indicators relevant to EMS performance can be calculated. In fact, what is peculiar to the FVG indicators set is that most of them are not simply based on data that are automatically captured by the system (e.g., dates and times), but used all the information avail- able, even if potentially affected by human recording or coding error. Although this implied that linkage of different databases and complex data management activities had to be carried out to clean and validate data and that some uncertainty remains in the esti- mates of various indicators, it had the advantage of allowing a more comprehensive audit of the whole emergency system in the Region. For example, at the national level, the unique available official indicator of mortality after AMI (in FVG, approximately 6.9% at 30 days)21 does not distinguish between STEMI and NSTEMI. With local data, on the other hand, we could look at the two different types of disease. In this article, KPI results were presented only briefly. However, within the Region, each single indicator was studied more in depth and all available details were provided to clinicians and policy makers to maximize usefulness. For example, it is inter- esting to notice that the proportion of patients admitted to the hos- pital with a main discharge diagnosis of STEMI who are treated with primary PCI is highly variable depending on the specific ICD-9-CM diagnosis code, ranging from 80% in case of AMI of the lateral wall (410.51, 51 total hospitalizations) or 70% in case of AMI of the inferior wall (410.41, 146 total hospitalizations) to less than 5% in case of AMI of unspecified site (410.91, 178 total hos- pitalizations), or that the median age of patients treated with pri- mary PCI was 12 years lower (66 years) than age of patients not undergoing primary PCI (78 years). In addition, in the feedback report that was provided to the heads of the regional clinical units involved in emergency care, most indicators were also stratified by geographical area or by center, to address the issue of geographic equity of care. After the distribution of the feedback report, various health professionals expressed satisfaction with this initiative since it was the first time they received an external feedback on their activities. They were aware of the questionable validity of some data, nonetheless, they believed it was useful to have many different aspects of emergency care depicted at the same time. According to the EASY-NET project plan, the next feedback, regarding emer- gency care activities during 2019, will be provided in a different way, though face-to-face presentations and workshops. In such events, the indicators will be divided by objectives and macro areas of interest, to focus the discussion on particular issues. As an alternative, different small events can be organized, one for each area of interest. In the future, a dashboard could be implemented where the KPI, presented by macro area, are accessible any time by emergency care professionals to monitor their activity. The indi- cators that are already included in the list of ministerial LEAs (which are used to monitor whether essential levels of care were provided to the population)27 will be promoted and particular emphasis will be given to improving the quality of data that are needed to calculate them. A continuous work is being conducted to deepen the targets of the indicators based on pre-established objectives. For example, to better assess the fulfillment of objectives regarding thrombolysis, we will need further indicators taking into account the date and time of symptoms onset and other clinical characteristics. New indicators will be included in the set as soon as the necessary infor- mation becomes available. The fact that the initial CREU working group did not include a neurologist is a limitation of our work, since the initial list used to feed the Delphi process might have been skewed away from neurological issues, which, in fact, were the least represented (only 5 indicators vs 53 cardiologic and 19 trauma indicators). Delphi panelists had a chance to propose addi- tional indicators between round I and round II and no one suggest- ed further indicators on stroke. Unfortunately, to maximize partic- ipation in the Delphi surveys, panelists could remain anonymous and we do not know the demographics and clinical area of those who responded. To overcome this limit, we will meet the regional neurology experts before starting the calculation of KPIs for the next EASY-NET audit. They will indicate clinically relevant objectives in their area of interest and help defining additional indicators to include in the regional set. Initiatives to improve data quality and richness should be con- current with actions to improve emergency care. Article Non -co mmerc ial us e o nly [Emergency Care Journal 2020; 16:8910] [page 107] References 1. Sørup CM, Jacobsen P, Forberg JL. Evaluation of emergency department performance - a systematic review on recommend- ed performance and quality-in-care measures. Scand J Trauma Resusc Emerg Med 2013;21:62. doi: 10.1186/1757-7241-21- 62. 2. Kelman S, Friedman JN. Performance improvement and per- formance dysfunction: an empirical examination of distor- tionary impacts of the emergency room wait-time target in the English National Health Service. JPART 2009;19:917–46. doi: 10.1093/jopart/mun028 3. Quality Health Care in America, Institute of Medicine. Crossing the quality chasm: a new health system for the 21st century. National Academy Press Washington, D.C.; 2001. 4. Beattie E, Mackway-Jones K. A Delphi study to identify per- formance indicators for emergency medicine. Emerg Med J 2004;21:47-50. 5. Schull MJ, Hatcher CM, Guttmann A, Leaver CA, Vermeulen M, Rowe BH, Anderson GM, Zwarenstein M. Development of a Consensus on Evidence-Based Quality of Care Indicators for Canadian Emergency Departments. ICES Investigative Report. Toronto: Institute for Clinical Evaluative Sciences; 2010. 6. Wakai A, O'Sullivan R, Staunton P, Walsh C, Hickey F, Plunkett PK. Development of key performance indicators for emergency departments in Ireland using an electronic modi- fied-Delphi consensus approach. Eur J Emerg Med 2013;20:109-14. doi: 10.1097/MEJ.0b013e328351e5d8] 7. Madsen MM, Eiset AH, Mackenhauer J, Odby A, Christiansen CF, Kurland L, Kirkegaard H. Selection of quality indicators for hospital-based emergency care in Denmark, informed by a modified-Delphi process. Scand J Trauma Resusc Emerg Med 2016;24:11. doi: 10.1186/s13049-016-0203-x. 8. Murphy A, Wakai A, Walsh C, Cummins F, O'Sullivan R. Development of key performance indicators for prehospital emergency care. Emerg Med J 2016;33:286-92. doi: 10.1136/emermed-2015-204793 9. Haugland H, Rehn M, Klepstad P, Krüger A; EQUIPE-collab- oration group. Developing quality indicators for physician- staffed emergency medical services: a consensus process. Scand J Trauma Resusc Emerg Med 2017;25:14. doi: 10.1186/s13049-017-0362-4. 10. Hsu CC, Stanford B. The Delphi Technique: Making Sense of Consensus. Practical Assess Res Eval 2007;12:1-8. 11. Jones P, Shepherd M, Wells S, Le Fevre J, Ameratunga S. Review article: what makes a good healthcare quality indica- tor? A systematic review and validation study. Emerg Med Australas 2014;26:113-24. doi: 10.1111/1742-6723.12195. 12. Madsen M, Kiuru S, Castrèn M, Kurland L. The level of evi- dence for emergency department performance indicators: sys- tematic review. Eur J Emerg Med 2015;22:298-305. doi: 10.1097/MEJ.0000000000000279 13. Howick J, Phillips B, Ball C, Sackett D, Badenoch D. Oxford centre for evidence-based medicine levels of evidence. Oxford: University of Oxford; 2009. 14. Parkinson JE. Throwing good money after bad: should research into Emergency Department performance indicators be prioritized? Eur J Emerg Med 2016;23:395-6. doi: 10.1097/MEJ.0000000000000348 15. Mohammadi E, et al. Emergency department performance evaluation indicators in pre-and post-health care reform in Kermanshah public hospitals. Res J Med Sci 2016;10:54-59. 16. Cohen K, Bruijns S. Describing key performance indicators for waiting times in emergency centres in the Western Cape Province, South Africa, between 2013 and 2014. S Afr Med J 2018;108:579-84. doi: 10.7196/SAMJ.2018.v108i7.12969. 17. Van Buuren M, Jan Kommer G, Rob van der Mei R, Bhulai S. EMS call center models with and without function differentia- tion: A comparison. Oper Res Health Care 2017;12:16–28. 18. Montassier E, Labady J, Andre A, Potel G, Berthier F, Jenvrin J, et al. The effect of work shift configurations on emergency medical dispatch center response. Prehosp Emerg Care 2015;19:254–9. 19. Penverne Y, Leclere B, Labady J, Berthier F, Jenvrin J, Javaudin F, Batard E, Montassier E. Key performance indica- tors' assessment to develop best practices in an Emergency Medical Communication Centre. Eur J Emerg Med 2018;25:335-40. doi: 10.1097/MEJ.0000000000000468 20. Regione autonoma Friuli Venezia Giulia. Piano dell’Emergenza Urgenza della Regione Friuli Venezia Giulia. Available from: http://mtom.regione.fvg.it/storage/ /2015_1674/Allegato%201%20alla%20Delibera%201674- 2015.pdf. Accessed: June 20, 2019. 21. Agenzia Nazionale per I Servizi Sanitari Regionali. Programma nazionale Esiti 2018. Available from https://pne.agenas.it/emur/default_emur.php 22. Laboratorio MeS - Istituto di Management Scuola Superiore Sant'Anna. Il sistema di valutazione della performance dei sis- temi sanitari regionali. Available from: http://performance. sssup.it/netval/start.php 23. Crilly J, Bost N, Thalib L, Timms J, Gleeson H. Patients who present to the emergency department and leave without being seen: prevalence, predictors and outcomes. Eur J Emerg Med 2013;20:248-55. doi: 10.1097/MEJ.0b013e328356fa0e 24. Trauma Services BC. 2017-18 Annual Report. Available from: h t tp : / /www.phsa .ca /our- se rv ices - s i t e /Documents / TSBC_Annual%20Report_2017-18.pdf 25. Valent F, Deroma L, Cocconi R, Picierno A, Sartor A. Hospital discharge diagnoses in patients with positive blood cultures in an Italian academic hospital. Ann Ist Super Sanita 2019;55:19- 25. doi: 10.4415/ANN_19_01_05 26. Di Chiara A, Clagnan E, Valent F. Epidemiology and mortality in an Italian region after the adoption of the universal defini- tion of myocardial infarction. J Cardiovasc Med (Hagerstown). 2020;21:34-9. doi: 10.2459/JCM.0000000000000893 27. Ministero della Salute. Servizio Sanitario Nazionale: I LEA. Available from: http://www.salute.gov.it/portale/lea/dettaglio ContenutiLea.jsp?lingua=italiano&id=1300&area=Lea&menu =leaEssn. Article Non -co mmerc ial us e o nly