Archives of Academic Emergency Medicine. 2022; 10(1): e71 OR I G I N A L RE S E A RC H Usability of Emergency Department Information System Based on Users’ Viewpoint; a Cross-Sectional Study Sohrab Almasi1, Nahid Mehrabi1∗, Farkhondeh Asadi2, Mahboobeh Afzali3 1. Department of Health Information Technology, School of Paramedical Sciences, AJA University of Medical Sciences, Tehran, Iran. 2. Department of Health Information Technology and Management, School of Allied Medical Sciences, Shahid Beheshti University of Medical Sciences, Tehran, Iran. 3. Department of Emergency Medicine, School of Paramedical Sciences, Aja University of Medical Sciences, Tehran, Iran. Received: June 2022; Accepted: July 2022; Published online: 1 September 2022 Abstract: Introduction: The emergency department is of special importance due to its emergency and vital services, the high volume of referrals, and the patients’ physical condition. Thus, it requires a well-designed information system with no usability problems. This study aimed to evaluate the usability of the emergency department information system from users’ perspectives. Methods: This was a cross-sectional study. The research setting was the emergency department of 3 hospitals. The research instrument was a 37-item questionnaire adapted from the USE and ISO Metrics questionnaires, consisting of five dimensions measuring the usefulness of the system, ease of use, ease of learning, user satisfaction, and suitability for the task. The content validity of the questionnaire was examined using the content validity ratio and content validity index, and its reliability was assessed using Cronbach’s alpha (α = 0.88). Results: Fifty questionnaires were administered in the three hospi- tals, and the response rate was 80%. According to the findings, 55% of the respondents were female. The highest mean scores belonged to usefulness in emergency department information system (EDIS) A, ease of use in EDIS B, ease of learning in EDIS A, user satisfaction in EDIS C, and suitability for the task in EDIS A. According to the usability evaluation criteria, ease of learning (3.66 ± 0.74), usefulness (3.53 ± 0.87), and suitability for the task (3.47 ± 0.96) received the highest scores, and the lowest scores belonged to user satisfaction (3.29 ± 1.01) and ease of use (3.12 ± 1.00). Conclusion: In terms of usability criteria, the emergency department information system is at a relatively good level. The usability of these systems can be further enhanced by considering the users’ working needs, improving software flexibility, customizing the software, using data visualization tools, observing consistency of features and standards, and increasing the quality of information and system services. Keywords: Informtation Systems; Hospital Information Systems; Emergency Service, Hospital Cite this article as: Almasi S, Mehrabi N, Asadi F, Afzali M. Usability of Emergency Department Information System Based on Users’ View- point; a Cross-Sectional Study. Arch Acad Emerg Med. 2022; 10(1): e71. https://doi.org/10.22037/aaem.v10i1.1635. 1. Introduction The emergency department is characterized by several major features, including unpredictable admission, patients’ phys- ical condition, and time constraints to perform diagnostic- therapeutic processes (1-4). These challenges, in turn, re- sult in irrecoverable consequences such as increasing patient mortality, heavy costs, increasing waiting time and patient dissatisfaction, increasing medical errors and the incidence ∗Corresponding Author: Nahid Mehrabi; Department of Health Information Technology, Aja University of Medical Sciences, Tehran, Iran. Phone number: (+98) 21 43822453, Fax: (+98) 21 8802 8364, Email: n.mehrabi@ajaums.ac.ir, ORCID: https://orcid.org/0000-0003-2840-056X. of violence, and disrupting medical services(5-7). Therefore, compared with other departments, the emergency depart- ment requires considerable flexibility and instant planning of resources(8). Persistent collection, and correct and timely processing of data may improve emergency department management (9). Emergency department information system (EDIS), a com- ponent of the hospital information system, plays a key role in information management and care, as well as management processes of the emergency department (10-12). The EDIS enjoys numerous advantages such as improving emergency department performance (13), providing easier and faster access to patient information (14), recording much better and more accurate clinical and managerial informa- This open-access article distributed under the terms of the Creative Commons Attribution NonCommercial 3.0 License (CC BY-NC 3.0). Downloaded from: http://journals.sbmu.ac.ir/aaem S. Almasi et al. 2 tion, assisting in decision-making (14), and eliminating the limitations of paper systems such as multiple users’ simulta- neous access to information and information illegibility (15). Despite its increasing use in recent years in different coun- tries (16), the EDIS is still not widely adopted and applied (17- 19). In some studies, poor display, a lack of workflow support, reduced efficiency of the emergency department, and con- gestion have been reported as the main disadvantages (20, 21). Given the vital importance of the emergency department’s activities, the emergency department’s information systems must be free of usability problems to prevent errors. In do- ing so, systems and applications must be designed and used appropriately in accordance with scientific principles (22). One method to ensure the proper design of applications and health information systems is evaluating their usability. Us- ability deals with various features of the software, including ease of learning, efficiency, ease of use, memorization, error prevention, and user satisfaction. Evaluation also plays a piv- otal role in software development (23, 24). According to var- ious studies, it is necessary to observe usability principles in designing the user interface of the EDIS (25, 26). The usabil- ity index evaluates the performance of a product in terms of user satisfaction and increased productivity (27-30). A com- mon method of evaluating the applicability of information systems is using the standard 9241/10 ISO Metric Question- naire and the USE Questionnaire, with approved validity and reliability (30, 31). Several studies have been conducted on the evaluation of hospital information systems’ subsystems in military hospitals. Based on results, the same maturity was observed in military and civilian hospitals, i.e., the third stage of the EMRAM model. Meanwhile, the potential bene- fits of these systems were not yet fully exploited in hospitals (32, 33). The present study aimed to evaluate EDISs from the perspec- tive of users in five dimensions of usefulness of the system, ease of use, ease of learning, user satisfaction, and suitability for the task. 2. Methods 2.1. Study design and setting This was a cross-sectional study conducted in 2021 in three hospitals affiliated with AJA University of Medical Sciences, Tehran, Iran, which employed an EDIS. The questionnaires were distributed from 6 to 15 November 2021. The data were collected five days later, from 20 to 27 November. The re- search setting was the emergency departments of hospitals affiliated with AJA University of Medical Sciences. Multi-stage sampling was carried out. The goal was to in- clude the emergency department of hospitals with 100 or more active beds, which led to the selection of the three hos- pitals. The emergency department system in two hospitals had been designed by two software companies, and in the third hospital, it had been developed by the hospital. The population comprised of users of EDISs (emergency depart- ment nurses and secretaries). A sample of 50 was selected from the mentioned centers through convenience sampling. To maintain confidentiality, the identities of vendors and hospitals remained anonymous and they were labeled as EDIS A, EDIS B, and EDIS C. This study was approved by the Ethics Committee of AJA Univer- sity of Medical Sciences, Iran (IR.AJAUMS.REC.1400.215). 2.2. Participants The inclusion criteria were as follows: having > five years of work experience and 1 year of experience with the EDIS. Fifty emergency information system users met the aforemen- tioned criteria in the three hospitals. 2.3. Data collection To evaluate the usability of EDIS, the standard 9241 ISO Met- ric Questionnaire (part 10) and the USE Questionnaire were administered (appendix 1). The 9241 ISO Metric Question- naire consists of 75 questions based on seven principles: suit- ability for tasks, self-description, controllability, error tol- erance, suitability for personalization (customization), and suitability for learning (34). The validity and reliability of the 9241 ISO Metric questionnaires have been confirmed in various studies (30, 31). The USE Questionnaire includes 30 questions that assess the following dimensions: usefulness, satisfaction, ease of use, and ease of learning (31). The valid- ity and reliability of the USE Questionnaire have also been confirmed in a number of studies (35, 36). The question- naire administered in this study was a combination of the two mentioned questionnaires, which covered five dimen- sions of system usefulness, ease of use, ease of learning, user satisfaction, and suitability for the task. The questionnaire encompassed a total of 37 questions on a five-point Likert scale ranging from completely disagree (score: 1) to com- pletely agree (score: 5). The number of questions for each dimension was as follows: usefulness of the system (eight questions), ease of use (10 questions), ease of learning (six questions), user satisfaction (seven questions), and suitabil- ity for the task (six questions). Experts determined the content validity of the questionnaire using the content validity ratio (CVR) and content validity in- dex (CVI), and its reliability was assessed using Cronbach’s alpha (α = 0.88). Then, the questionnaire was administered to 50 system users. 2.4. Statistical analysis We employed the SPSS 22 software (IBM SPSS, Armonk, NY, USA) for data analysis. Descriptive tests (mean, percentage, This open-access article distributed under the terms of the Creative Commons Attribution NonCommercial 3.0 License (CC BY-NC 3.0). Downloaded from: http://journals.sbmu.ac.ir/aaem 3 Archives of Academic Emergency Medicine. 2022; 10(1): e71 Table 1: Baseline characteristics of the respondents Variable Number (%) Gender Male 18 (45.0) Female 22 (55.0) Education Level High-school diploma 3 (7.50) Associate degree 2 (5.00) Bachelor’s degree 28 (70.00) Master’s degree 5 (15.00) Doctoral degree 1 (2.50) Occupation Nursing supervisor 3 (7.50) Nurse 26 (65.00) Secretary 11 (27.50) and frequency) were used to assess the data. For data analy- sis, the means and standard deviations (SD) were calculated, and then, the systems were compared against usability crite- ria. The relationship between the dimensions of usability and participants’ variables was assessed using Pearson correla- tion coefficient or Spearman correlation coefficient. 3. Results 3.1. Baseline characteristics of participants Fifty questionnaires were administered in the three hospitals (response rate = 80%). The baseline characteristics of the re- spondents are given in Table 1. 55% of the respondents were female and 70% had a bachelor’s degree. The mean age of the respondents was 29.8 ± 8.7 years. Also, their mean work experience and mean duration of involvement with the EDIS was 8 ± 5.1 and 7.1 ± 4.3 years, respectively. 3.2. Usefulness The highest mean scores for this criterion were reported for EDIS A (3.59 ± 0.78), EDIS B (3.54 ± 1.12), and EDIS C (3.48 ± 0.73), respectively. In EDIS A, the highest score belonged to the item "EDIS saves time (3.92 ± 0.49)", and the lowest score belonged to “EDIS meets your needs (3.3 ± 0.94)". As for EDIS C, the highest score was associated with "EDIS is useful and valuable (4 ± 0.42)", and the lowest score was related to "EDIS meets your needs (2.83 ± 0.93)" (Table 2). 3.3. Ease of use The highest mean scores for this criterion belonged to EDIS B (3.13 ± 1.07) and EDIS C (3.13 ± 0.90), respectively, while the lowest scores were reported for EDIS A (3.10 ± 1.05). In EDIS B, the highest score belonged to "Using the EDIS does not necessarily require much effort (3.53 ± 1.06)", whereas the lowest score pertained to “Using the EDIS, I can quickly and easily recover errors (2.8 ± 1.26)". As for EDIS C, the high- est score was associated with "EDIS is user-friendly (3.66 ± 0.65)" and the lowest score was related to "Using the EDIS, I can quickly and easily recover the errors (2.83 ± 0.83)" and "EDIS is flexible (2.83 ± 0.83)". With regard to EDIS A, the highest score belonged to "Using the EDIS is easy and simple (uncomplicated) (3.23 ± 0.83)" and the lowest score pertained to "I noticed no inconsistencies when using the EDIS (2.76 ± 1.02)". 3.4. Ease of learning For this criterion, the highest mean scores were obtained by EDIS A (3.77 ± 0.73), EDIS C (3.67 ± 0.70), and EDIS B (3.54 ± 0.81), respectively. In EDIS A, the highest score was related to "I will learn to work with the software quickly (4.00 ± 0.49)" and "If I do not use the emergency information system for a long time, I will most likely re-learn and re-use it easily (4.00 ±0.57)". As for EDIS B, the highest score belonged to "I quickly learn the re- quired skills to use the EDIS (4.00 ± 0.53)" and "I quickly learn to work with software (4.00 ± 0.53)", whereas the lowest score pertained to "In order to use the EDIS correctly, you have to remember a lot of details (2.60 ± 1.05)". Regarding EDIS C, the highest score was associated with "I learn to work with software quickly (4.25 ± 0.45)" and the lowest score was re- lated to "Learning the principles and instructions of working with software is easy (3.25 ± 0.75)". 3.5. User satisfaction For this criterion, the highest mean scores were obtained for EDIS C (3.33 ± 0.97), EDIS A (3.30 ± 1.08), and EDIS B (3.26 ± 0.98), respectively. In EDIS C, the highest score was related to "I feel that such a system is needed (3.66 ± 0.77)" and the lowest score was associated with "The EDIS works as expected (2.91 ± 0.90)". Regarding EDIS A, the highest score was related to "I feel that such a system is needed (3.69 ± 1.01)" and the lowest score pertained to "Using the emergency information system is fun (3 ± 1.15)". In the case of EDIS B, the highest score was allo- cated to "I feel such a system is needed (3.77 ± 0.77)" while the lowest score was reported for "The Emergency Informa- tion System is excellent (2.8 ± 1.08)". 3.6. Suitability for the task For this criterion, the highest mean scores were obtained for EDIS A (3.53 ± 0.94), EDIS C (3.49 ± 0.95), and EDIS B (3.41 ± 1.03), respectively. In EDIS A, the highest score belonged to "Software outputs (reports) are suitable for user tasks (3.92 ± 0.95)" and the lowest score was related to "Screen fields are fitted to user tasks (3.15 ± 0.89)". As for EDIS C, the highest score was related to "The information required by the user is available on the screens (3.66 ± 0.88)" while the lowest score This open-access article distributed under the terms of the Creative Commons Attribution NonCommercial 3.0 License (CC BY-NC 3.0). Downloaded from: http://journals.sbmu.ac.ir/aaem S. Almasi et al. 4 Table 2: Mean score of emergency department information systems (EDIS) usability criteria in three hospitals (A, B, and C) Systems Usability criteria Usefulness Ease of use Ease of learning User satisfaction Suitability for the task EDIS A 3.59±0.78 3.10±1.05 3.77±0.73 3.53±0.94 3.30±1.08 EDIS B 3.54±1.12 3.13±1.07 3.54±0.81 3.26±0.98 3.41±1.01 EDIS C 3.48±0.73 3.13±0.90 3.67±0.70 3.33±0.97 3.49±0.95 Total 3.53±0.87 3.12±1.01 3.66±0.74 3.29±1.01 3.47±0.96 Data are presented as mean ± standard deviation. Table 3: Relationship between the usability criteria of the emergency department information system (EDIS) and baseline characteristics of participants Variable Usefulness Ease of use Ease of learning User satisfactionr Suitability for task Education Level CC 0.01 -0.03 0.03 -0.04 -0.02 P-value 0.92 0.81 0.84 0.79 0.90 Occupation CC 0.12 0.14 0.27 0.21 0.16 P-value 0.42 0.36 0.08 0.18 0.32 Age CC -0.20 -0.37 -0.20 -0.36 -0.23 P-value 0.19 0.01 0.20 0.02 0.14 Work experience CC -0.32 -0.28 -0.30 -0.34 -0.33 P-value 0.03 0.07 0.05 0.02 0.03 Experience with EDIS CC -0.24 -0.34 -0.12 -0.24 -0.21 P-value 0.12 0.03 0.43 0.12 0.17 CC: correlation coefficient. was related "EDIS provides support for users’ daily activities (3.16 ± 1.02)". As for EDIS B, the highest score pertained to "EDIS provides support for users’ daily activities (3.66 ± 0.81)" and the lowest score pertained to "It is possible to ad- just the display of results (information) to the user’s needs (3.26 ± 1.16)". 3.7. Relationship between usability criteria and participants’ characteristics With increase in the level of education, age, work experience, or experience of working with the system, user satisfaction with each dimension of system usability declined. Evidence also shows that there is a stronger correlation between job and satisfaction with usability than between other variables (table 3). 4. Discussion Based on the findings of this study, the ease of use criterion received the lowest score among the criteria. Consistency in the displayed information and quick recovery of errors re- ceived the lowest scores as the sub-criteria of ease of use. In the study by Khajouei et al. (37), the most important recommendations for creating consistency and standards were: arranging numbers, fields, menu items, and the cursor equally and according to the standard on all pages; having an active exit button on each window; and using the same icon for buttons having the same function. As for the feature of helping users identify and correct errors, the most impor- tant recommendations were: presenting all the messages in one language (Persian) with accurate and consistent gram- mar, without using exclamation marks, and in positive state- ments. Indicating the severity of the error, the cause of the problem, and the necessary activities to recover the error are also recommended. In case of an error related to several data entry fields, after viewing and confirming it, the user should be directed to the relevant fields. Moreover, the software meeting the expectations and users enjoying working with the software had the lowest score in the user satisfaction criterion. In the study by Kalankesh et al., factors such as the quality of the information provided by the system, system quality, quality of services provided by the system, quality of vendor company support for the software and its users, and the compatibility of system performance with expectations were deemed useful for improving users’ satisfaction with the information systems (38). Various stud- ies show that paying attention to the design of the system’s This open-access article distributed under the terms of the Creative Commons Attribution NonCommercial 3.0 License (CC BY-NC 3.0). Downloaded from: http://journals.sbmu.ac.ir/aaem 5 Archives of Academic Emergency Medicine. 2022; 10(1): e71 user interface (in terms of allowing the background color, fit, order, and sequence of icons, and menus to be adjusted with the work processes of the emergency department), the use of data visualization tools, software speed, personalization, and system font size may increase user satisfaction with the EDIS (39, 40). In an applied research on the usability of hospital in- formation systems, Sadoughi et al. concluded that user satis- faction, perceived usefulness, meeting the expectations, and the intention to continue using the system were indicators of the success or failure of the system; thus, it is essential to resolve the deficiencies that reduce satisfaction with the sys- tem (41). Findings related to the suitability for the task criterion showed that the studied systems have a relatively optimal score in this dimension(3). Among the strengths of the sys- tems examined in the emergency departments were the suit- ability of software outputs (reports) for the user’s tasks, and the existence of information required by the user on the screen. Still, these systems scored poorly in terms of the fit of the screen fields and the possibility of personalizing the screen for user tasks. In the study by Farley et al., the poor user interface design was reported as a limitation of the EDIS, which caused difficulties for users in accessing the required information(12). In the study by Callen et al., despite improv- ing coordination and communication between providers, the use of this system had limitations in terms of quick access to patient information, increasing information registration time, and user-friendliness, which necessitated the use of in- novative methods for collecting and displaying emergency department information (42). Agharezaei et al. also found that from the users’ point of view, the systems are simple, working with these systems is not particularly complicated, and they are easy to learn, but they lack flexibility. It is thus suggested to pay more attention to this component when de- signing the system (43). Findings related to the usefulness criterion indicated that the systems under review received good scores regarding increasing efficiency and productivity, facilitating activities, and better control and management of tasks. The results re- lated to this criterion are in line with the findings of other studies examining ways to increase system usefulness (44, 45). Furthermore, the ease of learning criterion had the high- est score compared to the other criteria. The results of this study showed that features such as few learning details when working with the system and the small number of learn- ing instructions were positive features of the systems under study. The literature shows that educating users about using the system and paying attention to the design of these sys- tems to allow users to easily navigate the systems and learn them with little effort promote the learning of the system (46- 48). Therefore, Observing the principles of usability in health information systems can promote efficiency and user satis- faction, reduce errors, and ultimately improve patient safety and quality of care (49). This is even more critical in sys- tems used in departments such as the emergency depart- ment, where due to the critically ill status of patients, as well as numerous and unpredictable referrals, the users’ focus is diminished and they need easy-to-use systems (50, 51). 5. Limitations To evaluate the usability of EDIS, we administered a researcher-made questionnaire based on the standard 9241 ISO Metric Questionnaire part 10 and the USE Question- naire. The questionnaires had previously been used in other studies assessing the usability of hospital information sys- tems (28, 52). However, in this study, only the quantitative method was adopted to evaluate the usability of the EDIS. A mixed-methods study (quantitative and qualitative) is rec- ommended to gain better insight into the usability problems of the EDISs. Other limitations of this study include the lack of coopera- tion of some system users and users’ unfamiliarity with this system. Due to the existence of different processes and struc- tures in the emergency departments, the results of this study may not be generalizable to other settings. Still, out of the three examined information systems, two had been devel- oped by different companies, and one had been designed within the hospital, and this diversity may improve general- izability. To promote the usability of these systems, the following rec- ommendations are made: understanding the needs of users and the work environment, identifying the type of users and their diverse needs in a versatile environment, paying atten- tion to work processes and seemingly unimportant issues, designing the interface according to common principles and models, involving end-users in information system design, conducting usability tests, and applying final edits before im- plementing the system. 6. Conclusion The results of this study show that the studied emergency systems are at a desirable level in terms of usability. Never- theless, considering the following factors in designing these systems might increase the usability of EDIS: foreseeing users’ work needs, greater software flexibility, consistency of features and standards, similarity across the system, cus- tomizing users’ screens according to their needs, incorporat- ing data visualization tools, quality of the information pro- vided by the system, system quality, quality of after-sales ser- vices, software maintenance support, and displaying infor- mation such that users can easily understand them. This open-access article distributed under the terms of the Creative Commons Attribution NonCommercial 3.0 License (CC BY-NC 3.0). Downloaded from: http://journals.sbmu.ac.ir/aaem S. Almasi et al. 6 7. Declarations 7.1. Acknowledgments We would like to express our sincere gratitude towards the personnel of the emergency departments of hospitals affili- ated with AJA University of Medical Sciences. 7.2. Authors’ contributions S. Almasi, N. Mehrabi, F. Asadi and M. Afzali: Concept and design. S. Almasi: Literature search. S. Almasi: collecting the data. S. Almasi and N. Mehrabi: Data analysis and interpre- tation. S. Almasi: Manuscript drafting. N. Mehrabi, and F. Asadi and M. Afzali: Editing and critical review. All authors wrote and reviewed the manuscript. 7.3. Funding and supports This study was financially supported by AJA University of Medical Sciences. 7.4. Competing interests The authors declare that they have no competing interests. 7.5. Availability of data and materials The datasets used and/or analyzed in the current study are available from the corresponding author upon reasonable re- quest. References 1. Steptoe AP, Corel B, Sullivan AF, Camargo CA. Charac- terizing emergency departments to improve understand- ing of emergency care systems. International Journal of Emergency Medicine. 2011;4(1):42. 2. Khademian Z, Sharif F, Tabei SZ, Bolandparvaz S, Ab- baszadeh A, Abbasi HR. Teamwork improvement in emergency trauma departments. Iranian journal of nurs- ing and midwifery research. 2013;18(4):333-9. 3. Redley B, Botti M, Wood B, Bucknall T. Interprofessional communication supporting clinical handover in emer- gency departments: An observation study. Australasian emergency nursing journal : AENJ. 2017;20(3):122-30. 4. Soremekun OA, Takayesu JK, Bohan SJ. Framework for analyzing wait times and other factors that impact pa- tient satisfaction in the emergency department. The Journal of emergency medicine. 2011;41(6):686-92. 5. Morley C, Unwin M, Peterson GM, Stankovich J, Kins- man L. Emergency department crowding: A systematic review of causes, consequences and solutions. PloS one. 2018;13(8):e0203316. 6. Sun BC, Hsia RY, Weiss RE, Zingmond D, Liang LJ, Han W, et al. Effect of emergency department crowding on outcomes of admitted patients. Annals of emergency medicine. 2013;61(6):605-11.e6. 7. Stang AS, Crotts J, Johnson DW, Hartling L, Guttmann A. Crowding measures associated with the quality of emer- gency department care: a systematic review. Academic emergency medicine : official journal of the Society for Academic Emergency Medicine. 2015;22(6):643-56. 8. Chisholm CD, Collison EK, Nelson DR, Cordell WH. Emergency department workplace interruptions: are emergency physicians "interrupt-driven" and "multi- tasking"? Academic emergency medicine : official jour- nal of the Society for Academic Emergency Medicine. 2000;7(11):1239-43. 9. Bergs J, Vandijck D, Hoogmartens O, Heerinckx P, Van Sassenbroeck D, Depaire B, et al. Emergency depart- ment crowding: Time to shift the paradigm from predict- ing and controlling to analysing and managing. Interna- tional emergency nursing. 2016;24:74-7. 10. Clark LN, Benda NC, Hegde S, McGeorge NM, Guarrera- Schick TK, Hettinger AZ, et al. Usability evaluation of an emergency department information system proto- type designed using cognitive systems engineering tech- niques. Applied ergonomics. 2017;60:356-65. 11. Callen J, Li L, Georgiou A, Paoloni R, Gibson K, Li J, et al. Does an integrated Emergency Department Information System change the sequence of clinical work? A mixed- method cross-site study. International journal of medical informatics. 2014;83(12):958-66. 12. Farley HL, Baumlin KM, Hamedani AG, Cheung DS, Ed- wards MR, Fuller DC, et al. Quality and safety impli- cations of emergency department information systems. Annals of emergency medicine. 2013;62(4):399-407. 13. Landman A, Teich JM, Pruitt P, Moore SE, Theriault J, Dorisca E, et al. The Boston Marathon Bombings Mass Casualty Incident: One Emergency Department’s Infor- mation Systems Challenges and Opportunities. Annals of emergency medicine. 2015;66(1):51-9. 14. Handel DA, Wears RL, Nathanson LA, Pines JM. Using In- formation Technology to Improve the Quality and Safety of Emergency Care. Academic Emergency Medicine. 2011;18(6):e45-e51. 15. Stokes-Buzzelli S, Peltzer-Jones JM, Martin GB, Ford MM, Weise A. Use of health information technology to manage frequently presenting emergency department patients. The western journal of emergency medicine. 2010;11(4):348-53. 16. Schiro J, Pelayo S, Martinot A, Dubos F, Beuscart-Zéphir MC, Marcilly R. Applying a Human-Centered Design to Develop a Patient Prioritization Tool for a Pediatric Emer- gency Department: Detailed Case Study of First Itera- tions. JMIR human factors. 2020;7(3):e18427. 17. Shapiro JS, Baumlin KM, Chawla N, Genes N, Godbold This open-access article distributed under the terms of the Creative Commons Attribution NonCommercial 3.0 License (CC BY-NC 3.0). Downloaded from: http://journals.sbmu.ac.ir/aaem 7 Archives of Academic Emergency Medicine. 2022; 10(1): e71 J, Ye F, et al. Emergency department information system implementation and process redesign result in rapid and sustained financial enhancement at a large academic center. Academic emergency medicine : official jour- nal of the Society for Academic Emergency Medicine. 2010;17(5):527-35. 18. Kim MS, Shapiro JS, Genes N, Aguilar MV, Mohrer D, Baumlin K, et al. A pilot study on usability analysis of emergency department information system by nurses. Applied clinical informatics. 2012;3(1):135-53. 19. Landman AB, Bernstein SL, Hsiao AL, Desai RA. Emer- gency department information system adoption in the United States. Academic emergency medicine : official journal of the Society for Academic Emergency Medicine. 2010;17(5):536-44. 20. Gagnon MP, Desmartis M, Labrecque M, Car J, Pagliari C, Pluye P, et al. Systematic review of factors influencing the adoption of information and communication technolo- gies by healthcare professionals. Journal of medical sys- tems. 2012;36(1):241-77. 21. Inokuchi R, Sato H, Nakamura K, Aoki Y, Shinohara K, Gunshin M, et al. Motivations and barriers to imple- menting electronic health records and ED information systems in Japan. The American journal of emergency medicine. 2014;32(7):725-30. 22. Kushniruk AW, Triola MM, Borycki EM, Stein B, Kannry JL. Technology induced error and usability: the relation- ship between usability problems and prescription errors when using a handheld application. International jour- nal of medical informatics. 2005;74(7-8):519-26. 23. Gediga G, Hamborg K-C, Düntsch I. The IsoMetrics us- ability inventory: an operationalization of ISO 9241- 10 supporting summative and formative evaluation of software systems. Behaviour information technology. 1999;18(3):64-151. 24. Ahmadi M, Shahmoradi L, Barabadi M, Hoseini F. Usabil- ity evaluation of hospital information systems based on isometric 9241. Hakim Med J. 2011;13(4):226-33. [In Per- sian] 25. Dehdari T, Rahimi T, Aryaeian N, Gohari MR, Esfeh JM. Developing and testing a measurement tool for assessing predictors of breakfast consumption based on a health promotion model. Journal of nutrition education and be- havior. 2014;46(4):250-8. 26. Hosseini Teshnizi S, Hayavi Haghighi MH, Alipour J. Eval- uation of health information systems with ISO 9241-10 standard: A systematic review and meta-analysis. Infor- matics in Medicine Unlocked. 2021;25:100639. 27. Rostami S, Sarmad A, Mohammadi M, Cheleie M, Amiri S, Zardoei Golanbary SH. Evaluating hospital informa- tion systems from the point of view of the medical records section users in Medical-Educational Hospi- tals of Kermanshah 2014. Journal of medicine and life. 2015;8(Spec Iss 4):232-40. 28. Safdari R, Dargahi H, Shahmoradi L. Survey of quality er- gonomic of Iran’s hospital information system and com- parison with three other software from users’ point of view. Journal of Hospital. 2010;9(1):33-42. [In Persian] 29. Hamborg K-C, Vehse B, Bludau H-B. Questionnaire based usability evaluation of hospital information sys- tems. Electronic journal of information systems evalua- tion. 2004;7(1):21-30. 30. Machado Faria TV, Pavanelli M, Bernardes JL, edi- tors. Evaluating the Usability Using USE Questionnaire: Mindboard System Use Case. Learning and Collabora- tion Technologies; 2016 2016//; Cham: Springer Interna- tional Publishing. 31. Lund AM. Measuring usability with the use question- naire12. Usability interface. 2001;8(2):3-6. 32. Hamidi Farahani R, Sharifi M, Ayat M, Markazi Moghad- dam N. Hospital Information Systems in Iranian Mili- tary Hospitals: A Multiple Case Analysis. J Arch Mil Med. 2014;2(4):e22853. 33. Ayat M, Sharifi M. A Multiple Comparison Between Mili- tary Hospitals and Civilian Hospitals with a Look on the Maturity Situation of Hospital Information Systems in Iran. J Arch Mil Med. 2017;5(4):e13659. 34. International Organization for Standardization. Er- gonomic requirements for office work with visual display terminals (VDTs) – Part 11: Guidance on usability. Geneva, Switzerland: International Or- ganization for Standardization; 1998. (ISO 9241- 11:1998). Cited [20 February 2022]. Available at: https://www.iso.org/obp/ui/#iso:std:iso:9241:-11:ed- 1:v1:en 35. Dantas C, Jegundo A, Quintas J, Martins A, Queirós A, Rocha N. European Portuguese Validation of Usefulness, Satisfaction and Ease of Use Questionnaire (USE). Ad- vances in Intelligent Systems and Computing2017. 570. 36. Gao M, Kortum P, Oswald F. Psychometric Evaluation of the USE (Usefulness, Satisfaction, and Ease of use) Ques- tionnaire for Reliability and Validity. 2018;- 62:- 1418. 37. Khajouei R, Azizi A, Atashi A. Usability Evaluation of an Emergency Information System: A Heuristic Evaluation. Journal of Health Administration. 2013;16(52):61-72. 38. Kalankesh LR, Nasiry Z, Fein RA, Damanabi S. Factors In- fluencing User Satisfaction with Information Systems: A Systematic Review. Galen Med J. 2020;9:e1686-e. 39. Karahoca A, Bayraktar E, Tatoglu E, Karahoca D. Infor- mation system design for a hospital emergency depart- ment: A usability analysis of software prototypes. Journal of Biomedical Informatics. 2010;43(2):224-32. 40. McGeorge N, Hegde S, Berg RL, Guarrera-Schick TK, LaVergne DT, Casucci SN, et al. Assessment of Inno- This open-access article distributed under the terms of the Creative Commons Attribution NonCommercial 3.0 License (CC BY-NC 3.0). Downloaded from: http://journals.sbmu.ac.ir/aaem S. Almasi et al. 8 vative Emergency Department Information Displays in a Clinical Simulation Center. J Cogn Eng Decis Mak. 2015;9(4):329-46. 41. Sadoughi F, Khoshkam M, Farahi SR. Usability Evalua- tion Of Hospital Information Systems In Hospitals Af- filiated With Mashhad University Of Medical Sciences, Iran. health information management. 2012;9(3):310-7. [In Persian] 42. Callen J, Paoloni R, Li J, Stewart M, Gibson K, Georgiou A, et al. Perceptions of the effect of information and com- munication technology on the quality of care delivered in emergency departments: a cross-site qualitative study. Annals of emergency medicine. 2013;61(2):131-44. 43. Agharezaei Z, Khajouei R, Ahmadian L, Agharezaei L. Us- ability Evaluation Of A Laboratory Information System. health information management. 2013;10(2 (30)):1-12. [In Persian] 44. Vartak S, Crandall DK, Brokel JM, Wakefield DS, Ward MM. Transformation of Emergency Department pro- cesses of care with EHR, CPOE, and ER event tracking systems. Health information management : journal of the Health Information Management Association of Aus- tralia. 2009;38(2):27-32. 45. Palm J-M, Colombet I, Sicotte C, Degoulet P. Determi- nants of user satisfaction with a Clinical Information Sys- tem. AMIA Annu Symp Proc. 2006;2006:614-8. 46. Yu P, Li H, Gagnon MP. Health IT acceptance factors in long-term care facilities: a cross-sectional survey. Inter- national journal of medical informatics. 2009;78(4):219- 29. 47. De Leeuw JA, Woltjer H, Kool RB. Identification of Factors Influencing the Adoption of Health Information Technol- ogy by Nurses Who Are Digitally Lagging: In-Depth Inter- view Study. J Med Internet Res. 2020;22(8):e15630. 48. Dowding D, Randell R, Gardner P, Fitzpatrick G, Dykes P, Favela J, et al. Dashboards for improving patient care: re- view of the literature. International journal of medical in- formatics. 2015;84(2):87-100. 49. Hoot NR, Aronsky D. Systematic review of emergency de- partment crowding: causes, effects, and solutions. An- nals of emergency medicine. 2008;52(2):126-36. 50. Habib MI, Khan KM. Overcrowding and possible so- lutions for a busy paediatric emergency department. JPMA The Journal of the Pakistan Medical Association. 2017;67(9):1398-403. 51. Moghaddasi H, Rabiei R, Asadi F, Ostvan N. Evaluation of Nursing Information Systems: Application of Usability Aspects in the Development of Systems. Healthc Inform Res. 2017;23(2):101-8. 52. Ehteshami A, Sadoughi F, Saeedbakhsh S, Isfahani MK. Assessment of Medical Records Module of Health Infor- mation System According to ISO 9241-10. Acta Inform Med. 2013;21(1):36-41. This open-access article distributed under the terms of the Creative Commons Attribution NonCommercial 3.0 License (CC BY-NC 3.0). Downloaded from: http://journals.sbmu.ac.ir/aaem 9 Archives of Academic Emergency Medicine. 2022; 10(1): e71 Appendix 1: study questionnaire The present questionnaire is to evaluate the usability of the emergency department information system (EDIS) in critical situations at edu- cational and medical centers affiliated to the AJA University of Medical Sciences from the users’ viewpoints with a focus on five dimensions: usefulness, ease of use, ease of learning, user satisfaction and suitability for the task. The questionnaire includes phrases with the above- mentioned dimensions, please give your valuable opinion on each phrase by marking the available scales. This open-access article distributed under the terms of the Creative Commons Attribution NonCommercial 3.0 License (CC BY-NC 3.0). Downloaded from: http://journals.sbmu.ac.ir/aaem Introduction Methods Results Discussion Limitations Conclusion Declarations References