Archives of Academic Emergency Medicine. 2019; 7 (1): e34 REV I EW ART I C L E Applications of Machine Learning Approaches in Emer- gency Medicine; a Review Article Negin Shafaf1, Hamed Malek1 ∗ 1. Faculty of Computer Science and Engineering, Shahid Beheshti University, Tehran, Iran. Received: May 2019; Accepted: June 2019; Published online: 3 July 2019 Abstract: Using artificial intelligence and machine learning techniques in different medical fields, especially emergency medicine is rapidly growing. In this paper, studies conducted in the recent years on using artificial intelligence in emergency medicine have been collected and assessed. These studies belonged to three categories: prediction and detection of disease; prediction of need for admission, discharge and also mortality; and machine learning based triage systems. In each of these categories, the most important studies have been chosen and accuracy and results of the algorithms have been briefly evaluated by mentioning machine learning techniques and used datasets. Keywords: artificial intelligence; machine learning; emergency medicine; emergency service, hospital; triage Cite this article as: Shafaf N, Malek H. Applications of Machine Learning Approaches in Emergency Medicine; a Review Article. Arch Acad Emerg Mede. 2019; 7(1): e34. 1. Introduction A rtificial intelligence has been used in different health and medical fields. In particular, different artificial in- telligence and machine learning techniques have at- tracted attention in the recent years. Emergency department (ED) and triage are among the most important parts of any hospital in which diagnostic and therapeutic interventions should be performed both rapidly and effectively. As the number of people referring to ED increases, common tra- ditional techniques may not be sufficient. Thus, different methods of artificial intelligence such as natural language processing, data mining, clustering and classification algo- rithms should be used to significantly enhance the efficacy of hospital emergency system. Using artificial intelligence will bring some advantages; it will reduce human errors as well as time and expenses and improve the pace of provid- ing services. Moreover, machine learning techniques usually have a comparable and even better accuracy compared with medical staff of the hospital. Several studies have assessed application of artificial intelligence in ED triage from differ- ent points of view. Though, most of them can be divided into three categories, which will be further discussed in detail. ∗Corresponding Author: Hamed Malek; Shahid Beheshti University, Shahid Shahriari Square, Daneshjou Boulevard, Shahid Chamran Highway, Tehran, Iran. Email: h_malek@sbu.ac.ir Phone/Fax: +98 (21) 29904106 Machine learning systems have the potential of prediction and early detection of diseases in ED so that the disease can be treated more effectively and disease progression and oc- currence of inappropriate complications can be prevented. This issue is discussed in detail in part 3.1 by reviewing the most important studies in this regard. Increase of the num- ber of patients referring to ED will result in overcrowding, which decreases the efficiency of medical staff and increases the delay for patient visit. Consequently, artificial intelli- gence and machine learning techniques can be used in or- der to manage the ED, allocate money and equipment, and discharge patients more properly. A series of studies are pro- vided in part 3.2 in this regard, in which admission of patients referring to ED, their discharge or mortality is predicated. In part 3.3, studies on electronic triage methods based on machine learning techniques have been briefly reviewed. Studies presented in this part, prioritize patients based on different machine learning techniques, which is usually both faster and more accurate compared with traditional meth- ods and emergency severity index (ESI) (1). Faster triage not only improves patient satisfaction, but also improves perfor- mance of ED and prevents its overcrowding. 2. Machine learning techniques and its different types Machine learning is an important subset of artificial intelli- gence, which enables machines to learn and act on specific This open-access article distributed under the terms of the Creative Commons Attribution NonCommercial 3.0 License (CC BY-NC 3.0). Downloaded from: http://journals.sbmu.ac.ir/aaem N. Shafaf et al. 2 Table 1: Different models on disease prediction Disease Algorithm Result Year AKI boosted ensembles of decision trees AUC :0.72- 0.87 2018 (8) Logistic regression AUC: 0.77 2018 (9) Gradient Boosting Machine AUC:0.73-0.97 2018 (12) Deep Learning Accuracy: 99.1% 2018 (13) Logistic regression AUC : 0.74 2016 (10) Binary logistic regression Sensitivity :96.6% 2008 (11) Specificity: 95.7% Influenza Bayesian classifier (naive Bayes) AUC: 0.92-0.93 2015 (15) Bayesian network classifiers AUC: 0.79 2014 (14) Sepsis Gradient tree boosting AUC: 0.87-0.92 2018 (18) Support Vector Machine AUC: 0.86 2017 (19) COPD and Asthma Random forest C-statistic: 0.84 2018 (20) Logistic regression Accuracy: 89.1% 2017 (21) Naive Bayes Accuracy: 70.7% 2013 (23) Tree-based decision model AUC: 0.83 2010 (22) UTI Extreme gradient boosting AUC: 0.90 2018 (17) Appendicitis Rule base AP: 0.86 2013 (25) AKI: acute kidney injury; COPD: chronic obstructive pulmonary disease; UTI: urinary tract infection; AP: average precision and recall. tasks. In fact, machine learning consists of a set of techniques and algorithms which can predict some future events or clas- sify some data by learning the patterns in the existing data. Some of the most important algorithms in this field are logis- tic regression, support vector machines (SVM), Naive Bayes algorithm (2), decision trees, random forest, gradient boost- ing and deep learning. Logistic regression is a machine learning algorithm, which tries to find a linear model of the relations between variables by fitting a line on the curve of the given data (3). It can also be used for classification purposes. One of the frequently used algorithms for data classification is SVM algorithm (4). It has been proven to find the best data classifier for the given data. SVM can achieve a very good generalization perfor- mance. Decision tree is another tool for modeling data, which uses tree-like structures for classifying the decisions in order to output the class of the given data. When a number of deci- sion trees are employed, an ensemble learning method called random forest is produced (5). Gradient boosting method is also widely used in some machine learning problems (6). It produces an ensemble model of the data by employing some weak models. One of the advantages of this method is its abil- ity to reduce bias and variance in the model. Deep learning is another algorithm, which has recently been widely recognized as a successful method in some complex machine learning tasks (7). Deep learning is a part of another class of machine learning methods named artificial neural network (ANN). In ANN algorithms, a network of cells is pro- duced and the connections between the cells are adjusted in a way that the resulting network can learn the structure of the training data. In deep learning, the number of layers in the network is much higher than an ordinary ANN. This en- ables the algorithm to extract higher level features from the input data. 3. Review of studies In the following sections, the three most important issues in ED, including prediction and detection of disease; predic- tion of need to admission, discharge, and also mortality; and electronic triage will be discussed separately by reviewing the most important articles. 3.1. Disease prediction and detection A lot of information about patient’s demographic character- istics, symptoms, and disease appearance is available in EDs through Electronic Health Record (EHR), nursing reports, laboratory test results and patient profiles. This information could be used by machine learning techniques in order to predict and detect different diseases so that medical inter- ventions take place more rapidly and effectively. Some of the most important diseases, which can be detected through machine learning techniques, are presented below (table 1). Acute kidney injury Acute kidney injury (AKI) is a disease that can occur in a few hours to a few days and can lead to kidney failure if not man- aged properly and the patients would need dialysis for the rest of their lives and they may even die due to kidney failure. However, if this disease is diagnosed soon, it can be rapidly controlled to avoid complications. Artificial intelligence and machine learning techniques can help in fulfilling this objec- tive. This open-access article distributed under the terms of the Creative Commons Attribution NonCommercial 3.0 License (CC BY-NC 3.0). Downloaded from: http://journals.sbmu.ac.ir/aaem 3 Archives of Academic Emergency Medicine. 2019; 7 (1): e34 In 2018, a method was proposed, which was based on boosted ensemble of decision trees which was able to detect AKI at time of onset, as well as 12, 24, 48, and 72 hours before the onset of the disease (8). In this study, two databases were used, both of which contained information of patients above 18 years old. Stanford Medical Center dataset contains infor- mation of patients in all hospital wards and Beth Israel Dea- coness Medical Center contains information of patients in ICU ward collected from MIMIC-III database, which is avail- able. Efficiency of this algorithm was compared with Sequen- tial Organ Failure Assessment (SOFA) method. According to area under the curve (AUC), the accuracy of this method in prediction of AKI at the time of onset is 87%. Additionally, the accuracy of this method for 12, 24, 48, and 72 hours be- fore onset of AKI is 80%, 79%, 76%, and 72%, respectively. Another study assessed clinical notes in the first 24 hours af- ter admission in intensive care unit (ICU), which is extracted from MIMIC-III dataset (9). By means of natural language processing, meaningful words and representations of con- cept and embedding were produced and different supervised classifiers such as Multinomial naive Bayes (MNB), L1-/L2- regularization, SVM, Logistic Regression (LR), Random For- est (RF), Gradient Boosting and Decision Tree (GBDT) and ar- chitecture knowledge-guided deep learning were used to de- sign the model. Among these, the highest accuracy belonged to LR with 77.90%. In one other study in this regard, a model for prediction and detection of AKI in patients above 60 years old was proposed in which collected information of 25521 patients during one year was utilized (10). In this study, five machine learning techniques, including logistic regression, support vector ma- chines, decision trees, naive Bayes and ensemble were used for disease prediction and detection. These methods were compared with each other and based on AUC; logistic re- gression was the most accurate method for disease detection with 74% accuracy; while ensemble method was the most ac- curate for disease prediction with 66% accuracy 24 hours be- fore the onset of the disease. AKI is a major complication with a very high reported mor- tality rate for people trapped beneath the rubble following earthquakes. In these cases, AKI can be prevented through rapid and early fluid therapy. This issue is evaluated in a study, which presents two algorithms to help medical staff predict AKI on the first day after a disaster (11). This method is based on biochemical parameters and can be utilized very easily even in disasters. Early prophylactic hydration therapy for patients with kidney disease, who are susceptible to AKI, can reduce its occurrence during natural disasters. Since in these situations, a huge resource for fluid therapy might not be available, it is vital to separate those who are at risk of AKI from those who are not at risk. For this purpose, decision rules were designed by considering Receiver operating characteristic curve and Binary logistic regression, which have a sensitivity of 96.6% and specificity of 95.7% for AKI prediction. There was another method for AKI risk predication in patients admitted in hospitals. Data of patients were extracted from Electronic Health Record of academic medical centers from 2008 to 2016. Using Gradient Boosting Machine, AKI can be predicted with AUC of 0.73 to 0.97 in different conditions (12). Deep learning method was deployed in a study on prediction of AKI in ICU ward. This study was conducted on MIMIC-III database and was able to predict AKI with 99.1% accuracy (13). Influenza Influenza is an infectious disease affecting many people ev- ery year and it can cause epidemics due to its contagious na- ture. Thus, early diagnosis and prediction of influenza can prevent its epidemics and save people’s lives, reduce treat- ment costs and lessen patient reference to ED. A study in this regard has proposed an automatic method for influenza diagnosis based on EHR in which symptom extraction, feature selection, and their classification for in- fluenza diagnosis have been performed systematically (14). In fact, natural language processing and different classifiers have been used in a combined manner so that influenza dis- ease can be diagnosed automatically based on EHRs. More accurately, we can state that in this method, free-text re- ports of ED are processed using different language process- ing parsers such as Topaz, MedLEE, and an expert to extract the important information about influenza. Then Bayesian network classifier uses expert-defined-BN, BN-EMTopaz, or BN-EM-MedLEE for estimation of influenza probability. Fi- nally, for evaluation of this method, 9 experiments were con- ducted on three parsers and three classifiers and the results were compared with gold standard. Highest accuracy was achieved when an expert processed the reports and extracted influenza related information and BN-EMTopaz classifiers were used, AUC of which was 0.79 and showed the highest value among the 9 experiments. Free-text reports in ED can be helpful in early and real-time diagnosis of influenza. In a study, it is reported that in- fluenza is diagnosed in EDs using machine learning tech- niques based on these reports (15). The data used in this study included 31268 ED reports of 4 hospitals from 2008 to 2011. Then, by means of topaz, a natural language pro- cessing tool, features and terms related to influenza were extracted form reports and codified into three categories (acute, non-acute, and missing). Then, 7 classifiers includ- ing Naive Bayes, Bayesian network with the K2 algorithm, Efficient Bayesian Multivariate Classification, Artificial Neu- ral Networks, Logistic Regression, SVM, and Random forests were applied and the results were compared with each other. If there were missed data, Bayesian (naive base) showed bet- This open-access article distributed under the terms of the Creative Commons Attribution NonCommercial 3.0 License (CC BY-NC 3.0). Downloaded from: http://journals.sbmu.ac.ir/aaem N. Shafaf et al. 4 ter performance with AUC of about 92-93%. According to studies conducted to compare different meth- ods of natural language processing in influenza diagnosis, it can be concluded that by using all clinical notes, influenza can be diagnosed with 92.9% accuracy, while when only symptoms reported by the patient is processed, the accuracy of these methods drops to 70.3% (16). Urinary tract infection (UTI) Urinary tract infection is a common disease in EDs with a high rate of diagnostic error, since urine culture is not avail- able until 24 to 48 hours after the first visit. Diagnosis and de- cision making on medication prescription is based on symp- toms, physical examination findings, and results of labora- tory tests, which can cause overuse of antibiotics and resis- tance against antibiotics. Previous studies have shown that diagnostic efficacies of laboratory tests and individual pre- dictions are not enough. A study performed in this regard has proposed some models for prediction of UTI based on machine learning for different patients in ED (17). Data used in this study was collected from clinical reports of health profiles of patients above 18 years old admitted to 4 EDs between 2013 and 2016. Proposed models for disease prediction used 7 machine learning algorithms including Random forest, extreme gra- dient boosting, adaptive boosting, elastic net, support vector machine, logistic regression, and neural network. Results of this study showed that among the mentioned algorithms, XGBOOT algorithm provided the best efficacy with AUC of 0.90. Sepsis Sepsis is a severe infection with a high mortality rate and high therapeutic costs. Consequently, early diagnosis and treat- ment of sepsis can reduce mortality rate among the patients and reduce therapeutic costs for these patients. A machine learning based method has been proposed for prediction and diagnosis of sepsis, which can improve the treatment pro- cedure of patients (18). Using gradient tree boosting algo- rithm, three levels of sepsis are detected. Features used in this method include values of 6 vital signs in EDs, general wards, and ICU and eventually, area under the ROC value for sepsis and severe sepsis are 0.92 and 0.87, respectively. Another study suggests an automatic system by using ma- chine learning techniques for triage of sepsis patients in ED (19). In this study, a dataset including patient’s information and data including ED triage report note, triage vital signs, and ICD-9-CM codes was extracted from EHR. According to the reported results, it can be concluded that using free-text data from ED triage, as well as structured data such as vital signs and statistical information can significantly improve detection of patients suspected to infection. Two models, bag of words model and topic model were used for represen- tation of free-text data and then an SVM was used to design the predictor model. Applying SVM of bag of words is more effective compared with other methods and AUC value for test and train data are 0.86 and 0.89, respectively. Chronic obstructive pulmonary disease (COPD) and asthma Patients with chronic obstructive pulmonary disease and asthma, face the risk of exacerbation of their disease every day. Using special tools when necessary, the probability of exacerbation of the disease can be effectively reduced. By means of machine learning techniques, some methods have been proposed, which can be used for early diagnosis of exacerbation of the disease. Asthma condition and COPD exacerbation in EDs have been assessed in different studies using different machine learn- ing methods such as Lasso regression, random forest, and boosting, and deep neural network, and some models have been developed based on available data. These methods were compared based on C-statistic index, which showed that Random forest method has the highest efficacy with 84% accuracy (20). In another proposed algorithm, some features are deter- mined by physicians for both prediction of patient’s triage and diagnosis of disease exacerbation and data are labeled. Then, the algorithm is trained by different methods. Ac- cording to the obtained result for different classifiers of machine learning, Gradient-Boosted Decision Tree and Logistic Regression had the highest efficacy with an accuracy of 88.1% and 89.1%, respectively (21). Another study predicts the severity of asthma exacerbation among children in ED of Eastern Ontario Pediatrics Hospital. Data of children aged 1 to 17 years were used and a tree-based decision model was designed for prediction of severity of asthma exacerbation with AUC of 0.83 (22). Moreover, other machine learning methods have been used to predict this disease. In one of these studies, five different models were designed in this regard and Naive Bayes model was the most efficient model with 70.7% accuracy. Therefore, this can be used as a complementary model along with the traditional models such as Pediatric Respiratory Assessment Measure (PRAM) score (23). Appendicitis Appendicitis is one of the most common causes of abdomi- nal pain in patients referring to ED. A major challenge in di- agnosis of appendicitis is wrong diagnosis or delayed diag- nosis and perforation. Thus, early and accurate diagnosis of appendicitis is necessary (24). An automatic system has been designed in which major com- ponents are extracted from clinical notes in EDs and labora- This open-access article distributed under the terms of the Creative Commons Attribution NonCommercial 3.0 License (CC BY-NC 3.0). Downloaded from: http://journals.sbmu.ac.ir/aaem 5 Archives of Academic Emergency Medicine. 2019; 7 (1): e34 Table 2: Different models for prediction of disposition Prediction Algorithm result Year Need for admission XGBoost AUC: 0.92 2018(29) Need for admission gradient boosted machines AUC:0.85 2018 (30) ICU Readmission gradient boosted machine AUC: 0.76 2018 (33) In-hospital mortality LSTM AUC: 0.94 2018 (37) Length of ICU stay logistic regression AUC :0.93 2018 (37) Readmission Random forest AUC: 0.58 2018 (37) Need for admission Naive Bayes+ logistic regression AUC: 91.0 2017 (28) Need for admission Nu-Support Vector Machine F-score:0.77 2017 (36) In-hospital Mortality Random forest AUC: 0.86 2016 (34) Length of stay logistic regression Random forest > clinicians 2015 (35) Need for admission logistic regression AUC: 0.80- 0.89 2013 (32) Need for admission logistic regression Specificity=96.8% 2011 (31) Sensitivity=33.4% ICU: intensive care unit, LSTM: long short-term memory. tory test results. In this model, risk of appendicitis in children is classified to high risk, low risk, and equivocal (25). This method is based on machine learning techniques and natu- ral language processing in both of which structured informa- tion derived from EHRs (laboratory results) and clinical notes of ED are used. Firstly, the information about risk of appen- dicitis is extracted using natural language processing tech- niques. Then, a rule based method was used to classify ap- pendicitis risk to three classes (high risk, low risk, and equivo- cal). Finally, efficacy of this method was compared with gold standard method, which is designed manually by physicians. Mean precision and recall of this system were 38% and 86%, respectively. Electronic surveillance systems can detect the disease faster than diagnosis based systems based on chief complaint of patients. In this regard, a classifier is designed to collect free-text reports about chief complaint of patients in triage to classify patients in one of the 7 following categories: respi- ratory, botulinic, gastrointestinal, neurologic, rash, constitu- tional, and hemorrhagic syndromes (26). Final result of the study shows that for most of these syndromes, classification systems can detect nearly half of patients with a specificity more than 90% and positive predictive value of 12 to 44 per- cent, predicting the related syndrome with an overall accu- racy of 92.3% to 99.1% for these 7 syndromes. 3.2. Prediction of disposition and mortality Considering the increasing requests, boarding of patients ad- mitted to ED is an issue due to the overcrowding. Predic- tion of discharge and admission of patients can be performed automatically to improve this process. To achieve this goal, supervised machine learning methods and available health data are used to help in admission of new patients. Early prediction of admission can speed up allocation of re- sources and bed to the patient and shorten the boarding times. On the other hand, according to the results, it has been proved that in some cases, nurses are not sure when predicting patient admission and show lower performance compared with machine learning methods (27). Thus, ma- chine learning methods can be time-saving and improve out- comes in medical interventions and patient satisfaction, and reduce hospital costs. Different models have been proposed for early predicting of patient admission (table 2). In a study in this regard, a model was designed based on combined generative-discriminative approach (28). Number of variables has been reduced by means of naive Bayes (generative) and then a regression model (discriminative) is applied on the results of the previ- ous model. Using data of available EHRs, this model can pre- dict 73.4% of admissions with a specificity of 90% and 35.4% of admissions with a specificity of 99.5% (AUC=91%) in the first 30 minutes. Another study suggests a model based on machine learning in which, history of patients is used as well as the informa- tion collected in ED triage to predict patient admission or discharge (29). Three binary classifiers, logistic regression, gradient boosting (XGBoost), and deep neural networks, are applied on three kinds of datasets (first dataset only con- tains patient history, second dataset only contains triage in- formation, and third dataset includes both patient history and triage information). Results show that using patient his- tory in addition to triage information significantly improves efficacy of the prediction and XGBoost classifier provides the best efficacy with AUC of 0.92. In another study, available methods in data mining were ap- plied on electronic system data to form a prediction model for patient admission in ED triage (30). In this study, three algorithms including logistic regression, decision trees, and gradient boosted machines (GBM) have been used and re- sults proved that GBM method provides the best perfor- This open-access article distributed under the terms of the Creative Commons Attribution NonCommercial 3.0 License (CC BY-NC 3.0). Downloaded from: http://journals.sbmu.ac.ir/aaem N. Shafaf et al. 6 mance with AUC of 0.85 and accuracy of 80.31% compared with the other two methods. Yet, when it comes to inter- pretability of the data, adopting logistic regression model is a better choice. Many models developed for prediction of hos- pital admission in EDs are focused on a special group of pa- tients or patients with special diseases and just a few studies have discussed admission of all patients. A study suggests a model to predict risk level at the time of ad- mission. In this model, data of all patients with any disease in triage is assessed and data collected from the routine ex- aminations performed at the time of triage is used (31). This model helps nurses in triage to make faster decisions regard- ing whether the patient should be admitted or not, so that resources are allocated to those who should be admitted and ED crowding is reduced. Logistic regression is used to de- velop this method with a specificity of 96.8% and sensitivity of 33.4%. Additionally, result of a study showed that logistic regression method with AUC of 0.80 to 0.89 can be general- ized to different hospitals with different number of patients (32). In some cases, patients are transferred from ICU to general ward and then they are transferred to ICU which can cause problems such as longer ICU stay and increased costs. Also, it can increase mortality rate. This issue is discussed in a study and ICU readmission was predicted by machine learn- ing techniques using data of EHRs (33). Gradient boosted machine is used to design this model and the results are compared with two decision rules Stability and workload in- dex for transfer (SWIFT) score and Modified Early Warning Score (MEWS). It can be concluded from reported results that the proposed method with AUC of 0.76 is significantly bet- ter than MEWS method with AUC of 0.65 and SWIFT method with AUC of 0.58. Machine learning methods can also be helpful in prediction of patient mortality. Analyses and predictions performed in ED are often limited to clinical decision rules (CDRs), which use simple heuristics and scoring systems. A major problem of CDRs is that they are not generalizable and cannot be up- dated using new data. But, new techniques are based on ma- chine learning and can use many variables from EHRs. Mor- tality of patients with sepsis in ED is predicted using a ma- chine learning based method (34). A random forest model was developed using data of EHRs and was compared with models developed based on regression tree (CART) and lo- gistic regression models, where it showed better efficacy with AUC of 0.86. Moreover, machine learning approach can bet- ter predict mortality rate of patients with sepsis compared with available CDRs and traditional data analysis techniques. Also, machine learning can be used to predict hospital length of stay so that patients can be prioritized for discharge. Lo- gistic regression and random forest method have been used in a study (35). Comparison of these methods with predic- tion of hospital staff revealed that both methods have higher sensitivity and lower specificity. Several other studies have been conducted in prediction of admission or discharge of patients using different machine learning techniques. In one of these studies, prediction is performed by text mining of clinical reports by applying dif- ferent algorithms. Comparison of these methods showed that, Nu-Support Vector Machine provides the best perfor- mance with F1 score of about 0.77 in comparison to other algorithms (36). Another study has evaluated patient ad- mission, in hospital patient mortality, and length of stay on MIMIC-III database, separately. Different classic machine learning methods such as SVM, LR, MLP, Random forest, and Gradient Boost classifier and sequential models such as LSTM and CNN-LSTM have been used and the results are compared (37). In terms of patient mortality prediction, among classic models, MLP method with AUC of 0.85 and among sequential models, LSTM model with AUC of 0.94 showed the highest accuracies. In terms of length of stay prediction, among classic models, logistic regression method with AUC of 0.93 and among sequential models, LSTM model with AUC of 0.88 provided the best efficacies. Finally, in terms of patient admission, Random forest and LSTM models had provided the best results with AUC of 0.58. 3.3. Machine learning based triage systems Since EDs are overcrowded, the main problem in triage is to classify patients based on disease severity with a fast and accurate method to provide best services. Using machine learning techniques can improve pace and efficacy of patient management in triage compared with traditional methods. Studies performed in recent years are discussed in the fol- lowing paragraphs (table 3). Recently, a study has been conducted to suggest a method for proper classification of patients in triage into 5 groups based on disease severity (38). A new algorithm, namely Randomly Occurring Distributed Delayed Particle Swarm Optimization (RODDPSO), is proposed in this study, which is based on PSO evolutionary algorithms and is a method for clustering ED data suggested for patient classification. Efficacy of this method was finally evaluated based on mean silhouette value with two clustering algorithms K-means and FCM. Reported results showed that RODDPSO with a value of 0.31 for the mentioned index has a higher efficacy compared with the other two methods. Instead of using traditional protocols in triage, which are mostly based on disease symptoms, clinical decision support systems (CDSSs) may be used. CDSSs have been evaluated in a study for triage assistance using machine learning algorithms, which directly learn the model form data instead of an expert (39). There are different approaches for designing a model and the method should be chosen in a way that the result can be interpreted by physi- This open-access article distributed under the terms of the Creative Commons Attribution NonCommercial 3.0 License (CC BY-NC 3.0). Downloaded from: http://journals.sbmu.ac.ir/aaem 7 Archives of Academic Emergency Medicine. 2019; 7 (1): e34 Table 3: Different models used in machine learning based triage systems Algorithm Result Year Decision Tree Accuracy=84.0% 2018 (41) Support Vector Machine Accuracy=84.0% Random Forest AUC: 0.73-0.92 2018 (40) RODDPSO Mean silhouette value: 0.31 2018 (38) Naive Bayes classifier Accuracy: 87.9% 2008 (39) Bayesian network Accuracy:86.9% RODDPSO: randomly occurring distributed delayed particle optimization. cians. For instance, decision trees and Bayesian networks are appropriate methods. Furthermore, types of informa- tion used to develop the model are also important. Expert opinion or data can be used as information resources. Stud- ies suggest that using data and machine learning methods acts better compare with expert opinion in models. Also, Bayesian networks are more interpretable compared with rule-based models such as decision tree. Thus, developing models based on these methods can provide higher accuracy compared with other models. Bayesian network algorithms achieved accuracy of 87.9% and 86.9% by using Naive Bayes classifier and K2 algorithm, respectively, which are much higher compared with Decision tree method. Usually, standards of ED triage mostly depend on individual diagnosis and they have limited ability in identifying high risk patients. A method of assessment based on electronic triage using machine learning techniques is proposed in a study, which can predict high risk patient and separates patients properly (40). To be more accurate, this study classifies ESI level 3 patients so that prediction results can be more easily analyzed in triage. In this study, three decision tree learning models or in other words, random forest is used to predict in- tensive care, emergency procedure, and hospital admission for ED patients. Finally, the output of this model for each patient is one of these three categories. Results of this pre- diction has an AUC ranging from 0.73 to 0.92. This method classifies ESI level 3 patients more accurately. Another study on electronic triage of ED patients has pro- posed models with supervised machine learning algorithms and compared them (41). Classifiers such as Naive Bayes, Support Vector Machine, Decision Tree, and Neural Network have been deployed, among which Support Vector Machine and Decision tree provided highest efficacy with an accuracy of 84%. Another study proposed a calculative algorithm us- ing fuzzy logic and decision tree to classify patients in ED triage (42). 4. Conclusion Abilities of artificial intelligence and machine learning tech- niques can be used in medicine, especially in emergency medicine and in some important issues including disease prediction, admission or discharge prediction, and patient triage. By early prediction and diagnosis of high risk diseases such as AKI, sepsis, pneumonia, and contagious diseases such as influenza, necessary interventions can be performed more rapidly in ED to prevent multiple disease progression complications. In this regard, different machine learning algorithms such as Logistic regression, Bayesian network, deep learning etc. have been deployed, which generally have shown high accuracy ranging from 70% to 90%. Additionally, these algorithms and other methods can be helpful in predic- tion of patient admission and improve patient triage. 5. Appendix 5.1. Acknowledgements None. 5.2. Authors Contributions Negin Shafaf and Hamed Malek designed the study, devel- oped the methodology, collected the data, performed the analysis, and wrote the manuscript. Both authors reviewed the final manuscript. 5.3. Funding Support None. 5.4. Conflict of Interest The authors have no conflict of interest to declare. References 1. Stewart J, Sprivulis P, Dwivedi G. Artificial intelligence and machine learning in emergency medicine. Emer- gency Medicine Australasia. 2018;30(6):870-4. 2. Bonaccorso G. Machine learning algorithms: Packt Pub- lishing Ltd; 2017. 3. Kleinbaum DG, Dietz K, Gail M, Klein M, Klein M. Logis- tic regression: Springer; 2002. 4. Cortes C, Vapnik V. Support-vector networks. Machine learning. 1995;20(3):273-97. This open-access article distributed under the terms of the Creative Commons Attribution NonCommercial 3.0 License (CC BY-NC 3.0). Downloaded from: http://journals.sbmu.ac.ir/aaem N. Shafaf et al. 8 5. Koning M, Smith C. Decision Trees and Random Forests: A Visual Introduction for Beginners: Amazon Digital Ser- vices LLC - Kdp Print Us; 2017. 6. Natekin A, Knoll A. Gradient boosting machines, a tuto- rial. Frontiers in Neurorobotics. 2013;7(21). 7. LeCun Y, Bengio Y, Hinton G. Deep learning. nature. 2015;521(7553):436. 8. Mohamadlou H, Lynn-Palevsky A, Barton C, Chettipally U, Shieh L, Calvert J, et al. Prediction of acute kidney in- jury with a machine learning algorithm using electronic health record data. Canadian journal of kidney health and disease. 2018;5:2054358118776326. 9. Li Y, Yao L, Mao C, Srivastava A, Jiang X, Luo Y, editors. Early prediction of acute kidney injury in critical care set- ting using clinical notes. 2018 IEEE International Confer- ence on Bioinformatics and Biomedicine (BIBM); 2018: IEEE. 10. Kate RJ, Perez RM, Mazumdar D, Pasupathy KS, Nilakan- tan V. Prediction and detection models for acute kidney injury in hospitalized older adults. BMC medical infor- matics and decision making. 2016;16(1):39. 11. Najafi I, Van Biesen W, Sharifi A, Hoseini M, Rashid Farokhi F, Sanadgol H, et al. Early detection of patients at high risk for acute kidney injury during disasters: de- velopment of a scoring system based on the Bam earth- quake experience. J Nephrology 2008. 2008. 12. Koyner JL, Carey KA, Edelson DP, Churpek MM. The de- velopment of a machine learning inpatient acute kid- ney injury prediction model. Critical care medicine. 2018;46(7):1070-7. 13. Kanzawa M, Paul R, Weng N. Prediction of Acute Kidney Injury in the ICU. 14. Ye Y, Tsui F, Wagner M, Espino JU, Li Q. Influenza de- tection from emergency department reports using nat- ural language processing and Bayesian network classi- fiers. Journal of the American Medical Informatics Asso- ciation. 2014;21(5):815-23. 15. Pineda AL, Ye Y, Visweswaran S, Cooper GF, Wagner MM, Tsui FR. Comparison of machine learning clas- sifiers for influenza detection from emergency depart- ment free-text reports. Journal of biomedical informat- ics. 2015;58:60-9. 16. Elkin PL, Froehling DA, Wahner-Roedler DL, Brown SH, Bailey KR. Comparison of natural language process- ing biosurveillance methods for identifying influenza from encounter notes. Annals of Internal Medicine. 2012;156(1_Part_1):11-8. 17. Taylor RA, Moore CL, Cheung K-H, Brandt C. Predict- ing urinary tract infections in the emergency department with machine learning. PLoS One. 2018;13(3):e0194085. 18. Mao Q, Jay M, Hoffman JL, Calvert J, Barton C, Shimabukuro D, et al. Multicentre validation of a sep- sis prediction algorithm using only vital sign data in the emergency department, general ward and ICU. BMJ open. 2018;8(1):e017833. 19. Horng S, Sontag DA, Halpern Y, Jernite Y, Shapiro NI, Nathanson LA. Creating an automated trigger for sepsis clinical decision support at emergency de- partment triage using machine learning. PLoS One. 2017;12(4):e0174708. 20. Goto T, Camargo Jr CA, Faridi MK, Yun BJ, Hasegawa K. Machine learning approaches for predicting disposition of asthma and COPD exacerbations in the ED. The Amer- ican journal of emergency medicine. 2018;36(9):1650-4. 21. Swaminathan S, Qirko K, Smith T, Corcoran E, Wysham NG, Bazaz G, et al. A machine learning approach to triag- ing patients with chronic obstructive pulmonary disease. PLoS One. 2017;12(11):e0188532. 22. Farion K, Michalowski W, Wilk S, O’Sullivan D, Matwin S. A tree-based decision model to support prediction of the severity of asthma exacerbations in children. Journal of medical systems. 2010;34(4):551-62. 23. Farion KJ, Wilk S, Michalowski W, O’Sullivan D, Sayyad- Shirabad J. Comparing predictions made by a prediction model, clinical score, and physicians. Applied clinical in- formatics. 2013;4(03):376-91. 24. Elikashvili I, Spina L, Ayalin T, Cheng J, Morley EJ, Singh J. an Evidence-based Review Of Acute Appendici- tis In Childhood. Pediatric Emergency Medicine Practice. 2012;9(3):1-11. 25. Deleger L, Brodzinski H, Zhai H, Li Q, Lingren T, Kirk- endall ES, et al. Developing and evaluating an automated appendicitis risk stratification algorithm for pediatric pa- tients in the emergency department. Journal of the Amer- ican Medical Informatics Association. 2013;20(e2):e212- e20. 26. Chapman WW, Dowling JN, Wagner MM. Classification of emergency department chief complaints into 7 syn- dromes: a retrospective analysis of 527,228 patients. An- nals of emergency medicine. 2005;46(5):445-55. 27. Cameron A, Ireland AJ, McKay GA, Stark A, Lowe DJ. Pre- dicting admission at triage: are nurses better than a sim- ple objective score? Emerg Med J. 2017;34(1):2-7. 28. Barak-Corren Y, Fine AM, Reis BY. Early prediction model of patient hospitalization from the pediatric emergency department. Pediatrics. 2017;139(5):e20162785. 29. Hong WS, Haimovich AD, Taylor RA. Predicting hospi- tal admission at emergency department triage using ma- chine learning. PLoS One. 2018;13(7):e0201016. 30. Graham B, Bond R, Quinn M, Mulvenna M. Using data mining to predict hospital admissions from the emer- gency department. IEEE Access. 2018;6:10458-69. 31. Sun Y, Heng BH, Tay SY, Seow E. Predicting hospital ad- missions at emergency department triage using routine This open-access article distributed under the terms of the Creative Commons Attribution NonCommercial 3.0 License (CC BY-NC 3.0). Downloaded from: http://journals.sbmu.ac.ir/aaem 9 Archives of Academic Emergency Medicine. 2019; 7 (1): e34 administrative data. Academic Emergency Medicine. 2011;18(8):844-50. 32. Peck JS, Gaehde SA, Nightingale DJ, Gelman DY, Huck- ins DS, Lemons MF, et al. Generalizability of a sim- ple approach for predicting hospital admission from an emergency department. Academic Emergency Medicine. 2013;20(11):1156-63. 33. Rojas JC, Carey KA, Edelson DP, Venable LR, Howell MD, Churpek MM. Predicting intensive care unit read- mission with machine learning using electronic health record data. Annals of the American Thoracic Society. 2018;15(7):846-53. 34. Taylor RA, Pare JR, Venkatesh AK, Mowafi H, Melnick ER, Fleischman W, et al. Prediction of In-hospital Mortality in Emergency Department Patients With Sepsis: A Lo- cal Big Data–Driven, Machine Learning Approach. Aca- demic emergency medicine. 2016;23(3):269-78. 35. Barnes S, Hamrock E, Toerper M, Siddiqui S, Levin S. Real-time prediction of inpatient length of stay for dis- charge prioritization. Journal of the American Medical Informatics Association. 2015;23(e1):e2-e10. 36. Lucini FR, Fogliatto FS, da Silveira GJ, Neyeloff JL, An- zanello MJ, Kuchenbecker RdS, et al. Text mining ap- proach to predict hospital admissions using early med- ical records from the emergency department. Interna- tional journal of medical informatics. 2017;100:1-8. 37. Tang F, Xiao C, Wang F, Zhou J. Predictive modeling in ur- gent care: a comparative study of machine learning ap- proaches. JAMIA Open. 2018;1(1):87-98. 38. Liu W, Wang Z, Liu X, Zeng N, Bell D. A novel parti- cle swarm optimization approach for patient cluster- ing from emergency departments. IEEE Transactions on Evolutionary Computation. 2018. 39. Abad-Grau MM, Ierache J, Cervino C, Sebastiani P. Evolu- tion and challenges in the design of computational sys- tems for triage assistance. Journal of biomedical infor- matics. 2008;41(3):432-41. 40. Levin S, Toerper M, Hamrock E, Hinson JS, Barnes S, Gardner H, et al. Machine-learning-based electronic triage more accurately differentiates patients with re- spect to clinical outcomes compared with the emer- gency severity index. Annals of emergency medicine. 2018;71(5):565-74. e2. 41. Olivia D, Nayak A, Balachandra M, editors. Machine Learning Based Electronic Triage for Emergency De- partment. International Conference on Applications and Techniques in Information Security; 2018: Springer. 42. Mondragon N, Istrate D, Wegrzyn-Wolska K, Garcia JC, Sanchez JC. Patient classification algorithm at urgency care area of a hospital based on the triage system. Jour- nal of Medical Informatics & Technologies. 2013;22. This open-access article distributed under the terms of the Creative Commons Attribution NonCommercial 3.0 License (CC BY-NC 3.0). Downloaded from: http://journals.sbmu.ac.ir/aaem Introduction Machine learning techniques and its different types Review of studies Conclusion Appendix References