Emergency. 2017; 5 (1): e28 OR I G I N A L RE S E A RC H Role of Feedback during Evaluation in Improving Emer- gency Medicine Residents’ Skills; an Experimental Study Ali Vafaei1, Kamran Heidari1, Mohammad-Ali Hosseini2, Mostafa Alavi-Moghaddam3∗ 1. Department of Emergency Medicine, Loghman Hakim Hospital, Shahid Beheshti University of Medical Sciences, Tehran, Iran. 2. Rehabilitation Management Department, University of Social Welfare and Rehabilitation Sciences, Tehran, Iran. 3. Department of Emergency Medicine, Imam Hossein Hospital, Shahid Beheshti University of Medical Sciences, Tehran, Iran. Received: October 2016; Accepted: November 2016; Published online: 10 January 2017 Abstract: Introduction: Evaluation of students’ learning in clinical education system is one of the most important and challenging issues that facilities in this field have been facing. The present study aimed to evaluate the role of feedback during evaluation in increasing emergency medicine residents’ clinical skills. Method: The present experimental study was performed on all second year emergency medicine residents of two educational hospi- tals, Tehran, Iran, with switching replications design and before-after method. They were randomly divided to two groups (with or without feedback) and evaluated three times regarding chest ultrasonography for trauma patients, using direct observation of procedural skills (DOPS) and valid and reliable checklist. Data were ana- lyzed using SPSS 20. Results: 30 emergency medicine residents with the mean age of 36.63 ± 30.30 years were devided to two equal groups (56.7% male). Studied groups were similar regarding the baseline characteristics. In both groups, obtained scores showed a significant increase from the first to the third evaluation (p < 0.001). Mean scores of first and second evaluations were 10.24 ± 0.77, 17.73 ± 0.46 in feedback receivers and 9.73 ± 0.77 and 12.13 ± 0.47 in others (p < 0.001). Mean third evaluation scores after switching replication were 18.53 ± 0.22 in feedback receivers and 18.99 ± 0.22 in others (p = 0.213). Conclusion: Based on the findings of the present study, giving feedback after evaluating the second year emergency medicine residents regarding chest ultrasonography for trauma patients, led to a significant improvement in their scores in future evaluations and consequently their skill. Keywords: Education, medical, graduate; emergency medicine; learning; formative feedback © Copyright (2017) Shahid Beheshti University of Medical Sciences Cite this article as: Vafaei A, Heidari K, Hosseini MA, Alavi-Moghaddam M. Role of Feedback during Evaluation in Improving Emergency Medicine Residents’ Skills; an Experimental Study. Emergency. 2017; 5 (1): e28. 1. Introduction Training of clinical skills is one of the essential parts of med- ical education. Medical education facilities have been de- veloped for creating measurable changes in clinical practice (1). Evaluation of students’ learning in clinical education sys- tem is one of the most important and challenging issues that facilities in this field have been facing (2). In most clinical courses, evaluation methods do not have sufficient efficiency for assessment of clinical skills (3-5). Common evaluation methods have failed to assess students’ clinical skills accu- ∗Corresponding Author: Mostafa Alavi-Moghaddam; Emergency Depart- ment, Imam Hossein Hospital, Shahid Madani Street, Imam Hossein Square, Tehran, Iran. Tel: +989122967248 E-mail: mosalavi@yahoo.com rately, they are not considered as tools for learning. To solve prominent problems, possible strengths and weaknesses of educational facilities were counted and then positive aspects of the educational facilities were improved and their short- comings were eradicated (6). Investigators in medical ed- ucation field assessed some methods of clinical evaluation such as performance observation, 360 degree evaluation, ob- jective structured clinical examination, mini-clinical evalu- ation exercise (Mini-CEX) and direct observation of proce- dural skills (DOPS), which have been used for evaluation of medical students and their clinical environments. Effective assessment of medical students’ clinical skills can improve their motivation and help professors have a more accurate measurement of their actual skills (7). DOPS is known as a new procedure for evaluation of clinical practice skills of medical residents and students, which provides a suitable This open-access article distributed under the terms of the Creative Commons Attribution NonCommercial 3.0 License (CC BY-NC 3.0). Downloaded from: www.jemerg.com A. Vafaei et al. 2 feedback tool for medical professors (2). Use of formative feedback has positive effects on professional behavior, atti- tude and communication for medical students (8). Emer- gency department is one of the critical hospital wards in which residents must have clinical skills to be prepared for primary health care procedures, diagnosis and treatment of referred patients. Based on the above-mentioned points, the present study aimed to evaluate the role of feedback during evaluation in increasing emergency medicine residents’ clin- ical skills. 2. Methods 2.1. Study design and setting The present experimental study was performed on all sec- ond year emergency medicine residents of two educational hospitals affiliated with Shahid Beheshti University of Med- ical Sciences (Imam Hossein and Loghman hakim), Tehran, Iran, from June to November 2015, with switching replica- tions design and before-after method. The study aimed to evaluate the role of feedback during evaluation (FDE) in in- creasing residents’ skills regarding chest ultrasonography for trauma patients. All researchers adhered to ethical recom- mendations of Helsinki protocol and confidentiality of par- ticipants’ information during the study period. 2.2. Ethical Consideration Individuals participated in the study by giving informed con- sent, in addition, data were analyzed as a pool, and the name of individuals was not mentioned. The researchers promised to share the results with the participants on demand. All the patients that participated in rating the skills of residents were selected based on the indications provided in references and were given sufficient information regarding the method of the study. Any patient that wanted to leave the study or the hospital was excluded from the study. Replacements were se- lected for the 8 patients that decided to leave the study or the hospital. This study did not interfere with the routine diag- nostic and treatment process of the patients or impose addi- tional costs on them. 2.3. Participants Study participants were selected from second year emer- gency medicine residents, who had passed their first year promotion exam, using census sampling. All participants were trained regarding chest ultrasonography for trauma pa- tients during their first year of the residency program. They were randomly divided into 2 groups of Loghman Hakim (group 1) or Imam Hossein Hospital (group 2), using simple randomization method. In each studied hospital, one emer- gency medicine professor, who was informed regarding the study design and DOPS method, was responsible for evalu- ation of residents. Both attends, who were responsible for evaluation of the participants, were informed regarding the method of the study and its details in a 4-hour coordinator session. 2.4. Procedure After preparing essential tools for doing the procedure, par- ticipants were evaluated regarding chest ultrasonography for blunt trauma patients using DOPS method. Initially, all the participants (separately in both study hospitals) were evalu- ated using a checklist designed for this purpose and DOPS method in off-duty hours and their scores were recorded. At this stage, after the evaluation was finished, the residents in Loghman Hakim Hospital were given feedback on their weak and strong points for 5 minutes but those in Imam Hossein Hospital were evaluated traditionally and without feedback. After 2 months, the evaluation was repeated for residents of both hospitals. However, this time evaluation method was switched between the 2 groups and this time residents of Imam Hossein Hospital were given feedback and those in Loghman Hakim Hospital took were evaluated traditionally and without feedback. Finally, about 2 months later, all the second year residents of both hospitals were evaluated again, for the third time. Duration of evaluation was 15 minutes in all 3 stages, and when the residents were supposed to receive feedbacks it would be organized in 5 minutes. Effort was made to provide similar environmental conditions such as time, place, ultrasonography device, evaluated patients, etc. for both groups. 2.5. Data gathering Data gathering tool was a standard (20-item) checklist pre- pared according to evidence-based texts regarding the re- quirements that should be met in this procedure (appendix 1). Items of the checklist were designed to evaluate 3 gen- eral parts of the procedure including preparation (4 items), carrying out the procedure (13 items), and post-procedure measures (3 items). Each correct answer was given 1 point and for wrong answers, no point was given. Consequently, the maximum and minimum obtainable scores for each par- ticipant were 20 and 0, respectively. A score between 0 and 10 was considered as fail and a score between 10 and 20 was considered as pass. Score above 17 was classified as an excel- lent score. Validity of the prepared checklist was approved by 10 emergency medicine professors with more than 5 years of experience from various universities. Reliability of the tools was calculated in a pilot study on 15 people and its correla- tion coefficient was estimated to be 85% based on Cronbach’s alpha. This open-access article distributed under the terms of the Creative Commons Attribution NonCommercial 3.0 License (CC BY-NC 3.0). Downloaded from: www.jemerg.com 3 Emergency. 2017; 5 (1): e28 Appendix 1: Evaluation checklist of chest ultrasonography for trauma patients using DOPS method Items Proper performance Points* Preparation Yes No 1. Correctly identifying the patient 2. Introducing self and being professional 3. Knowing indications of the procedure 4. Proper position of patient and device Ultrasonography performance 1. Preparing proper tools 2. Correct selection of probe 3. Evaluation regarding pneumothorax 4. Identifying and evaluating the 3 zones 5. Evaluating all intercostal spaces 6. Identification of pleural cavity 7. Lung point sign evaluation 8. Pleural sliding evaluation 9. Evaluation of comet tails/B-line 10. Evaluation of seashore sign in M-mode 11. Evaluation regarding hemothorax 12. Right side evaluation of hemothorax 13. Left side evaluation of hemothorax Post-procedure 1. Saving and printing the image 2. Interpretation of findings 3. Proper decision based on findings *: Each right answer gets 1 point and wrong answers receive no point (minimum 0 and maximum 20). 2.6. Statistical analysis Study variables were analyzed by SPSS software version 20. Mean ± standard deviation or frequency and percentage were used for quantitative and qualitative variables, respec- tively. Paired sample t-test and chi-square were used for comparing quantitative and qualitative variables between study groups. The Greenhouse-Geisser Correction test was used for assessing the impact of repeatedly performing DOPS. All P-values less than 0.05 were considered as signif- icant. 3. Results: 3.1. Baseline characteristics 30 emergency medicine residents with the mean age of 36.63 ± 30.30 (29 - 47) years participated (56.7% male). They were randomly allocated to either Loghman Hakim (group 1) or Imam Hossein hospitals (group 2). Table 1 shows the baseline characteristics of the participants. The two studied groups were similar regarding the baseline characteristics. 25 (83.4%) residents performed chest ultrasonography more than five times (56.7% more than 10 times) and 25 (83.3%) residents believed that chest ultrasonography for trauma pa- tients had medium to high difficulty level. Figure 1: Trend of scores (minimum 0 and maximum 20) in three evaluations of studied groups (p < 0.001, based on Greenhouse- Geisser Correction). 3.2. Outcomes Mean scores of the residents in the first, second, and third evaluations were 10.00 ± 0.54, 14.93 ± 0.32, and 18.72 ± 0.15, respectively. Figure 1 shows the scores obtained by the res- idents for the 3 evaluations done based on the hospital. In both groups, obtained scores showed a significant increase from the first to the third evaluation (p < 0.001) according to Greenhouse-Geisser Correction test. Mean score obtained for preparation, scan and post-scan items among all resi- This open-access article distributed under the terms of the Creative Commons Attribution NonCommercial 3.0 License (CC BY-NC 3.0). Downloaded from: www.jemerg.com A. Vafaei et al. 4 Table 1: Baseline characteristics of participants Variable Loghman Hakim Imam Hossein p Mean age (year) 34.93 ± 5.40 38.33 ± 5.23 0.091 Sex Male 9 (60) 8 (53.3) 0.71 Female 6 (40) 7 (46.7) Previous experience1 (time) 1 -4 3 (20) 2 (13.3) 5- 9 4 (26.7) 4 (26.7) 0.879 ≥ 10 8 (53.3) 9 (60) Background attitude2 (difficulty) Low 4 (26.7) 1 (6.7) Medium 10 (66.7) 11 (73.3) 0.241 High 1 (6.7) 3 (20) Mean promotion exam score3 81.06 ± 6.29 82.42 ± 6.27 0.565 Data were presented as mean ± standard deviation or frequency (percentage); 1: Previous experience of chest ultrasonography; 2: Attitude of the resident regarding the difficulty of the procedure; 3: Score in exam taken for promotion from year 1 to year 2 of the residency program (out of 100). Table 2: Comparison of mean evaluation scores between groups Evaluation Loghman Hakim Imam Hossein P-value Preparation (0 - 4) First 3.00 ± 0.75 2.1 ± 0.92 0.009 Second 4.00 ± 0.00 3.00 ± 0.85 < 0.0001 Third 3.93 ± 0.26 4.00 ± 0.00 0.326 Scan (0 - 13) First 6.87 ± 2.39 6.27 ± 2.43 0.501 Second 11.60 ± 0.99 7.33 ± 1.95 < 0.001 Third 12.53 ± 0.74 12.26 ± 0.59 0.287 Post scan (0 - 3) First 0.40 ± 0.63 1.20 ± 0.67 0.002 Second 2.13 ± 0.35 1.80 ± 0.86 0.176 Third 2.53 ± 0.64 2.27 ± 0.71 0.287 Total (0 - 20) First 10.26 ± 2.99 9.73 ± 2.94 0.626 Second 17.73 ± 1.09 12.13 ± 2.29 <0.001 Third 18.93 ± 0.96 18.53 ± 0.74 0.213 Data were presented as mean ± standard deviation. Table 3: Comparison of score quality between groups Evaluation Loghman Hakim Imam Hossein P-value First Failed 8 (53.3) 5 (33.3) Passed 7 (46.7) 10 (66.7) 0.462 Excellent 0 (0) 0 (0) Second Failed 0 (0) 2 (13.3) Passed 3 (20) 13 (86.7) < 0.001 Excellent 12 (80) 0 (0) Third Failed 0 (0) 0 (0) Passed 15 (100) 15 (100) NA Excellent 0 (0) 0 (0) Data were presented as frequency and percentage; NA: not applicable. This open-access article distributed under the terms of the Creative Commons Attribution NonCommercial 3.0 License (CC BY-NC 3.0). Downloaded from: www.jemerg.com 5 Emergency. 2017; 5 (1): e28 dents were 3.34 ± 0.91, 9.48 ± 3.16, and 1.72 ± 0.97, respec- tively. Tables 2 and 3 compare the scores obtained by each group in the 3 tests qualitatively and quantitatively. Mean scores of first and second evaluations were 10.24 ± 0.77, 17.73 ± 0.46 in feedback receivers and 9.73 ± 0.77 and 12.13 ± 0.47 in others (p < 0.001). Mean third score after switching groups were 18.53 ± 0.22 in feedback receivers and 18.99 ± 0.22 in others (p = 0.213). As can be seen, residents in Loghman Hakim Hospital were significantly better in the second exam both quantitatively and qualitatively. 4. Discussion: Based on the findings of the present study, giving feedback after evaluating the second year emergency medicine resi- dents regarding chest ultrasonography for trauma patients, led to a significant improvement in their scores in future eval- uations and consequently their skill. Previous studies have shown that assessment of trainees via DOPS can significantly improve skill learning among medical students and residents in some countries (9-11). In most medical education cur- riculums, DOPS was used for evaluation of main clinical skills such as intravenous cannulation, lumbar puncture, and endotracheal intubation. These procedures were observed in different clinical settings including outpatient clinics and emergency departments. Barton et al. and Thomas-Gibson et al. studies indicated that DOPS method causes an increase in residents’ competence in the fields of colonoscopy and en- doscopy (12, 13). According to the findings of Hauck S et al., use of DOPS for medical students in ultrasound training field caused a significant increase in their knowledge, con- fidence, self-motivation and understanding of anatomical structures. It is recommended that medical trainers observe medical students while doing the procedure on the patients, and ask them about indications, complications and post pro- cedure cares and immediately give them feedback for future improvement of their skills. Charleen Liu, who is active in the emergency medicine training field in the United King- dom, believes that the recent trend in medical education is rapidly moving from traditional routine examinations to col- lecting evidence of clinical competence and professional be- havior observed in clinical environments (work-based learn- ing) through methods such as DOPS, mini-CEX, and case- based discussion (CbD). This is consistent with the highest level of the Miller education pyramid (14). It is believed that some factors in DOPS such as more practice in implemen- tation of procedure, evaluations in several stages, and giv- ing feedback in each stage can explain improvement of clin- ical skill among study participants. Based on these results of Holmboe et al., DOPS leads to significant changes in stu- dents’ behavior and competence improvement, as well as in- crease of teachers’ confidence and satisfaction compared to traditional methods (15, 16). The results of the current study revealed the significant effect of giving feedback during edu- cational evaluation. Scores of the second evaluation was sig- nificantly higher both qualitatively and quantitatively in the group that had received feedback regarding their strong and weak points compared to those who had not. However, af- ter switching the method of evaluation in the second phase, the scores of both groups were at the same level in the third evaluation. Among the secondary outcomes of the study was that it seems that regular evaluation using DOPS itself can improve the skill score of the residents. 5. Limitations Among the limitations of the present study are the small sam- ple size and evaluation of the residents by 2 separate attends. In addition, considering the previous familiarity of attends with residents we cannot be sure if a conflict of interest ex- isted in their scoring or not. However, in this regard all efforts were made to minimize these limitations by randomly divid- ing the residents and giving explanations to the evaluators. 6. Conclusion: Based on the findings of the present study, giving feedback after evaluating the second year emergency medicine resi- dents regarding chest ultrasonography for trauma patients, led to a significant improvement in their scores in future eval- uations and consequently their skill. 7. Appendix 7.1. Acknowledgements The authors would like to thank all the staff of Medical Edu- cation Faculty of Shahid Beheshti University of Medical Sci- ences and Clinical Research Development Center of Logh- man Hakim Hospital for their cooperation. 7.2. Authors contribution All authors passed four criteria for authorship contribution based on recommendations of the International Committee of Medical Journal Editors. 7.3. Conflict of interest None. 7.4. Funding None. References 1. Abedini SAS, Aghamolaee T, Jome Zadeh A, Kamjo A. Clinical education problems from the perspective of This open-access article distributed under the terms of the Creative Commons Attribution NonCommercial 3.0 License (CC BY-NC 3.0). Downloaded from: www.jemerg.com A. Vafaei et al. 6 nursing and midwifery students Hormozgan University of Medical Sciences. Journal of Hormozgan University ofMedical Sciences. 2009;12:249-53. 2. Crossley J, Humphris G, Jolly B. Assessing health profes- sionals. Medical education. 2002;36(9):800-4. 3. Nouhi SFA, Foroud A, editors. Investigation of the prob- lems in clinical evaluation according to clinical teachers of medicine and dentists in Kerman University of Medi- cal Sciences 6th Medical Education Congress. 6th Medi- cal Education Congress S; 2003; Tehran: hahid Beheshti University of Medical Sciences. 4. Rushforth HE. Objective structured clinical examination (OSCE): review of literature and implications for nursing education. Nurse education today. 2007;27(5):481-90. 5. Schoonheim-Klein M, Walmsley AD, Habets L, van der Velden U, Manogue M. An implementation strategy for introducing an OSCE into a dental school. European journal of dental education : official journal of the Asso- ciation for Dental Education in Europe. 2005;9(4):143-9. 6. Smith-Strom H, Nortvedt MW. Evaluation of evidence- based methods used to teach nursing students to crit- ically appraise evidence. The Journal of nursing educa- tion. 2008;47(8):372-5. 7. Franko DL, Cousineau TM, Trant M, Green TC, Rancourt D, Thompson D, et al. Motivation, self-efficacy, physical activity and nutrition in college students: randomized controlled trial of an internet-based education program. Preventive medicine. 2008;47(4):369-77. 8. Heinzow HS, Friederichs H, Lenz P, Schmedt A, Becker JC, Hengst K, et al. Teaching ultrasound in a curricular course according to certified EFSUMB standards during undergraduate medical education: a prospective study. BMC medical education. 2013;13(1):1. 9. Veloski J, Boex JR, Grasberger MJ, Evans A, Wolfson DB. Systematic review of the literature on assessment, feed- back and physicians’ clinical performance: BEME Guide No. 7. Medical teacher. 2006;28(2):117-28. 10. Beard J, Strachan A, Davies H, Patterson F, Stark P, Ball S, et al. Developing an education and assessment frame- work for the Foundation Programme. Medical education. 2005;39(8):841-51. 11. Beard JD, Jolly BC, Newble DI, Thomas WE, Donnelly J, Southgate LJ. Assessing the technical skills of surgical trainees. The British journal of surgery. 2005;92(6):778- 82. 12. Barton JR, Corbett S, van der Vleuten CP, Programme EBCS. The validity and reliability of a Direct Observation of Procedural Skills assessment tool: assessing colono- scopic skills of senior endoscopists. Gastrointestinal en- doscopy. 2012;75(3):591-7. 13. Thomas-Gibson S, Bassett P, Suzuki N, Brown G, Williams C, Saunders B. Intensive training over 5 days improves colonoscopy skills long-term. Endoscopy. 2007;39(09):818-24. 14. Liu C. An introduction to workplace-based assessments. Gastroenterology and Hepatology from bed to bench. 2012;5(1):24. 15. Holmboe ES, Hawkins RE, Huot SJ. Effects of training in direct observation of medical residents’ clinical compe- tence: a randomized trial. Annals of Internal Medicine. 2004;140(11):874-81. 16. Chen W, Liao S-c, Tsai C-h, Huang C-c, Lin C-c, Tsai C- h. Clinical skills in final-year medical students: the re- lationship between self-reported confidence and direct observation by faculty or residents. ANNALS-ACADEMY OF MEDICINE SINGAPORE. 2008;37(1):3. This open-access article distributed under the terms of the Creative Commons Attribution NonCommercial 3.0 License (CC BY-NC 3.0). Downloaded from: www.jemerg.com Introduction Methods Results: Discussion: Limitations Conclusion: Appendix References