Microsoft Word - NarrativeFeedbackinFacultyArticle.docx Published by University of Minnesota Libraries Publishing Narrative Feedback in Faculty Development Ralitsa Akins, MD, PhD DOI: https://doi.org/10.24926/jrmc.v2i2.1220 Journal of Regional Medical Campuses, Vol. 2, Issue 2 (2019) z.umn.edu/JRMC All work in JRMC is licensed under CC BY-NC Ralitsa Akins, MD, PhD; Provost, Des Moines University Corresponding author: ralitsa_akins@yahoo.com; 515-271-1505 All work in JRMC is licensed under CC BY-NC Volume , Issue 6 (2019) Journal of Regional Medical Campuses Original Reports Narrative Feedback in Faculty Development Ralitsa Akins, MD, PhD; Provost, Des Moines University Abstract Research on faculty development and its generalizability is lagging compared to other areas of research in medical education. Providing feedback has been identified as a skill in need of improvement for medical educators, both in the classroom and at the clinical bedside. Surprisingly, little has been published on faculty skills in providing feedback during faculty development sessions, and more specifically, providing narrative feedback. An IRB-approved study analyzed the outcomes of 73 faculty development sessions conducted within one academic year. A qualitative study of the narrative portion of end-of-session evaluations examined type and scope of narrative feedback provided to presenters about their presentation skills as well as about the perceived quality and usefulness of the faculty development sessions. The findings from this study suggest that further and more in-depth professional development in providing feedback is warranted, preferably early in faculty's professional development. Introduction Research on faculty development and its generalizability is lagging compared to other areas of research in medical education.9 Providing feedback has been identified as a skill in need of improvement for medical educators, both in the classroom and at the clinical bedside.12,14 Surprisingly, little has been published on faculty skills in providing feedback during faculty development sessions, and more specifically, providing narrative feedback aiming to guide improvement of presenters’ skills or session organization. In general, faculty development includes activities used to assist faculty in their roles as teachers, researchers, clinicians, administrators, and leaders.3 Traditionally, needs assessment utilizing surveys or focus groups have been used to determine the needs for faculty development.10 It has been suggested that in creating faculty development sessions, the session “developers” use the processes of negotiation, construction, and attuning of knowledge to actively interact with the environment and respond to faculty development needs.1 The need for improved faculty development involving teaching, communication, and practice behaviors is well documented.5,15 Objective Structured Teaching Evaluations (OSTEs) have been used to assess teaching and assessment skills, and to provide feedback to faculty participating in professional development programs.6 While the role of medical teachers in giving feedback to learners has been more extensively explored,4 little is known about whether teachers use the same conceptual principles in providing feedback related to faculty development. Providing peer review in teaching has been a challenging task, and peer-review evaluations may be difficult to implement.11 Materials and Methods An IRB-approved study (WSU No.16228, 2017) examined the outcomes of faculty development sessions to explore the skills of faculty in providing narrative feedback as assessed by the narrative portion of faculty development end-of-session evaluations. Seventy-three faculty development sessions were conducted on 4 campuses of one medical school during one academic year. Three hundred and three individuals attended one or more sessions and provided end-of-session evaluations that included narrative portions about: 1) the attributes of the presenter(s), and 2) the perceived usefulness of the professional development offerings. The narratives from the evaluations were analyzed with the help of qualitative software (NVivo Professional 11.x64 by QSR). Cluster analysis, frequency queries, and tree map analysis were performed. A word cloud was created to visually demonstrate the narrative word frequencies. Results Often participants provided one-word narrative feedback, where the words "great", "good", and "helpful" were the top 3 choices used, as visualized in the word cloud (Figure 1). Figure 1. Word cloud presenting a visual of the most frequent word choices in providing narrative feedback in faculty development session evaluations DOI: https://doi.org/10.24926/jrmc.ADDHERE Journal of Regional Medical Campuses, Vol. 1, Issue 6 Original Reports This finding was also demonstrated in the narrative tree map analysis of the total of 1006 distinctive words found in the narratives from the feedback notes. These words were used with varied frequencies, and showed specific patterns of use, as shown on Figure 2. Figure 2. Clustering of feedback narratives Although separate fields were available in the session feedback form to provide responses about the quality of the session and the characteristics of the presenter(s), session participants mixed the types of feedback they were asked to provide in the 2 separate fields. For example, comments about the presenter’s attributes would be written in the field asking about session usefulness: “Loved presenter’s teaching style and approach.” “Presenter was very organized.” “Was very responsive to the needs of the faculty.” Vice versa, comments about the session’s usefulness would be found in the field asking about presenter’s attributes, as demonstrated in the following comments: “Both the lecture and the workshop were excellent.” “Too much information squeezed into 1 ½ hours. Perhaps one well-developed program per section of talk would suffice.” “This session was great! Practical, informative, and great discussion.” Useful feedback that could aid future session or presenter skills improvement were seen less often than expected. Many feedback narratives consisted of description of feelings (e.g., "Loved it", “Impressed”) or writing "Thank you". Examples of comments that did not guide to the specific activity that the participants appreciated included, “Great job!”, “Great session”, “Very good!”, “Well done!” and “Great work!”. Session participants also used the opportunity to provide feedback outside of the areas of interest (presenter’s attributes and session usefulness), writing about unmet needs and expectations: “We need more support for coordination of course meetings and timelines.” “I’m still concerned about information overload for faculty and students.” “I didn’t get specific directions or parking information FYI.” The skills of faculty attending professional development sessions in providing feedback about session usefulness and presenter skills varied vastly between session attendees as related to usability of provided feedback and ability to focus answers on the questions asked. Narrative comments made it evident that there could be a possible disconnect between the goals of the requested feedback (e.g. goal to improve the session usefulness and presenter skills) and the nature of the written comments (exploring unmet needs and describing feelings and offering non-specific comments such as “Great job!”). Overall, narrative feedback suggested that faculty preferred small group, interactive sessions. Faculty were appreciative of the opportunity to participate in professional development sessions, as demonstrated in the comment, “Thank you for these very well organized sessions – great information, great food, great colleagues.” Discussion Faculty’s skills have been shown to improve after a participation in a faculty development program.2 In one study, faculty-novices to peer evaluations in teaching benefited from structured training on providing frame-of- reference feedback when assessing a performance at the workplace.8 Another study showed that greater agreements between ratings from novices and experts were documented upon completion of focused hands-on training, placing the process of assessment and feedback beyond the teacher- learner framework, and within providing feedback-to-peers realm. Both performance-dimension training, and frame-of- reference training positively influenced faculty’s approach in providing feedback.7 Therefore, introducing faculty development sessions on providing peer-to-peer feedback early in the professional development program could improve the usefulness of the participants’ comments which will better aid improvement of future sessions and presenters’ skills. Participants in the faculty development sessions included in this study confirmed the perceived usefulness of the faculty development offerings and/or the perceptions that their skills improved: “They challenged me to reframe the way I was thinking about things and it helped me fill in a few ‘gaps’ that I was unable to ‘articulate’ prior to the session.” DOI: https://doi.org/10.24926/jrmc.ADDHERE Journal of Regional Medical Campuses, Vol. 1, Issue 6 Original Reports “This didactic was relevant to what I needed training on. This will be helpful now/tomorrow. It is helpful to learn by role playing and the “workshop teaching style.” “Thank you – I learned things and I will try to incorporate them.” Giving feedback to the participants in a faculty development activity (e.g. sending the results from an end-of-session satisfaction survey back to the session participants) has been perceived to enhance future participation as well as the credibility of the people/institution conducting the activity/survey. Feedback provided back to the participants is known as a “reciprocal” feedback, and it entails sending to the participants a summary of the findings. Importantly, reciprocal feedback could enhance future participation, future feedback reliability, and serve as an incentive to participate in change.13 Reciprocal feedback was a concept used for the sessions in this study. The office organizing the faculty development sessions provided summary of the received feedback and what actions were taken, if any, back to the session participants. For example, if questions were asked during a session that were not immediately answered, the session organizers and the presenters provided additional information and responses in a follow-up communication to the session participants. In addition, all narratives, as originally written by the session participants, were sent as a collated document back to the presenter(s) to aid future improvement. While the narratives were provided by session participants in an anonymous way, some comments clearly identified the writers and were redacted to delete a person’s or department’s name, as applicable. In turn, session participants and presenters offered spontaneous, unsolicited follow-up feedback that they valued the opportunity to review the outcomes and appreciated the actions taken to respond to participants’ needs. Conclusion This study was conducted within 4 campuses of one institution over one academic year, and this may limit the study’s generalizability. The study suggested that faculty in medical educational programs would need focused skills development in providing narrative feedback early in their professional careers to include providing meaningful feedback to peers and in faculty development sessions. More studies including multiple institutions could lead to a better understanding of the needs in faculty development about narrative feedback. References 1. Baker, L., Leslie, K., Panisko, D., Walsh, A., Wong, A. Stubbs, B., & Mylopoulos, M. (2018). Exploring Faculty Developers’ Experiences to Inform Our Understanding of Competence in Faculty Development. Academic Medicine, Vol. 93(2): 265- 273. 2. Branch, W. T. Jr., Frankel, R., Gracey, C. F., Haidet, P. M., Weissmann, P. F., Cantey, P., Mitchell, G. A., & Inui, T. S., (2009). A Good Clinician and a Caring Person: Longitudinal Faculty Development and the Enhancement of the Human Dimensions of Care. Academic Medicine 84(1):117-125. 3. Centra J.A. (1978). Types of faculty development programs. J Higher Educ (49)151–162. 4. Cote, L. & Bordage, G. (2012). Content and Conceptual Frameworks of Preceptor Feedback Related to Residents' Educational Needs. Academic Medicine 87(9):1274-1281. 5. DaRosa, D. A., Skeff, K. Friedland, J. A., Coburn, M. Cox, S. Pollart, S. O'Connell, M. & Smith, S. Barriers to Effective Teaching. Academic Medicine 86(4):453- 459. 6. Julian, K., Appelle, N., O'Sullivan, P., Morrison, E.H., & Wamsley, M. (2012). The impact of an objective structured teaching evaluation on faculty teaching skills. Teach Learn Med. 24(1):3-7. 7. Kogan, J.R., Conforti, L.N., Bernabeo, E., Iobst, W., & Holmboe, E. (2015). How faculty members experience workplace-based assessment rater training: a qualitative study. Med Educ. 49(7):692- 708. 8. Newman, L. R., Brodsky, D., Jones, R. N., Schwartzstein, R. M., Atkins, K. M., & Roberts, D. H. (2016). Frame-of-Reference Training: Establishing Reliable Assessment of Teaching Effectiveness. Journal of Continuing Education in the Health Professions 36(3):206-210. 9. O'Sullivan, P.S., & Irby, D.M. (2011). Reframing research on faculty development. Acad Med. 86(4):421-8. 10. Pololi, L. H., Dennis, K., Winn, G. M., & Mitchell, J. (2003). A needs assessment of medical school faculty: Caring for the caretakers. Journal of Continuing Education in the Health Professions 23(1):21-29. 11. Strafford, K., Tartaglia, K., Mahan, J., Nagel, R., Verbeck, N., & Davis, J. (2016). Description and Evaluation of Peer Review of Teaching in an Integrated Medical School Curriculum. Obstetrics & Gynecology 128 Supplement 1:52S-53S, October 2016. 12. Telio S., Ajjawi, R., & Redehr, G. (2015). The “Educational Alliance” as a Framework for Reconceptualizing Feedback in Medical Education. Academic Medicine (90): 609-614. 13. Watson, R.A. (2015). Reciprocal Feedback: Closing the Loop on Postactivity Surveys. Journal of Continuing Education in the Health Professions 35(4):284-285. DOI: https://doi.org/10.24926/jrmc.ADDHERE Journal of Regional Medical Campuses, Vol. 1, Issue 6 Original Reports 14. Weinstein, D. F. (2015). Untying the Gordian Knot. Academic Medicine 90(5): 559-61. 15. Wilkes, M. S., Hoffman, J. R., Usatine, R. & Baillie, S. (2006). An Innovative Program to Augment Community Preceptors' Practice and Teaching Skills. Academic Medicine 81(4):332-341.