Microsoft Word - BeginwiththeEndinMindarticle.docx Published by University of Minnesota Libraries Publishing Begin with the End in Mind: Designing and Implementing an Effective System for Evaluation and Feedback of Learners in Regional Medical Campus Residency Settings Bernadette Miller, MD, FACP and Brent W. Beasley, MD, MBA DOI: https://doi.org/10.24926/jrmc.v3i1.2259 Journal of Regional Medical Campuses, Vol. 3, Issue 1 (2020) z.umn.edu/JRMC All work in JRMC is licensed under CC BY-NC Brent W. Beasley, MD, MBA; The University of Oklahoma—Tulsa Bernadette Miller, MD, FACP; The University of Oklahoma—Tulsa Corresponding author: Brent W. Beasley, MD, MBA; The University of Oklahoma—Tulsa, 4444 E. 41st St. Tulsa Okla, 74135 Cell: 816- 678-4122 Phone: 918-660-3456 Email: brent-beasley@ouhsc.edu All work in JRMC is licensed under CC BY-NC Volume 3, Issue 1 (2020) Journal of Regional Medical Campuses Perspectives Begin with the End in Mind: Designing and Implementing an Effective System for Evaluation and Feedback of Learners in Regional Medical Campus Residency Settings Bernadette Miller, MD, FACP and Brent W. Beasley, MD, MBA Abstract Developing an evaluation system within a medical education program can be daunting and confusing. The authors present a step by step approach, incorporating education theory, recent trends, and the many facets required by accreditation organizations. Introduction Regional Medical Campus residency programs began documenting progressive achievement of milestones soon after the accreditation organizations propagated the new platform.1,2 The work residency programs and their supporting organizations have done on defining the milestones and competencies is commendable; however, the effort required for an individual residency program at a regional medical campus to implement these can feel overwhelming. Warm et al documented their experience and recommendations for mapping Entrustable Professional Activities into residency assessment and evaluation systems.3,4 Their labors demonstrate the tension in making practical all items that must be accomplished within the parameters provided by accrediting bodies. To add to these, here is one program’s step-by-step approach built upon prior concepts. The purpose of this paper is to describe the 4 phases involved in creating and applying a complete system for the evaluation of learners in any regional medical campus clinical setting. By sharing these methods, educators will be able to: 1) identify and outline evaluation goals and expectations for learners, 2) develop written evaluations linked to these goals with descriptors that result in a “shared mental model” of entrustable professional activities, competencies, and milestones for learners and evaluators,5 3) parcel assessments into everyday clinical situations, creating calendars for evaluators and learners, and 4) implement evaluation sessions, providing face-to-face feedback through the ADAPT6 or the R2C27 feedback model, and assessments using online evaluations. Creating the Evaluation System A rule of thumb for educators and evaluation developers is summed up in Covey’s “Seven Habits of Highly Successful People”: begin with the end in mind.8 For a program director, 2 end goals must be met. Our primary job is to develop residents into qualified physicians. Second, we must meet our accreditation requirements. Throughout this paper, we use as an example the creation of an outpatient general internal medicine continuity clinic evaluation for residents. Our residents spend every fifth week throughout the year in this clinic. We will demonstrate our process by building a summative evaluation of their continuity clinic progress to be used twice yearly. Identify evaluation goals and expectations Using the “Five W’s” (where, why, what, who, when, and how), having already stated our where (i.e. GIM Continuity Clinic), next we identify: • the overall purpose of this evaluation, • the content being assessed, • who will be the evaluator, • what timing intervals are required or desired, and • how the evaluations should be performed and the specific curricular requirements to be met. First, we turn to our accrediting body to ensure we meet their requirements, building adherence into the daily tools we use.9 Pulling verbiage from the accreditation requirements, we must assess our residents in data gathering, clinical reasoning, patient management, and procedures, by direct observation and with feedback. The timing is at least semiannually, with multiple evaluators, and must be progressive. Create the Expectations Document: Linking evaluations to learning goals The next step is to create our Expectations Document to guide both the evaluator and the learner, and to know what suppositions will be tracked. It should provide: 1) answers to the questions above (the who, where, what, how, when), 2) a narrative description of the type of patient encounter to be observed, 3) a narrative description of the verbal feedback session that should follow the observed encounter, and 4) a DOI: https://doi.org/10.24926/jrmc.ADDHERE Journal of Regional Medical Campuses, Vol. 3, Issue 1 Perspectives sample of the expected written summative evaluation. The document should be written so that both parties, the evaluator and the learner, can read the same expectations and anticipate what ought to occur during the evaluation. Here is an example: Figure 1: Documenting Learner and Evaluator Expectations Create the Evaluation Form To create the Evaluation Form, begin by sequencing each evaluation's timing in the overall training learning curve. To do so, answer the following checklist: • Should the content areas be assessed chronologically? • Must a trainee learn something basic (novice level) before learning new content (knowledge building)? • Is mastering a certain skill required before learning others? Can the knowledge or skill be assessed at any time during training? • At what training intervals is it appropriate to reassess? To find the answers to these questions we must determine what is considered the standard “entry level” abilities. As reflected in the AAMC Entrustable Professional Activities for graduating medical students, primary care internal medicine interns begin clinic with baseline competence in data gathering, clinical reasoning, patient management, and general procedures of a physician.10 Next, data gathering is generally mastered prior to clinical reasoning and patient management and should be evaluated earlier during training. Procedures, however, occur throughout residency and may be assessed at any time. Finally, at least 2 evaluations in each area are necessary throughout the year to demonstrate progressive improvement. By knowing these parameters, an educator can design and sequence the evaluations: one evaluation form to be administered at least twice to show progression in each learning area as required by the accreditation requirements. The evaluation form will be populated with knowledge, attitudes, and skill descriptors (KAS), observable actions and behaviors expected of a licensed practitioner by the completion of training. For GIM Continuity Clinic, we approached our outpatient faculty to describe the KAS required to perform their job. Also, we mined the ACGME’s 6 core competencies, subdivided into 22 milestones, as a major source of evaluative descriptors. We recommend no more than 6-8 descriptors per evaluation to avoid evaluator response fatigue. Our final 8 descriptors are in Figure 2. Figure 2: Selecting the 6-8 Evaluation Descriptors and the Competency Milestones They Meet Link Descriptors to Milestones Next, the descriptors must be linked to specific competency assessment milestones.11 This will be important for demonstrating to the accrediting body that the residency tracks competence in these areas. For instance, the descriptor “the resident elicited and documented appropriate history of present illness for an acute problem from patient” links to the PC-1 milestone for patient care “gathers and emphasizes essential and accurate information to define each patient's clinical problem.” Looking further at other milestones, the descriptor also links to MK-1 for medical knowledge. Programs should link each descriptor to as many milestones as appropriate and maintain documentation of these links. See our example in Figure 3. DOI: https://doi.org/10.24926/jrmc.ADDHERE Journal of Regional Medical Campuses, Vol. 3, Issue 1 Perspectives Figure 3: Linking Evaluations to Competency Milestones As previously described, our accrediting body requires trainee evaluations demonstrate “progressive” learning: the lowest level for the novice learner and the highest for “aspirational competence.” As the evaluation form is populated with the KAS descriptors, we use a 1 to 5-point scale grounded by progressive designations: “critical deficiency,” “Intern/PGY-1 Level,” “Senior/PGY-2 Level,” “Senior/PGY-3 Level,” and “Aspirational Level.” A partial example of our final continuity clinic evaluation is shown below in Figure 4. Figure 4: Final Continuity Clinic Evaluation (partially shown) As each evaluation is developed, we maintain a Tracking Map demonstrating where each milestone is evaluated in a progressive fashion throughout the course of the entire residency. Our example of our Residency Milestone Evaluation Map in an early stage of documentation is seen below in Figure 5. Figure 5: Residency Milestone Evaluation Map Organized Implementation: Scheduling Evaluations Creating an academic evaluation schedule is important, including when your evaluations are going to be administered, which evaluators are needed, how many learners you have, and how many evaluators are required. Sometimes, all portions of an evaluation may NOT be completed in one session and might require separating them into shorter mini evaluations. For example, in our academic calendar during the first year, we assess interns on “data gathering” in the first 6 months, and again in the following 6 months; however, “clinical reasoning” will be assessed at the beginning of their second year and again at the beginning of their third year. Additionally, it may be advantageous to have one particular evaluator assess a resident’s clinical reasoning at 2 to 3 different points in time to ensure the resident is progressing. Therefore, we maintain a chronological evaluation schedule. To achieve this, the evaluation appointment must be scheduled on individual evaluators’ calendars. If evaluators are unaware that they are expected to evaluate a resident, it will not happen. We distribute our expectation documents prior to the scheduled evaluation sessions and set reminders. Reminders should be sent soon after the evaluation encounter if deadlines are not met and repeated at short intervals to ensure evaluators recall the resident’s performance when completing the evaluation. We recommend providing faculty development sessions prior to asking faculty to evaluate your residents, training them in the ADAPT or the S2C2 feedback model.12 Additionally, the faculty development should provide time for faculty to discuss the required skills and behaviors and the levels of performance for each post-graduate year to improve inter- rater reliability. Discussion The time required to implement this system depends on the type and number of evaluations needed for a given training experience and whether only one person or a team is working on it. For a longitudinal evaluation of the Ambulatory Clinic Experience as described above, it required we consider learner advancement along the continuum of training. We require our faculty to “check out” each resident in the room in front of the patient, often reviewing their history, DOI: https://doi.org/10.24926/jrmc.ADDHERE Journal of Regional Medical Campuses, Vol. 3, Issue 1 Perspectives exam, and assessments in real time with allowance for the progression through residency, so that interns provide us more detailed information in the room than third year residents. This provides each faculty member with a “daily” understanding and knowledge of how our residents perform. We believe this likely precludes a “game-day” change in how our residents behave. We developed the evaluations sequentially, first for interns, then PGY2, then PGY3. The whole development process required 5-15 hours of work, resulting in 6 different evaluations for Ambulatory Clinic throughout residency. Overall, revamping the evaluation system for the whole residency is a large effort that warrants a team/committee to take this on with a one-year aim for completion. Conclusion We believe the benefits of this system are substantial. First, it highlights the formative progression of a resident through training. Second, it ensures the training program is in compliance with accreditation regulations around evaluation and feedback. Third, it demarcates where and when resident milestones and competencies are expected to be achieved and demonstrated. Fourth, it results in greater objectivity in evaluations rather than subjective impressions of trainees by evaluating faculty. Finally, faculty assignments and clear expectations can be scheduled and tracked for the department. References 1. Weinberger SE, Pereira AG, Iobst WF, Mechaber AJ, Bronze MS; Alliance for Academic Internal Medicine Educaiton Redesign Task Force II. Competency-based education and training in internal medicine. Ann Intern Med. 2010;153:751-6. 2. Hauer KE, Vandergrift J, Lipner RS, Holmboe ES, Hood S, McDonald FS. National Internal Medicine Milestone Ratings: Validity Evidence From Longitudinal Three-Year Follow-up. Acad Med. 2018 Aug;93(8):1189-1204. 3. Warm EJ, Mathis BR, Held JD, Pai S, Tolentino J, Ashbrook L, Lee CK, Lee D, Wood S, Fichtenbaum CJ, Schauer D, Munyon R, Mueller C. Entrustment and mapping of observable practice activities for resident assessment. J Gen Intern Med. 2014 Aug;29(8):1177-82. 4. Warm EJ, Held JD, Hellmann M, Kelleher M, Kinnear B, Lee C, O'Toole JK, Mathis B, Mueller C, Sall D, Tolentino J, Schauer DP. Entrusting Observable Practice Activities and Milestones Over the 36 Months of an Internal Medicine Residency. Acad Med. 2016 Oct;91(10):1398-1405. 5. Carraccio C, Englander R, Holmboe ES, Kogan JR. Driving Care Quality: Aligning Trainee Assessment and Supervision Through Practical Application of Entrustable Professional Activities, Competencies, and Milestones. Acad Med. 2016 Feb;91(2):199-203. 6. Fainstad T, Mcclintock A A, Van Der Ridder M J, et al. (December 11, 2018) Feedback Can Be Less Stressful: Medical Trainee Perceptions of Using the Prepare to ADAPT (Ask-Discuss-Ask-Plan Together) Framework . Cureus 10(12): e3718. 7. Sargeant J, Lockyer J, Mann K, Holmboe E, Silver I, Armson H, Driessen E, MacLeod T, Yen W, Ross K, Power M. Facilitated Reflective Performance Feedback: Developing an Evidence- and Theory-Based Model That Builds Relationship, Explores Reactions and Content, and Coaches for Performance Change (R2C2). Acad Med. 2015 Dec;90(12):1698-706. 8. Covey, S. R. (1989). The seven habits of highly effective people: Restoring the character ethic. New York: Simon and Schuster. 9. Accreditation Council for Graduate Medical Education. ACGME Program Requirements for Graduate Medical Education in Internal Medicine. https://www.acgme.org/Portals/0/PFAssets/ProgramReq uirements/140_internal_medicine_2017-07-01.pdf effective: July 1, 2017. Accessed March 14, 2019. 10. Association of American Medical Colleges (AAMC). Core entrustable professional activities for entering residency: curriculum developers’ guide. AAMC iCollaborative, Washington, DC. 2014. https://www.mededportal.org/icollaborative/resource/8 87. Accessed 1 Jan 2016. 11. The Internal Medicine Milestone Project. https://www.acgme.org/Portals/0/PDFs/ Milestones/ InternalMedicineMilestones.pdf?ver=2017-07-28- 090326-787. Published July 2015. Accessed March 13, 2019. 12. Sargeant J, Lockyer JM, Mann K, Armson H, Warren A, Zetkulic M, Soklaridis S Könings KD, Ross K, Silver I, Holmboe E, Shearer C, Boudreau M. The R2C2 Model Residency Education: How Does It Foster Coaching and Promote Feedback Use? Acad Med. 2018 Jul;93(7):1055- 1063.