MANUSCRIPT TYPE CLINICIAN EXPERTISE COMMENTARY 26 Copyright © by Indiana State University Clinical Practice in Athletic Training All rights reserved. ISSN Online 2577-8188 Volume 7 – Issue 2 July 2024 Standardized Patient Evaluation Tool: A Valid Measure of the Core Competencies Lindsey E. Eberman PhD, LAT, ATC* & Jessica Edler Nye PhD, LAT, ATC** *Indiana State University, Terre Haute, IN **Grand View University, Des Moines, IA Content Focus Health Professions Education Correspondence 567 N 5th Street, DAT Suite, Room A-10, Terre Haute, IN 47809 E-mail: Lindsey.Eberman@indstate.edu Twitter: NA. Reference Eberman LE, Edler Nye J. Standardized Patient Evaluation Tool: A Valid Measure of the Core Competencies. Clin Pract Athl Train. 2024;7(2):26-34. https://doi.org/10.31622/2024/0007.02.4. Author Characteristics Dr. Lindsey Eberman is the Program Director for the post-professional doctorate in athletic training, bachelors degree in leadership and professional development, and Doctor of Philosophy in instructional leadership at Indiana State University. She earned her bachelors degree from Northeastern University in Boston, MA and her post-professional masters in advanced athletic training and doctorate in curriculum and instruction from Florida International University in Miami, FL. Dr. Eberman is an expert in pedagogy and assessment. Dr. Eberman has been recognized with the several awards recognizing her commitment to teaching, scholarship, and service to the profession at the local, state, regional, and national level. She facilitates doctoral student scholarship that improves the practice of athletic training and maintains a robust research agenda of her own amassing over 200 published manuscripts. Dr. Jessica Edler Nye is an Associate Professor and Program Director of the Master of Science in Athletic Training at Grand View University. She earned her bachelors degree from Central College and her post-professional masters and doctorate in curriculum and instruction from Indiana State University. Dr. Edler Nye is an expert in pedagogy and assessment. COMMENTARY In 2017, Dr. Jessica Edler Nye and I had the opportunity to develop and validate a standardized patient evaluation tool (SPET) (Appendix A) to measure the Institute of Medicine’s healthcare core competencies using a Delphi-panel review process.1 Although these core competencies (Table 1)2 have historically existed in the Commission on Accreditation of Athletic Training Education (CAATE) post-professional program standards and, until recently, the residency standards, they have also now been added to the professional education standards. There is also a crosswalk available between these core competencies and the AT Milestones, which were developed from the ACGME core competencies (Table 2).3 The SPET was developed because no high-quality mechanism existed to measure these outcomes, and as the program director of a CAATE-accredited post-professional Doctor of Athletic Training (DAT) program, it was my responsibility to ensure we were using a valid measure in our standardized patient encounters (SPEs) to show that program graduates were, in fact, advanced practice clinicians upon graduation. To establish validity, we first created five cases, representative of all the core competencies, each with a primary focus of one of the core competencies (Case 1: patient-centered care, Case 2: interprofessional and collaborative practice, Case 3: evidence-based practice, Case 4: quality improvement, Case 5: healthcare informatics). We then used a panel of 22 experts to conduct a 4-round Delphi panel review to establish consensus on the content validity of the SPET and its ability to apply to the SP cases. We established consensus on the evaluative criteria as appropriate (mean=3.6±0.6 out of 4, 65% strongly agree) and its use across https://doi.org/10.31622/2024/0007.02.4 Standardized Patient Evaluation Tool: A Valid Measure of the Core Competencies 27 Copyright © by Indiana State University Clinical Practice in Athletic Training All rights reserved. ISSN Online 2577-8188 Volume 7 – Issue 2 July 2024 the SPEs to evaluate the healthcare core competencies (mean=3.5±0.8 out of 4, 65% strongly agree) (Table 3). Experts indicated, through the Delphi-panel process, that quality improvement and healthcare informatics may be difficult to assess during a single SPE. We also worked to establish reliability but learned that more advanced training was necessary to ensure raters are consistently scoring learners. To create SPEs for the raters to evaluate, we implemented six different continuing professional development sessions, which were eligible for continuing education units and consisted of a 45-minute interactive lecture, a maximum 30-minute SPE that was video recorded, and a 15- minute group debriefing session. We then trained three independent raters (30-minutes) on the SPET and its evaluative criteria and asked them to review the SPEs (n=41). We established moderate reliability for the overall performance score (ICC=0.641) and each of the specific competencies demonstrated variable reliability (ICC: patient-centered care=0.778; interprofessional and collaborative practice=0.581; evidence-based practice=0.274; quality improvement=0.390; healthcare informatics=0.232). The raters also stated that there was limited opportunity to assess quality improvement and healthcare informatics during the SPEs. After establishing the SPET, the DAT program has implemented the tool over 1000 times as part of its clinical education curricula. This includes an initial SPE to establish baseline performance, and an additional 4 SPEs at the end of each semester, with a culminating progressive SPE experience (3 encounters with one patient case [onset to discharge]). This curricula also includes 4 virtual SPEs, which use only 2 individual competencies and the overall performance measure instead of the entire tool. Upon enrolling students in the newly- accredited Masters in Athletic Training (MAT) program at Indiana State, we also began integrating the SPET into their SPE evaluations, with broad use of the tool consistently across semesters in the program. The lessons learned from the regular use of the tool across both programs are that foundational training, as well as continued maintenance, are necessary to ensure accuracy. As DAT faculty, we meet once per semester to watch 1 to 2 SPEs, score them, calculate inter-rater reliability, and then discuss. We also have an established bias checklist to help with repeated use of the SPET when we are evaluating multiple encounters in a row. When new evaluators are introduced to the team, we do preliminary training and integration into virtual (formative) SPEs first before integrating them into the evaluative SPEs, with feedback provided to new evaluators throughout this process. In addition to training evaluators, we also provide performance feedback to the learners. For each encounter, they receive a copy of their SPET, a checklist of behaviors performed (consistent with simulation best practices), and a summary of how the core competencies, including how quality improvement and healthcare informatics could be integrated into the encounter. The summary outlines competent performance for that case. In both the MAT and DAT programs, we have learned that it is critical to train the learner in interpreting the results and responding to the feedback. Historically in athletic training education, we have misused the terms competent and proficient, and interchanged competency and proficiency. Moreover, the Board of Certification examination and program graduation add additional elements of confusion around competence. These missteps and misunderstandings make it difficult for learners to comprehend the behavior expectations around entering and continuing competence in the profession. In addition, competence is a continuum, not a “place achieved after investing time and effort.” Competence ebbs and flows due to use and disuse and a more complete understanding of the construct is critical for learners to embrace, so that they can use the feedback for self-improvement. The SPET is a staple tool in our post-professional, CAATE-accredited DAT program and helps us document performance across 6 semesters. We remind learners regularly that is not uncommon for competence to be fluid, especially when you don’t consistently apply the core competencies in practice. Therefore, it is not atypical for students to see drastic improvements in patient-centered care between the first and second Standardized Patient Evaluation Tool: A Valid Measure of the Core Competencies 28 Copyright © by Indiana State University Clinical Practice in Athletic Training All rights reserved. ISSN Online 2577-8188 Volume 7 – Issue 2 July 2024 semester, but in the third, when the focus is on interprofessional and collaborative practice, they may slip back into past behaviors. However, the ultimate goal is that the learners are demonstrating competent to proficient behavior in all core competency areas upon graduation. REFERENCES 1. Eberman LE, Edler JR. Developing, validating and establishing reliability of a standardized patient evaluation tool to measure healthcare core competency. J Athl Train. 2020;55(6):S-240. doi:10.4085/1062-6050-55.6s.S-1 2. The Core Competencies Needed for Health Care Professionals. Institute of Medicine. Health Professions Education: A Bridge to Quality. Washington, DC: The National Academies Press, 2003. 3. Sauers EL, Laursen RM, Pecha F. Walusz HJ. The Athletic Training Milestones. https://www.atmilestones.com/milestones.html. Accessed on April 24, 2024. https://www.atmilestones.com/milestones.html Standardized Patient Evaluation Tool: A Valid Measure of the Core Competencies 29 Copyright © by Indiana State University Clinical Practice in Athletic Training All rights reserved. ISSN Online 2577-8188 Volume 7 – Issue 2 July 2024 TABLE 1. Healthcare Core Competencies (IOM) Core Competency Description Provide patient- centered care − identify, respect, and care about patients’ differences, values, preferences, and expressed needs − relieve pain and suffering − coordinate continuous care − listen to, clearly inform, communicate with, and educate patients − share decision making and management − continuously advocate disease prevention, wellness, and promotion of healthy lifestyles, including a focus on population health Work in interdisciplinary teams − cooperate, collaborate, communicate, and integrate care in teams to ensure that care is continuous and reliable Employ evidence- based practice − integrate best research with clinical expertise and patient values for optimum care, and participate in learning and research activities to the extent feasible Apply quality improvement − identify errors and hazards in care − understand and implement basic safety design principles, such as standardization and simplification − continually understand and measure quality of care in terms of structure, process, and outcomes in relation to patient and community needs − design and test interventions to change processes and systems of care, with the objective of improving quality Utilize informatics − communicate, manage knowledge, mitigate error, and support decision making using information technology Standardized Patient Evaluation Tool: A Valid Measure of the Core Competencies 30 Copyright © by Indiana State University Clinical Practice in Athletic Training All rights reserved. ISSN Online 2577-8188 Volume 7 – Issue 2 July 2024 Table 2. CAATE Core Competency to AT Milestone Crosswalk CAATE Core Competency AT Milestones General Competencies & Sub-competencies Patient-Centered Care Patient-Care and Procedural Skills (Sub-competencies 1,2) Interpersonal and Communication Skills (Sub-competencies 1,2) Interprofessional Practice and Education Patient-Care and Procedural Skills (Sub-competency 7) Interpersonal and Communication Skills (Sub-competency 3) Professionalism (Sub-competency 2) Systems-Based Practice (Sub-competencies 1-4) Evidence-Based Practice Practice-Based Learning and Improvement (Sub-competency 1) Medical Knowledge (Sub-competency 3) Quality Improvement Practice-Based Learning and Improvement (Sub-competencies 2-4) Systems-Based Practice (Sub-competencies 1-4) Health Care Informatics Interpersonal and Communication Skills (Sub-competency 4) Systems-Based Practice )Sub-competency 6) Professionalism Professionalism (Sub-competencies 1-4) Standardized Patient Evaluation Tool: A Valid Measure of the Core Competencies 31 Copyright © by Indiana State University Clinical Practice in Athletic Training All rights reserved. ISSN Online 2577-8188 Volume 7 – Issue 2 July 2024 Table 3. Delphi Panelists Perceptions about SPET Evaluative Criteria Representing the Core Competencies Case 1 Case 2 Case 3 Case 4 Case 5 Patient-Centered Care 3.9 ± 0.3 3.8 ± 0.4 3.8 ± 0.4 3.9 ± 0.2 4.0 ± 0.0 Interprofessional and Collaborative Practice 3.5 ± 0.6 3.2 ± 0.7 3.9 ± 0.3 3.3 ± 0.7 3.2 ± 0.6 Evidence-Based Practice 3.7 ± 0.5 3.9 ± 0.3 3.6 ± 0.6 3.7 ± 0.5 3.7 ± 0.5 Quality Improvement 3.5 ± 0.6 3.3 ± 0.8 3.2 ± 0.7 3.4 ± 0.7 3.4 ± 0.7 Healthcare Informatics 3.6 ± 0.6 3.2 ± 0.6 3.4 ± 0.5 3.3 ± 0.7 3.1 ± 0.7 Likert Scale: 1=Strongly disagree, 2=Disagree, 3=Agree, 4=Strongly agree CLINICIAN EXPERTISE COMMENTARY 32 Copyright © by Indiana State University Clinical Practice in Athletic Training All rights reserved. ISSN Online 2577-8188 Volume 7 – Issue 2 July 2024 Appendix A. Standardized Patient Evaluation Tool (SPET) Evaluative Criteria:  Critically Deficient – These learner behaviors are not within the spectrum of developing competence. Instead they indicate significant deficiencies in performance.  Novice – A novice has a basic understanding of the knowledge and skill and applies the information within a specific set of rules or steps that they have learned. Skills are applied as taught, although typically without efficiency. Critical thinking is limited or not present. Understanding, deciding, and the outcomes are free of the context in which they are being applied.  Advanced Beginner – An advanced beginner has begun to develop a framework for the application of knowledge and skills. He or she begins to recognize situational aspects and applies appropriate skills. Practice efficiency is minimally evident, and critical thinking is implemented at certain points of the encounter. The advanced beginner demonstrates, to a degree, that their understanding and decision-making are contextual, but they do not experience personal responsibility for the situation.  Competent – A competent clinician is able to identify situational aspects that are important and those that can be ignored, making understanding and decision-making easier. The clinician practices in an efficient manner and critical thinking is present throughout the encounter. Skill application is appropriate. The competent clinician demonstrates that their understanding and decision-making are contextual and that they take personal responsibility for the outcome of the situation.  Proficient – A proficient clinician identifies goals of the encounter early in the process and is able to critically apply skills and knowledge in an efficient manner. The proficient clinician uses intuition and prior experiences to develop practice patterns and critical thinking, versus traditional rule following.  Expert – An expert is able to determine what needs to be achieved and they also immediately see a path to achieving the specific goal. Evidence of efficient clinical practice with critical thinking application is demonstrated throughout the encounter. Decision-making is intuitive. An expert is involved and committed to the understanding, decision-making, and outcomes of the situation. Standardized Patient Evaluation Tool: A Valid Measure of the Core Competencies 33 Copyright © by Indiana State University Clinical Practice in Athletic Training All rights reserved. ISSN Online 2577-8188 Volume 7 – Issue 2 July 2024 Critically Deficient Novice Advanced Beginner Competent Proficient Expert Provide Care that is Patient-Centered Patient-centered care includes sharing the responsibility of decision making with the patient. The clinician communicates with the patient in a fully open manner and also takes into account the patient’s individuality, emotional needs, values, and life issues. Work in Interdisciplinary Teams Working in interdisciplinary teams requires understanding about other team members’ expertise, background, knowledge/values, roles and processes to work collaboratively. The clinician works to ensure that accurate and timely information is shared with the patients/families and other providers, customizes care, ensures continuity of care between providers using a shared language. The clinician plays an active role in resolving conflicts with other members of the team. Employ Evidence-Based Practice Evidence-based practice refers to the integration of best research evidence, clinical expertise, and patient values in decision making. The clinician demonstrates the ability to identify and interpret best evidence and the ability to integrate the evidence to guide clinical practice decisions, with respect for patient values. Standardized Patient Evaluation Tool: A Valid Measure of the Core Competencies 34 Copyright © by Indiana State University Clinical Practice in Athletic Training All rights reserved. ISSN Online 2577-8188 Volume 7 – Issue 2 July 2024 Critically Deficient Novice Advanced Beginner Competent Proficient Expert Apply Principles of Quality Improvement Quality improvement refers to a continual process to understand and measure the quality of care by analyzing the structure, process, and outcomes of a healthcare unit. The clinician systematically acts to identify errors and areas for improvement in care and the health status of the patient(s). Utilize Informatics Informatics is the application of technology systems to solve problems and answer questions in healthcare. The clinician uses data from patient- practitioner interactions and available electronic resources (internal and external databases and the internet) to find information, make decisions, mitigate error, and communicate with patients and other providers while securing data and ensuring ethical and legal practices are followed. The clinician is able to assess the veracity of the data and the data sources before making decisions and implementing practices. Overall Impression Please use the above evaluative criteria to describe the behaviors consistent with the learner’s performance during this patient encounter. This should align with scoring of the aforementioned healthcare competencies. Comments: