











































MANUSCRIPT TYPE


CLINICIAN EXPERTISE COMMENTARY 

 

26 
Copyright © by Indiana State University                        Clinical Practice in Athletic Training  
All rights reserved. ISSN Online 2577-8188                                                                                  Volume 7 – Issue 2 July 2024 

Standardized Patient Evaluation Tool: A Valid Measure of the Core 
Competencies 
Lindsey E. Eberman PhD, LAT, ATC* & Jessica Edler Nye PhD, LAT, ATC** 
*Indiana State University, Terre Haute, IN 
**Grand View University, Des Moines, IA 

 
Content Focus 
Health Professions Education 
 
Correspondence 
567 N 5th Street, DAT Suite, Room A-10, Terre Haute, IN 47809  
E-mail: Lindsey.Eberman@indstate.edu 
Twitter: NA.  
 
Reference  
Eberman LE, Edler Nye J. Standardized Patient Evaluation Tool: A Valid Measure of the Core Competencies. 
Clin Pract Athl Train. 2024;7(2):26-34. https://doi.org/10.31622/2024/0007.02.4.  
 
Author Characteristics 
Dr. Lindsey Eberman is the Program Director for the post-professional doctorate in athletic training, bachelors degree in leadership 
and professional development, and Doctor of Philosophy in instructional leadership at Indiana State University.  She earned her 
bachelors degree from Northeastern University in Boston, MA and her post-professional masters in advanced athletic training and 
doctorate in curriculum and instruction from Florida International University in Miami, FL. Dr. Eberman is an expert in pedagogy 
and assessment.  

Dr. Eberman has been recognized with the several awards recognizing her commitment to teaching, scholarship, and service to the 
profession at the local, state, regional, and national level.  She facilitates doctoral student scholarship that improves the practice of 
athletic training and maintains a robust research agenda of her own amassing over 200 published manuscripts.  

Dr. Jessica Edler Nye is an Associate Professor and Program Director of the Master of Science in Athletic Training at Grand View 
University. She earned her bachelors degree from Central College and her post-professional masters and doctorate in curriculum 
and instruction from Indiana State University. Dr. Edler Nye is an expert in pedagogy and assessment.  

 
COMMENTARY 

In 2017, Dr. Jessica Edler Nye and I had the opportunity to develop and validate a standardized patient 
evaluation tool (SPET) (Appendix A) to measure the Institute of Medicine’s healthcare core competencies 
using a Delphi-panel review process.1 Although these core competencies (Table 1)2 have historically existed 
in the Commission on Accreditation of Athletic Training Education (CAATE) post-professional program 
standards and, until recently, the residency standards, they have also now been added to the professional 
education standards. There is also a crosswalk available between these core competencies and the AT 
Milestones, which were developed from the ACGME core competencies (Table 2).3 The SPET was developed 
because no high-quality mechanism existed to measure these outcomes, and as the program director of a 
CAATE-accredited post-professional Doctor of Athletic Training (DAT) program, it was my responsibility to 
ensure we were using a valid measure in our standardized patient encounters (SPEs) to show that program 
graduates were, in fact, advanced practice clinicians upon graduation.  

To establish validity, we first created five cases, representative of all the core competencies, each with a 
primary focus of one of the core competencies (Case 1: patient-centered care, Case 2: interprofessional and 
collaborative practice, Case 3: evidence-based practice, Case 4: quality improvement, Case 5: healthcare 
informatics). We then used a panel of 22 experts to conduct a 4-round Delphi panel review to establish 
consensus on the content validity of the SPET and its ability to apply to the SP cases. We established consensus 
on the evaluative criteria as appropriate (mean=3.6±0.6 out of 4, 65% strongly agree) and its use across 

https://doi.org/10.31622/2024/0007.02.4


Standardized Patient Evaluation Tool: A Valid Measure of the Core Competencies 

 

27 
Copyright © by Indiana State University                        Clinical Practice in Athletic Training  
All rights reserved. ISSN Online 2577-8188                                                                                  Volume 7 – Issue 2 July 2024 

the SPEs to evaluate the healthcare core competencies (mean=3.5±0.8 out of 4, 65% strongly agree) (Table 
3). Experts indicated, through the Delphi-panel process, that quality improvement and healthcare informatics 
may be difficult to assess during a single SPE. 

We also worked to establish reliability but learned that more advanced training was necessary to ensure 
raters are consistently scoring learners. To create SPEs for the raters to evaluate, we implemented six 
different continuing professional development sessions, which were eligible for continuing education units and 
consisted of a 45-minute interactive lecture, a maximum 30-minute SPE that was video recorded, and a 15-
minute group debriefing session. We then trained three independent raters (30-minutes) on the SPET and its 
evaluative criteria and asked them to review the SPEs (n=41). We established moderate reliability for the 
overall performance score (ICC=0.641) and each of the specific competencies demonstrated variable 
reliability (ICC: patient-centered care=0.778; interprofessional and collaborative practice=0.581; 
evidence-based practice=0.274; quality improvement=0.390; healthcare informatics=0.232). The raters 
also stated that there was limited opportunity to assess quality improvement and healthcare informatics 
during the SPEs.   

After establishing the SPET, the DAT program has implemented the tool over 1000 times as part of its clinical 
education curricula. This includes an initial SPE to establish baseline performance, and an additional 4 SPEs 
at the end of each semester, with a culminating progressive SPE experience (3 encounters with one patient 
case [onset to discharge]). This curricula also includes 4 virtual SPEs, which use only 2 individual competencies 
and the overall performance measure instead of the entire tool. Upon enrolling students in the newly-
accredited Masters in Athletic Training (MAT) program at Indiana State, we also began integrating the SPET 
into their SPE evaluations, with broad use of the tool consistently across semesters in the program. 

The lessons learned from the regular use of the tool across both programs are that foundational training, as 
well as continued maintenance, are necessary to ensure accuracy. As DAT faculty, we meet once per semester 
to watch 1 to 2 SPEs, score them, calculate inter-rater reliability, and then discuss. We also have an 
established bias checklist to help with repeated use of the SPET when we are evaluating multiple encounters 
in a row. When new evaluators are introduced to the team, we do preliminary training and integration into 
virtual (formative) SPEs first before integrating them into the evaluative SPEs, with feedback provided to 
new evaluators throughout this process. In addition to training evaluators, we also provide performance 
feedback to the learners. For each encounter, they receive a copy of their SPET, a checklist of behaviors 
performed (consistent with simulation best practices), and a summary of how the core competencies, including 
how quality improvement and healthcare informatics could be integrated into the encounter. The summary 
outlines competent performance for that case. In both the MAT and DAT programs, we have learned that it 
is critical to train the learner in interpreting the results and responding to the feedback. Historically in athletic 
training education, we have misused the terms competent and proficient, and interchanged competency and 
proficiency. Moreover, the Board of Certification examination and program graduation add additional 
elements of confusion around competence. These missteps and misunderstandings make it difficult for learners 
to comprehend the behavior expectations around entering and continuing competence in the profession. In 
addition, competence is a continuum, not a “place achieved after investing time and effort.” Competence 
ebbs and flows due to use and disuse and a more complete understanding of the construct is critical for 
learners to embrace, so that they can use the feedback for self-improvement. 

The SPET is a staple tool in our post-professional, CAATE-accredited DAT program and helps us document 
performance across 6 semesters. We remind learners regularly that is not uncommon for competence to be 
fluid, especially when you don’t consistently apply the core competencies in practice. Therefore, it is not 
atypical for students to see drastic improvements in patient-centered care between the first and second 



Standardized Patient Evaluation Tool: A Valid Measure of the Core Competencies 

 

28 
Copyright © by Indiana State University                        Clinical Practice in Athletic Training  
All rights reserved. ISSN Online 2577-8188                                                                                  Volume 7 – Issue 2 July 2024 

semester, but in the third, when the focus is on interprofessional and collaborative practice, they may slip 
back into past behaviors. However, the ultimate goal is that the learners are demonstrating competent to 
proficient behavior in all core competency areas upon graduation. 

 

  

REFERENCES 

1. Eberman LE, Edler JR. Developing, validating and establishing reliability of a standardized patient 
evaluation tool to measure healthcare core competency. J Athl Train. 2020;55(6):S-240. 
doi:10.4085/1062-6050-55.6s.S-1 

2. The Core Competencies Needed for Health Care Professionals. Institute of Medicine. Health 
Professions Education: A Bridge to Quality. Washington, DC: The National Academies Press, 2003. 

3. Sauers EL, Laursen RM, Pecha F. Walusz HJ. The Athletic Training Milestones. 
https://www.atmilestones.com/milestones.html.  Accessed on April 24, 2024.  

  

https://www.atmilestones.com/milestones.html


Standardized Patient Evaluation Tool: A Valid Measure of the Core Competencies 

 

29 
Copyright © by Indiana State University                        Clinical Practice in Athletic Training  
All rights reserved. ISSN Online 2577-8188                                                                                  Volume 7 – Issue 2 July 2024 

TABLE 1. Healthcare Core Competencies (IOM)  

Core Competency Description 
Provide patient-
centered care 

− identify, respect, and care about patients’ differences, values, 
preferences, and expressed needs 

− relieve pain and suffering 
− coordinate continuous care 
− listen to, clearly inform, communicate with, and educate patients 
− share decision making and management 
− continuously advocate disease prevention, wellness, and promotion of 

healthy lifestyles, including a focus on population health 
Work in 
interdisciplinary 
teams 

− cooperate, collaborate, communicate, and integrate care in teams to 
ensure that care is continuous and reliable 

Employ evidence-
based practice 

− integrate best research with clinical expertise and patient values for 
optimum care, and participate in learning and research activities to 
the extent feasible 

Apply quality 
improvement 

− identify errors and hazards in care 
− understand and implement basic safety design principles, such as 

standardization and simplification 
− continually understand and measure quality of care in terms of 

structure, process, and outcomes in relation to patient and community 
needs 

− design and test interventions to change processes and systems of care, 
with the objective of improving quality 

Utilize informatics − communicate, manage knowledge, mitigate error, and support 
decision making using information technology 

  



Standardized Patient Evaluation Tool: A Valid Measure of the Core Competencies 

 

30 
Copyright © by Indiana State University                        Clinical Practice in Athletic Training  
All rights reserved. ISSN Online 2577-8188                                                                                  Volume 7 – Issue 2 July 2024 

Table 2.  CAATE Core Competency to AT Milestone Crosswalk 

CAATE Core 
Competency 

AT Milestones General Competencies & Sub-competencies 

Patient-Centered 
Care 

Patient-Care and Procedural Skills (Sub-competencies 1,2) 
Interpersonal and Communication Skills (Sub-competencies 1,2) 

Interprofessional 
Practice and 
Education 

Patient-Care and Procedural Skills (Sub-competency 7) 
Interpersonal and Communication Skills (Sub-competency 3) 
Professionalism (Sub-competency 2) 
Systems-Based Practice (Sub-competencies 1-4) 

Evidence-Based 
Practice 

Practice-Based Learning and Improvement (Sub-competency 1) 
Medical Knowledge (Sub-competency 3) 

Quality 
Improvement 

Practice-Based Learning and Improvement (Sub-competencies 2-4) 
Systems-Based Practice (Sub-competencies 1-4) 

Health Care 
Informatics 

Interpersonal and Communication Skills (Sub-competency 4) 
Systems-Based Practice )Sub-competency 6) 

Professionalism Professionalism (Sub-competencies 1-4) 
 

  



Standardized Patient Evaluation Tool: A Valid Measure of the Core Competencies 

 

31 
Copyright © by Indiana State University                        Clinical Practice in Athletic Training  
All rights reserved. ISSN Online 2577-8188                                                                                  Volume 7 – Issue 2 July 2024 

Table 3. Delphi Panelists Perceptions about SPET Evaluative Criteria Representing the Core 
Competencies  

 Case 1 Case 2 Case 3 Case 4 Case 5 

Patient-Centered Care 3.9 ± 0.3 3.8 ± 0.4 3.8 ± 0.4 3.9 ± 0.2 4.0 ± 0.0 

Interprofessional and 
Collaborative Practice 

3.5 ± 0.6 3.2 ± 0.7 3.9 ± 0.3 3.3 ± 0.7 3.2 ± 0.6 

Evidence-Based Practice  3.7 ± 0.5 3.9 ± 0.3 3.6 ± 0.6 3.7 ± 0.5 3.7 ± 0.5 

Quality Improvement 3.5 ± 0.6 3.3 ± 0.8 3.2 ± 0.7 3.4 ± 0.7 3.4 ± 0.7 

Healthcare Informatics  3.6 ± 0.6 3.2 ± 0.6 3.4 ± 0.5 3.3 ± 0.7 3.1 ± 0.7 

Likert Scale: 1=Strongly disagree, 2=Disagree, 3=Agree, 4=Strongly agree 



CLINICIAN EXPERTISE COMMENTARY 

 

32 
Copyright © by Indiana State University                        Clinical Practice in Athletic Training  
All rights reserved. ISSN Online 2577-8188                                                                                  Volume 7 – Issue 2 July 2024 

Appendix A. Standardized Patient Evaluation Tool (SPET) 

Evaluative Criteria:  

 Critically Deficient – These learner behaviors are not within the spectrum of developing competence. Instead they indicate 
significant deficiencies in performance. 

 Novice – A novice has a basic understanding of the knowledge and skill and applies the information within a specific set of 
rules or steps that they have learned.  Skills are applied as taught, although typically without efficiency. Critical thinking is 
limited or not present. Understanding, deciding, and the outcomes are free of the context in which they are being applied. 

 Advanced Beginner – An advanced beginner has begun to develop a framework for the application of knowledge and skills.  
He or she begins to recognize situational aspects and applies appropriate skills. Practice efficiency is minimally evident, and 
critical thinking is implemented at certain points of the encounter. The advanced beginner demonstrates, to a degree, that their 
understanding and decision-making are contextual, but they do not experience personal responsibility for the situation. 

 Competent – A competent clinician is able to identify situational aspects that are important and those that can be ignored, 
making understanding and decision-making easier. The clinician practices in an efficient manner and critical thinking is present 
throughout the encounter. Skill application is appropriate. The competent clinician demonstrates that their understanding and 
decision-making are contextual and that they take personal responsibility for the outcome of the situation. 

 Proficient – A proficient clinician identifies goals of the encounter early in the process and is able to critically apply skills and 
knowledge in an efficient manner. The proficient clinician uses intuition and prior experiences to develop practice patterns and 
critical thinking, versus traditional rule following.  

 Expert – An expert is able to determine what needs to be achieved and they also immediately see a path to achieving the 
specific goal. Evidence of efficient clinical practice with critical thinking application is demonstrated throughout the encounter. 
Decision-making is intuitive.  An expert is involved and committed to the understanding, decision-making, and outcomes of the 
situation.  

 



Standardized Patient Evaluation Tool: A Valid Measure of the Core Competencies 

 

33 
Copyright © by Indiana State University                        Clinical Practice in Athletic Training  
All rights reserved. ISSN Online 2577-8188                                                                                  Volume 7 – Issue 2 July 2024 

 Critically 
Deficient 

Novice Advanced 
Beginner 

Competent Proficient Expert 

Provide Care that is Patient-Centered 

Patient-centered care includes sharing the responsibility of decision making 
with the patient. The clinician communicates with the patient in a fully open 
manner and also takes into account the patient’s individuality, emotional 
needs, values, and life issues. 

      

Work in Interdisciplinary Teams 

Working in interdisciplinary teams requires understanding about other team 
members’ expertise, background, knowledge/values, roles and processes to 
work collaboratively. The clinician works to ensure that accurate and timely 
information is shared with the patients/families and other providers, 
customizes care, ensures continuity of care between providers using a 
shared language. The clinician plays an active role in resolving conflicts 
with other members of the team. 

      

Employ Evidence-Based Practice 

Evidence-based practice refers to the integration of best research 
evidence, clinical expertise, and patient values in decision making. The 
clinician demonstrates the ability to identify and interpret best evidence 
and the ability to integrate the evidence to guide clinical practice decisions, 
with respect for patient values. 

      

  



Standardized Patient Evaluation Tool: A Valid Measure of the Core Competencies 

 

34 
Copyright © by Indiana State University                        Clinical Practice in Athletic Training  
All rights reserved. ISSN Online 2577-8188                                                                                  Volume 7 – Issue 2 July 2024 

 Critically 
Deficient 

Novice Advanced 
Beginner 

Competent Proficient Expert 

Apply Principles of Quality Improvement 

Quality improvement refers to a continual process to understand and 
measure the quality of care by analyzing the structure, process, and 
outcomes of a healthcare unit. The clinician systematically acts to identify 
errors and areas for improvement in care and the health status of the 
patient(s). 

      

Utilize Informatics 

Informatics is the application of technology systems to solve problems and 
answer questions in healthcare. The clinician uses data from patient-
practitioner interactions and available electronic resources (internal and 
external databases and the internet) to find information, make decisions, 
mitigate error, and communicate with patients and other providers while 
securing data and ensuring ethical and legal practices are followed. The 
clinician is able to assess the veracity of the data and the data sources 
before making decisions and implementing practices. 

      

Overall Impression 

Please use the above evaluative criteria to describe the behaviors 
consistent with the learner’s performance during this patient encounter. This 
should align with scoring of the aforementioned healthcare competencies. 

      

Comments:  

 


