MANUSCRIPT TYPE EDITORIAL 1 Copyright © by Indiana State University Clinical Practice in Athletic Training All rights reserved. ISSN Online 2577-8188 Volume 7 – Issue 2 July 2024 The Value of Using Validated Competency-Based Assessments Across the Career Span of an Athletic Trainer Matthew Drescher, PhD, DAT, LAT, ATC *; Lindsey Eberman, PhD, LAT, ATC† *North Dakota State University, Fargo, ND; †Indiana State University, Terre Haute, IN Content Focus Health Professions Education Correspondence 1340 Administration Ave, Fargo, ND 58102 E-mail: matthew.drescher@ndsu.edu Reference Drescher MD, Eberman LE. The Value of Using Validated Competency-Based Assessments Across the Career Span of an Athletic Trainer. Clin Pract Athl Train. 2024; 7(2):1-4. https://doi.org/10.31622/2024/0007.02.1 COMMENTARY Background In educating students, there is an inherent need to quantify and track student performance over time. This quantification is typically done with an outcome of some type, such as an assessment rubric or grading criteria, from which subjective student performance is translated to an objective value. Traditionally, this objective value has been quantified as grades (on a numeric or alpha scale). To quantify those grades, assessments are given scores that contribute to the overall grade in a course or area of study. This method has proven so effective that it is the predominant way students are assessed in the United States, with high school and undergraduate students being categorized by their grade point average as a method to average a student’s performance over time or across content.1 Grades and their subsequent derivatives can be an effective method of quickly and easily expressing student performance and aptitude in a particular area of study. We must, though, examine the assumptions supporting the system's use of grading performance. First, we must define what is meant by “measure of learning.” Learning, as a concept is difficult to define, with conflicting definitions from different schools of thought.2 A more modern consensus is that learning is a process of gradual changes in behavior, mental processes, and critical thinking.2 How, then, do we objectively measure a continuous process? Historically, measures of learning were considered to be methods such as memorization, knowledge assessment, and behavioral observation, as these were seen to be analogs to learning as a permanent change in behavior.2 When discussing a more holistic measure of learning from a modern understanding, we should also consider including competency-based assessment and overall performance assessment as well as measures of knowledge and behavior. These measures reflect the more modern definition of learning and create a more complete representation of a student’s learning process. This is the foundational idea behind competency- based education. When implementing competency-based education, there is an assumption that grades can still be applied as an accurate, objective measure of learning, whether using holistic student assessment or otherwise.1 However, this assumption relies on an assessment method's reliability and the assessor's objectiveness, which can vary widely. Take, for example, an assessment of student knowledge on a traditional exam as a measure of learning. Such an assessment could objectively quantify a student’s learning in a numeric grade (and even transferred to an alpha-coded grade), given that the items on the exam were accurate and encompassed https://doi.org/10.31622/2024/0007.02.1 The Value of Using Validated Competency-Based Assessment Across the Career Span of an Athletic Trainer 2 Copyright © by Indiana State University Clinical Practice in Athletic Training All rights reserved. ISSN Online 2577-8188 Volume 7 – Issue 2 July 2024 expected content. Now consider the student’s final grade in the course. If this final grade is an average of all graded assessments in the course, then this should be an accurate assessment of their competence relative to the learning outcomes. However, indirect and subjective measurements, such as student participation or time-on-task, are frequently included in a final grade, making the final grade less reflective of knowledge and rather a reflection of both subjective and objective factors, diminishing its accuracy as an objective measure of learning. Other factors may also influence the objectiveness of grades, such as the study habits of students, different approaches to grading, or even students' motivations to receive a particular score or grade (e.g. contract grading). These factors indicate that grades might measure learning in certain situations but are not, and should not, be considered all-encompassing. To be clear, we do not suggest that all education should be moved to a competency-based education model and that traditional grading should be removed altogether. In higher education, numerous barriers block the adoption of such a model, from federal classification standards to educator training and buy-in. Still, in fields and professions where competency in a skill is a necessary requirement for practice, integrating competency- based assessment strategies into more traditional education models can help bridge the gap between student course performance and competency in the field. In recent times, healthcare education programs have embraced the adoption of competency-based assessment methods, in lieu of traditional grading systems, to evaluate clinical competence. Competency-based assessment approaches are deemed more effective in evaluating the clinical application of skills in healthcare students.3 However, it is imperative to acknowledge that these approaches carry certain assumptions that must be considered for their effective implementation in practice. Namely, we must assume that competency-based assessment criteria are accurate and encompass the totality of a particular clinical skill or behavior. However, even when an assessment tool is critically appraised for its accuracy, there remains a level of subjectivity and bias within the tool, originating from its creator and assessor. To establish the validity of an assessment tool, it should ideally be based on a scientifically proven theory, and it should be validated. In athletic training, assessment tools have been created and validated based on the Dreyfus Model of Skill Acquisition,4,5 which is an established method for categorizing and evaluating skill acquisition.6 The Dreyfus model uses a 5-point scale, with each point characterizing a particular level of skill acquisition, from novice to expert.6 Using the Dreyfus model as a framework helps to validate the theoretical development of a competency-based assessment tool. For a model like this to work, however, a list of the skills and behaviors that correlate to practice must be identified and paired with the framework. A team of experts, or those who are knowledgeable about the skills and behaviors required to perform a task, should be consulted to develop the list of competencies that constitute the comprehensive practice, in this case, the comprehensive practice of athletic training. The use of experts, rather than an individual creator, helps to prevent bias and ensure that all aspects of a skill are accounted for. This can seem like a lot of work for an educator to implement into an existing program. However, much of the groundwork has already been completed. The Accreditation Council for Graduate Medical Education (ACGME) has created a list of core competencies, in alignment with the healthcare core competencies, that provide list of behaviors for each broad competency and subcompetency. The ACGME then applied the Dreyfus model framework to characterize the range of performance from novice to expert for each competency and sub-competency. This has also been done in the athletic training profession, whereby 4 leaders in the profession developed The Athletic Training Milestones (AT Milestones). The AT Milestones were built similarly to the ACGME Milestones with the Dreyfus model framework applied across the core competencies and sub-competencies and are still in progress for some of the specialty areas of athletic The Value of Using Validated Competency-Based Assessment Across the Career Span of an Athletic Trainer 3 Copyright © by Indiana State University Clinical Practice in Athletic Training All rights reserved. ISSN Online 2577-8188 Volume 7 – Issue 2 July 2024 training practice. Subsequently, a different group of researchers validated the AT Milestones using a conservative approach to establishing a content validity index using 12 additional experts in the field. The overall scale index was very high (0.99) and the 28 subcompetency items ranged from 0.83 to 1.00.4 One of the benefits of the AT Milestones is that they identify skills and establish the behaviors consistent with competence for all athletic trainers, not just athletic training students. This provides students with a clear pathway to competence and provides practicing clinicians with effective tools for assessing their clinical practice. These milestones should be used across the continuum of practice, from education through retirement, as a guidepost to competent clinical practice.4 As with any assessment tool, assessment personnel should be trained. Untrained individuals might incorrectly evaluate, which could lead to a false sense of competence. An understanding of the rankings and their requisite behaviors, along with knowledge of the profession and activity being performed, is critical to accurate assessment. Proper training on the use of these tools helps to ensure alignment with tool design and to provide the individual being assessed with effective feedback on performance. In athletic training education, tools such as the AT Milestones are not frequently used,7 thus they are not yet a key metric for progression and matriculation. A validated tool can give educators confidence in assessing student performance and ensure that the totality of the behavior or skill is accounted for prior to graduation. Such a tool also provides students with a path to mastery and helps to show a progression of skills that can help them continue to learn based on their capability rather than their knowledge alone.3 A validated tool can also be used for student education by showing them what skills and behaviors are expected of them in the profession. Further, a competency-based assessment tool can be used across the career of a provider helping to ensure alignment with best practices across time. Finally, using a validated competency-based assessment tool allows students to shift their mindset from acquiring knowledge, as is promoted by traditional grading schemes, to applying such knowledge to clinical practice.8 REFERENCES 1. Cain J, Medina M, Romanelli F, Persky A. Deficiencies of traditional grading systems and recommendations for the future. Am J Pharm Educ. 2022;86(7):8850. doi:10.5688/ajpe8850 2. Sehic S. Redefining the definition of learning from an educational aspect. Int J Inf Res Rev. 2020;7:6940- 6942. 3. Mace KL, Bacon CEW. The future of health professions education: Considerations for competency-based education in athletic training. Athl Train Educ J. 2019;14(3):215-222. doi:10.4085/1403215 4. Bacon CEW, Anderson BE, Cavallario JM, Van Lunen BL, Eberman LE. Content validation of the athletic training milestones: A report from the Association for Athletic Training Education Research Network. J Athl Train. 2023;58(5):483-487. doi:10.4085/1062-6050-0332.22 5. Eberman, LE, Edler, JR. Developing, Validating and establishing reliability of a standardized patient evaluation eool to measure healthcare core competency. J Athl Train. 2020;55(6s):S-240. doi:10.4085/1062-6050-55.6s.S-1 6. Rousse B, Dreyfus S. Revisiting the six stages of skill acquisition. Teach Learn Adult Ski Acquis Appl Dreyfus Dreyfus Model Differ Fields. Published online 2021:3-28. 7. Imanipour M, Ebadi A, Monadi Ziarat H, Mohammadi MM. The effect of competency-based education on clinical performance of health care providers: A systematic review and meta-analysis. Int J Nurs Pract. 2022;28(1):e13003. doi:10.1111/ijn.13003 The Value of Using Validated Competency-Based Assessment Across the Career Span of an Athletic Trainer 4 Copyright © by Indiana State University Clinical Practice in Athletic Training All rights reserved. ISSN Online 2577-8188 Volume 7 – Issue 2 July 2024 8. Welch Bacon CE, Anderson BE, Cavallario JM, Van Lunen BL, Eberman LE. Perceptions and use of the athletic training milestones in education: A report from the AATE Research Network. Clin Pract Athl Train. 2024;7(2);XX-XX. doi: