9 SA JournAl of PhySiotherAPy 2011 Vol 67 no 2 Research Article Exploring the Key Performance Areas and Assessment Criteria for the Evaluation of Students’ Clinical Performance: A Delphi study Corresponding Author: Conran Joseph Department of Physiotherapy University of the Western Cape Private Bag x 17 Bellville 7530 E­mail: cjoseph@uwc.ac.za AbSTrAcT: Background: evaluating students’ clinical performance is an integral part of the quality assurance in a physiotherapy curriculum, however, the objectivity during clinical examination have been questioned on numerous occasions. The aim of this study was to explore the essential key clinical performance areas and the associated assessment criteria in order to develop a reliable clinical assessment form. Methods: A Delphi study was used to obtain consensus on the development of a reliable clinical performance assessment tool. The study population consisted of purposively selected academic physiotherapy staff from the University of Western Cape as well as supervisors and clinicians involved in the examination of physiotherapy students from the three Universities in the Western Cape. Findings from the Delphi rounds were analysed descriptively. Fifty percent or higher agreement on an element was interpreted as an acceptable level of consensus. Results: Eight key performance areas were identified with five assessment criteria per key performance area as well as the weighting per area. It was evident that evaluators differed on the expectations of physiotherapy students as well as the criteria used to assess them. Conclusions: The Delphi panel contributed to the formulation of a clinical assessment form through the identification of relevant key performance areas and assessment criteria as they relate to undergraduate physiotherapy training. Consensus on both aspects was reached following discussion and calculation of mean ranking sores. Implications: This process of reaching consensus in determining clear criteria for measuring key performance areas contributes to the objectivity of the process of examinations. Key wordS: key PeRFoRMAnCe AReAS (kPA), CRITeRIA, DelPhI. Joseph C, BSc Physiotherapy,1 Hendricks C, BSc Physiotherapy,1 Frantz J, PhD,1 1 University of the Western Cape evaluation process, training experience and the importance of specific aspects of clinical performance could influence the coherence of the students’ assessments (Meldrum et al 2008). There is a lack of research on stan­ dardised clinical performance assess­ ments of physiotherapy students in Africa. Meldrum et al (2008) found that the current clinical performance assessment tools lack reliability and consistency. The latter authors found that the key performance areas such as patient evaluation and management, student development, organisation and time management are standard con­ structs for measuring clinical perform­ ance amongst physiotherapy under­ graduates. However, there are limited guidelines and descriptions of the beha­ viours associated with each key perfor­ mance area. The lack of clear guidelines iNTRODUCTiON Assessment of clinical performance in undergraduate students has always been regarded as an essential component of health professional education and practice (Hobbs et al 2000; Mitchell et al 2009). However, a major concern of health educators has been the difficulty of standardising on objective assessment of students’ clinical performance (Cross and Hicks 1997; Lewis et al 2008). Measuring physiotherapy students’ clinical performance has always been subjective, and the profession acknowl­ edges the arbitrary criteria for evaluating students’ clinical performance (Christie et al 2003; Epstein et al 2002) for year level promotion and completion of the degree. The role of the educator and supervisor greatly contributes to the subjectivity. Certain factors such as their existing knowledge regarding the and asso ciated criteria prompts the need to develop specific assessment tools in order to determine a student’s compe­ tence in a clinical setting. This would contribute to the formulation of clear and uniform expectations from the students, and could thus minimise the bias from the supervisor/instructor regarding what should be evaluated as part of the forma­ tive examination in the clinical arena. Cross and Hicks (1997) carried out a series of studies exploring beha ­ viours students should exhibit during 10 SA JournAl of PhySiotherAPy 2011 Vol 67 no 2 cli nical practice to determine their level of proficiency. Their recommendations advocate for assessment tools to include a description of the concept of profes­ sional competence and domains or key performance areas, such as knowledge, skills and attitudes. Cross and Hicks (1997) further recommended, based on the study findings, that domains should be categorised more explicitly, which facilitates the assessment of the level of students’ performance. Even though specific guidelines and criteria may exist, individual assessors tend to have different views on what constitutes achievement and may use their own identified criteria or attributes to assess clinical performance (Meldrum et al 2008). Such tendencies are acknowl­ edged by the Chartered Society of Physiotherapy (CSP) in its guidelines on developing effective assessment procedures: “Assessment is a subjective process... assessors are likely to apply hidden or unconscious criteria in assess­ ing students’ performance” (CSP, 1996, p.10). Conversely, it is important to note that guidelines for assessing performance in students in clinical practice may not contribute to the content­and construct validity and of the assessment process, but has been found as an indicator for obtaining more reliable results (Lewis et al 2008). It is the belief of the authors that the current assessment tools being used to evaluate clinical performance at physio­ therapy training institutions in South Africa possess some of the key perfor­ mance areas that are unified with other standardised clinical assessment tools. However, they may differ on the impor­ tance of each performance area (Cross 1999; Freeman and Rogers 2010). Currently no explicit guidelines exist on the behaviours that a student should demonstrate within the various key per­ formance areas and assessment criteria to determine the level of perfor mance. After reviewing the literature and evalu­ ating the status quo of the existing clini­ cal assessment forms with regards to the KPA’s and assessment criteria, the purpose of the study was to develop a clinical performance assessment form through exploring the essential KPA’s essential for the evaluation of students’ clinical performance, as well as the asso ciated criteria and weighting of each KPA’s. METHODS The study was conducted in two phase. The first was to identify the key per­ formance areas that physiotherapy stu­ dents should exhibit during their clinical years, including criteria and weighting for each area. Secondly, the reliability of the new tool will be determined among academics and clinical super­ visors. This paper will report phase one of the study. The Delphi technique was used to obtain an informed or refined consensus from a group of experts or knowledgeable informants. The Delphi surveying technique can be used to turn an opinion into consensus by asking content experts questions that are then coded into themes. These issues are then re­presented to the participants for further comments (Hassan et al 2000). Participants Participants who were knowledgeable in the field of clinical education and prac­ tice were purposively selected to serve on the panel and written informed con­ sent was obtained from each of them. The criteria for selection into the Delphi panel were academics involved in the training and supervision of students as well as clinicians and supervisors involved in training of physiotherapy students of all three training institutions in the Western Cape. The applicability of the purposively sampling strategy ensured that the Delphi panel consisted of participants from all three universi­ ties offering the undergraduate physio­ therapy programme. Literature indicated that the emphasis of KPA’s differs from institution to institution, therefore the selected participants are thought to be a representation of the views and opinions regarding KPA’s and assessment criteria at each university. instruments Prior to the Delphi study, participants completed a self­administered question­ naire. The aim of the electronic­based, self­administered questionnaire was to determine participants’ experience (in years) and role in clinical education and practice, their field of supervision and level of health care that they supervise physiotherapy students. This assisted in ensuring that the Delphi panel included key stakeholders. The questionnaire was reviewed by two independent research­ ers in the field of clinical education for face­ and content validity, which led to minor changes being made. It was recommended to change “institutions supervised” to level of health care super­ vised in the past and the correction of grammatical errors. The table 1 opposite illustrates the characteristics of selected participants based on the aim of the questionnaire Data collection and procedure Four rounds of the Delphi survey were conducted via email. Each question for the round was sent as an e­mail attach­ ment along with a message that provided general information about the question­ naire and explained how the information would be handled. The general informa­ tion consisted of an opening statement giving a description of what current practice is for clinical examination and the inconsistencies highlighted in litera­ ture. Thereafter, specific questions fol­ lowed in an attempt to reach consensus. The purpose of the first round of the Delphi study was to explore whether or not key performance areas should be the same or different for third and fourth year physiotherapy students. The secon­ dary purpose of this question was to establish whether participants regard the importance of KPA’s the same for the different year group. The importance of this observation will be important since third year students are exposed to the formal clinical “real­life situation” for the first time. Subsequent to the par­ ticipants opinion whether KPA’s should be similar or different for the two year groups, they were asked to list the rele­ vant key performance areas for both third and fourth year students. Data from round 1 were collated and summarised by the lead researcher and sent to the co­authors who reviewed the deductions regarding the KPA’s identified for third and fourth years for correctness. 11 SA JournAl of PhySiotherAPy 2011 Vol 67 no 2 The second survey (round 2) had 2 main purposes. Firstly, participants were asked to review the list of key perfor­ mance areas, summarised by the research­ ers based on the responses from the previous round, and rank them accord­ ing to importance with 1 being most important and 8 being least important. Secondly, they were asked to list at least five associated assessment criteria that they thought are essentially linked to each key performance area. A list of all the assessment criteria was compiled for each key performance area and consen­ sus was reached by the researchers based on descriptive analysis of the data on the most common criterions identified by the Delphi. The rank order of the identified key performance areas was determined by calculating the mean. The purpose of round 3 of the Delphi study was to clarify the importance of KPA for third and fourth years by resending a section of the second round of the Delphi. This further served as quality assurance and validation of responses from participants. Participants Table 1: information regarding Delphi panel no Practising exp (yrs) work settings Supervision exp (yrs) clinical areas supervised level of healthcare where you are supervising or co-ordinating students 1 33 Academic & Student supervision 20 Respiratory Neurology OMT Paediatrics Primary Secondary Tertiary Specialised (Schools) 2 22 Government & Private hospital Student supervision 15 Adult Neurology Secondary Tertiary 3 24 Student supervision 13 Adult Neurology Specialised centres 4 17 Academic 15 OMT Sport Rehabilitation Intensive Care Unit Primary Tertiary Specialised 5 18 Private Hospital Academic Student supervision 5 Intensive Care Unit OMT Primary Tertiary 6 12 Government Hosp. Academic Student supervision 7 Paediatrics Community OMT Neurology Respiratory Primary Specialised 7 5 Academic Student supervision 4 Community Primary Secondary Specialised (Schools) 8 6 Academic 3 Community OMT Primary Secondary Specialised 9 16 Academic 14 OMT Respiratory Orthopaedics Primary Secondary 10 28 Government Hosp. 22 OMT Neurology Paediatrics Community Primary 11 17 Government hospital 10 Rehabilitation Tertiary 12 35 Government Hosp Academic Supervisor 20 OMT Primary Tertiary 12 SA JournAl of PhySiotherAPy 2011 Vol 67 no 2 were asked to rank the key perfor­ mance areas again and then to also rank the criteria, identified by the participants during the previous round, for each key performance area on a scale of 1 – 4, with 1 being “Strongly disagree” and 4 being “Strongly agree”. The median for each section was used as a guide for final inclusion into the list of criteria. The purpose of round 4 was to accept the assessment criteria for each KPA and determine the weighting of each KPA for third and fourth year physiotherapy students respectively. The participants were asked to include a weighting for each key performance area in both the evaluation (A) of an unknown patient and the treatment (B) of a known patient for third and fourth years respectively. They were requested to clearly indicate in cases where the weighting of the key performance areas differed and also instructed not to exceed a cumulative percentage of 100. So the assessment criteria were not sent to them again – thus they did not reach consensus on the most important assessment criteria. Ethical clearance for the study was obtained from the University of the Western Cape. All ethical considera­ tions in the Helsinki declaration were adhered to during the study period (World Medical Organization 1996). Written informed consent was sought from selected participants after a clear overview of the study was given to them. Full anonymity could not be guaranteed because the electronic submission of responses could not be blinded to the researchers. The only threat to anonym­ ity was that the addresses could be linked participants, but anonymity was ensured in the reporting of study findings. Data analysis The responses were collated by one per­ son (the lead researcher) and the list was cross­checked by the other two authors in order to address potential bias. Data were captured on an excel sheet and the frequencies with which items were mentioned were tabulated. The frequen­ cies with which a KPA and criterion were entered in order of preference were determined in order to establish the ranking of importance of each aspect. A summary list which was reflective of each aspect and round of the Delphi was then developed and submission RESULTS Description of participants Twelve participants (four males, eight females) were purposively selected and invited to participate in the study. The participants included in the study participated in supervision or clinical examinations at the three universities in the Western Cape. The mean number of years of practice experience was 19.4 years (5­35years) and the mean number of years supervising was 12 years (3­22years). All of the participants were involved in student clinical supervision and 67% in academic teaching at either the postgraduate or undergraduate level. More than half of the participants have supervised in two or more domains (neurology, paediatrics, respiratory and orthopaedics) of physiotherapy and the level of health care (primary, secondary, tertiary and specialised). Round 1: The response rate for the first round was 100%. Ten of the 12 participants indi­ cated that the key performance areas among third and fourth year physiothe­ rapy students should be the same. The key performance areas identified by the panel were knowledge, communication, professionalism, planning, clinical rea­ soning, patient management, reflection and time and organisational skills. Responses of the second round Delphi In the second round, participants ranked the key performance areas in order of importance. Nine responses were received, indicating a response rate of 69%. The agreement level was set at 50% (meaning that half or more of the participants agreed on the level of impor­ tance of a specific KPA) for inclusion of each KPA at the desired level of impor­ tance. The recommended agreement level was only reached once for time and organisational management as a KPA. The aforementioned was classified the least important KPA in the assessment of clinical performance according to the panel. In addition, the mean rank­ ing scores were calculated to report the ranking of performance areas. Since the KPA’s were classified dif­ ferently for third and fourth years, the question on the ranking of KPA’s in higher order of importance was resent as part of the subsequent round, in order to gain consensus. Responses of round 3 and 4 of Delphi study: For the selection and inclusion of the assessment criteria under each KPA, a 50% agreement among participants was required. In cases where the agreement level was not met, the most commonly listed criteria were selected. For the pur­ pose of quality assurance, participants were given another opportunity to com­ ment on the assessment criteria where 50% agreement in the subsequent round Table 2: Ranking of key performance areas in third and fourth year physiotherapy students by the participants (Round 2) Third years Fourth years Key Performance Areas Mean ranking Ranking Mean ranking Ranking Knowledge 3 1 3 1 Communication 3.3 2 3.9 3 Professionalism 3.8 3 4.1 4 Planning 4.9 5 4.3 5 Clinical reasoning 5.3 6 3.6 2 Patient management 4.1 4 4.6 6 Reflection 5.3 6 5.9 7 Time management 6.2 7 6.7 8 13 SA JournAl of PhySiotherAPy 2011 Vol 67 no 2 Table 3: Key performance areas with the associate assessment criteria and weighting. Key Performance Area Assessment criteria Third and fourth year weighting Assessment (%) Treatment (%) Knowledge Knowledge about interviewing techniques Knowledge about assessment techniques Planning Evidence based practice Anatomy Conditions Precautions/Contra-indications Referral 20 15 Communication Ability to convey information Use of appropriate language with different stakeholders Appropriate body language Listening skills Written skills (documentation) Ability to give constructive feedback 10 10 Professionalism Demonstrates responsibility towards patients during assessment and treatment Demonstrates respect towards patient (cover pt, address, etc) Works within safety precautions Respect the rights of the patient Punctuality 10 10 Planning Ability to identify short-term and long-term goals Ability to prioritize goals Ability to choose appropriate assessment and treatment techniques Able to gather information within a reasonable time limit Logical planning of assessment and treatment 10 10 Clinical reasoning Understands the “why” and the reasoning behind “why” Able to interpret the findings Ability to adapt Ability to justify why There is a thread of logic throughout session 20 15 Patient management Able to assess and manage the patient as a whole: pt physical, social and functional goals Quality of techniques both assessment and treatment Ability to choose relevant techniques Ability to adapt during sessions Ability to educate Decision making process to refer 20 25 Reflection Ability to reflect after treatment on effects of treatment Ability to reflect after assessment on progress Ability to reflect and decide on action plan 5 10 Time and organisational management Ability to manage appropriate caseload Plan and organise equipment needed for the management of the patient Plan treatment sessions Plan time for documentation Be on time for sessions 5 5 14 SA JournAl of PhySiotherAPy 2011 Vol 67 no 2 was not obtained. Participants were also expected to weight the key performance areas according to the perceived impor­ tance of each area. Although ranking of KPA’s was different between third and fourth years, the weighting of the KPA’s by the participants were found to be the same for third and fourth years The average of the participants’ responses is shown in Table 3. DiSCUSSiON The evaluation of health­care students includes the assigning of grades to various components of their clini­ cal per formance. In order to be valid, this evaluation should assess students’ demonstrated competency in several areas of clinical practice, focusing on the most relevant clinical skills. In order to achieve this, the value attached to different aspects of practice should be reassessed periodically, to align the evaluation with changes in emphasis in health care policy and physiotherapy practice norms. The results of this study provide a guideline for evaluators to assess the clinical competencies that are expected from physiotherapy students, along with basic assessment criteria. Knowledge, clinical reasoning and patient manage­ ment were rated the top key performance areas, similar to the results of Sanford et al (1993), who found that patient management, professional behaviour and clinical reasoning were important com­ petencies. Even more recently, Jette et al (2007) also identified similar attributes to the findings of the current study. Researchers have long highlighted the importance of key performance areas and competencies that are required in the clinical setting (Berner and Bender 1978). In a study by Simuzingile (1998) at the University of the Western Cape, disparities and inconsistencies were found when determining essential clinical competencies in physiotherapy. Similarly, almost a decade and half later, the current study that was also car­ ried out at the University of the Western Cape, found disparities in the ranking of each key performance area in order of perceived importance among third and fourth year students respectively. These differences in perception among edu­ cators may influence the objectivity of the evaluation process and reduce the reliability and content validity of the evaluation tool. This perceived difference can be attributed toward their knowledge regarding the evaluation process, their training experience and their personal or unconscious bias (Meldrum et al 2008). This study highlighted the process of acquiring consistent consensus around what should be evaluated and the impor­ tance thereof, yet inconsistency is still common. The weighting of key performance areas has been found to assist in high­ lighting the importance of these areas. A study done by Sanford et al (1993) found that different weightings are applied to different clinical areas to stress the impor­ tance thereof. The results of the current study are concurrent with the latter study in demonstrating the discrepancies in the weighting of key performance areas, thus the importance placed on certain aspects of clinical performance may differ from institution to institution or the broadened scope of physiotherapy as part of the multi­disciplinary team. In order to prevent examiner bias the researchers are advocating for weighting of KPA’s prior to examination. Although the order of importance of the key performance areas differed between third and fourth year students in the current study, the mean weighting remained the same for both an evaluation and treatment exam for the two year groups. This inconsis­ tency in acknowledging the importance of certain KPA above others and not applying the relative weighting demon­ strate that examiners are not constantly aware of what they are evaluating, thus the objectivity of each examination can be questioned. In the last decade, a radical shift from the medical model to the biopsycho­ social model has occurred. The biopsy­ chosocial model is based on the premise that “...physical health and well­being are shaped by the interactions between biological, psychological, and social fac­ tors” (Suls and Rothman, 2004, p.121). This requires a deeper understanding of the influence the health condition has on the functioning and quality of life of patients. Thus it is important that assess­ ment of students incorporates a holistic approach in evaluating the clinical skills of students. The institutions offering physiotherapy training are well aware that patients are more critical and scep­ tical of medical treatment and that they may question the integrity of students. Thus, students are trained to conduct themselves in a professional manner to establish patient confidence within the practitioner, establish trust and to avoid possible litigations (Simuzingili 1998). One of the goals of physiotherapy education is to produce competent prac­ titioners. In order to achieve the afore­ mentioned, physiotherapists should be able to critically reflect on their per­ formance, if need be, adapt interven­ tions according to the need of the patient (Clouder 2000). Literature suggests that learning occurs through constant critical reflection. In the field of allied health, critical reflection is perceived as the bridge between theory and prac­ tice (Wellard and Bethune 1996). Since undergraduate students are in the pro­ cess of acquiring new knowledge and consolidating the knowledge, reflection is essential to the personal meaningful learning process. Thus, the profession is at the forefront of promoting and encouraging activities and assess­ ment tasks that are centred towards the develop ment of reflection. This research paper described the essential KPA’s and assessment criteria that were needed for the development of a new clinical assessment form. The next step in the process of determining the applicability of the assessment form would be to conduct a pilot study and to establish the reliability of the newly formulated form. Conclusion and implications for practice The results of this study can be used as a basis to develop a new clinical assess­ ment form that reflects the competencies that physiotherapy students should pos­ sess in a South African context that takes into account current norms and clinical standards. This process of establishing a reliable tool can be used in all health professional disciplines. In addition, the study further contributes towards reduc­ ing subjective bias among clinical edu­ cators and assessors by providing them 15 SA JournAl of PhySiotherAPy 2011 Vol 67 no 2 with clear guidelines regarding what to assess, which could ultimately improve the validity of assessing student out­ comes. Limitation(s) of the study The study did not include all the physio­ therapy training schools in South Africa however, the information can be used to guide the practices in all the schools. References Berner ES, Bender JK 1978 Determining how to begin, in evaluating clinical competencies in the health professions. Morgan. M & Irby. D eds. Mosby Co, St Louis Chartered Society of Physiotherapy 1996c (pg 10) Developing effective assessment procedures, CSP, London Christie CR, Bowen DM, Paarman CS 2003 Curriculum evaluation of ethical reasoning ad professional responsibility. Dental Education 67; 55­63 Clouder L 2000 Reflective practice in physi­ otherapy education: a critical conversation. Studies in Higher education 25: 211­223 Cross V 1999 The same but different. A Delphi study of clinician’s and academics’ perceptions of physiotherapy undergraduates. Physiotherapy 85; 28­39 Cross V, Hicks C 1997 What do clinical educators look for in physiotherapy students? Physiotherapy 83: 249­260 Epstein RM, Hundert EM 2002 Defining and assessing professional competence. Journal of the American Medical Association 287; 226­235 Freeman J, Rogers J 2010 A comparison of rank ordered professional attributes by clinical supervi­ sors and allied health students. Internet Journal of Allied health Sciences and Practice. 8: 1­4 Hassan F, Keeney S, McKenna H 2000 Research guidelines for the Delphi survey technique. Journal of advanced nursing 32: 1008­1015 Hobbs C, Henley E, Higgs J, Williams V 2000 Clinical education program strategies for challeng­ ing times. Focus on Health Professional Education: A Multidisciplinary Journal 2; 1­17 Jette DU, Bertoni A, Coots R, Johnson H, Mclaughlin C, Weisbach C 2007 Clinical instruc­ tors perceptions of behaviours that comprise entry­ level clinical performance in physical therapist student: A Qualitative study. Physical Therapy 87; 833­843 Lewis L, Stiller K, Hardy F 2008 A clinical assessment tool used for physiotherapy students: is it reliable? Physiotherapy Theory and Practice 24: 121­134 Meldrum D, Lydon A, Loughnane M, Geary F, Shanley L, Sayers K, shinnik E, Filan D 2008 Assessment of undergraduate physiotherapy clinical performance: Investigation of educator inter­rater reliability. Physiotherapy 94: 212­219 Mitchell M, Henderson A, Groves M, Dalton M, Nulty D 2009 The objective structured clini­ cal examination (OSCE): Optimising its value in the undergraduate nursing curriculum. nurse education Today 29: 398­404 Sanford J, Stratford P, Solomon P 1993 Clinical evaluation; Physiotherapists ranking of competen­ cies. Medical Teacher 15:121­127 Simuzingili T 1998 Identification of clinical competencies in physiotherapy education at the University of the Western Cape. University of the Western Cape, South Africa Suls J, Rothman A 2004 Evolution of the Biopsychosocial Model: Prospects and Challenges for Health Psychology 23: 119­125 Wellard SJ, Bethune E 1996 Reflective journal writing in nurse education: whose interests does it serve? Journal of Advanced Nursing 24: 1077­1082 World Medical Organization 1996 Declaration of Helsinki. British Medical Journal 313: 1448­1449