16 May 2015, Vol. 7, No. 1 AJHPE Research OSPE is an abbreviation for Objective Structured Practical Examination. In the literature the terms OSPE and OSCE (Objective Structured Clinical Examination) are sometimes interchanged.[1] For the purpose of this article, the term OSPE will be used, as this method is solely applied in the assessment of practical skills and not in the application of these skills in the clinical setting. The OSPE consists of a circuit of stations that tests a variety of techniques to establish practical competence. It has been researched and shown to be an effective, valid, reliable and defendable assessment method in emergency medical training,[1] nursing,[2,3] physiology[4] and oral surgery.[5] At the University of the Witwatersrand, Johannesburg, South Africa, the OSCE method is currently used in the graduate entry medical programme (medicine) and in the undergraduate nursing programme. The traditional, unstructured method of practical skills assessment has three variables, which have the potential to increase the subjectivity of the method[4,5] and consequently interfere with the assessment of the student. These variables include the student, examiner and technique.[1] The OSPE method is an attempt to control examiner and technique variability. An attempt is also made to standardise the environment and process of the practical skills test.[4] Currently, the students’ peers act as models during a practical skills test, which introduces a certain amount of standardisation as they are free from comorbidities that may complicate the assessment. The structured nature of the OSPE decreases the variability of the examiner, which is especially important as they often have different levels of experience.[5] Second-year students have not yet been exposed to the clinical area and during a practical skills examination, where a peer is used as a model, the outcome or effect on the patient cannot be judged.[6] At second-year level, the focus is on competency in technique rather than effectiveness of treatment. Students are still learning the elements needed to execute the skills safely and effectively.[7] This makes it possible to control the practical skills test, including the examiner, technique and environment, to improve objectivity, consistency and fairness to all students. Traditionally, physiotherapy practical skills have been assessed by a method that leaves room for subjective interpretation of competency and at times lacks the formative benefits of assessment. The OSPE method attempts to control for the variability of the examination by providing examiners with a checklist that contains the micro-skills required from the student to be able to effectively complete the practical technique and by providing clear instructions to examiners, students and models. The influence that a change in method will have on student performance is as yet unclear. This study sought to evaluate the effect that a change in method of assessment would have on student performance and level of satisfaction. As such, the objective of this study was to describe and compare students’ and examiners’ perceptions of the OSPE and traditional mark sheet in the assessment of students’ practical skills. A further aim was to compare the examiners’ ratings of students’ performances when using the OSPE mark sheet with those of the traditional mark sheet. Methods This was a quantitative, descriptive and comparative study. Ethical clearance was granted by the Human Research Ethics Committee of the University of the Witwatersrand. All second-year physiotherapy students and examiners participating in the practical tests were invited to participate in the study. Informed consent was obtained from the students and examiners. Students and examiners involved in supplementary practical tests were excluded. Student and examiner satisfaction were assessed with self-administered questionnaires. Student and examiner satisfaction questionnaires were developed.[8,9] Background. Traditionally, physiotherapy practical skills have been assessed by a method that relies on the subjective interpretation of competency by the examiner and lacks the formative benefits of assessment. Objective. To describe and compare student performance and satisfaction and examiner satisfaction with regard to the Objective Structured Practical Examination (OSPE) and traditional mark sheets during the practical skills assessment. Method. Students and examiners taking part in the second-year physiotherapy practical skills test were invited to participate by completing a series of questionnaires. Performance of techniques was marked using both the OSPE and traditional mark sheets. Results. Sixty-seven students and nine examiners participated in the study. Students scored an average of 4.6% (SD ±16.4) better when using the traditional mark sheet. Nonetheless, students and examiners expressed a preference for the OSPE mark sheet. Conclusion. The OSPE mark sheet allows for increased objectivity, as the specific micro-skills are clearly listed and appropriately weighted. This resulted in increased satisfaction, but a decrease in marks obtained. By assessing the effect of implementation of the OSPE method on performance and satisfaction, change in the current situation can be monitored. Afr J Health Professions Educ 2015;7(1):16-21. DOI:10.7196/AJHPE.228 The implementation of the Objective Structured Practical Examination (OSPE) method: Students’ and examiners’ experiences B Olivier, PhD; V Naidoo, MSc Physiotherapy; W Mudzi, PhD; H van Aswegen, PhD; J Potterton, PhD; H Myezwa, PhD; R Roos, PhD; L Godlwana, MSc Physiotherapy; D Maleka, PhD, MPH; S Mtshali, MSc Physiotherapy; V Ntsiea, PhD, MPH; A Stewart, PhD; M Romm, MSc Pain; C Humphries, MSc Physiotherapy; B Watt, BSc Physiotherapy Department of Physiotherapy, Faculty of Health Sciences, University of the Witwatersrand, Johannesburg, South Africa Corresponding author: B Olivier (benita.olivier@wits.ac.za) May 2015, Vol. 7, No. 1 AJHPE 17 Research Content and construct validity and reliability were established. A group of 10 physiotherapists (not participating in the main study) were asked to critique the content of the questionnaires. The group consisted of academics, clinicians and students. Suggestions were incorporated and questionnaires were modified accordingly. A pilot study was performed to establish the time taken to complete the questionnaires and to establish their reliability. Each of the 10 students and the examiners participating in the pilot study were invited to complete the questionnaires twice over a period of five days. The development of the OSPE mark sheets was done according to the following steps:[1] • Second-year students were divided into groups of 10. • Each group was given a technique to break down into micro-skills or steps. • A discussion took place between the groups. • Where necessary, the list of micro-skills was added to or modified, or the sequence of performance changed. • This list was given to the lecturer who weighted each item according to her own perception of importance or difficulty. The weighting of micro-skills was performed by the lecturer and not by the students, as it was felt that experience was needed to judge certain micro- skills as being more important than others. • Two weeks before each practical test, all examiners involved discussed the technique and weighting of each micro-skill. • During the practical test, each examiner made notes with regard to the original set of micro- skills for the technique being examined. Possible concerns regarding the specific technique were to be discussed after the examination. • All techniques used during the practical test were re-evaluated and modified where necessary. During each practical test, there were six stations of five minutes each. One technique was examined at each station. Stations 1 - 3 were assessed by one examiner at each station using the traditional mark sheet. Stations 4 - 6 were assessed by two examiners at each station – one examiner used the traditional mark sheet and the other the OSPE mark sheet. The OSPE examiners moved to stations 1 - 3 halfway through the practical test. These two examiners were positioned at separate tables and not allowed to share their experiences during or immediately after the assessment. The behaviour of the examiners and the procedures were standardised at each station (Addendum A). Students were made aware of the information stated in the instructions to examiners (Addendum A). For each micro-skill, a limited number of ratings was available to improve reliability.[8] A rubric was provided for clarity on the allocation of marks: 0/6=0% (incompetent); 2/6=33% (poor performance); 4/6=66% (satisfactory performance); 6/6=100% (excellent); and 2/4=50% (competent). If there was any discrepancy in the marks given by the traditional compared with the OSPE mark sheet, the higher mark was awarded to the student. This ensured that students were not disadvantaged by participating in this study. Directly after the practical test or examination, all students and examiners were given the first student and examiner satisfaction questionnaires, respectively, to complete. Students and examiners were directed to separate rooms where they completed the questionnaires. Participants were required not to discuss the questions or answers with one another, but merely to give honest answers. Two days after the marks and practical mark sheets were returned to the students, they were asked to complete the second student satisfaction questionnaire, as it was argued that students’ perceptions may change after they received their marks. The same procedures were followed as described above for completing the first post-practical questionnaire. Descriptive statistics were used to analyse the demographic details of the study sample. Student and examiner satisfaction and student performance were analysed using frequencies and presented in tables as appropriate. Student performance during the practical test using the traditional and OSPE mark sheets was presented using means and standard deviations, while student performance using the two mark sheets was compared using a paired t-test. Performance was compared at stations where a student was examined by using both the OSPE and traditional mark sheets at the same time. Results Sixty-seven students took part in the study. The average age was 21.3 years (SD ±2.4). There were 10 (15%) male and 57 (85%) female students in the study sample. Nine examiners participated in the study. Of these, 3 (33%) were male and 6 (66%) were female. The overall student performance when using the OSPE and traditional mark sheets is shown in Table 1. The difference in student marks when using the OSPE and traditional mark sheets (combined) is shown in Table 2. The mean student mark was 4.6% higher when using the traditional mark sheet than with the OSPE mark sheet. The student satisfaction with the practical test (soon after the test but before knowing their marks) is shown in Table 3. Table 1. The overall student performance when using the OSPE and traditional mark sheets (combined) (N=563)* Marks allocated via: Mean, % ±SD ±SEM Correlation p-value Traditional mark sheet 64.4 20.4 0.86 0.7 0.000 OSPE mark sheet 59.8 18.8 0.79 SD = standard deviation; SEM = standard error of mean. *Marks allocated to the performance of different techniques were added together. Table 2. The difference in student marks when using the OSPE and traditional mark sheets (combined) (N=563)* Marks allocated via: Mean difference, % ±SD ±SEM 95% CI t-value p-value Traditional mark sheet minus OSPE mark sheet 4.6 16.4 0.69 3.3 - 6.7 6.7 0.000 SD = standard deviation; SEM = standard error of mean; CI = confidence interval. *Marks allocated to the performance of different techniques were added together. Table 3. General student satisfaction with regard to the practical test (N=67)* Item Agree, n (%) Disagree, n (%) The practical test was fair 65 (97) 2 (3) In general I’m satisfied with the way the practical test was conducted 65 (97) 2 (3) *Feedback from students directly after the practical tests, before they had access to their marks. 18 May 2015, Vol. 7, No. 1 AJHPE Research The majority of students thought that the practical tests were fair. Those who disagreed on the fairness and general conduct of the practical test indicated that they perceived it as too rushed and were not given enough time ‘to think’. Students’ views on the use of the two mark sheets are shown in Table 4. On average, more students thought that the marks from the traditional mark sheets matched their own perception of their performance than the marks on the OSPE sheet. However, on average more students thought using the OSPE mark sheet was fairer compared with the traditional sheet. The student satisfaction with the two mark sheets after receiving their marks is shown in Table 5. The students were satisfied with both the traditional and OSPE mark sheets after receiving their marks. Some students felt that the traditional mark sheet allowed for better marks to be obtained and that the examiners ‘think more about what you deserve rather than just giving ticks and crosses’. Students who preferred the OSPE mark sheet felt that the specific micro- skills that were listed made the process much more objective and ‘specific’. The examiners’ views on the use of the traditional mark sheet are shown in Table 6. Both the traditional and OSPE examiners were satisfied with the general conduct of the practical examination. The examiner who indicated dissatisfaction with the way in which the practical test was conducted stated that more time was needed in between students to add up the marks. The examiners’ satisfaction with the two mark sheets is shown in Table 7. More examiners were satisfied with the OSPE mark sheet than the traditional one. The examiner who was dissatisfied with the OSPE mark sheet indicated that the weighting of some of the micro-skills should be adapted to obtain a better reflection of the students’ overall performance. Discussion Before this study was conducted, practical tests in the university’s physiotherapy department were done where students’ ability to perform certain skills or techniques was evaluated. However, these practical tests were conducted in a partially unstructured manner, where the technique, examiner and environment were not controlled adequately. The traditional mark sheets left much room for the examiner’s subjective interpretation of components or micro-skills to be assessed and weighting of the former (Addenda B and C). The marks awarded when using the OSPE mark sheet were on average 4.6% lower than when using the traditional mark sheet; however, a relatively good correlation of 0.7 was found between the two types of mark sheets. The difference in marks may be because the OSPE sheet has numerous micro- skills listed, each with a predetermined weighting. With the traditional mark sheet it is therefore possible that examiners may not notice if a student omitted a micro-skill, or that it may have been done in an incorrect manner. For example, when a contract relax technique of the knee is the being tested, the student may position the model’s leg incorrectly or may have forgotten to explain the findings to the model. Each of these two micro-skills appear on the OSPE mark sheet and should therefore be assessed in a uniform manner by all examiners. Each micro-skill is allocated a predetermined weighting; therefore, all examiners will subtract the same amount of marks for a skill that is poorly executed or omitted. According to the literature, the OSPE mark sheet gives a more valid presentation of a student’s true ability to perform a technique.[4,9] Traditionally, students did not receive optimal formative benefits when the traditional mark sheet (Addendum B) was used. This lack of benefit may be attributed to the lack of specificity as explained above and the need for extensive writing within the time allocated at each station. The time was not always enough to thoroughly assess the performance of the technique and write in enough detail what the student did correctly and incorrectly. Table 4. Student views on the use of the two mark sheets (traditional and OSPE) (N=59)* Item Agree, n (%) Disagree, n (%) The OSPE marks I received matched my own perception of my performance 36 (61) 23 (39) The traditional marks I received matched my own perception of my performance 42 (71) 17 (29) Mark allocation using the OSPE mark sheet is fair 48 (81) 11 (19) Mark allocation using the traditional mark sheet is fair 36 (61) 23 (39) The OSPE mark sheet should be used in the practical examination in future 43 (73) 16 (27) The traditional mark sheet should be used in the practical examination in future 31 (53) 28 (47) *Eight students did not complete the second student satisfaction questionnaire as they were absent on the day that the questionnaires were handed out. Table 5. Student satisfaction with the traditional and OSPE mark sheets (N=59)* Total Mark sheet Satisfied, n (%) Dissatisfied, n (%) Traditional 46 (78) 13 (22) OSPE 48 (81) 11 (19) *Feedback from students after they received their practical test marks. Table 6. Examiner satisfaction regarding the general conduct of the practical test (N=9) Total Question Agree, n (%) Disagree, n (%) In general I’m satisfied with the way the practical test was conducted 8 (89) 1 (11) Table 7. Examiner satisfaction with the traditional and OSPE mark sheets (N=9) Total Mark sheet Satisfied, n (%) Dissatisfied, n (%) Traditional 5 (56) 4 (54) OSPE 8 (89) 1 (11) May 2015, Vol. 7, No. 1 AJHPE 19 Research Feedback to the students was therefore often inadequate. Larsen and Jeppe- Jensen[10] found that one of the greatest benefits of the OSPE was immediate feedback. Feedback motivated students and improved their learning. In this study, feedback was given a week after the practical test. By reviewing this delayed time frame, one can improve on formative benefits. In this study, students felt that the practical test was fair and were in general satisfied with the manner in which it had been conducted, even though they did not have access to their marks at that point in time. The abovementioned feedback is highlighted as students’ opinion had not yet been biased by the marks that they received for the practical tests. This finding is supported by Ryan et al.[3] Although student satisfaction could not be compared with that of previous years, the controlled environment and behaviours of examiners may have contributed to the positive attitude of the student. The negative impact of external factors, such as the effect of examiner behaviour on student performance, was emphasised by Larsen and Jeppe-Jensen.[10] They also highlighted the importance of a positive atmosphere. Furthermore, all students were marked by the same examiners, which decreased the variability in marks owing to mark differences. All these factors can contribute to student satisfaction. On average, students felt that the traditional mark sheet reflected their performance better than the OSPE sheet. This may be the result of the marks showing a better performance when marked with the traditional mark sheet. Students were satisfied with both the traditional and OSPE mark sheets. Even though their marks were lower when marked with the OSPE sheet, they did report that they felt the OSPE was fairer and should be used in practical tests in the future. Feedback given in studies done by Menezes et al.,[11] Larsen and Jeppe-Jensen[10] and Abraham et al.[4] confirm that students were in favour of the OSPE. They also found that the OSPE mark sheet was described as fair owing to the increased objectivity, which results from the specific micro-skills being clearly listed and appropriately weighted in each of the OSPE mark sheets. The OSPE also increases the inter-rater reliability when less experienced examiners are involved in marking practical tests, and in cases when examiners are marking stations on content that they have not taught the students.[8] Human resource constraints makes it impossible for examiners to mark only those stations that are testing skills that they taught the students. Chenot et al.[8] found moderate to good reliability when the mark allocation of less experienced examiners was compared with that of more experienced examiners when using the OSPE and stated that training of examiners may improve reliability. Examiners were satisfied with the conduct of the practical examination, regardless of which mark sheet was used. This finding is important as it excludes bias towards the practical tests that may not be related to a specific utilised mark sheet. It may also indicate that the specific guidelines given to examiners (Addendum A) may have contributed to decrease uncertainty with regard to factors such as prompting and time keeping. Improvement of these factors will increase inter-rater reliability.[8] Larsen and Jeppe-Jensen[10] and Qureshi[12] found that examiners perceived the OSPE favourably and as a good test of clinical relevance. Examiners in this study were satisfied with the OSPE mark sheet, more so than with the traditional one. The process whereby OSPE mark sheets were developed gave students the opportunity to learn, as it contributed to their development. The OSPE mark sheet will be refined in future research, as Chenot et al.[8] found greater reliability between micro-skills where mark allocation is dichotomous. The lower the number of options available, the lower the leeway for interpretation. They furthermore suggested that training of examiners can improve reliability in an OSPE.[8] A traditional and an OSPE examiner were present at only three of the six stations owing to human resource constraints. The best possible solution to this limitation was to move the three OSPE examiners to different stations halfway through the practical test. Participating examiners should preferably remain in their specific stations. Examiners had different levels of experience, including clinical and practical examination experience. Olivier et al.[13] found that there was a high correlation between examiners with a similar number of years of experience. To overcome the different levels of experience[1] all staff underwent a briefing session on behaviour, the practical test process and the mark sheet before the practical test was undertaken. Conclusion Practical examinations will always contain an element of subjectivity, but the amount of subjectivity can be limited by using the OSPE mark sheet during practical tests. The clearly operationalised list of items that forms part of the OSPE method of assessment makes it the most objective method available to assess the practical competence of students. Although students and staff were satisfied with the traditional manner in which practical skills were assessed, the satisfaction arising from introducing an evidence-based, educationally sound method of assessment by using the OSPE mark sheet in practical tests is shown. References 1. O’Connor HM, McGraw RC. Clinical skills training: Developing objective assessment instruments. Med Educ 1997;31:359-363. 2. Kurz JM, Mahoney K, Martin-Plank L, et al. Objective structured clinical examination and advanced practice nursing students. J Prof Nurs 2009;25:186-191. [http://dx.doi.org/10.1016/j.profnurs.2009.01.005] 3. Ryan S, Stevenson K, Hassell AB. Assessment of clinical nurse specialists in rheumatology using an OSCE. Musculoskeletal Care 2007;5:119-129. 4. Abraham RR, Raghavendra R, Surekha K, et al. A trial of the objective structured practical examination in physiology at Melaka Manipal Medical College, India. Adv Physiol Educ 2009;33:21-23. [http://dx.doi. org/10.1152/advan.90108.2008] 5. Macluskey M, Hanson C, Kershaw A, et al. Development of a structured clinical operative test (SCOT) in the assessment of practical ability in the oral surgery undergraduate curriculum. Br Dent J 2004;196:225-228. 6. Medley D. Teacher Competency Testing and the Teacher Educator. Ablex: New Jersey, 1984. 7. Miller GA, Galanter E, Pribram KH. Motor Skills and Habits. Harmondsworth: Penguin, 1970. 8. Chenot JF, Simmenroth-Nayda A, Koch A, et al. Can student tutors act as examiners in an objective structured clinical examination? Med Educ 2007;41:1032-1038. 9. Patricio MF, Juliao M, Fareleira F, et al. Is the OSCE a feasible tool to assess competencies in undergraduate medical education? Med Teach 2013;35(6):503-514. [http://dx.doi.org/10.3109/0142159X.2013.774330] 10. Larsen T, Jeppe-Jensen D. The introduction and perception of an OSCE with an element of self- and peer- assessment. Eur J Dent Educ 2008;12:2-7. [http://dx.doi.org/10.1111/j.1600-0579.2007.00449] 11. Menezes RG, Nayak VC, Binu VS, et al. Objective structured practical examination (OSPE) in forensic medicine: Students’ point of view. J Forensic Leg Med 2011;18:347-349. [http://dx.doi.org/ 10.1016/j.jflm.2011.06.011] 12. Qureshi NS. Examiners’ perceptions of the objective structured clinical examination in colposcopy. J Obstet Gynaecol 2013;33:188-190. [http://dx.doi.org/10.3109/01443615.2012.737050] 13. Olivier B, Naidoo V, Humphries C, et al. Inter-examiner reliability when using the Objective Structured Practical Examination (OSPE) mark sheet for physiotherapy practical examinations. S Afr J Physiotherapy 2013;Wits Special Edition:21-28. 20 May 2015, Vol. 7, No. 1 AJHPE Research Addendum A. Instructions to examiners Reading the question Give the student time to read their question in silence before they start (±30 sec). Greeting and politeness Examiners should greet the students politely. Put the student at ease by having a neutral expression on your face. Prompting students If a student leaves out a step of the technique, you should prompt him/her. However: • You should wait until s/he has completed the whole skill before prompting, in case s/he remembers by her/himself • Only prompt once • S/he will lose at least half the marks for that step Keeping time: 5 minutes per station Guide the candidate in terms of time – one prompt per station when needed. Please send students away the moment the bell rings, even if they have not finished. General information • Students have to talk to the patient/model throughout the exam – explain what they are going to do, their findings, etc. • Whenever the student is doing a procedure which will not be relevant to discuss with the patient/model, the student has to tell the examiner what they are doing while they are doing it. • Make sure that students don’t just talk without doing a technique. It is however important that they talk through the technique so that we don’t miss important steps that may not be so clear just from observing their actions. • Some patients/models help students indirectly, e.g. by positioning themselves correctly. If you notice this please reprimand the model. • Some lecturers wish to use the opportunity to teach students while they are examining them. Please do not do this since it gives students tips for the following stations and it takes up time. Students will receive their mark sheets back and will be able to learn from the feedback. • The student can make an appointment to discuss his/her performance with the examiner at a later stage. • Examiners should write a short report after the prac test/exam on common errors made by students, as well as other problems encountered at their station. Addendum B. Physiotherapy examination form PHYSIOTHERAPY EXAMINATION FORM NAME: ……………………………………………… DATE: ……………………… QUESTION/PROBLEM:………………………………………………………….………………………………… Possible marks Marks awarded 1. General 1.1 Professional appearance & conduct 1.2 Preparation of patient & equipment (including positioning) 1.3 Interaction with patient (explanation, motivation, physical handling, respect & use of voice. 5 COMMENTS 2. Technique 2.1 Correct choice 2.2 Demonstration 2.3 Application of technique (appropriate hand position, ROM, use of body weight, depth, sequence etc) 2.4 Effectiveness of technique 40 COMMENTS 3. Background knowledge and recording 5 TOTAL 50 PERCENTAGE: ………………………………………….. SIGNATURE: ……………………………………… May 2015, Vol. 7, No. 1 AJHPE 21 Research Addendum C. Example of OSPE mark sheet PHYSIOTHERAPY EXAMINATION FORM Name: ______________________________________________ Date: ___________ Question/problem: Demonstrate use of the contract relax technique to improve his knee flexion range of movement 1. General /5 Professional appearance & conduct 0 1 Preparation of area & equipment 0 1 2 Interaction with patient (explanation, motivation, physical handling, respect & use of voice) 0 1 2 Comments 2. Technique /40 Screening for contra-indications 0 2 4 Positioning of patient – high sitting or prone 0 1 2 Student places segment at the end point of limitation within the movement pattern 0 2 4 6 Resistance is then given either to the restricted agonist (direct contraction) or to the antagonist (reciprocal relaxation) 0 2 4 6 Allow a few degrees of motion to ensure that all the muscles in that group have been recruited 0 2 4 6 Duration and intensity of contraction should be sufficient to generate a strong contraction (approx 5 sec ) 0 2 4 Ask pt. to completely relax after which segment is passively/actively taken into new available ROM 0 2 4 Repeat procedure 0 1 2 Explanation of findings to patient 0 2 4 General impression 0 1 2 Comments: 3. Background knowledge and recording Difference between the Contract Relax and Hold Relax Techniques? 0 2.5 5 The contract relax method uses an isotonic contraction while the hold relax uses an isometric contraction Comments /5 TOTAL /50 PERCENTAGE Name of examiner: __________________________ Signature: _____________