40 Fisioterapie, M ei 1986, d e e l 42 no 2 The Application a n d Development o f the Objective Structured P ractical Examination in Evaluating Physiotherapy Students’ P erform ance a t the University o f Cape Town SHERRIL HULETT, NARINA GILDER SUMMARY The purpose of this paper is to present the development and application of the Objective Structured Practical Examination (OSPE) in evaluating physiotherapy students’ practical performance. This is discussed in terms of station allocation, the circuit of tests and the method of evaluating these tasks. The effects on teaching and examining practical skills are analysed. Recommendations are made for implementation of the system. The goals of greater objectivity and efficiency in examining practical skills seem to be achieved by using OSPE, but these have as yet not been validated scientifically. Obje cti ve s t r u c t u r e d pr ac t ic al e x a m i n a t i o n ( O S P E ) o r i g i n a t e s f r o m th e O b je c ti v e S t r u c t u r e d C li n ic a l E x a m i n a t i o n ( O S C E ) 1,2 which was i n t r o d u c e d d u r i n g a visit to S o u t h Africa in 1981 by P ro fe ss or H a rd e n , H ea d o f th e Me d ic al E d u c a t i o n U n it at t h e University o f D u n d e e . T h e need fo r m o r e objectivity2,3'4 an d effi­ ciency in e xa m in in g students was highlighted, an d O S P E a p p e a r s to offer a pra ctical s ol ut io n to these needs. OPSOMMING Die doel van hierdie referaat is om die ontwik- keling en toediening van die Objektiewe Gestruktuurde Praktiese Eksamen (OSPE) voor te stel in die evaluasie van die praktiese ver- toning van fisioterapie studente. Dit word bespreek deur middel van stasie aanwysing, die omtrek van toetse en die metode van evalu- ering van hierdie take. Die uitwerking op die onderwys en eksaminering van praktiese bekwaamhede word ontleed. Voorstelle word gemaak vir die implementering van die stelsel. Die doelwitte van groter objektiwiteit en nuttig- heid by die eksam inasie van praktiese bekwaamheid word blykbaar bereik deur die gebruik van OSPE, maar is tot nog toe nie wetenskaplik bekragtig nie. O S P E A T U C T In 1982 the D e p a r t m e n t o f P h y s i o t h e r a p y a t U C T i n t r o d u c e d O S P E as p a r t o f the e x a m i n a t i o n system. It a t t e m p t s to e va lu a te pra ct ic al o r technical skills r a t h e r t h a n th e a p p li c a ti o n o f skills to specific pa ti en ts . T he la t te r w o ul d co n st it u te clinical skills. S tu d e n t s o f ot h e r y ear s are used as “ m o d e l s ” t o w h o m the ex am in ee s ap p ly their technical skills. a Sherril Hulett, B.Sc. (Physiotherapy), Adv. Dip.T.P. Lecturer, Department of Physiotherapy, University of Cape Town Narina Gilder, B.Sc. (Physiotherapy), Dip. Ed. Physiother. Senior Lecturer, Head of Department, Dept, of Physio­ therapy, University of Cape Town D E S C R IP T IO N T h e O S P E consists o f a cir cuit o f st at io n s m u c h like th e id e a o f a cir cu it o f exercises in c ir c ui t tr ai n in g. A t each st a ti o n the st u d e n t is req u ir ed to p e rf o rm a specific ta s k in five mi n ut es . A g o n g goes a n d th e st u d e n ts mo ve to the n e x t s t a ti o n a n d re ad the n e x t t a s k fo r on e m in u te . A n o t h e r g on g goes a n d the st u d e n ts p e rf o rm R ep ro du ce d by S ab in et G at ew ay u nd er li ce nc e gr an te d by th e P ub lis he r (d at ed 2 01 3. ) Physiotherapy, M ay 1986, vol 42 no 2 41 the tas k fo r five mi nu tes an d so on until all the tasks have been pe rf o rm e d by each st ud ent . At U C T a p r a g m a ti c a p p r o a c h is being used to develop a w o rk in g m ode l o f O S P E in o r d e r to suit the p a rt ic u l a r d e p a r t m e n t a l needs. This m od el m us t be cons id ere d as being in the e x p e ri m en t al stage. A L L O C A T IO N O F S T A T IO N S T h e c o n t e n t o f th e ta s k s is su bj ect -ba sed an d the al lo ca ti o n o f st at io ns was originally d o n e o n the basis o f the n u m b e r o f h o u rs fo r wh ich ea ch subject was t a u g h t d ur in g the year. T h e u n d e rl y in g a s s u m p t i o n was t h a t the n u m b e r of h o u r s in d ic at ed th e i m p o r t a n c e o f the subject. This has b e e n r e j e c t e d a n d t h e s t a t i o n s a r e n o w a l l o c a t e d ac c o rd i n g to the syllabus. E ac h syllabus is given fair representation. (A syllabus includes one or m or e subjects — see Table). T h e o r g a n i s a ti o n o f time a n d space limits the n u m b e r o f st at i o n s to ten. O S P E is held three times ^ y e a r l y in J u n e , S e p t e m b e r an d N o v e m b e r. T h e n u m b e r i ) o ( st at i o n s pe r year in di cat ed in the T a b le is d is tri bu te d over the th r ee e x a m i n a t i o n s ac c o rd i n g to th e a m o u n t o f w o r k covered by the time they occur. T A S K S E x a m p l e s o f the tasks are the following: “ D e m o n s t r a t e the ap p li c a ti o n o f u lt r a s o u n d to the me di al as pec t o f th e knee usi ng th e dose 0,25 w a tt s / c m 2 fo r on e m i n u t e ”. “ D e m o n s t r a t e the ve rt e b ra l ar te ry insufficiency tests an d th e n d e m o n s t r a t e cervical ro t a t i o n to the right gr a d e II ( M a it l a n d ) for 30 se c o n d s”. E a c h ta s k is ev a lu a te d by m e an s o f a checklist. T he check list c onsi sts o f a b r e a k d o w n o f th e ta s k int o the c o m p o n e n t s k i l l s . F o r e x a m p l e t a s k o n e a b o v e , e x a m i n i n g the te ch n iq ue o f u lt r a s o u n d , co ul d be ev al u ­ ated in te rm s of: 1. T he p r e p a r a t i o n o f the model 2. Testing the mac hi ne 3. M a i n t e n a n c e o f the m a c h i n e ’s h ea d in c o n ta c t with the ( s t u d e n t ’s) m o d e l ’s skin. 4. T he speed with wh ich the h e a d is m o v e d and 5. A ccuracy o f the dose applied T H E C H E C K L IS T T he checklist is qua n ti fi ed by a scor ing system for ea ch o f the c o m p o n e n t s , reflecting the re q u i re m e n ts o f ex e c u t io n , as well as th e qu al it y o f the e x e c u t io n o f the skills where a p p r o p r i a t e . A g ai n , using the ta s k on u l t r a s o u n d as an e x a m p l e , the checklist cou ld ta k e the following form: 1. P r e p a r a t i o n o f mod el 0 1 2 3 2. Te sting th e ma ch in e 0 1 3. M a i n te n a n c e o f the head in co n tr a c t with m o d e l ’s skin 0 1 2 3 4. T h e speed o f the tec hn iq u e 0 1 2 5. Ac cur acy o f the dose 0 1 (0 a n d I in di cat e “n o t d o n e ” o r “d o n e ” respectively while ad d it io n a l m a r k s are scor ed for qu ality o f the ex e c u t io n in so m e skills). S u b m i n i m u m scores o r penalties can be built int o the checklist. In the ch ecklist used above, pena lties for o m i t t i n g the m a c h i n e test an d in ac c u ra c y o f the dose co u ld be m in u s th r ee f o r each. Th is w ou ld e ns u r e th a t the s t u d e n t do es n o t pass the st a ti o n if these impe rat ive p re c a u t io n s are n o t perf orm ed. Co mp il in g checklists is time- con sum in g an d originally th e ch eck li st o f skills was s h o r t a n d general. T h e need fo r m o r e de tai l be c am e o b v io u s an d m o r e specific checklists were compiled. This pr o d u c e d so m e checklists o f th e o p p o s i te e x tr e m e in w h ic h u p to fifty items were ex p ec t ed to be e x a m i n e d in five minutes. Thfs was u n a c c e p t a b le a n d ac ce n t u a te d the need fo r a more d et ai le d analysis o f th e skills to be ev al u a te d, as well as se tt in g m o r e specific go als as to the type o f skills th a t are requi red fo r testing. T H E A N A L Y S IS O F T A S K S A n a ly si ng the ta s ks int o c o m p o n e n t skills to be tes ted , en ab le s o n e to c o n tr o l the e m p h a s is o n the skills to be ev al u a te d an d a g re a te r cross sect ion of skills c a n th u s be tested. T he incr eased so ph i st i c a ti o n Table. Technique IB Year I Syllabus one Movement Technique 6 Rehabilitation 5 Splints and bandaging 3 Syllabus two Massage 5 Syllabus three Passive Movement 5 Mobilisations Syllabus fo u r Electrotherapy Syllabus fiv e Basic Facilitations P N F Technique Technique Total per IIB IIIB Syllabus Year I I Year 3 17 3 — 11 6 — 16 5 6 15 7 8 16 3 5 3 5 R ep ro du ce d by S ab in et G at ew ay u nd er li ce nc e gr an te d by th e P ub lis he r (d at ed 2 01 3. ) 42 Fisioterapie, M ei 1986, deel 42 no 2 o f p h y s i o th e ra p y techn iq ues and the vast increase in syllabus c o n t e n t d e m a n d s a m or e skill-based m e t h o d of teac hin g an d exa m in in g. In satisfying these d e m a n d s the em ph as is can be placed on principles o f tech ni que w hen e x a m i n i n g prac tic al skills. O S P E a p p e a r s to p r o ­ vide a m e an s o f achieving these objectives. T H E E F F E C T O N T E A C H IN G Th e need to def in e skills in m o r e deta il in he r en t in O S P E co u ld result in clea rer te ac hi ng objectives and m o r e s e q u e n t i a l l y s t r u c t u r e d t e a c h i n g pr oce ss . T h e results o f O S P E also pr o v i d e a m e an s o f eva lua ting tea c hi ng efficiency a n d identifying specific in st ruc tio n and learning prob le ms . T he validity of, “are we teaching w h a t we w a n t to t e a c h ” can be ev alu at ed by assessing the con sistency o f results obt a in ed by th e stud ent s in each task. P o o r p e r f o r m a n c e o f an en tire class in any specific tas k could imply either tea c h er o r lear ning pro b le m s, scor ing p r o b l e m s , or q u es ti on m i s i n te rp r e ta t io n and the lecturer could then investigate the underlying reasons m or e easily. O S P E identifies weaknesses in p a r t ic u l a r s t u d e n t s ’ p e r f o r m a n c e o f basic technical principles and specific an d direct fee d- ba ck to st ude nts is p o s s ib le .1,4 C O M P A R T M E N T A L IS A T IO N O S P E d oes , how eve r, have its p ro bl e m s. It c o m p a r t ­ me nt al is es 2 aspects o f tr e a t m e n t , it rem ove s the st u d en t f r o m th e real s i tu a t io n an d it inhi bit s a holistic view of p ati en t m a n a g e m e n t. In redressing the p ro b l e m o f co m - p a r t m e n t a l i s a t i o n , U C T will a t t e m p t in th e f u t u r e to organ is e the O S P E for each g r o u p o f s t u de n ts a r o u n d a theme. F o r e x a m p l e the techn ica l skills ex a m i n e d will be app lie d to a specific p a r t o f the bo d y so t h a t as far as possible c o n ti n u it y in tr e a tm e n t is achieved. F u r t h e r ­ more, O S P E is co m p l e m en t e d by r eg ul ar clinical testing and a final clinical e x a m i n a to n . In these tests and e x a m i n a ti o n s , the s t u d e n t s ’ ability to a p p r o a c h the p a ti en t as a wh ole is ev alu a ted by c o n si d er in g p a ti e n t assessment, p la n n i n g the t r e a tm e n t, e x ec ut io n o f tec h­ nique, in te rpe rs ona l relationships and p a ti e n t educ atio n. E F F IC IE N C Y T h e entire process is a co ns id era bl e u n d e r t a k i n g and respo nsi bi li ty fo r the st aff m e m b e r c o - o r d i n a t i n g 2,3 and orga nis ing O S P E , a n d the co m pi li ng o f checklists is ti m e - co n su m in g fo r all involved. O nc e a b a n k o f O S P E q ue st io ns an d checklists has been bu ilt up an d a m o r e c o n s t a n t m o d u s o p e ra n d i fo r the e x e c u t io n o f O S P E is esta blish ed , th e efficiency o f this p a r t o f O S P E will improve. A lt h o u g h close a t t e n t i o n to a n u m b e r o f st ud ent s p e rf o rm i n g th e sam e tas k d u r i n g the e x a m i n a t i o n is d e m a n d i n g a n d tiring fo r th e e x a m i n e r 2, O S P E enable s a large n u m b e r o f st u d e n ts to be e x a m i n e d in a r e p r e ­ sent ative cro ss sect ion o f skills in on e day. Th is is ob vious ly pre fe rab le to the f o r m e r e x a m i n a t i o n system which w ou ld req u ir e day s o f ex am i n in g fo r the same n u m b e r o f s t u de n ts . W it h the increase in the n u m b e r of s t u d e n ts e n te r in g p h y s i o t h e r a p y schoo l it is im pe rat ive to use a m or e efficient m e t h o d o f testing skills. F o r this r e as o n, in 1984 U C T e x p e r i m e n t e d with a d u p li c a te cir cu it system in wh ich tw o ide ntical circuits o f stations were set up in a d jo in in g ro o m s. Bo th circuits had eight to ten stations. This system en abl es up to for ty s tu d en ts to be ex a m i n e d in on e m o r n i n g i.e. in two roun ds , each ro u nd exa mi ni ng up to twenty students in up to ten tasks and the whole process ta k in g a p p r o x i ­ m a t e ly tw o an d a h a l f ho u rs . T h e tw o g r o u p s of st u d e n ts are pre ven te d fr o m co n ta c ti n g one a n o t h e r by th e seco nd g r o u p arrivin g be fore the end o f r o u n d one a nd being isolated in a se p ar ate roo m. P R O B L E M S T w o pr o b l e m s arose fr o m the ab ov e system. T he first was the co nsi ste ncy o f scoring by ex a m i n er s assessing the sa m e tas ks in dif fe re nt circuits. Th is was partia ll y s o lv ed by m a k i n g a v a i l a b l e t h e c h e c k l i s t to the e x a m i n e r s a few day s p r i o r to th e e x a m i n a t i o n in or d e r t h a t the y fam iliaris e th emselv es with the items and clarify an y pro bl em s. R e h e ar sa l time fo r e x a m i n e r an d m ode l on the d ay o f the e x a m i n a t i o n was also increased for the same reasons. T h e seco nd p r o b l e m was th e ava ilability o f 16-20 e x a m i n e r s pe r O S P E . T h e s o l u ti o n to this m a y lie in the w o r k o f H ila r y W a d s w o r t h . 5 S h e con c lu d e s on the basis o f her study t h a t final year st ud ent s, using detailed checklists, will m a k e the sam e as se ssm en t o f p e r f o r m ­ an ce o f a tas k as a lectur er . T hi s is achie ved o nl y by intensive p r e p a r a t i o n o f the s t u d e n t assessor in terms o f o b s e r v a ti o n a l and affective skill training. Th is w ould be very ti m e c o n s u m i n g initially, b u t as a w o r k in g model bec om e s est ab li sh e d, so the efficiency will im p r o v e an d the system pro v e w orthwhile. R E C O M M E N D A T IO N S T h e a u t h o r s m a k e the fol lowing r e c o m m e n d a t i o n s for d e p a r t m e n t s ho pi n g to include O S P E int o their system o f e v a lu a tio n o f stu de n ts pra ctical p e rf or m an c e: 1. S et very specific goals fo r the system used a cc or di ng to d e p a r t m e n t a l needs and resources. 2. A s c e r t a in as s o o n as poss ible w h e th e r these g o a l ^ . are being met. 3. E v a lu a t e skills tested in th e clinical s i tu a t io n so as best to c o m p l e m e n t th e pra ct ic al skills tested by O S P E . 4. E m p h a s i s e the p r e p a r a t i o n o f assessors o f p e r f o r m ­ ance an d the mod el s involved in the e x a m i n a ti o n . C O N C L U S IO N T h e goals o f objectivity an d efficiency in O S P E are slowly being me t b u t are yet to be scientifically vali­ da te d . Specific goals as to the em p h a s is o f skills tested in O S P E are in the proces s o f being set. O S P E seems best suited to test p s y c h o m o t o r skills. T h e ability o f the s t u d e n t to t r e a t the p a ti e n t as a w h ol e an d pr og re ss t r e a t m e n t a p p r o p r i a t e l y is best e va lu a te d in the clinical s i tu a t io n . As in - d e p th e v a lu a ti o n o f the skills req u ir ed o f stud ent s at different levels o f co m pet en ce will enh anc e R ep ro du ce d by S ab in et G at ew ay u nd er li ce nc e gr an te d by th e P ub lis he r (d at ed 2 01 3. ) Physiotherapy, M ay 1986, vol 42 no 2 43 th e va lu e o f O S P E as a m e t h o d o f e v a lu a ti n g p s y c h o ­ m o t o r skills and as a c o m p l e m e n t to clinical p e r f o r m ­ ance testing. H o w ev er p r o b l e m a t i c O S P E m a y seem to be, its p o te n t ia l to satisfy the m a j o r e x a m i n a t i o n go als of object ivi ty an d efficiency is im m e n s e an d w a r r a n t s the time a n d energy sp ent on perfecting the system. References 1. Ha rde n R M, Stevenson M, Downie WW, Wilson GM. Assessment of clinical competence using objective struc­ tured clinical examination. Brit M ed J 1975; 1: 447-451. 2. Harden RM , Gleeson FA. Assessment of clinical com­ petence using an objective structured clinical exam in a­ tion (OSCE). M ed Educ 1979; 13: 41-54. 3. Stokes JF . The Clinical E xam ination (Medical Educa­ tion Booklet No. 2). Dundee: Association for the study of Medical Education, 1974. 4. Kent AP, Lazarus J. An objective medical student examination in obstetrics. S A M J 1983; 64: 388-389. 5. Wads wo rth H. Students as assessors in a performance test. Report on a pilot study. Physiotherapy 1981; 67: 364-367. The Program m e o f Clinical Experience an d Evaluation Employed a t the University o f Cape Town M. J. FUTTER SUMMARY This paper provides a brief description of the system of clinical evaluation used at the Univer­ sity of Cape Town. It gives the objectives of the clinical programme and the method of clinical supervision employed to achieve them. Some general problems of evaluation are discussed. 1.0 IN T R O D U C T IO N In o r d e r to fulfil the re q u i re m en ts for th e degree of B.Sc. P h y s i o t h e r a p y a n d e n su r e t h a t the pu bli c is p r o ­ tected, s tu de nt s have to be assessed acc u ra te ly in the clinical situ at io n so th a t thei r level o f co m p et en ce can be te s te d .1 2.0 O B J E C T IV E S O F T H E C L IN IC A L P R O G R A M M E A t the end o f the clinical p r o g r a m m e the st u d en t is M. J. Futter, B.A., Dip. (Phys), D.S.E., Assistant Lecturer, Dept, of Physiotherapy, University of Cape Town | OPSOMMING | Hierdie referaat gee 'n kort beskrywing van die ! kliniese evaluasie stelsel wat by die Universiteit i i van Kaapstad gebruik word. Die doelwitte van [ die kliniese program en die metode van kliniese 1 j toesig wat gebruik word om hulle te bereik, I word uiteengesit. Sommige algemene probleme | van evaluasie word bespreek. exp ec te d to have th e kn ow led ge, technical skills, and a tt itu de s to: 2.1 Int eg ra te an d ap pl y th eo r et i ca l kn ow led ge intelli­ gently in the clinical situa tion . 2.2 E x t r a c t th e r e l e v a n t f a c t s f r o m th e p a t i e n t ’s med ica l re co rd s an d i n t e r p r e t p e r t in e n t special inves tigations accurately. 2.3 Ev alu at e a p a ti en t co mprehensively. 2.4 Pl an an d prog re ss an a p p r o p r i a t e t r e a t m e n t p r o ­ g ram me . 2.5 P e r f o r m an effective a n d safe t r e a t m e n t a c c o r d ­ ing to h i s / h e r level o f study. 2.6 Keep a d e q u a t e d o c u m e n t a t i o n o f th e tr ea tm e nt s. 2.7 R e c o g n i s e c o m p l i c a t i o n s a s s o c i a t e d w it h th e p a t i e n t ’s c o n d it io n and avo id d a n g e r o u s practice. R ep ro du ce d by S ab in et G at ew ay u nd er li ce nc e gr an te d by th e P ub lis he r (d at ed 2 01 3. )