Editorial Setting standards in health sciences education – a wake-up call Scarpa Schoeman E-mail: schoemanfhs@ufs.ac.za Standard setting is the process of determining what the minimum require- ments are to be deemed knowledgeable or competent to practise.1 As the second decade of the 21st century progresses, we as health sciences edu- cators in Africa need to ask ourselves whether we are keeping pace with current international practice. Determining the cut score or pass mark of an examination has rightfully been coined the ‘holy grail’ in assessment,2 and it is therefore not surprising that many methods have been described over the past 50 years.3-5 Despite the wealth of literature on determining cut scores the ‘most widely used one (i.e. the holistic fixed percentage pass mark) is the least defensible!’2 A holistic fixed percentage pass mark is commonly still used by African institutions for both undergraduate and postgraduate as- sessments. By convention it is set at 50% in South Africa, but ranges vary from 50% to 60% at some European and North American Universities.6 Why then are the vast majority of medical schools and health sciences faculties in Africa still employing a holistic pass mark of 50%? Where does this number come from? Why is it not 52% or 55% or 45%? What is the scientific rationale for using a score of 50%? Unfortunately, the an- swer to all these questions is the same: We don’t know! Thankfully, this is not an African problem alone – the holistic fixed percentage method is still widely used in many universities around the globe. We have been using 50% (or whichever fixed percentage) since memory began. As one author put it, ‘It was pulled from the air’.3 So, why is this tried and trusted method inherently flawed? There are two key reasons: (i) we are unable to explain the rationale for using an empirically derived fixed score to our stakeholders and (ii) the method does not take into account test difficulty. Cut scores have profound ef- fects on stakeholders including the student, the university, health profes- sions registration bodies, the Department of Health and most importantly the patients we serve. Setting the pass mark is clearly entrenched in this process of maintaining and setting standards for graduating health pro- fessionals. Our aim must surely be to strive to eliminate false positives (passing the incompetent) and false negatives (failing the competent)1,3 from our assessment systems. To be able to do this, we need to factor in the issue of test difficulty, among other things. It is logical that no two tests or ex- aminations have the same difficulty. Small, minor variances in the level of difficulty are acceptable between papers, but we have all experienced that ‘horribly difficult’ or ‘ridiculously easy’ paper. Reasons why major variances in test difficulty occur are plentiful and commonly occur due to a lack of moderation and repeated use of old questions. Therefore, any standard setting process used must be able to absorb these variances and adapt the pass mark accordingly.1,7 As previously mentioned there is no gold standard for determining the cut score in all circumstances.1-3,7 How- ever, there is always an appropriate method for your institution, consider- ing the intricate details of your situation and resources.2 In 2010, a new and exciting method for determining the pass mark for written papers was described by Cohen-Schotanus and Van der Vleu- ten from the Netherlands.6 They called it the ‘Cohen method’ and it offers a fresh perspective and approach to standard setting for written papers, in particular. In this method, the top performing students are used as a point of reference to set the pass mark. Essentially the performance of the top 95th percentile of the test scores is used as the benchmark and the pass mark is set as 60 - 70% of the 95th percentile. This sets the pass mark as a function of the performance of the top candidates who offer a real reflection of the difficulty of the examination. The 95th percentile is used because this top cohort of students is usually stable and performs equally well between different year groups compared with the mean test score which is usually dragged down by poorly performing students. This method, as well as the Borderline Regression method for OSCEs, will be the topic at the author’s workshop on standard setting at the 2011 SAAHE conference in Potchefstroom. In conclusion, appropriate and transparent standard setting is a criti- cal element of good assessment and educational practice.2,8 We as health sciences educators need to make this issue a top priority and move to- wards implementing explicable, defensible and stable standard setting methods2 to the benefit all stakeholders in health sciences education. Conflict of interests: None Scarpa Schoeman Guest Editor 1. Cusimano MD. Standard setting in medical education. Acad Med 1996;71(10 Suppl):S112-120. 2. Schuwirth LWT, Van der Vleuten CPM. How to design a useful test: Principles of assessment. In: Swanwick T, editor. Understanding Medical Education: Evidence, Theory, Practice. 1st ed. Wiley-Blackwell, 2010:204-205. 3. Barman A. Standard setting in student assessment: is a defensible method yet to come? Ann Acad Med Singapore 2008;37(11):957-963. 4. Downing SM, Tekian A, Yudkowsky R. Procedures for establishing defensible absolute passing scores on performance examinations in health professions educa- tion. Teaching and Learning in Medicine 2006;18:50-57. 5. Norcini JJ. Setting standards for educational tests. Med Educ 2003;37:464-469. 6. Cohen-Schotanus J, van der Vleuten CP. A standard setting method with the best performing students as point of reference: practical and affordable. Med Teach 2010;32(2):154-160. 7. Bandaranayake RC. Setting and maintaining standards in multiple choice exami- nations: AMEE Guide No. 37. Med Teach 2008;30(9-10):836-845. 8. Norcini J, Anderson B, Bollela V, et al. Criteria for good assessment: consensus statement and recommendations from the Ottawa 2010 Conference. Med Teach 2011;33(3):206-214. 2 June 2011, Vol. 3, No. 1 AJHPE Scarpa Schoeman Guest Editor