SQU Med J, December 2009, Vol. 9, Iss. 3, pp. 219-223 , Epub. 19th Dec 2009 Submitted - 26th October 09 Accepted - 8th November 09 In the august 2009 issue of SQUMJ 2009, Professor C.B. Hazlett wrote about the “Prerequisite for Enhancing Student Learning Outcomes in Medical Education”.1 Basically, he discussed the three requirements for good outcomes in medical education. These include Active versus Passive learning, and how active learning material needs to be constructed, contextualised and cooperatively acquired. He then went on to discuss the importance of the two types of assessment: summative and formative. Summative assessments determine whether the individual should pass or be promoted, while Formative assessments promote lifelong learning if accompanied by appropriate feedback. Feedback is of paramount importance for this to happen, a point which Professor Hazlett underlined. Professor Hazlett then pointed out that the third requirement is the Alignment of curriculum, course objectives, teaching and assessments: they must complement each other. Kudos goes to the College of Medicine and Health Sciences (COM&HS) at Sultan Qaboos University (SQU) for producing a set of “Assessment Policy & Guidelines 2009”2 soon after the implementation of the new curriculum for the Medical School. On the postgraduate front, congratulations are due to the Oman Medical Specialty Board (OMSB) which is currently in the midst of changing its teaching to competency-based curricula and creating detailed guidelines for resident assessment. To stress the importance of residents’ training and assessment, the Royal College of Physicians & Surgeons of Canada recently devoted four days to resident education, at an international conference “Innovations in Residency Education”, (ICRE) held in Victoria, British Columbia, 23-26 September 2009. It is thus appropriate that the COM&HS of SQU and the OMSB both are now directing their energies towards ensuring the alignment of the curriculum, course objectives, training and assessment. This will help ensure that we produce quality assessment tools. Only such efforts can assure the Triple A doctors that we all would like to produce.1 We have to embrace the alignment of these four elements. The two levels of doctors’ training (undergraduate and postgraduate) have to work in concert to achieve that goal, and they have to involve the stakeholders such as the Ministry of Health, the health centres, the teaching and community hospitals, and the public in general. Together, they need to decide what kind of doctors we need, thus establishing what the country needs. This, in turn, will dictate the type of Core Competencies that need to be established for future Omani graduating doctors. In this era of quality assurance and accompanying accountability, it is imperative that we change the emphasis from structure and process to outcome- based education. We then have to develop or utilise 1Editor-in-Chief, SQU Medical Journal, College of Medicine & Health Sciences, Sultan Qaboos University, Muscat, Sultanate of Oman; 2Department of Radiology & Molecular Imaging, College of Medicine & Health Sciences, Sultan Qaboos University, Muscat, Sultanate of Oman, and Oman Medical Specialty Board, Muscat, Sultanate of Oman *To whom correspondence should be addressed. Email: mjournal@squ.edu.om أدوات التقييم للدراسة الطبية التخصصية هل حنتاج إىل أدوات “صنع يف ُعمان”؟ ملك اللمكي، نيلة اللمكي editorial Evaluation Tools in Postgraduate Medical Education Do we need “Made in Oman” tools? *Lamk Al-Lamki1 and Al-Neela Lamki2 Evaluation Tools in Postgraduate Medical Education Do we need “Made in Oman” tools? 220 | SQU Medical Journal, December 2009, Volume 9, Issue 3 the existing assessment tools necessary to ensure that tomorrow’s young doctors achieve the required levels of core competencies. The examples of assessment tools recently cited in the “Assessment Policy, Regulations and Guidelines” of the COM&HS at SQU2 are mainly Summative tools. While these constitute an excellent beginning, they need to be supplemented by some more Formative assessment tools, e.g. those that can more comprehensively assess the 7 Domains [Figure 1] and their accompanying core competencies suggested by the Institute for International Medical Education (IIME) as the “Global Minimum Essential Requirements in Medical Education”.3,4 These are the expected competencies amongst medical school graduates, but unfortunately few medical schools have assessment tools specific for these 7 Domains. Using Summative assessments alone is not adequate, especially for the more senior clinical students, as they alone cannot ensure fulfillment of the teaching mission of the medical school. To supplement medical school training and to achieve an expert doctor after postgraduate training, we need clearly defined core competencies and appropriate assessments to go with them. Such an undertaking in the postgraduate arena belongs to OMSB, which must step in to define the essential core competencies and the assessment tools they want trainers to employ. OMSB is currently discussing whether to have “made in Oman” core competencies and accompanying assessment tools or adopt what is available elsewhere, such as those from the Royal College of Physicians and Surgeons of Canada or those from the Accreditation Council for Graduate Medical Education (ACGME) of USA. The Canadian Royal College “7 core competencies” were established in 1996 as CanMEDS (Canadian Medical Education Directives for Specialists)5 and then modified in 2005 [Figure 2]. CanMEDS are well established and have been adopted by many countries including Canada, Australia and Denmark and some Far Eastern medical schools. ACGME modified the CanMEDS and came up with “6 core competencies” in 1999.6 The two are basically fairly similar [Table 1]. It seems that the American ACGME 6 competencies are not unlike the 7 Canadian ones, but the ACGME has dropped the Scholar, perhaps believing that it is incorporated in Medical Knowledge (Medical Expert), and the names of the competencies are different. Generally speaking, OMSB postgraduate medical training is more aligned to Canadian than American training largely because most new trainers have received their postgraduate training in Canada. It will not be easy for Oman to develop its own core competencies as it took the Canadians four years (1993 to 1996) to develop CanMEDS. It thus stands to reason that we adopt the CanMEDS like many other countries have done.7 Traditionally, “competency in a physician” Critical �inking Clinical Skills Scientific Foundations Population Health Professional Values, Attitudes Information Management Communication Skills Figure 1: The 7 Domains of the International Institute for Medical Education to be achieved by Medical School Graduates Figure 2: 7 Core Competencies of the Canadian Royal College to be achieved by the end of Post-graduate Residency training Lamk Al-Lamki and Neela Al-Lamki editorial | 221 revolved around being a medical expert; however, the patient-centered care that a physician with CanMEDS core competencies can provide will better meet the needs of Omani patients and society. If we can assure ourselves that our residents are not only good Medical Experts, but are also competent Communicators, Managers, Health Advocates, Scholars and are Professional then we are winners. Thus it is logical that Oman adopts these 7 core competencies from Canada. To ensure that our residents acquire the 7 competencies of CanMEDS, we need to have good set of assessment tools. The Canadians have been developing guidelines on how to assess residents and came up with “Preferred Tools” for each of the CanMED core competencies.8 However, this is an evolving process, the tools will continue to evolve and become more specific for each competency. The different universities that train residents in Canada are still struggling to come up with more definitive tools.9,10 The ACGME got together with American Board of Medical Specialists (ABMS) and produced a “Toolbox of Assessment Methods”11 which is also still evolving. They have so far come up with the following tools: 1. 360-degree evaluation (by faculty/supervisors peers, subordinates, patients families, nurses, allied health professionals, other residents) 2. Chart simulated recall oral examination (30 minutes by each of two examiners) 3. Checklist evaluation (for specific competency components and behaviours) 4. Global rating of live or recorded performance (e.g. patient care skills, communication) 5. Objective structured clinical examination (OSCE): standardised patient encounter stations and data interpretation 6. Procedure/case logs (documentation of procedures performed) 7. Patient surveys (include questions about physician competency and empathy) 8. Portfolios (case logs with learning evidence and resident reflection) 9. Record/chart review (review of patients’ medical records by the resident) 10. Simulations and models (wide array of options resembling reality) 11. Standardised patient examination (examiner uses checklist or rating form during observation) 12. Written examinations: multiple choice questions (MCQs), modified essay questions (MEQs) or short essay examination In some form or another, most of these 12 tools are used universally and the Canadians have supplemented them with some more specific assessment tools, such as, the “In Training Evaluation Report” (ITER) done during and at the end of the monthly rotation. They have also added the CEx (Clinical Evaluation Exercise) and MiniCEx (Mini Clinical Evaluation Exercise) which perhaps can be regarded as a complement to the Chart Stimulated Oral examination. In the MiniCEx, the faculty member observes and rates a trainee performing a focused clinical task (e.g. history taking or physical exam). Perhaps now is a good time for Oman to join the effort to create the ideal assessment tools by supplementing the currently accepted tools Table 1: Comparison of Canadian Medical Education Directives for Specialists (CanMEDS) and (ACGME) Accreditation Council for Graduate Medical Education Core Competencies (adapted from Ref. 5) CanMEDS ACGME Medical Expert Medical knowledge Communicator Communication and interpersonal skills Collaborator Patient care Manager System-based practice Health Advocate Practice-based learning and improvement Scholar Professionalism Professionalism Evaluation Tools in Postgraduate Medical Education Do we need “Made in Oman” tools? 222 | SQU Medical Journal, December 2009, Volume 9, Issue 3 described above. In the process, we can tailor them for the assessment of young Omani doctors and so help in moulding future medical specialty care in Oman. All the above assessment tools need criteria to indicate their reliability, reproducibility, validity and educational benefit. Also, how labour intense will it be to implement any of these tools? Also, do they give the residents an opportunity for feedback?12 This has to be an essential companion to all assessment tools. The next question is when to employ which tools, and at what stage of residency training? To approach this question more appropriately, one has to refer to Miller’s pyramid for “The assessment of clinical skills, competence and performance”[Figure 3].13 The lowest level “Knows” refers to the basic knowledge needed by undergraduates and junior residents. Hence, the assessment tools will be those that test knowledge, such as written examinations, MCQs, etc. This is followed by “Knows how” which refers to competence; here the assessment tools have to test the competence of the skills and performance expected of mid-level residents. Such tools include standardised patients, simulated direct observation and oral examinations. “Shows How” refers to performance and is associated with senior residents. These assessment tools have to assess high-level competence and performance; tools such as the OSCE, log book, and the Australian “Structured Assessment using Multiple Patient Scenarios” (STAMPS)14 are all used at this level. “Does” is at the highest level in post-graduate education. In clinical skills, it involves testing performance or Action. It is applicable to the learned skills of senior residents and fellows and has also been applied to physicians in practice. The possible assessment tools at this level of Miller’s pyramid are multiple, yet difficult to authenticate, and several parties are working on this.15 Assessment tools that are commonly used include direct observation, MiniCEx, Multi- source Feedback (MSF) and 360 degree evaluation. Standardised patients, computer-based simulation, model-driven simulation and virtual reality/haptic devices can all be used to assess “Does” and also some aspects of “Shows How”. All these tools are available to us. What Oman needs to do now is to select which of the above tools to adopt. Overall, we suggest that Omani postgraduate medical educators should adopt the assessment tools of CanMEDS core competencies for now, but, if necessary, these can be modified and further developed in the future to suit Oman’s needs. For now, however, postgraduate medical education in Oman has to adopt formally some core competencies before we can decide on specific tools. This will not only help us clarify what competencies we want our young Omani doctors to develop, but it will also encourage us to develop assessment tools that are truly relevant to us for the adopted competencies. Either way, it is imperative that we DOES (Action) SHOWS HOW (Performance) KNOWS HOW (Competence) KNOWS (Basic Knowledge) Direct Observation MSF Standardised Patients OSCE Oral Examination Simulations Written Examinations MCQs Legend: MSF= Multi-source feedback; OSCE= Objective structured clinical examination; MCQs= Multiple choice questions Figure 3: Examples of assessment tools to be used for various levels of Miller’s pyramid Lamk Al-Lamki and Neela Al-Lamki editorial | 223 give guidance to our trainers and medical educators on the competencies that we are targeting for our young doctors and on the assessment tools to be used. As indicated, adopting the CanMEDS core competencies will almost inevitably mean automatic adoption of all or some of the accepted and attested assessment tools. However, let us look into what it is like to create assessment tools for our specific needs i.e. “Made in Oman Tools”. To do so, we have to find specific weaknesses amongst our residents or areas of necessary improvement if we want them to achieve CanMEDS core competencies like the residents in western countries like USA, Canada - countries that have adopted the core competencies and accepted the assessment tools. For example, Omani residents need to enhance markedly their self-directed learning, their communication skills, their motivation to acquire knowledge, and their work ethics and also overcome other possible weaknesses. These four main weaknesses of many Omani residents need to be addressed in training as well as in assessment. The “Made in Oman” assessment tools that will have to be created to improve these weaknesses need to be of a formative type so that the residents can learn through assessment. We need to appreciate that the four examples cited as weaknesses among our residents belong to different core competencies and thus their respective Omani tools of evaluation, when developed, will have to be classified accordingly. For example, self learner belongs to “Scholar” and work ethic belongs to “Professionalism”. Each core competency will need its own assessment tools and these will vary according to the level of the resident as per Miller’s pyramid. It is now a most appropriate time for Oman to assess the postgraduate assessment tools as the SQU medical school has recently implemented its new curriculum and has also developed its assessment tools,2 while OMSB is in the final stages of decision on core competencies and revision of its assessment tools. Acknowledgments Figures 1 and 2 are reproduced with permission. References 1. Hazlett CB. Prerequisite for Enhancing Student Learning Outcomes in Medical Education. SQU Med J 2009; 9:119-23. 2. College of Medicine & Health Sciences, SQU Assessment Policy, Regulations, & Guidelines. First Edition 2009. From http://www.squ.edu.om/ medicine-health/tabid/1638/default.aspx Accessed Oct. 2009. 3. Global Minimum Essential Requirements in Medical Education. From http://www.iime.org/documents/ gmer.htm Accessed Oct. 2009. 4. The assessment of global minimum essential requirements in medical education. From http:// w w w. i n f o r m a w o r l d . c o m / s m p p / 2 1 3 1 0 3 3 6 9 - 74340499/content~db=all~content=a713626660 Accessed Oct. 2009. 5. CanMEDS. From http://rcpsc.medical.org/canmeds/ index.php Accessed Oct. 2009. 6. ACGME Outcome Project. From http://www.acgme. org/outcome/comp/compMin.asp Accessed Oct. 2009. 7. Are some of the challenging aspects of the CanMEDS roles valid outside Canada? From http:// www3.interscience.wiley.com/journal/118729511/ abstract?CRETRY=1&SRETRY=0 Accessed Oct. 2009. 8. The CanMEDS Assessment Tools Handbook. From http://www.rcpsc.edu/canmeds/resources/ handbook_e.php Accessed Oct. 2009 9. Mickelson JJ, MacNeily AE. Tools for implementing the CanMEDS competencies in Canadian urology residency training From http://www.ncbi.nlm.nih. gov:80/pmc/articles/PMC2532541/ Accessed Oct. 2009. 10. University of Toronto, Faculty of Medicine, Post Graduate Medical Education. From http://www. p g m e . u t o r o nt o . c a / R e s o u r ce s / C a n M E D S . ht m Accessed Oct. 2009. 11. ACGME Outcome Project - Toolbox of Assessment Methods. From http://www.acgme.org/Outcome/ assess/Toolbox.pdf. Accessed Oct. 2009. 12. McAleer S. Choosing Assessment Instruments. In: Dent JA, Harden RM, Eds. A Practical Guide for Medical Teachers. New York: Churchill Livingstone, Elsevier, 2009. Section 6. 13. Miller GE. The Assessment of Clinical Skills/ Competence/performance. Acad Med 1990; 65: s63-7. 14. Rural and Remote Health Australia. From http:// www.rrh.org.au/publishedarticles/article_print_805. pdf Accessed Oct. 2009. 15. Association for Medical Education in Europe (AMEE). From http://www.amee.org/index. asp?tm=22 Accessed Oct. 2009.