JIIMC June 2016.cdr ORIGINAL ARTICLE ABSTRACT Objectives: ! To investigate the assessment tools and processes used to assess students of MHPE programs in Pakistan for achieving the learning outcomes of the program. ! To Compare this process in developed (United Kingdom) and developing country (Pakistan). Study Design: It is a qualitative archival study designed to provide an insight into the type and level of assessment tools used in MHPE programs. st th Place and Duration of Study: The study was conducted in IIMC from 1 October 2013 to 30 July 2014. Materials and Methods: The study method chosen was archival research. The data was collected from the official websites of the program. Purposive and convenient sampling method was used to select 08 programs, 04 each from United Kingdom and Pakistan. As programs in Pakistan offer only blended programs, hence only blended programs were included in the study. Manifest conventional content analysis of the data was done using NVIVO 10. Results: Common assessment tools used to assess competence were assignments and dissertation submission in all the programs. Programs in UK also used portfolios whereas in comparison only summative examination using MCQ, SEQ's and OSTE was employed in Pakistan. All programs in the study assessed student at 'shows how' level except 02 programs in UK, which assessed Meta competency (Does level). Conclusion: Distant learning blended programs in HPE assess competency at different levels of competence. This results in variability of level of assessment and hence affects the eventual outcome. A uniform method of assessment should exist for master programs in health professions education to ensure uniform learning and outcome. Key Words: Assessment Tools, Distant Learning Programs, Health Professions Education, level of Assessment. Assessing Competency in Distant Learning Master Programs in Medical Education: A Qualitative Analysis 1 2 3 Rehan Ahmed Khan , Omer Awwab Khan , Madiha Sajjad JIIMC 2016 Vol. 11, No.2 Correspondence: Dr. Omer Awwab Khan Associate Professor, Medicine Islamic International Medical College Riphah International University, Islamabad E-Mail: omer.awab@riphah.edu.pk 1 2 3 Department of Surgery/ Medicine / Pathology Islamic International Medical College Riphah International University, Islamabad Funding Source: NIL ; Conflict of Interest: NIL Received: Feb 25, 2016; Revised: Apr 15, 2016; Accepted: May 06, 2016 satisfying both national and international needs of 5 medical education. As we know that 'Assessment 6,7 drives learning' , implementation of the learning outcomes of the M H P E programs have direct relationship to the type and level of assessment that is being employed in these programs. The main objectives of this study were: ! To investigate the assessment tools and processes used to assess students of MHPE programs in Pakistan for achieving the learning outcomes of the program. ! Compare this process in developed (United Kingdom) and developing countries (Pakistan). Materials and Methods It is a Qualitative archival study in which archival method of research has been used. In archival research, the researcher collects data stored in archives. The archives can be in the forms of newspapers, articles of journals, books or websites. The reason for choosing archival research was its simplicity, ease of gathering data and easy Introduction MHPE programs aim to produce effective leaders in 1 - 4 medical education. These programs are an emerging field and mushrooming at a fast pace. The reason for this surge is understandable. However w h a t n e e d s t o b e a s c e r t a i n e d i s t h e p r o f e s s i o n a l i z a t i o n o f t h e s e p r o g r a m s . Professionalization is defined as the process of giving a professional character, identity, or status to 'Health Professions Education' as a profession. It involves establishing a suitable and accreditable qualification, 85 Competency in Distant Learning Programs availability of the archives. More over because of the type of objectives of the study the data required to analyze was present in the archives available via internet and did not require to collect data through other methods as questionnaires, interviews or emails. 8 To collect data, first the website of the program was found through . The next step was www.google.com to locate the information provided about the program. The data about the MHPE program was either present in pdf format or information given on the webpages itself. All the concerned pdf files and webpages were downloaded and opened in NVIVO. Convenience and Purposive sampling technique was 9,10 used. Eight MHPE programs were selected. Out of t h e s e , f o u r p r o g r a m s w e r e f r o m P a k i s t a n (developing country) and four were from the United Kingdom (developed country). Eight programs were included in the study as they yielded enough data to ensure validity and reliability in a qualitative study. It was also taken into consideration that data from 08 programs had to be transcribed and coded into themes. In qualitative case studies, handling this amount of data requires considerable time and work. The programs were selected each from well- established and newly established institutes running these programs. Results All programs in UK and Pakistan used assignments at the end of a module or a semester. All programs included in the study from UK required dissertation for the award of master's program except one university. Two universities from UK used portfolio as their tool of assessment. All programs in Pakistan used written assignments and dissertation for assessment purposes. However, two universities also had a summative exit assessment. The detailed results are reported below: Discussion The global distribution of MHPE programs is not uniform. The reported number of MHPE programs was 121 Up to 2013.The maximum number of programs are offered in USA (32) followed by UK 11 (31). The number of programs in Pakistan was only 12 two until 2012 however it has increased to 6 now. All these HPE programs share many features in common but they have a lot of differences as well. Table I: Dura�on and mode of delivery of the content (UK= United Kingdom, we=Well Established, ne= newly established, PK= Pakistan) Table II: Type and dura�on of contact session Table III: Exits in the program 86 JIIMC 2016 Vol. 11, No.2 Competency in Distant Learning Programs different formats including contacts sessions using web boards, Skype, Adobe connect, and physical face 1-4, 13-15 to face contact sessions. In UK, the MHPE programs offered by Universities are either on campus, blended or complete distance learning. In Pakistan the programs are only blended, which includes an average of 6-8 weeks of contact session, which offers very little flexibility to the students. In authors observation, one of the main reason for the difference in flexibility of these programs is that complete distance learning p r o g r a m s a r e s t i l l n o t r e c o g n i z e d b y t h e accreditation body in Pakistan. Learning outcomes in masters in Health professions education are assessed by variety of assessment tools. Most of the programs use both formative and summative assessment methods. The tools used are pen and paper methods; OSCE, OSTE, Short Essay Questions (SEQ's), Multiple choice questions (MCQ's), portfolios, assignments 11,12 and oral defense of thesis. These tools assess 'shows how', to assessment strategies that evaluate 'Does' of the Miller's Pyramid. All programs have aligned learning outcomes and learning strategies with assessment tools to achieve the desired outcomes. Programs in United Kingdom are more flexible as they offer exits at different levels, optional and specialized modules to study and different modes (complete distant learning, blended and on campus) of program to choose from. They also aim to produce capable medical educationists (at the 'does' level), who can perform in real life situations. Programs in Pakistan are in their infancy, however their learning outcomes, assessment and learning strategies share a lot of features in common with programs in UK. The main contrasting feature though is minimal flexibility and assessment at the 'shows how' level. Conclusion Distant learning blended programs in HPE assess competency at different levels of competence. This results in variability of level of assessment and hence affects the eventual outcome. Guidelines and sta n d a rd s f ro m Re g u l ato r y a n d A c c re d i t i n g authorities should stress on uniform method of a s s e s s m e nt fo r m a ste r p ro g ra m s i n H e a l t h Professionals Education to ensure uniform learning and outcome throughout world. Table IV: Details of Assessment tools used Table V: Classifica�on of assessment tools according to Miller's Pyramid The flexibility of a program is evident from the optional modules it offers to its students. The optional modules are only offered in UK programs which give the students freedom to choose from topics of their interest. Programs have options for surgeons, anesthetists and gynecologists. A lot of emphasis in these programs is on the impact of medical education in clinical training. This is lacking in Pakistani programs where only one program offers options of electives. Flexibility is further enhanced in UK programs by offering exit at multiple levels. Duration of these program is variable extending from 1 to 5 years. The mode of delivery varies from complete distant learning (DL) to distant learning with contact sessions. Contact sessions are offered in 87 JIIMC 2016 Vol. 11, No.2 Competency in Distant Learning Programs Blackwell companion to organizations. 2002: 805-28. 9. Dawson C. Practical research methods: a user-friendly guide to mastering research techniques and projects. United Kingdom: How to Books Ltd. 2002; P. 48-9. 10. Fraenkel J, Wallen RN, E. How to design and Evaluate Research in Education. New York: McGraw-Hill; 2010; P. 97- 8. 11. Tekian A, Artino Jr AR. AM Last Page: Master's Degree in Health Professions Education Programs. Academic Medicine. 2013; 88: 1399. 12. Tekian A, Harris I. Preparing health professions education leaders worldwide: A description of masters-level programs. Med Teach. 2012; 34: 52-8. 13. T Ekian A. Doctoral programs in health professions education. Medical teacher. 2014; 36: 73-81. 14. Pugsley L, Brigley S, Allery L, Macdonald J. Counting quality because quality counts: differing standards in master's in medical education programmes. Med Teach. 2008; 30: 80- 5. 15. Pugsley L, Brigley S, Allery L, MacDonald J. Making a difference: researching master's and doctoral research programs in medical education. Med Educ. 2008; 42: 157- 63. REFERENCES 1. Tekian A, Dwyer M. Lessons for the Future: Comparison and Contrasts of the Master of Health Professions Education Programs Offered in China and Egypt. Teaching and Learning in Medicine. 1998; 10: 190-5. 2. Tekian A, Harris I. Preparing health professions education leaders worldwide: A description of masters-level programs. Medical teacher. 2012; 34: 52-8. 3. Tekian A, Roberts T, Batty HP, Cook DA, Norcini J. Preparing leaders in health professions education. Med Teach. 2014; 36: 269-71. 4. Al-Subait R, Elzubeir M. Evaluating a masters of medical education program: attaining minimum quality standards? Med Teach. 2012; 34: 67-74. 5. Tekian A. Professionalization in Health Professions Education. Accessed on 14-03-2014fromhttp://www.ae- me.org/PlenaryPresentation-Abstruct-AaraTekian.php. 2014. 6. McLachlan JC. The relationship between assessment and learning. Medical education. 2006; 40: 716. 7. Wood T. Assessment not only drives learning, it may also help learning. Medical Education. 2009; 43: 5-6. 8. Ventresca MJ, Mohr JW. Archival research methods. The 88 JIIMC 2016 Vol. 11, No.2 Competency in Distant Learning Programs