PhiliPPine Journal of otolaryngology-head and neck Surgery Vol. 36 no. 1 January – June 2021 PhiliPPine Journal of otolaryngology-head and neck Surgery Vol. 36 no. 1 January – June 2021 PhiliPPine Journal of otolaryngology-head and neck Surgery 1514 PhiliPPine Journal of otolaryngology-head and neck Surgery ORIGINAL ARTICLES ABSTRACT Objective: To determine the self-reported assessment of initial implementation of the 3 domains of Outcome-Based Education in accredited Otolaryngology - Head and Neck Surgery residency training programs in the Philippines by consultants and residents and explore any associations between their demographic profiles and assessments. Methods: Design: Mixed Method Research Design Setting: Multicenter - 30 accredited ORL-HNS residency training institutions in the Philippines- National Capital Region (NCR) 19, Luzon 7, Visayas 2, and Mindanao 2. Participants: A total of 129 consultants and 82 second to fourth year residents in training were included in the study by convenience sampling. First-year residents who started their residency training in January 2020 were excluded. Respondents answered self-reported questionnaires to assess implementation of the 3 domains of OBE: intended learning outcomes (ILO), teaching and learning activities (TLA) and assessment tasks (AT) using the 4-point scale score from “fully implemented” (4) to “not implemented” (1). Results of questionnaires were confirmed using open-ended questions on the challenges of OBE with a focused group discussion among 4 consultants and 1 resident. Results: The self-reported assessment of respondents on OBE implementation was “fully implemented” in the 3 domains. However, low numerical scores were seen for “managing community health and social need” in the ILO and “laboratory activities and workshops” in the TLA for both consultants and residents, in the assessment task (AT “multisource feedback by nurses and administrative staff ” for the consultants, and “direct observation of performance skills for patient encounter” for residents. Among the 7 modules, “research methodology” had the lowest score for both consultants’ and residents’ self-perception. Challenges of OBE revealed included “mastery,” “time” and “data keeping.” Consultants younger than 60 years of age who had been in the department longer than 3 years and residents who attended an OBE workshop / lecture tended to give higher scores. Conclusion: Two years after distribution of the manual on OBE to ORL-HNS residency training institutions, the consultants’ and residents’ self-reported assessment on implementation in all the 3 domains of OBE was “fully implemented.” Keywords: outcome-based education; outcome process assessment; health care; otolaryngology; health plan implementation; formative feedback; internship and residency; teaching; medical graduate education Self-Reported Assessment of Outcome-Based Education in Philippine Otolaryngology - Head and Neck Surgery Residency Training Programs by Consultants and Residents Maria Natividad A. Almazan, MD, MSc, PhD School of Graduate Studies Manila Central University Correspondence: Dr. Maria Natividad A. Almazan School of Graduate Studies Manila Central University Research and Development Office Filemon D. Tanchoco Medical Foundation EDSA, Caloocan City 1400 Philippines Phone: (632) 8367-2031 local 1226 Email: natividadarceoalmazan@gmail.com The author declared that this represents original material that is not being considered for publication or has not been published or accepted for publication elsewhere in full or in part, in print or electronic media; that the requirements for authorship have been met by the author, and that the author believes that the manuscript represents honest work. Disclosures: The author signed a disclosure that there are no financial or other (including personal) relationships, intellectual passion, political or religious beliefs, and institutional affiliations that might lead to a conflict of interest. Funding: No funding support was received for this study. This paper is condensed from the Dissertation submitted to the Faculty of the School of Graduate Studies of the Manila Central University - Filemon D. Tanchoco Medical Foundation in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy in Education last July 30, 2020. Philipp J Otolaryngol Head Neck Surg 2021; 36 (1): 15-23 c Philippine Society of Otolaryngology – Head and Neck Surgery, Inc. Creative Commons (CC BY-NC-ND 4.0) Attribution - NonCommercial - NoDerivatives 4.0 International PhiliPPine Journal of otolaryngology-head and neck Surgery Vol. 36 no. 1 January – June 2021 PhiliPPine Journal of otolaryngology-head and neck Surgery Vol. 36 no. 1 January – June 2021 PhiliPPine Journal of otolaryngology-head and neck Surgery 1716 PhiliPPine Journal of otolaryngology-head and neck Surgery ORIGINAL ARTICLES Outcome-Based Education (OBE) emphasizes the output and defines what the learner is answerable for any teaching and learning program.1 OBE is precisely designed towards the global call for transformative scale up of health professions education and to respond to society’s needs.2 In the United States, the Accreditation Council for Graduate Medical Education (ACGME) mandated new structural changes in residency education with its newly created core competencies and an emphasis on outcomes-based education.3 Canadian higher education institutions started to show a significant shift towards OBE in order to stay globally competitive.4 The Association of Southeast Asian Nations (ASEAN) Joint Coordinating Committee on Medical Practitioners (JCCMP) decided on regional economic community integration in 2014 with outcome based training in residency training. Specifically, it means transforming the member-countries into a single market production base, highly competitive economic region with equitable economic development that is fully integrated into the global economy. The Enhanced Basic Education Act of 2013, the Philippine Qualifications Framework, the Commission on Higher Education Memorandum Order Number 46 and the global call for transformative education serve as precursors of Outcome Based Education (OBE) in the Philippines.2 In preparation for ASEAN integration, the Professional Regulation Commission Board of Medicine (PRC BOM) communicated with the different specialty board societies in 2015. The Philippine College of Surgeons (PCS) sent the bases for the specialty curriculum formed by the Joint Coordinating Council on Accreditation and Certification (JCCAC) to the Philippine Society of Otolaryngology - Head and Neck Surgery (PSO-HNS) in January 2016. An ad hoc Committee on Outcome Based Education was formed by the Philippine Board of Otolaryngology - Head and Neck Surgery (PBO- HNS) and the Philippine Society of Otolaryngology - Head and Neck Surgery (PSO-HNS) together with the different heads of the subspecialty groups convened to come up with the ORL-HNS OBE manual.5 This was sent to the Professional Regulation Commission Board of Medicine (PRC BOM) in 2016 and distributed to all 34 ORL residency training institutions in 2017 to standardize the OBE program and focus on its aim to graduate locally and internationally competent and competitive general otolaryngologists. More than 2 years after the distribution of the OBE manual, there is a need to assess perception of the consultants and residents on initial implementation of the 3 domains of OBE in the ORL-HNS residency training institutions. These domains are the intended learning outcomes (ILO), teaching and learning activities (TLA) and assessment tasks (AT). The knowledge on its implementation will benefit the stakeholders: the administrators of the accredited ORL-HNS training departments, their OBE coordinators, consultants, residents, ad hoc Committee on OBE, the PSO-HNS, and the patients. This is the first study on implementation of OBE program in ORLHNS that was introduced only in the middle of 2017 and results of the paper will be used to create guidelines to monitor progress of OBE program. The objective of this study is to determine the self-reported assessment of initial implementation of the 3 domains of Outcome Based Education in accredited Otolaryngology -Head and Neck Surgery residency training programs in the Philippines by consultants and residents and explore any associations between their demographic profiles and assessments. METHODS This is a multicenter study that sought to involve the 34 accredited residency-training programs in ORL-HNS all over the Philippines from January to June 2020. The eligible respondents were the consultants and second to fourth year residents in training who experienced the OBE program prepared by the PBO-HNS and the PSO-HNS in 2017. First year residents were excluded because they only started residency training in 2020. The study focused on the consultants’ and residents’ self-reported assessment on implementation and the challenges of OBE in the residency training of ORL-HNS. Institutional Review Board (IRB) /Ethics Committee approval was granted at the following institutions: Manila Central University FDTMF (2020-03), East Avenue Medical Center (2020-06) and Veterans Memorial Medical Center (2020-08). Finally, since this was an academic research paper, it received a certificate of exemption from the Single Joint Research Ethics Board (SJREB-2020-12) of the Department of Health. Informed consent with data privacy was obtained from the respondents. The study used a mixed method research design. The first part was quantitative (descriptive, correlational, comparative); the second part was qualitative. The descriptive process depicted consultants’ and residents’ profiles and their self-reported assessments on implementation of the 3 domains of OBE: intended learning outcome (ILO), teaching and learning activities (TLA), and assessment tasks (AT). Correlational analysis was used to determine associations between the profiles of the consultants and residents with their self-reported assessment on implementation. Comparisons were used to determine differences between the consultants’ and residents’ profiles and their assessment on implementation of OBE in ORL-HNS residency training. The qualitative process utilized an interview among selected consultants and residents to confirm the quantitative data. The study was carried out as a qualitative work based on a descriptive phenomenological approach. Purposive sampling was utilized for the recruitment of participants selected from several accredited hospital institutions. PhiliPPine Journal of otolaryngology-head and neck Surgery Vol. 36 no. 1 January – June 2021 PhiliPPine Journal of otolaryngology-head and neck Surgery Vol. 36 no. 1 January – June 2021 PhiliPPine Journal of otolaryngology-head and neck Surgery 1716 PhiliPPine Journal of otolaryngology-head and neck Surgery ORIGINAL ARTICLES Open-ended questions on the challenges of OBE were asked during a one-on-one interview via Viber platform. The investigator wrote down the answers and data collected were grouped based on pre-planned coding. Data analysis used the saturation method coming up with the themes and subthemes. For the quantitative part of the study, there were 2 sets of researcher- made questionnaires, one for consultants and one for residents in training. The first part obtained demographic data of the respondents and the second part focused on implementation of the 3 domains of OBE taken from the manual of the OBE in ORL-HNS in 2017. The questionnaires were constructed in consonance with the constructive alignment of the 3 domains proposed by Dr. William Spady, father of OBE:6 (Figure 1) First, the focal point of the learning process are the intended learning outcomes (ILO). All teaching and learning activities (TLA) and assessment tasks (AT) must be aligned with the learning outcomes. This supports the learners in their progress (formative assessment) and validates the achievement of the intended learning outcomes at the end of the process (summative assessment).7 The first domain of intended learning outcomes (ILO) included 12 core competencies recommended by the Professional Regulation Commission Board of Medicine (PRC BOM) for all residency training programs in the Philippines that were incorporated in the 7 modules (6 subspecialties and research methodology) of ORL-HNS residency training. These were all reflected in the questionnaires. The second domain of teaching and learning activities (TLA) covered 8 activities: case-based discussion conferences; lectures; laboratory activities and workshops; patient encounters at the out-patient department, emergency room, wards and operating room; ward rounds; subspecialty and multidisciplinary meetings; specialty rotations; and instructional courses and workshops. The third domain covered the 12 assessment tasks (AT) such as “direct observation of performance skills for patient encounter” and “multisource feedback form.” These tasks are reflected in 12 formative assessment forms (1 to 12) that are designed to elicit feedback to improve learning throughout the year while a summative assessment is performed on a summary of these forms at the end of the year for each resident, and placed in their learning portfolio. The questionnaires also mentioned the objective of the study, instructions how to answer, and that results would be fed back to respondents and stakeholders for recommendations on creation of guidelines to monitor the OBE program. The questionnaires were validated for face and content validity by 3 experts, a psychometrician and a grammarian. From the pre-validated questions, 41 questions were revised, 7 added and 20 accepted. Pilot testing was done among 16 consultants and residents at a government ORL-HNS Department in Luzon. The computed Cronbach’s alpha for internal and external reliability testing was .969. The self-reported researcher-made questionnaires were answered by the respondents in their respective institutions in hard copies at the weekly and national conferences of the ORL-HNS Departments after the informed consent was obtained, with data privacy assured. Electronic mails using Google forms were used to send the questionnaire to those who were not able to answer in hard copies. All efforts were exhausted to include 34 institutions in the study. The results of the self-reported questionnaires were used to prepare open-ended questionnaires for the focus group discussion (FGD) among a small number of 4 consultants and 1 resident to confirm the quantitative data. The “saturation principle” established the themes on the challenges and problems of OBE implementation. The questionnaire sought the demographic profile of the respondent consultants and residents of the Departments of ORL-HNS of the accredited institutions. For the consultants, these included the following: age, sex, training institution category of hospital, number of years in the department, appointment designation, number of hospital affiliations and attendance in a workshop on OBE. For the residents in training, these included: age, sex, residency year level, training institution category and attendance in an OBE workshop. The questionnaire assessed the implementation of OBE in the 3 domains of education: intended learning outcome (ILO), teaching and learning activities (TLA) and assessment tasks (AT) among consultants and residents as well as the difference between the 2 groups. Subsequent data analysis explored associations between variables in the demographic profiles of consultants and residents, and their assessments on implementation of OBE. Data analysis also determined differences in consultants’ and residents’ assessments on implementation when grouped according to their demographic profiles. Associations and differences were tested using a criterion of p ≤ 0.05 level of significance. Sample size was computed using the proportion of doctors who have attended the OBE workshop (set at 50%) based on a preliminary study. However, since the sample size of 462 participants was too big compared to the source population size of 327 (200 and 127 consultants and residents respectively), a finite population correction (FPC) was applied to the computed sample size. After FPC, a sample size of at least 192 was computed. The computed number of respondents allocated proportionately to consultants and residents was at least 118 consultants and 74 residents respectively to meet the computed sample size of 192 participants. For the validation interviews, the respondents were chosen from both consultants and residents in government and private institutions who consented to answer questions of exact wording and sequence on their OBE experience. Purposive sampling was utilized for the PhiliPPine Journal of otolaryngology-head and neck Surgery Vol. 36 no. 1 January – June 2021 PhiliPPine Journal of otolaryngology-head and neck Surgery Vol. 36 no. 1 January – June 2021 PhiliPPine Journal of otolaryngology-head and neck Surgery 1918 PhiliPPine Journal of otolaryngology-head and neck Surgery ORIGINAL ARTICLES recruitment of 4 consultants and 1 resident. The participants were selected from 4 Metro Manila institutions and one from Visayas after the quantitative data were collected and analyzed. To validate the quantitative results of the study and expound on the challenges of the OBE, open-ended questions were asked during a one on one interview privately between the researcher and the participant via Viber call. The researcher obtained a permission to record the answers in writing. Data collected was collated based on the pre-planned coding on the raw data on the challenges of the 3 domains of OBE, the Intended learning Outcome, Teaching Learning Activities (TLA) and the Assessment Tasks (AT). Data was analyzed using the saturation principle coming up with the themes and subthemes. Data collection and analysis were performed using Microsoft Office Excel 2007 Version 12.0.4518.1014 (Microsoft Corp., Redmond WA USA). Statistical analysis was performed using SPSS version 10.0.5 722680 (IBM Corp., Armonk NY USA) with the following statistical tools: descriptive statistical tool of frequency and percentage for the profile of the respondents, weighted mean for the computation of the assessment on implementation of OBE using a 4-point rating scale, chi- square for the inferential relationship of the profile of the respondents and the assessment of the implementation of the OBE, independent-T test for the inferential difference of the implementation of OBE between the 2 groups of respondents when grouped according to their profile and difference between the 2 groups and lastly, analysis of variance (ANOVA) for the difference between the profile and the assessment of implementation of OBE among the consultants and residents. RESULTS A total of 211 respondents participated in this study, 129 consultants (65%) and 82 residents (65%). They represented 30 out of the 34 accredited ORL-HNS residency training institutions in the Philippines: 19 in the National Capital Region (NCR), seven in Luzon, two in the Visayas and two in Mindanao. Most of the consultants (47; 36.43%) were between 40-49 years of age, followed by 35 (27.13%) who were between 50-59 years of age. There were more males (85; 65.89%) than females (44; 34.11%). Most (74; 57.36%) were in the government sector and 121 (93.8%) had been with their department for more than 3 years. In terms of appointment designation, 80 (62.02%) were tenured and had a permanent item, mostly in government hospitals (74; 57.36%). There were 80 (62.02%) with only one hospital affiliation, 39 (30.23%) with 2-3 hospital affiliations, and 10 (7.5%) with 4 hospital affiliations. Eighty-one (62.79%) had administrative positions in the department: Chairman (23; 17.83%), Vice- Chairman (6; 4.65%), Training Officer / OBE coordinator (25; 19.38%) and Subspecialty Head (27; 20.93%). The number of respondents with attendance in an OBE workshop or lecture was 78 (60.47%). More than half of the residents (45; 54.88%) were between 25- 29 years of age, newly - licensed physicians of the Philippine Board of Medicine. There was a similar number of males (43; 52.44%) and females (39; 47.56%). Except for less sophomores, the distribution by year level was equivalent, with 30 (36.59%) second year, 21 (25.61%) third year, and 30 (36.59%) fourth year residents. There was one fifth year respondent who was the chief resident in one institution. The respondents were mostly from government hospitals (69; 84.15%). There were 51 respondents (62.20%) had attended a workshop or lecture on OBE. The self-reported assessments of the consultants and residents on implementation of OBE in ORL-HNS rated the first domain of Intended Learning Outcome (ILO) as “fully implemented” (FI). (Table 1) The weighted means of the 4-point rating scales for perceptions of the consultants ranged from 3.47 to 3.78. Among the residents, the weighted means ranged from 3.54 to 3.81. The highest score was the ability to demonstrate clinical and surgical competence and engaging in professional development among the consultants. Among the residents, it was exhibiting Professionalism. The core competency on “managing community health and social needs’ had the lowest score among both consultants and residents, at 3.47 and 3.54, respectively. Table 2A shows the self-reported assessment of consultants and residents on implementation of OBE in ORL-HNS for the second domain, teaching and learning activities (TLA). The consultants and residents perceived this domain as “fully implemented” (FI). The weighted means of the 4-point rating scales for perceptions of consultants ranged from 3.47 to 3.88. Among the residents, the weighted means ranged from 3.46 to 3.79. The highest score on the Teaching and Learning Activities was Case-Based Discussion Conferences among the consultants. Among the residents, it was Patient Encounter at the OPD, ER and Ward. Lowest scores were laboratory activities and workshops for both consultants and residents, at 3.47 and 3.46 respectively. Table 2B shows the self-reported assessment of consultants and residents on implementation of the eight (8) teaching and learning activities (TLA) for the seven (7) modules of ORL-HNS residency training (6 subspecialties and the research methodology). The consultants and residents alike perceived this domain as “fully implemented.” The weighted means on the 4-point rating scale on the perception among the consultants ranged from 3.57 to 3.85. Among the residents, the weighted means ranged from 3.51 to 3.82. The highest score on the 7 modules was on general otorhinolaryngology among the consultants and among the residents, it was Otology, Neuro-otology and Audiology. The activity on the module of research methodology had the lowest PhiliPPine Journal of otolaryngology-head and neck Surgery Vol. 36 no. 1 January – June 2021 PhiliPPine Journal of otolaryngology-head and neck Surgery Vol. 36 no. 1 January – June 2021 PhiliPPine Journal of otolaryngology-head and neck Surgery 1918 PhiliPPine Journal of otolaryngology-head and neck Surgery ORIGINAL ARTICLES score among the consultants and the residents’ perception, 3.57 and 3.51 respectively. Table 3 shows the self-reported assessment of consultants and residents on implementation on the third domain, assessment tasks (AT). This domain was perceived as “fully implemented” by both consultants and residents. For the consultants’ perception of implementation for all the 12 assessment task forms, the weighted means ranged from 3.28 to 3.74. The highest score was on Presentation skill assessment and the lowest score was on the multisource feedback form, accomplished by the nurses or the administrative staff. For the residents’ perception, the weighted means ranged from 3.32 to 3.69. The highest score was the basic lecture assessment and the lowest score was on Direct Observation of Performance Skills for Patient Encounter. Among the consultants’ demographic profiles, there was a significant association (x2=10.10, df=3, p<.01) between age and perception of implementation for all the 3 domains of OBE - intended learning outcome (ILO), teaching and learning activities (TLA) and assessment tasks (AT). Consultants aged between 40 to 49 years old comprised the majority of respondents (47/129; 36.43%) who answered “fully implemented” (FI). Consultants younger than 60 years tended to give higher scores in the implementation of OBE in ORL-HNS. A significant correlation was seen in the following: 30-39 years vs. 60 years and above (X2=7.24, df=1, p=.01); 40-49 years vs 60 years and above (X2=6.18, df=1, p=.01), and 50-59 years vs. 60 years and above (X2=6.10, df=1, p=.01). There was also a significant correlation between consultants’ number of years in the department and their perception of implementation of the third domain of OBE, the Assessment Tasks (AT ) (X2=15.52, df=6, p<.01). Consultants who had been in the department longer than 3 years tended to give higher ratings ( positive correlation) for the Assessment Tasks (AT ). On the other hand, sex (X2=3.96, df=3, p<.27), sector category of the hospital (X2=4.401, df=6, p<.62), appointment designation (X2=3.39, df=3, p<.34), number of training hospital affiliations (X2=9.83, df=3, p<.13) and attendance to lecture or workshop on OBE (X2=3.42, df=3, p<.33) were not significantly associated with perception of OBE implementation among the consultants. Among the residents, there was a significant association between attendance in a workshop or lecture on OBE and the perception of OBE implementation for the domain of Intended Learning Outcome (ILO) (X2=6.009, df=2, p<.05). Therefore, there was a positive correlation. There was no significant association between the demographic profile of residents in terms of age (X2=5.45, df=4, p<.24), sex (X2=2.28, df=2, p<.32), residency training level (X2=2.47, df=2, p<.87), category sector, (X2=0.23, df=2, p<.89) and position in the department (X2=0.31, df=2, p<.86) and their perception of implementation of OBE in ORL-HNS residency training in the Intended Learning Outcome (ILO). The study also checked on the comparison between consultants’ and residents’ self-reported assessments of implementation of OBE in ORL-HNS residency training. Using t-test at 0.05 level of significance, there was no significant difference noted in the assessment of Intended Learning Outcome (ILO) (t=0.23, df=209, p=.82), Teaching and Learning Activities (TLA) (t=0.85, df=209, p=.402), and Assessment Tasks (AT) (t=- 0.49, df=209, p=.62) between consultants and residents. Both of them reported the assessment of the OBE as “fully implemented (FI) in the 3 domains. A qualitative study based on descriptive phenomenological approach showed the following results. Throughout the interview, in-depth, semi-structured open questionnaires were asked “tell me about some of the challenges and problems in OBE implementation in general”, “What are the challenges in implementation in terms of the 1st Domain on Intended Learning Outcome as recommended by PRC BOM and the different modules?” “What are the challenges in terms of the Learning Objectives?” and “what are the challenges in terms of the Assessment Tasks?” To motivate the participants to share deeper information, probing questions were also asked, such as “tell me more” and “can you give me an example.” Immediately after conducting each interview, the researcher actively transcribed and reviewed the words of the participants in the interview. Statements and phrases were extracted and the core meaning was collected in terms of the 3 domains of OBE and its challenges. Data saturation was achieved after interviews with 5 participants. Three main themes were extracted from the interviews and emerged in the participants’ experiences. The 3 themes were 1) “Mastery of Intended Learning Outcome”, 2) “Time factor contributing to the implementation of the teaching and learning activities” and, 3) “ Data keeping of the assessment forms.” Quotations from the participants were also used writing the descriptions and the explanations of the 3 subthemes of for each of the 3 themes for a total of 9 subthemes. (Table 4) DISCUSSION The Philippine Board of Otolaryngology and Philippine Society of Otolaryngology Head and Neck Surgery complied with the PRC BOM mandate on outcome-based training for ASEAN integration with a manual on OBE distributed to the accredited institutions in 2017. After 2 years, there was a need for consultants and residents to assess the implementation of OBE in ORL-HNS residency training and form guidelines for better monitoring to graduate competent and competitive ORL-HNS doctors. The sample respondents in this study represent trainers, trainees and residency training programs in the PhiliPPine Journal of otolaryngology-head and neck Surgery Vol. 36 no. 1 January – June 2021 PhiliPPine Journal of otolaryngology-head and neck Surgery Vol. 36 no. 1 January – June 2021 PhiliPPine Journal of otolaryngology-head and neck Surgery 2120 PhiliPPine Journal of otolaryngology-head and neck Surgery ORIGINAL ARTICLES residents on all 3 domains of OBE using the 4-point scale score was “fully implemented.” The first domain on intended learning outcome (ILO) included all 12 core competencies mandated by the PRC BOM to all postgraduate residency training. Among these core competencies, the expected result was that managing community health and social needs will have the lowest score. Although ORL-HNS is a surgical specialty with residency training in a hospital setting, the perception of the respondents was that exposure to this core competency occurs only in medical and surgical missions outside the hospital. Other means of community and social needs like advocacies in ORL-HNS like silip tenga, silip lalamunan, and community hearing evaluation in the hospitals are Table 1. Consultants’ and Residents’ Assessment of the Extent of Implementation of the OBE in the ORLHNS Residency Training in terms of Intended Learning Outcome (ILO) Intended Learning Outcomes (ILO) Core Competencies Residents (N = 82) WM/VI Consultants (N = 129) WM/VI 3.78/FI 3.73/FI 3.58/FI 3.54/FI 3.74/FI 3.54/FI 3.78/FI 3.83/FI 3.53/FI 3.57/FI 3.47/FI 3.68/FI 3.65/FI Ability to demonstrate Clinical and Surgical Competence Displaying Effective communication and Interpersonal skills Exhibiting Effective leadership and management skills Engaging in evidence-based practice Exhibiting inter-professionalism Engaging in Health Systems-Based Practice Engaging in Continuing Professional Development Exhibiting Professionalism Demonstrating nationalism and Internationalism Practicing Social Accountability Managing Community Health and Social Needs Competency in the Curricular Goals divided into 3: Introduce, Practice and Demonstrate the different modules depending on the year level of residency training. Overall Weighted Mean/FI 3.71/FI 3.73/FI 3.57/FI 3.56/FI 3.70/FI 3.62/FI 3/63/FI 3/81/FI 3.58/FI 3.54/FI 3.51/FI 3.63/FI 3.63FI WM – Weighted Mean; VI – Verbal Interpretation Fully Implemented (FI) 3.26 - 4.00; Partially Implemented (PI) 2.51 - 3.25; Minimally Implemented (MI) 1.76 - 2.50 Not Implemented (NI) 1.0 – 1.75 Table 2A. Consultants’ and Residents’ Assessment of the Extent of Implementation of the OBE in the ORLHNS residency training in terms of Teaching and Learning Activities (TLA) Teaching and Learning Activities (TLA) Extent of Implementation Residents (N = 82) WM/VI Consultants (N = 129) WM/VI 3.88D/FI 3.77/FI 3.47/FI 3.81/FI 3.77/FI 3.57/FI 3.58/FI 3.70/FI Case-Based Discussion Conferences Lectures Laboratory activities and workshops Patient Encounter at the OPD, ER, Ward and OR Ward Rounds Subspecialty and multidisciplinary Meetings Specialty rotations Instructional courses and workshops 3.76/FI 3.64/FI 3.46/FI 3.79/FI 3.76/FI 3.70/FI 3.61/FI 3.63/FI WM – Weighted Mean; VI – Verbal Interpretation Fully Implemented (FI) 3.26 - 4.00; Partially Implemented (PI) 2.51 - 3.25; Minimally Implemented (MI) 1.76 - 2.50 Not Implemented (NI) 1.0 – 1.75 Table 2B. Consultants’ and Residents’ Assessment of the Extent of Implementation of the OBE in the ORLHNS residency training in terms of Teaching and Learning Activities (TLA) in the seven modules The Teaching and Learning Activities (TLA) in 7 Modules Residents (N = 82) WM/VI Consultants (N = 129) WM/VI 3.85/FI 3.78/FI 3.77/FI 3.77/FI 3.82/FI 3.81/FI 3.57/FI 3.73/FI General Otorhinolaryngology Otology, Neuro-Otology, Audiology Rhinopharyngology, Allergy, Oral Cavity and Sleep Surgery Laryngobronchoesophagology Head and Neck Surgery Craniomaxillofacial, Plastic and Reconstructive Surgery Research Methodology Overall Weighted Mean (OWM) 3.77/FI 3.82/FI 3.67/FI 3.68/FI 3.76/FI 3.73/FI 3.51/FI 3.69/FI WM – Weighted Mean; VI – Verbal Interpretation Fully Implemented (FI) 3.26 - 4.00; Partially Implemented (PI) 2.51 - 3.25; Minimally Implemented (MI) 1.76 - 2.50 Not Implemented (NI) 1.0 – 1.7 Philippines with the aim to standardize residency training program and to graduate competent and competitive medical specialists locally and internationally. The study was done at 30 accredited institutions in the Philippines from January to May 2020. The evaluation tool used was a researcher- made questionnaire with self-reported assessment of the consultants and residents on implementation of the OBE in ORL-HNS. A focused group discussion among 4 consultants and one resident with open- ended questionnaire confirmed the results on the assessment of implementation and added information on the challenges of OBE in ORL-HNS. Constructive alignment is defined as the teaching design for outcome-based education clearly and explicitly linked in 3 domains: intended learning outcome (ILO), teaching and learning activities (TLA) and, assessment task (AT).6,8 The self-reported perception of the consultants and the residents’ assessment on implementation of OBE in ORL-HNS used the 4-point rating scale evaluation tool. The self-reported assessment of both the consultants and the PhiliPPine Journal of otolaryngology-head and neck Surgery Vol. 36 no. 1 January – June 2021 PhiliPPine Journal of otolaryngology-head and neck Surgery Vol. 36 no. 1 January – June 2021 PhiliPPine Journal of otolaryngology-head and neck Surgery 2120 PhiliPPine Journal of otolaryngology-head and neck Surgery ORIGINAL ARTICLES part of the intended learning outcome at manuals on OBE for all.5 The ORL-HNS residency training adapted this in 2017 with the distribution of the manual of OBE in ORL-HNS. For the second domain on 8 teaching and learning activities (TLA), the perception of both consultant and resident respondents on implementation using the 4-point scale score was “fully implemented.” The lowest score among these activities was reported on the laboratory activities and workshops. In ORL-HNS residency training, surgical excellence is of utmost importance so without the laboratory activities and workshops, opportunity to introduce simulation is missed. Simulation in a surgical specialty is a preparation to be adept and avoid complications in actual patients. Good simulations offer immediate opportunities for feedback and additional crucial “soft’ skills. Not all simulation workshops are offered in all training institutions so there is a need to strengthen in-house activities at all accredited institutions. The teacher’s fundamental task to get students engaged in learning activities that are likely to result in their achieving specific learning outcomes for all the 7 modules in ORL-HNS is important in the constructive alignment of the teaching activities and the outcomes. OBE enables students to learn the desired outcome in an effective manner.11 The learning process is fully enhanced with the application considered a part of the core competency on managing community health and social needs so there is a need to reiterate this to the stakeholders for better understanding. As stated by the Accreditation Council for Graduate Medical Education, the core competencies ensure residents are properly trained to practice medicine successfully.9 As seen in this study, the perception of the trainers and trainees on the core competencies with high scores were on engagement in continuing professional development and demonstration of clinical and surgical competence. The surgical training programs in the USA and in Canada also emphasized these competencies with medical and technical knowledge, clinical decision-making and motor skills in surgical expertise.10 With the OBE in place, The PRC BOM mandated the core competencies in the residency training programs in 2015 to be Table 4. Main Theme and Secondary Themes of OBE Challenges as Perceived by the Consultants and Residents (Interview Results) Main Theme Secondary Theme Mastery of Intended Learning Outcome in the participants’ experiences Time factor contributing to the implementation of the Teaching and learning activities Data keeping of the assessment forms 1. Comprehension of the core competencies set by the PRCBOM 2. Familiarity of the set core competencies and the modules from the ORLHNS manual 3. Knowledge of the different modules and the core competencies from both the PRCBOM and the ORLHNS 1. Conflict of time to teach and practice among consultants 2. Duration of time spent for the learning activities among residents 3. Time management of both consultants and residents for all the learning activities for the different modules 1. Record –keeping of the assessment forms by residents 2. Safeguard and protection of data of the assessment forms accomplished by consultants 3. Monitoring of data kept in each Portfolio of the residents Table 3. Consultants’ and Residents’ Assessment of the Extent of Implementation of the OBE in the ORLHNS residency training In Terms of Assessment Tasks (AT) Assessment Tasks (AT) Extent of Implementation Residents (N = 82) Verbal Interpretation Consultants (N = 129) Weighted Mean 3.74/FI 3.63/FI 3.58/FI 3.47/FI 3.28/FI 3.57/FI 3.46/FI 3.45/FI 3.47/FI 3.45/FI 3.48/FI 3.38/FI 3.43/FI 3.49/FI Presentation Skill Assessment (Form 1) Basic Lecture Assessment (Form 2) Direct Observation of Performance Scale (DOPS) on Procedural Skills (Form 3) DOPS for Patient Encounter (Form 4) Multi-Source Feedback (Form 5) Accomplished by nurses and administrative staff DOPS for Operative Performance (Form 6) Research Skills Assessment (Form 7) The Self Global Assessment (Form 9) give Opportunity for self-reflection and insight of own performance of the residents The Peer Global Assessment (Form 10) articulates the residents’ perception of the performance of their co-trainees The Global Assessment for Consultants (Form 11) provides feedback of their contribution to the residency training. The Residency Training Program Global Assessment (Form 12) by the consultants and residents helpful in planning strategies in implementation The formative assessment (Forms 1-12) allows feedback for better learning Summative Assessment is done at the end of the year for each resident by the department Overall Weighted Mean (OWM) 3.67/FI 3.69/FI 3.45/FI 3.32/FI 3.40/FI 3.41/FI 3.54/FI 3.64/FI 3.61/FI 3.56/FI 3.56/FI 3.49/FI 3.52/FI 3.53/FI WM – Weighted Mean; VI – Verbal Interpretation Fully Implemented (FI) 3.26 - 4.00; Partially Implemented (PI) 2.51 - 3.25; Minimally Implemented (MI) 1.76 - 2.50 Not Implemented (NI) 1.0 – 1.75 PhiliPPine Journal of otolaryngology-head and neck Surgery Vol. 36 no. 1 January – June 2021 PhiliPPine Journal of otolaryngology-head and neck Surgery Vol. 36 no. 1 January – June 2021 PhiliPPine Journal of otolaryngology-head and neck Surgery 2322 PhiliPPine Journal of otolaryngology-head and neck Surgery ORIGINAL ARTICLES of OBE instruction in improving communication skills and acquiring a lot of fun with the learning and teaching activities, both for the trainers and the trainees.12 Among the teaching and learning activities, the consultants’ self-reported assessment showed high scores in case-based discussion conferences and among the residents, patient encounter at the emergency room, outpatient department, operating room and the wards. Again, the strength of training was clinical and surgical exposure to different ORL-HNS cases. Among the 7 modules, the perception of the respondents on the assessment was “fully implemented” but Research Methodology had the lowest score. Research in the residency training needs cooperation of the training institution and department to provide support in terms of time, training, facilities, and finances to the residents in training. Residency training in university hospitals has research incentives and funding with time off to do research. However, in most training institutions, research perks are not available. According to the ACGME, the research curriculum program already takes into account scholarly activity requirements for residents and teaching faculty. Some programs have additional requirements to improve training in research methodology13 and this has to be monitored among the different accredited institutions. For the third domain on assessment tasks (AT), the perception of the consultants on implementation using the 4-point scale score was “fully implemented.” Among the assessment tasks, the lowest score was the multi-source feedback accomplished by nurses and administrative staff. Its goal is to look at the residents’ work from a variety of perceptions that will give a complete picture of performance, not only from the colleagues.14 The doctors in residency training are still not used to being assessed by non-colleagues at this point in time in the Philippines. The evaluation of other co-workers is a way to balance work relationships among the health care workers for the benefit of the patients. The perception of the residents on implementation of the assessment task (AT) was “fully implemented” but the lowest score was on direct observation of performance skills at the operating room, outpatient department, emergency room and the wards. Before the advent of OBE, evaluation and grading of residents’ performance relied more on later recall of the trainers and were not immediately documented regularly in writing. With the introduction of the OBE, documentation of each domain from the teaching and learning activities to assessment on the intended learning outcomes are documented and kept in a portfolio for each resident. The paper trail was mandated by the Accreditation Council for Graduate Medical Education to fully integrate assessment tools into the OBE curriculum and provide a comprehensive evaluation of all areas of the core competencies. 9 Among the assessment tasks, the highest scores were seen on presentation skills and on basic lectures. These were done frequently at grand rounds and regularly meetings of the department. Evaluation and assessment has become an integral process of any educational institution toward an improved and quality learning experience.15 The 3 domains on OBE, the intended learning outcomes (ILO), teaching and learning activities (TLA) and the assessment tasks (AT) was constructively aligned in ORL-HNS. The consultants and the residents’ perception on implementation after 2 years are “fully implemented” among the accredited ORL-HNS residency institutions in the Philippines. Although all domains were fully implemented, the low scores have to be monitored and improved for the success its aim in ORL-HNS residency training and, that is to graduate competent and competitive doctors. This study also showed that there was no difference in the consultants and residents self-reported assessment on implementation of the OBE in ORL-HNS two years after the introduction to the departments. The focused group discussion among a small number of respondents confirmed the results that the OBE was “fully implemented.” The challenges of the OBE were grouped to the 3 themes on mastery and familiarization of the intended learning outcome, the time spent for the teaching and learning activities and lastly is the data keeping of the assessment forms. The perception on implementation of OBE was associated with age in all the 3 domains of OBE. Most of those who answered the questionnaire were young consultants in the 40 to 49 year-old-age group because ORL-HNS is a relatively young specialty. At this age group, educators are more likely to be optimistic, positive or open to educational changes like the introduction of OBE. In a local study of an English class, teachers’ age and number of years in teaching were positively related to the their attitude towards Outcome-Based Education (OBE).16 This was seen in our study. There was also an association of the consultants ‘number of years in the department and the implementation of the Assessment Tasks (AT).’ Most of those who answered the survey were already present in the department for more than 3 years. It takes time to master a new program and the longer and more frequent they practice answering the assessment forms, the more it becomes a habit. The habit then becomes part of the system of the department. Among the residents, there was an association of the attendance to OBE workshop or lecture and the perception on implementation. OBE program in the ORL-HNS residency program is something new with a lot of documentation and paperwork to assess the residents in training. Commitment to a new curriculum is important. Preparing teachers for change precedes training. When there is a need for change in the PhiliPPine Journal of otolaryngology-head and neck Surgery Vol. 36 no. 1 January – June 2021 PhiliPPine Journal of otolaryngology-head and neck Surgery Vol. 36 no. 1 January – June 2021 PhiliPPine Journal of otolaryngology-head and neck Surgery 2322 PhiliPPine Journal of otolaryngology-head and neck Surgery ORIGINAL ARTICLES REFERENCES 1. Hague M. Outcome based medical education – A brief commentary. Natl J Physiol Pharm Pharmacol. 2017 Apr 09; 7(9): 881-885. DOI: 10.5455/njppp.2017.7.0411203052017. 2. Sana EA, Roxas AB, Reyes ALT. Introduction of Outcome Based Education in Philippine Health Professions Education Setting. Phil J Health Res Dev. 2015 Mar; 19(1):60-74. 3. Accreditation Council for Graduate Medical Education (ACGME) general competencies. [Cited 2020 Aug 7] Available from: https//www.acgme.org. 4. Mousavihejazi B, Borja de Mozato B. Value of Design Competencies within an Outcome Based Education. In: 11th European Academy of Design Conference. France: Boulogne Billancourt; 2015. 5. Outcome Based Education Manual in Residency Training in Otorhinolaryngology-Head and Neck Surgery (ORL-HNS). Phil. Society of Otolaryngology Head and Neck Surgery and Philippine Board of Otolaryngology Head and Neck Surgery, 2016. 6. Spady WG, Uy FA. Outcome-Based Education: Critical Issues and Answers: Maxor Publishing House, Inc., 1994; ISBN 978-971-0167-41-8. 7. Crespo R, Najjar J, Dernti M, Leony D, Neumann S, Oberhuemer P, et al. Aligning Assessment with Learning Outcomes in Outcome Based Education. IEEE Xplore. 2010 May; 1239-1246. DOI: 10.1109/EDUCON.2010.5492385. 8. Biggs J. Aligning Teaching and Assessing to Course Objectives. Teaching and Learning in Higher Education: New Trends and Innovations. University of Aveiro, 2003 April; 13-17. 9. Bancroft GN, Basu CB, Leong M, Mateo C, Hollier LH, Stal S. Outcome-Based Residency Education: Teaching and Evaluating the Core Competencies in Plastic Surgery. Plast Reconstr Surg. 2008 Jun; 121(6): 441e-448e. DOI: 10.1097/PRS.0b013e318170a778; PubMed PMID: 18520871. 10. Grober E, Jewett M. The concept and trajectory of “Operative competence” in Surgical Training. Can J Surg. 2006 Aug; 49(4): 238-40. PubMed PMID: 16948880; PubMed Central PMCID: PMC3207565. 11. Caguimbal D, Delacion D, Medina AG, Mendoza M. Mendoza RJ, Sanche M. Level of Awareness of The Maritime Students on the Outcomes-Based Education. Educ Res Int. 2013 Aug; 2(1); 7-12. 12. An IL. Impact of Outcome Based Education Instruction to Accountancy Students in an Asian University. Asia Pacific Journal of Education, Arts and Sciences. 2014 Nov; 1(5): 48-52. Available from: http://apjeas.apjmr.com/wp-content/uploads/2014/11/APJEAS-2014-1-085.pdf. 13. Accreditation Council for Graduate Medical Education (ACGME) Common Program Requirements (Residency). ACGME-approved focused revision: February 3, 2020; effective July 1, 2020. 14. Lockyer J. Multisource Feedback in the Assessment of Physician Competencies. J Contin Educ Health Prof. Winter 2003; 23(1): 4-12. doi: 10.1002/chp.1340230103; PMID: 12739254. 15. Bay BE, Subido, H. DREEM is Real: Dental Students Learning Environment in an Asian University, International Journal of Academic Research in Business and Social Sciences. 2014 Aug; 4(7); 620- 635. DOI: 10.6007/IJARBSS/v4-i7/1060. 16. Ortega RA, Ortega, Dela Cruz R. Educators’ Attitude towards Outcomes-Based Educational Approach in English Second Language Learning. Am J Educ Res. 2016; 4(8): 597-601. DOI: 10.12691/education-4-8-3. http://pubs.sciepub.com/education/4/8/3. 17. Shamsuddin A. Chan CM, Suratkon A. Implementation of OBE in a MoT Program. Lesson Learned. Proceedings of the 2015 International Conference on Operations Excellence and Service Engineering. 2015 Sept; 10-11. curriculum, teachers had to be empowered.17 The activity of the PSO- HNS in reaching out to the different accredited institutions to guide the departments with a lecture and guide to the manual was indeed effective in the understanding of the new OBE program. Getting a good background of the constructive alignment of the 3 domains of OBE and familiarization of the manual given by the ad hoc committee on OBE in ORL-HNS showed that there was more understanding of what is expected in the assessment of residents in training. The following are the limitations of the study. There are few published studies on OBE in postgraduate training (which was introduced only about 5 years ago compared to OBE in medical education). The accuracy measures with a self-reported questionnaire can be quite low so a focused group discussion in a small number of respondents was done to confirm the results and add other questions on challenges on implementation. The author is a member of the Philippine Board of Otolaryngology Head and Neck Surgery, accrediting body of the residency training, but not a member of the ad hoc Committee on OBE in ORL-HNS. The OBE coordinators or the chief residents of the accredited departments supervised the conduction of the data collection. Due to time constraints and lockdown because of the COVID 19 pandemic, data gathering from 4 out of 34 institutions were not done, however the number of respondents exceeded the recommended sample size of 192. It is recommended that because the self-reported implementation may or may not necessarily correspond to the true implementation extent, there is a need to monitor the lowest scores in the core competency of community health and social needs, laboratory activities and workshops, module on research methodology and assessment of multisource feedback and direct observation of performance skills of patient encounter at the operating room, emergency room, out-patient department and wards. The challenges on mastery, time and data keeping in a portfolio as revealed on the focus group discussion among the respondents will be relayed to the Committee on OBE in ORL-HNS in the creation of guidelines to monitor the program. Future researches on knowledge, attitudes and perception among stakeholders as well as use of measured monitoring tools on implementation are also recommended. In conclusion, this study showed that two years after the distribution of the manual on OBE to the ORL-HNS residency training institutions, the consultants and residents’ self-reported assessment is that implementation in all the 3 domains of OBE has been “fully implemented.”