ORIGINAL�ARTICLE ABSTRACT Objective: To evaluate the effectiveness of faculty development workshop on “Introduction & development of student's portfolio” using first two levels of Kirkpatrick (KP) model. th Study Design: A quantitative evaluation study based on evaluation of one day workshop, conducted on 15 November 2019. Place and Duration of Study: The study was conducted at Islamic International Medical College at Department of Medical Education. Materials and Methods: The study aimed to evaluate the efficacy of faculty development workshop on Kirkpatrick's model of program evaluation. A 4-hours workshop was carried at Islamic International Medical th College on 15 November 2019. Sampling technique was purposive (n=21). The reaction of the study participants about the training experience was evaluated through a feedback evaluation proforma. The assessment of knowledge was done through MCQs and reflective writing skill was assessed through a checklist based upon framework of Gibbs reflective cycle. Both knowledge and skill were assessed before and after the workshop. The content of pre-test and post-test was same. Results: The knowledge of the participants about portfolio was remarkably increased, from 3.95 ± 1.35 (median=4) in pre-test to 6.23 ±1.17 (median=6) in post-test. The improvement in reflective writing skill was also significant 2.14 ± 0.727 (median=2) in pre-workshop reflection to 4.23 ± 0.70 (median=4) in post-workshop reflection. The Wilcoxon signed rank test showed a significant increase in both knowledge and skill (p-value <0.000). The reaction of the participants showed high percentages regarding content delivery (95%), learning environment (100%) and increase in knowledge (95%) measured on 5-point Likert scale. Conclusion: Faculty development workshop on student's portfolio was highly valued by faculty members. Most of the participants were convinced and ready to start portfolio for learning and assessment of students in the institution. Key Words: Faculty Development, Kirkpatrick Model, Portfolio, Reflective Writing, Students Learning. 1 from experience by the process of reflection. The reflection is a metacognitive process that improves future course of action by creating deeper understanding about one's self and situation. It 2 makes students strategic life-long learners. The reflective writing is a fundamental skill required for development of portfolio which is not necessarily inherent for teachers and learners. Hence the medical educators need to devise the strategies necessary to develop this skill among medical 3 teachers. Thus teachers training on reflective writing is directly related to portfolio development by the students. For faculty development different activities can be arranged, among which workshops are most popular and desired ones. This may be due to an interactive ambiance in a small group, an ease of 4 communication and flexible environment. Introduction Portfolio is one of the most useful cognitive learning and assessment tool. It stimulates deeper learning Effectiveness of Faculty Development Workshop on Introduction and Development of Student's Portfolio: A Pilot Project 1 2 3 4 5 Lubna Rani Faysal , Raheela Yasmin , Ambreen Ansar , Shawana Sharif , Attia Zaman Correspondence: Dr. Lubna Rani Faysal Assistant Professor Department of Dermatology Islamic International Medical College Riphah International University, Islamabad E-mail: lubna.faysal@gmail.com 1 2 Department of Dermatology/Medical Education Islamic International Medical College Riphah International University, Islamabad 3 Department of Community Medicine Wah Medical College, Wah 4 Department of Dermatology Rawalpindi Medical College, Rawalpindi 5 Department of Biochemistry Federal Medical and Dental College, Islamabad Funding Source: NIL; Conflict of Interest: NIL Received: July 25, 2020; Revised: December 14, 2020 Accepted: January 28, 2021 Faculty Development on Student's PortfolioJIIMC 2021 Vol. 16, No.1 37 After the successful implementation of any workshop the next challenge is to measure its impact on the learning of the participants. The most suitable framework to evaluate the workshop training programs is the Kirkpatrick's model. The evaluation of reaction, learning, behavior and results through different levels of the Kirkpatrick model-KP is an invaluable tool to assess the achievement of an 5 outcome. The literature review has shown that the trainings on the development of reflective writing capacity are lacking in Pakistan and this is the reason due to which the institutions are unable to use the effective tool of portfolio both for students learning and 6 assessment. Portfolio is a regular part of undergraduate curriculum in majority of medical schools worldwide and the teachers are formally trained to achieve excellence in the skill of reflective writing through series of ongoing training programs. The purpose of current study is to evaluate the effectiveness of faculty development workshop on introduction and development of student's portfolio. The article will provide guidelines on faculty training in context of reflective writing in developing countries like Pakistan. It will also be helpful to start with portfolio as an effective tool of learning and assessment in a curricular program. Materials and Methods It was a quantitative evaluation study carried at the department of Medical Education, Islamic International medical college, Rawalpindi on 15th November 2019. It was 4 hours workshop on Introduction and Development of Student's Portfolio The workshop evaluation was based on KP model. The workshop was attended by 21 participants and the sampling technique was purposive sampling. Approval from the institutional ethical review committee was taken and informed written consent was obtained from all participants. The study participants were both from basic and clinical sciences, including professors, associate professors, assistant professors and senior registrars with certificate or master's degree in medical education. Newly inducted faculty members and those without any background of medical education were not included in the workshop. The content delivered in the workshop was about the introduction of student's portfolio, the process of developing a sample portfolio and its importance in learning and student's assessment. The pre-existing knowledge of the participants about portfolio was assessed through a pretest consisting of 10 MCQs. It was followed by a reflective writing activity by the participants on any of the incident from their academic life experience. This was to assess the reflection writing skill of the participants before attending the workshop. The evaluation of learning of knowledge was done through MCQs and the reflective writing skill was assessed through a checklist based on Gibbs reflective cycle (table I). Both knowledge and skill were assessed by the workshop facilitator. The MCQs & reflective writing assessment checklist were approved from the expert in medical education. After the workshop, MCQs and reflective writing activity was repeated as post-test. MCQs and reflective writing activity were kept same in both pre-test and post-test. Both the pre- and post-test MCQs and reflective writing scores were compared to see any change in the learning of participants. The reaction of the participants to workshop was collected on a pre-designed feedback evaluation proforma, given to the participants at the end of workshop. The feedback (reaction) was assessed on 5-point Likert scale, where 1=strongly disagree (SD), 2=disagree (D), 3=do not know (DK) 4=agree (A) and 5=strongly agree (SA). For analysis of data, SPSS version 20 was used. The frequencies & percentages were calculated for the feedback evaluation of participants while mean & median scores were calculated for MCQs & reflective writing. For comparison of pre-test and post-test scores, Wilcoxon signed-rank test was used as the data was non-parametric. Table I: Checklist for Evaluation of Reflective Writing Skill Results The workshop was attended by 21 participants. There were 3 (14%) males & 18 (86%) female Faculty Development on Student's PortfolioJIIMC 2021 Vol. 16, No.1 38 participants, all appeared in pre & post-test. The reaction of the participants assessed through feedback evaluation proforma is given in figure 1. Except for one domain “appropriate time allocation”, all the responses fall into the categories of 4 (A) & 5 (SA) on Likert scale. Both category 4 and 5 were considered positive response and sum of the percentages in each of these categories show a high satisfaction of the participants to the workshop regarding content delivery (95%), learning environment (100%), teaching strategy (90%) and increase in knowledge (95%). The knowledge of the participants about portfolio was remarkably increased, from 3.95 ± 1.35 (median=4) in pre-test to 6.23 ±1.17 (median=6) in post-test. The improvement in reflective writing skill was also significant 2.14 ± 0.727 (median=2) in pre- workshop reflection to 4.23 ± 0.70 (median=4) in post-workshop reflection figure 2. The Shapiro-Wilk test showed the non-normal distribution of the scores and Wilcoxon signed rank test confirmed remarkable improvement (p=0.000) in the post- workshop knowledge and reflective writing skill of the participants about student's portfolio (table-II). Discussion The knowledge of participants about portfolio and skill of reflective writing were remarkably improved after attending the workshop. The highly satisfactory response of the study participants further proved the effectiveness of workshop. Such activities are necessary to develop the reflective writing skill of the medical teachers which is fundamental to the 7 process of portfolio development. The biggest challenge for health educators is the pedagogical tool to help students to write reflectively. Portfolio is one such tool to develop reflective capacity of students to foster their critical thinking and problem-solving skills to prepare them 8 for informed clinical judgments. It helps them to perform better in written assessments and to develop decision making skills as a clinician. The implementation of any new strategy either for teaching or assessment of students is always very challenging. One of the biggest challenges is faculty resistance. The people do not feel comfortable to carry out the new responsibilities either due to inadequate exposure or lack of training. Hence it is imperative to do capacity building of teachers at 9 first. The current study has implications for many institutions across the country, planning to begin with portfolio as an integral part of students teaching and assessment. There is not enough literature on the reflective capacity building of teachers in Pakistan. The concept is relatively new in teachers training programmed on our set up. In a study conducted by M Ilyas, Shawana Fazal & M Amin, the comparison of perceptions of students and university teachers in Pakistan and UK showed that majority of teachers and students here were not even well aware about 10 the notion of reflection. Teachers represent one of the key resources for the academic success and investment in their training results in unceasing 11 success despite all challenges. Teachers training Fig 1: Reaction of The Participants to the Workshop on 5-Point Likert Scale 1=Pre-test, 2=post-test Table II: Improvement in Knowledge and Reflective Writing Skill Fig 2: Descriptive Statistics of Pre and Post Workshop Knowledge and Skill Scores Faculty Development on Student's PortfolioJIIMC 2021 Vol. 16, No.1 39 always results in improvement of skills, motivation, and performance. It helps to boosts up one's self- confidence and brings positive behavioral changes among teachers for educating students. In this study, an outcome-based workshop was conducted as apart of faculty development program, with targeted audience. The workshop was evaluated to assess the outcome at level I and level II of Kirkpatrick's model for reaction, improvement of knowledge and reflective writing skill of the participants. The study showed statistically significant increase in knowledge and reflective writing skill of all the participants after the workshop on student's portfolio. The pre workshop reflection was very ordinary, there was no framework followed by the participants, the coherence and flow of events was missing. After the workshop, all reflective writings improved significantly. The participants were trained to reflect using Gibbs cycle of reflection and assessment was done on a checklist based on it. Gibbs reflective cycle is an effective model for 12 teaching of reflection. After workshop, there was good reflection of description of situation along with feelings, critical analysis of the context and action plan. The evaluation of reaction of participants also showed good response with 95% satisfaction with content delivery and increase in knowledge. The participants valued and enjoyed all teaching and learning activities in the workshop on student's portfolio with 100% satisfaction on learning environment. They acknowledged the activities were interesting, informative, and contextual with less lecturing and more hands-on work. These findings highlighted the effective planning of the workshop. The pro-active measures in relation to expectations of the participants are of utmost 13 importance for an effective workshop. There are various other tools which can be used as valuable avenues to promote reflective writing. In another study conducted by Karkabi K et al., the use of abstract paintings and narrative literature was used to develop reflective capacity in a multinational 14 faculty development workshop. The participants found this arts-narrative based workshop very useful to facilitate their reflective writing in context of educator-learner relationship. Qualitative analysis of participants feedback was done. The sample size was small and no objective quantitative evaluation of the net gain in knowledge and skill about reflective writing of the participants before and after the workshop. Also change in behavior was not evaluated. In another study by Donald Boudreau J et al., a workshop on reflective writing for clinical teachers, the evaluation of outcome was limited only to level I of Kirkpatrick's model, quantitative data focused on patient satisfaction with qualitative analysis of their 15 narrative writings. Narrative methods had been used successfully to nurture self-reflection through a half day interactive workshop. In comparison to current study, objective assessment of pre- and post- workshop reflective writings was lacking. In this study, the quantitative data interpretation was done before and after the workshop for both knowledge and reflective skills of participants along with their feedback evaluation. Use of multiple tools like MCQs and reflective writing skill evaluation in a 4-hours workshop is another strength of this study. Present study strongly emphasizes to initiate and continue these workshops for teachers training to promote their reflective writing skill. This is not only essential for portfolio development, but it is the need of hour to help our students to become critical thinkers and reflective life-long practitioners for 16 better health care services. Conclusion The workshop on student's portfolio development was highly valued by the participants. There was remarkable improvement in knowledge and reflective writing skill of the participants required to develop a portfolio. Such workshops are immensely needed for building reflective capacity of teachers which is one of the core competencies all health care professionals. Limitations The results are generalized from the single workshop, which is main limitation of the study. Further research is recommended with use of different validated qualitative and quantitative tools. Disclaimer This manuscript has not been previously presented or published in any conference and it is not a part of research, PhD or thesis project. Conflict of interest It is declared that there are no personal, financial, and professional interests involved. Faculty Development on Student's PortfolioJIIMC 2021 Vol. 16, No.1 40 Funding Disclosure There are no funding sources to disclose. REFERENCES 1. Van Tartwijk J, Driessen EW. Portfolios for assessment and learning: AMEE Guide no. 45. Medical teacher. 2009 Jan 1;31(9):790-801. 2. Huang JJ, Yang SJ, Chiang PY, Tzeng LS. Building an e- portfolio learning model: Goal orientation and metacognitive strategies. Knowledge Management & E- Learning: An International Journal. 2012;4(1):16-36. 3. Wald HS, Reis SP. Beyond the margins: reflective writing and development of reflective capacity in medical education. Journal of general internal medicine. 2010 Jul 1;25(7):746- 9. 4. Steinert Y. Faculty development: from workshops to communities of practice. Medical teacher. 2010 Jan 1;32(5):425-8. 5. Rouse DN. Employing Kirkpatrick's evaluation framework to determine the effectiveness of health information management courses and programs. Perspectives in health information management. 2011;8. 6. Ashraf H, Rarieya JF. Teacher development through reflective conversations–possibilities and tensions: a Pakistan case. Reflective practice. 2008 Aug 1;9(3):269-79. 7. Tsingos C, Bosnic-Anticevich S, Smith L. Reflective practice and its implications for pharmacy education. American journal of pharmaceutical education. 2014 Feb 12;78(1). 8. Embo MP, Driessen E, Valcke M, Van Der Vleuten CP. Scaffolding reflective learning in clinical practice: A comparison of two types of reflective activities. Medical teacher. 2014 Jul 1;36(7):602-7. 9. Raza SA, Majid Z, Zia A. Perceptions of Pakistani university students about roles of academics engaged in imparting development skills: Implications for faculty development. Bulletin of Education and Research. 2010 Dec 1;32(2). 10. Fazal S, Amin M. Reflection in Teacher Education Programmes in Pakistan and the UK: A Comparison. Journal of Research & Reflections in Education (JRRE). 2014 Dec 1;8(2). 11. Breckwoldt J, Svensson J, Lingemann C, Gruber H. Does clinical teacher training always improve teaching effectiveness as opposed to no teacher training? A randomized controlled study. BMC medical education. 2014 Dec 1;14(1):6. 12. Adeani IS, Febriani RB, Syafryadin S, Mangana AV, Kurniawan E, Ananda J, Maryam S. Using GIBBS reflective cycle in making reflections of literary analysis. Indonesian EFL Journal (IEFLJ).;6(2):139-48. 13. Behar-Horenstein LS, Garvan CW, Catalanotto FA, Su Y, Feng X. Assessing faculty development needs among Florida's allied dental faculty. American Dental Hygienists' Association. 2016 Feb 1;90(1):52-9. 14. Karkabi K, Wald HS, Castel OC. The use of abstract paintings and narratives to foster reflective capacity in medical educators: a multinational faculty development workshop. Medical Humanities. 2014 Jun 1;40(1):44-8. 15. Boudreau JD, Liben S, Fuks A. A faculty development wo rks h o p in n a rrat ive- b a s ed ref lec t ive w rit in g. Perspectives on medical education. 2012 Aug 1;1(3):143- 54. 16. Burdick W, Amaral E, Campos H, Norcini J. A model for linkage between health professions education and health: FAIMER international faculty development initiatives. Medical teacher. 2011 Aug 1;33(8):632-7. Faculty Development on Student's PortfolioJIIMC 2021 Vol. 16, No.1 41