81 m e d ic a l e d u c a t io n iSSn 2413-6077. iJmmR 2020 Vol. 6 issue 1 doI 10.11603/IJMMR.2413-6077.2020.1.11504 WEB-BASED 5-DIMENSIONAL ELECTRONIC PORTFOLIO (5DEP) AS A COMPETENCY-BASED ASSESSMENT TOOL IN POST-GRADUATE MEDICAL TRAINING *Z. Bokhua1, K. Chelidze1,2, K. Ebralidze1 1 – TBILISI STATE MEDICAL UNIVERSITY (TSMU), TBILISI, GEORGIA 2 – THE FIRST UNIVERSITY CLINIC OF TSMU, TBILISI, GEORGIA Background. New challenges of permanently changing context of healthcare system requires new methods of medical education and new assessment tools, as well. Competency-based Medical Education (CBME), framework which has been adopted as a new approach in medical education, needs appropriate assessment tool such as portfolio. Portfolio is learner-centered assessment instrument which evaluates learner’s progression towards outcomes and enables both residents and teachers to engage in a process of learning through assessment. Objective. In this paper we aim to share our successful experience of an effective use of web-based 5-Dimensional Electronic Portfolio (5DeP) as an assessment tool in the Pilot Group. Methods. Pilot Group of sixteen residents (six first year residents of Obstetrics/Gynecology and ten first year residents of Internal Medicine. Tbilisi State Medical University Institute of Postgraduate Medical Education and Continuous Professional Development) and twelve mentors (four Obstetrics/Gynecology mentor and eight Internal Medicine mentors) reported some feedback about 5-Dimensional Electronic Portfolio (5DeP) as a new assessment tool. Results. Feedback about 5-Dimensional Electronic Portfolio (5DeP) as a new assessment tool from mentors and students demonstrated efficiency of the program. It enables assessment within a framework of transparent and declared criteria and learning objectives; provides a model for lifelong learning and continuing professional development; increases competence in a wider context with benefits to both professional and personal roles; improves organizing skills. Conclusions. 5DeP have been recognized as an extremely effective assessment tool. KEY WORDS: Competency-based Medical Education (CBME); postgraduate education; electronic portfolio; feedback. International Journal of Medicine and Medical Research 2020, Volume 6, Issue 1, p. 81-87 copyright © 2020, TNMU, All Rights Reserved *Corresponding author: Zaza Bokhua MD. Ph.D., Associate Professor of Reproductive Health Department, Director of In- stitute of Postgraduate Medical Education and Continuous Professional Development, Tbilisi State Medical University, Tbilisi, Georgia. E-mail: z.bokhua@tsmu.edu Z. Bokhua et al. Introduction The fundamental changes in health care and the complexity of health problems have radically changed the role of Physicians in the health care system and mounted different challenges in terms of their under- and post- graduate education, and continuous profes- sional development. Over the last two decades, Competency‐ Based Medical Education (CBME) frameworks have been internationally adopted as the pri- mary educational approach in medicine. CBME is a learner-centered, active, and lifelong ex- perience that incorporates feedback between the teacher and the learner to fulfill the desired competency outcomes [1]. Despite standardized core competencies of medical education, there are no standardized assessment methods to determine learner's achievement in all the core competencies prior to completion of residency training [1]. Com- petency-based education needs continuous, comprehensive feedback and assessment systems [2]. The use of one of most popular assessment instruments, portfolio can facilitate the above-mentioned goals of assessment by collecting information about the learner’s progression towards outcomes [3]. Portfolio- based assessment tool is at the apex of Miller’s pyramid, because it provides performance- based assessment in real context by analysis of actions [4, 5]. Smith and Tillema made four different types of portfolios [6]: 1. The dossier portfolio, containing man- dated records of achievement. 2. The training portfolio, containing a man- dated collection of acquired skills and compe- tencies, in a fixed format, with some reflective comments on selected evidence. 82 m e d ic a l e d u c a t io n iSSn 2413-6077. iJmmR 2020 Vol. 6 issue 1 3. The reflective portfolio, containing a pur- poseful collection of evidence for personal growth and development, and 4. The personal development portfolio, con- taining reflective and personal evaluations of progress in time, enabling discussion and va- luing of these activities. In this paper, we aim to provide guidance to program directors for implementation of an electronic portfolio (e-portfolio) by reporting our successful experience of an effective use of web-based e-portfolio system in Internal Me- dicine and Obstetrics/Gynecology residency programs of Tbilisi State Medical University, where the 5-Dimensional Electronic Portfolio (5DeP) was introduced with support of United Nations Population Fund (UNFPA) in 2015. Methods The Pilot Group of sixteen residents (six first year residents of Obstetrics/Gynecology and ten first year residents of Internal Medicine. Tbilisi State Medical University Institute of Postgraduate Medical Education and Conti- nuous Professional Development) and twelve mentors (four Obstetrics/Gynecology mentor and eight Internal Medicine mentors) reported some feedback about 5-Dimensional Electronic Portfolio (5DeP) as a new assessment tool. Design and structure of 5-Dimensional Electronic Portfolio (5DeP) The new web-based 5-Dimensional Elec- tronic Portfolio (5DeP) is designed in simple and intuitive manner for users with minimal experience and skills with information tech- nologies. The three modules with different levels of access by individual username and password are designed for (Fig.1): • Resident (Module 1, access level A), to: � upload/confirm evidences of performance/ achievements during training course, approved by the Mentors � view of Training Course Assessment Form (TCAF) and � view of Final Portfolio Assessment Form (FPAF) and narratives • Mentor (Module 2, access level B), for resident training place assessment (TPAF); • Program Director/Experts Panel (Module 3, access level C), for monitor of training process and final assessment of portfolio (IPAF/FPAF). Module 1 – Collected Evidences (CE) This module covers all the four types of Smith’s & Tillema’s stratification [6] and is intended for residents to: • upload evidences collected during all training courses: � achievement of learning outcomes; � clinical (inpatients/outpatients) cases, write-ups and shift records; � acquired essential skills; � self-evaluation of progress in time. • view Training Course Assessment Form (see below) after completion each training course; • self-monitor of dynamics on each dimen- sion during the residency training process. Module 2 – Training Course Assessment Form (TCAF) This module is intended for mentors (clinical trainers) to assess resident on following five dimensions at the end of each training course: • Communication – by evaluation of a resident’s interpersonal communication skills, ability to work as integral part of a multidis- ciplinary medical team, to prevent and manage of communication barriers. This dimension also looks at the strengths of resident’s medical record keeping skills; • Self-development – by evaluation of resident’s attitude and responsibility for his/ her own personal development, ability to take active part in learning opportunities, and effec- tive use of reflection; This dimension also looks at the resident’s ability to follow patient/staff/ self-safety and security policy; • Professionalism – by evaluation of re si- dent’s attitude to work, professional appearan- ce, timekeeping, professional boundaries, quality assurance; ability to maintain patient confidentiality, and recognize and respect people’s diversity, preferences, choices and beliefs; • Assessment and planning – by evaluation resident’s ability to use basic knowledge and analytical skills in patient assessment and clinical reasoning, collaborate with patient in decision making process and elaborate appro- priate management plan; • Intervention and treatment – by eva- luation resident’s ability to identify specific precautions/contraindications to the interven- tion/treatment, deliver intervention/treatment with skill and care, and take appropriate actions in case of complications. Each dimension contains marking guide table with detailed graded descriptors. Choo- sing specific descriptor that mostly accurately describes the resident’s performance (Excellent: 5 points; Very good: 4 points; Good: 3 points; Z. Bokhua et al. 83 m e d ic a l e d u c a t io n iSSn 2413-6077. iJmmR 2020 Vol. 6 issue 1 Average: 2 points; Below average: 1 point) the module calculates final score for specific dimension and turns the score into an appro- priate grade with percentage mark (A –Excel- lent: 91-100% of the maximum grade; B – Very good: 81-90% of the maximum grade; C – Good: 71-80% of the maximum grade; D – Satisfactory: 61-70% of the maximum grade; E – Sufficient: 51-60% of the maximum grade; Fx – Fail: 41-50% of the maximum grade and F – Fail: >50% of the maximum grade). At the end of the training course the mentor should complete and approve (without further permission of editing) the Training Course Assessment Form (TCAF) with the Net Diagram (D-Net) for marks and grades visualization Figure 1. 5-Dimensional Electronic Portfolio (5DeP). Design and structure. CE: Collected evidences (form training courses 1, 2 and X). TCAF: Training Course Assessment Form (made by the Tutor 1, 2 and X of training courses 1,2 and X, respectively); FPAF&N: Final Portfolio Assessment Form and Narratives. Solid line: to upload/edit/confirm; Dotted line: to view; Long dashed line: to approve/ confirm. 3 Figure 1. 5-Dimensional Electronic Portfolio (5DeP). Design and structure. CE: Collected evidences (form training courses 1, 2 and X). TCAF: Training Course Assessment Form (made by the Tutor 1, 2 and X of training courses 1,2 and X, respectively); FPAF&N: Final Portfolio Assessment Form and Narratives. Solid line: to upload/edit/confirm; Dotted line: to view; Long dashed line: to approve/confirm. CE-1 Resident TCAF-1 Mentor 1 CE-2 CE-XTCAF-2 Mentor 2 TCAF-X Mentor X Training Course XTraining Course 2Training Course 1 Program Director/Assigned Expert IPAF/FPAF&N Z. Bokhua et al. 84 m e d ic a l e d u c a t io n iSSn 2413-6077. iJmmR 2020 Vol. 6 issue 1 (Fig. 2). After completion of the training course mentor will meet with the resident to discuss his/her overall performance. During the meeting should be discussed in which areas the resident has been deficient and why. Module 3 – Interim Portfolio Assessment Form/Final Portfolio Assessment Form (IPAF/ FPAF) This module is designed for the Program Director to perform consistent monitoring of training process and provide: • interim portfolio assessment at the end of each residency year • final portfolio assessment at the end of residency course chairing the Experts Panel • report with analysis of resident’s perfor- mance and progression towards outcomes during the training courses and final D-Net. Results The Pilot Group of sixteen residents (six first year residents of Obstetrics/Gynecology and ten first year residents of Internal Medicine. Tbilisi State Medical University Institute of Postgraduate Medical Education and Conti- nuous Professional Development) and twelve mentors (four Obstetrics/Gynecology mentor and eight Internal Medicine mentors) reported some feedback about 5-Dimensional Electronic Portfolio (5DeP) as a new assessment tool. The mentors found that 5DeP: • is customized to meet the needs of the residency programs; • is easy to understand and efficient to use; • deals with the inherent deficiencies of traditional system of evaluation; • enables assessment within a framework of transparent and declared criteria and learning objectives; • evaluates and stimulates progress towards educational and professional outcomes; • evaluates learning outcomes not easily assessed by other methods; • provides summative assessment of pro- gress and formative evaluation; • improves mentor awareness of student’s need and support required for students; • enhances of interactions between resi- dents and mentors; • focusses on resident’s personal attributes; • motivates mentors to focus on the training objectives; 5 Figure 2. Example of D-Net. The Resident’s excellent performance in all dimensions is visualized by D-Net, except “Self-development”. Module 3 – Interim Portfolio Assessment Form/Final Portfolio Assessment Form (IPAF/FPAF) This module is designed for the Program Director to perform consistent monitoring of training process and provide: • interim portfolio assessment at the end of each residency year • final portfolio assessment at the end of residency course chairing the Experts Panel • report with analysis of resident’s performance and progression towards outcomes during the training courses and final D-Net. Results The Pilot Group of sixteen residents (six first year residents of Obstetrics/Gynecology and ten first year residents of Internal Medicine. Tbilisi State Medical University Institute of Postgraduate Medical Education and Continuous Professional Development) and twelve mentors (four Obstetrics/Gynecology mentor and eight Internal Medicine mentors) reported some feedback about 5-Dimensional Electronic Portfolio (5DeP) as a new assessment tool. The mentors found that 5DeP: • is customized to meet the needs of the residency programs; • is easy to understand and efficient to use; • deals with the inherent deficiencies of traditional system of evaluation; • enables assessment within a framework of transparent and declared criteria and learning objectives; • evaluates and stimulates progress towards educational and professional outcomes; • evaluates learning outcomes not easily assessed by other methods; • provides summative assessment of progress and formative evaluation; Figure 2. Example of D-Net. The Resident’s excellent performance in all dimensions is visualized by D-Net, except “Self-development”. Figure 2. Example of D-Net. The Resident’s excellent performance in all dimensions is visualized by D-Net, except “Self-development”. Z. Bokhua et al. 85 m e d ic a l e d u c a t io n iSSn 2413-6077. iJmmR 2020 Vol. 6 issue 1 • can accommodate evidence of learning from a range of different contexts; • allows a range of learning styles to be used according to the preferences of the student; • provides a model for lifelong learning and continuing professional development. The students found that 5DeP: • allows to understand learning objectives, using specific targets; • enables the consolidation of the connection between theory and practice, and apply abstract principles to practical contexts; • recognizes and encourages the autono- mous and reflective learning that is an integral part of professional education and development; • enhances self-learning and self-develop- ment; • enables to correct errors and remedies deficiencies; • focusses on resident’s personal attributes; • evaluates and stimulates progress towards educational and professional outcomes; • enhances of interactions between resi- dents and mentors; • motivates both residents to focus on the training objectives; • helps to update of knowledge and skills in existing and new areas of practice; • increases competence in a wider context with benefits to both professional and personal roles; • improves organizing skills; • helps to estimate potential to progress to other levels or courses. Discussion The use of portfolio as an assessment instrument enables residents and teachers to engage in a process of learning through assessment. The use of portfolio broadens the scope of assessment and introduces several educational benefits [7, 8, 9, 10]: • Contribution to: � the assessment of learning outcomes � the provision of evidence of performance � the representation of evidence collected over a period � resident progression towards the learning outcomes � summative and formative assessment • Focus on personal attributes • Enhancement of interactions between residents and teachers • Stimulation of the use of reflective stra- tegies • Encouragement of a holistic and integrative approach to medical practice. The portfolio assessment remains relatively unknown among clinical teachers and residents [11, 12, 13, 14, 15]. Conclusions 5-Dimensional Electronic Portfolio (5DeP) have been recognized as an extremely effective assessment too that could provide motivation to study, can assist students in forming positive attitudes toward learning, to enable students to individualize and personalize their learning by supporting and encouraging active parti- cipation. Conflict of interest The authors declare no conflict of interest. Funding This work was supported by United Nations Population Fund (UNFPA). Author’s contribution Zaza Bokhua – contributed to the data curation, formal analysis, funding acquisition, investigation, resources and writing original draft; Kakhaber Chelidze – contributed to the con ceptualization, formal analysis, metho- dology, project administration, supervision, visualization, validation and review and editing of original draft; Ketevan Ebralidze – contributed to the formal analysis, and review and editing of original draft. П’ЯТИВИМІРНЕ ЕЛЕКТРОННЕ ПОРТФОЛІО (5DEP) ЯК ІНСТРУМЕНТ ОЦІНКИ КОМПЕТЕНТНОСТЕЙ У ПІСЛЯДИПЛОМНОМУ МЕДИЧНОМУ НАВЧАННІ *Z. Bokhua1, K. Chelidze1,2, K. Ebralidze1 1 – TBILISI STATE MEDICAL UNIVERSITY (TSMU), TBILISI, GEORGIA 2 – THE FIRST UNIVERSITY CLINIC OF TSMU, TBILISI, GEORGIA Вступ. Сучасні виклики у системі охорони здоров’я, зумовлені постійними її змінами, вимагають новітніх методів навчання та інструментів їх оцінки. Компетентнісний підхід до навчання, котрий було взято за основу сучасних стандартів професійної медичної освіти, потребує відповідних методів оцінки, таких як портфоліо. Портфоліо – це інструмент, сконцентрований на простежуванні прогресу Z. Bokhua et al. 86 m e d ic a l e d u c a t io n iSSn 2413-6077. iJmmR 2020 Vol. 6 issue 1 набутих знань особою, що навчається, який дозволяє зацікавити і залучити до процесу не лише резидента, а й викладача. Мета – поділитися успішним досвідом застосування 5-вимірного електронного портфоліо (5DeP) у якості інструменту оцінки набуття професійних компетентностей у пілотній групі. Методи. Пілотну групу склали 16 резидентів (шість з них – резиденти першого року навчання зі спеціальності Акушерство/гінекологія, ще десять – першого року навчання зі спеціальності Внутрішня медицина Інституту післядипломної медичної освіти та безперервного професійного розвитку, Тбіліський державний медичний університет) та дванадцять менторів-наставників (чотири з акушерства/гінекології та вісім – з Внутрішньої медицини). Учасники пілотного проекту ділилися відгуками щодо оцінки застосування 5-вимірного електронного портфоліо (5DeP) у якості інструменту оцінки набуття професійних компетентностей Результати. Отримані у процесі застосування 5-вимірного електронного портфоліо (5DeP) відгуки та зворотній зв'язок від учасників пілотної групи (менторів та резидентів) продемонстрували ефективність програми як інструменту оцінки набуття професійних компетентностей. Портфоліо забезпечує прозорі та чіткі критерії оцінки знань та компетентностей, забезпечує модель безперервного професійного розвитку на “навчання протягом усього життя”, підвищує рівень як професійних так і особистісних компетентностей, покращує організаційні навики. Висновки. 5DeP – надзвичайно ефективний інструмент оцінки набутих знань та компетентностей. КЛЮЧОВІ СЛОВА: компетентнісно орієнтована медична освіта (CBME); післядипломна освіта; електронне портфоліо; зворотній зв’язок. Information about the authors Zaza Bokhua – MD, Ph.D., Associate Professor of Reproductive Health Department, Director of Institute of Postgraduate Medical Education and Continuous Professional Development, Tbilisi State Medical University, Tbilisi, Georgia. ORCID https://orcid.org/0000-0003-4974-4513, e-mail: z.bokhua@tsmu.edu Kakhaber Chelidze – MD, Ph.D., Professor, Chair of Internal Medicine in The First University Clinic of TSMU, The Head of Department of Internal Medicine, Tbilisi State Medical University, Tbilisi, Georgia. ORCID https://orcid.org/0000-0002-3906-9967, e-mail: k.chelidze@tsmu.edu Ketevan Ebralidze – MD, Ph.D., Director of Internal Medicine Residency Program, Head of the Organizational Service of Institute of Postgraduate Medical Education and Continuous Professional Development, Invited Teacher of Department of Physiology, Tbilisi State Medical University, Tbilisi, Georgia. ORCID https://orcid.org/0000-0003-3880-3733, e-mail: k.ebralidze@tsmu.edu References 1. NEJM Knowledge+ Team. What Is Competency- Based Medical Education? NEJM Knowledge+. June 15th, 2017. doi: 10.1093/clinchem/63.12.1912 2. Holmboe ES, Sherbino J, Long DM, Swing SR, Frank JR, International CBME Collaborators. The role of assessment in competency-based medical edu- cation. Medical teacher. 2010 Aug 1;32(8):676-82. doi: 10.3109/0142159X.2010.500704 3. ten Cate O, Scheele F. Competency-based post graduate training: can we bridge the gap between theory and clinical practice? Academic Medicine. 2007 Jun 1;82(6):542-7. doi: 10.1097/ACM.0b013e31805559c7 4. Challis M. AMEE Medical Education Guide No. 11 (revised): Portfolio-based learning and assessment in medical education. Medical teacher. 1999 Jan 1;21(4):370-86. doi: 10.1080/01421599979310 5. David MF, Davis MH, Harden RM, Howie PW, Ker J, Pippard MJ. AMEE Medical Education Guide No. 24: Portfolios as a method of student assessment. Medical Teacher. 2001 Jan 1;23(6):535-51. doi: 10.1080/01421590120063321 6. Smith K, Tillema H. Clarifying different types of portfolio use. Assessment & Evaluation in Higher Education. 2003 Dec 1;28(6):625-48. doi: 10.1080/0260293032000130252 7. David MF, Davis MH, Harden RM, Howie PW, Ker J, Pippard MJ. AMEE Medical Education Guide No. 24: Portfolios as a method of student assessment. Medical Teacher. 2001 Jan 1;23(6):535-51. doi: 10.1080/01421590120090952 8. Davis MH, Ponnamperuma GG. Portfolios, projects and dissertations. In:Dent JA & Harden RM, eds. A practical guide for medical teachers. London: Elsevier Churchill Livingstone; 2006. Z. Bokhua et al. 87 m e d ic a l e d u c a t io n iSSn 2413-6077. iJmmR 2020 Vol. 6 issue 1 9. Roberts C, Newble DI, O'Rourke AJ. Portfolio- based assessments in medical education: are they valid and reliable for summative purposes? Medical education. 2002 Oct 1;36(10):899-900. doi: 10.1046/j.1365-2923.2002.01288.x 10. Davis MH, Ben-David MF, Harden RM, Howie P, Ker J, McGhee C, Pippard MJ, Snadden D. Portfolio assessment in medical students’ final exa- minations. Medical Teacher. 2001 Jan 1;23(4):357-66. doi: 10.1080/01421590120063349 11. Gómez SS, Ostos EM, Solano JM, Salado TF. An electronic portfolio for quantitative assessment of surgical skills in undergraduate medical education. BMC medical education. 2013 Dec 1;13(1):65. doi: 10.1186/1472-6920-13-65 12. Heeneman S, Driessen EW. The use of a portfolio in postgraduate medical education–reflect, assess and account, one for each or all in one? GMS journal for medical education. 2017;34(5). 13. Hawkins RE, Welcher CM, Holmboe ES, Kirk LM, Norcini JJ, Simons KB, Skochelak SE. Imple- mentation of competency‐based medical education: are we addressing the concerns and challenges? Medical Education. 2015 Nov;49(11):1086-102. doi: 10.1111/medu.12831 14. Fernsten L, Fernsten J. Portfolio assessment and reflection: Enhancing learning through effective practice. Reflective Practice. 2005 Jan 1;6(2):303-9. doi: 10.1080/14623940500106542. 15. Koole S, Dornan T, Aper L, Scherpbier A, Valcke M, Cohen-Schotanus J, Derese A. Does reflec- tion have an effect upon case-solving abilities of undergraduate medical students? BMC Med Educ. 2012;12(1):75. doi: 10.1186/1472-6920-12-75 Received 26 Mar 2020; revised 09 Apr 2020; accepted 12 June 2020. This is open access article distributed under the Creative Com- mons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Z. Bokhua et al.