Pak J Ophthalmol. 2021, Vol. 37 (4): 394-398 394 Original Article Comparison of Online Assessment with Traditional Assessment of Ophthalmology Students: Experience from Public Sector Medical College in Pakistan Muhammad Hammad Ayub 1 , Muhammad Hassaan Ali 2 , Uzma Hamza 3 , Kashif Jahangir 4 Department of Ophthalmology, 1-4 Allama Iqbal Medical College, Jinnah Hospital, Lahore ABSTRACT Purpose: To perform online assessment of undergraduate ophthalmology students and compare the online examination results with conventional in-person examination during COVID-19 pandemic. Study Design: Comparative, cross-sectional study. Place and Duration of Study: Department of Ophthalmology, Allama Iqbal Medical College, Lahore, Pakistan. From October 2019 to December 2020. Methods: We used online Google Assessment Forms to administer the online ophthalmology send-up examination to 4th-year MBBS students. The exam consisted of 50 test items with varying difficulty levels and pictures were incorporated into them. The examination results were compared with the conventional face-to-face send-up ophthalmology examination conducted a year ago, which consisted of 50 multiple-choice questions (MCQs). Results: There were 50 questions in the online examination with categorization into C1:C2:C3 levels of cognition. A total of 336 students submitted their responses for online assessment with a mean score of 36.28 out of 50 marks (73.0%) (Range: 7 – 45 marks). C1, C2, and C3 questions were correctly answered by 95.3%, 45.1%, and 11.0% of the students, respectively. The conventional examination was taken by 326 students and consisted of 50 MCQs with C1:C2:C3 questions. The online exam yielded more mean scores than the in-person exam (73.0% versus 56.0%, p-value: 0.001). The rate of failure with scores less than 50% marks was significantly higher in students appearing in conventional exam versus online exam (16.6% versus 1.5% respectively, p-value: 0.0412). Conclusion: Online assessment of undergraduate ophthalmology students is a possible alternative to the conventional examinations during this pandemic. Key Words: Ophthalmology, COVID-19 Pandemic. Medical education. How to Cite this Article: Ayub MH, Ali MH, Hamza U, Jahangir K. Online Assessment of Undergraduate Ophthalmology Students and Its Comparison with Traditional Assessment: Experience from Public Sector Medical College In Pakistan. Pak J Ophthalmol. 2021, 37 (4): 394-398. Doi: 10.36351/pjo.v37i4.1330 Correspondence: Muhammad Hammad Ayub Allama Iqbal Medical College Jinnah Hospital, Lahore Email: drhammadayub@gmail.com Received: September 01, 2021 Accepted: September 23, 2021 INTRODUCTION COVID-19 pandemic has impactedall spheres of human life. But its impact on education and particularly medical education, has been tremendous. 1,2 Although e-learning and telemedicine/ telehealth concepts are not new, these ideas are pretty OPEN ACCESS mailto:drhammadayub@gmail.com Muhammad Hammad Ayub, et al 395 Pak J Ophthalmol. 2021, Vol. 37 (4): 394-398 novel from Pakistan's perspective. Our medical education has evolved during this era of the COVID- 19 pandemic and more contemporary concepts are shaping into reality.Online teaching, which was suddenly thrust upon teachers and students alike, is now becoming a norm. The concepts of e-learning are being improvised constantly to replace face-to-face learning. 3-5 This has led to another challenge of planning and rapidly implementing assessment methods to determine whether the learning objectives have been achieved successfully. 6-9 The medical students of 4th-year MBBS, in the subject of ophthalmology, are traditionally evaluated in their finalprofessional examination both in written, in the form of multiple-choice questions (MCQs) and short essay questions (SEQs) and clinically in the form of objectively structured practical examination (OSPE). The Department of Ophthalmology devised an online formative assessment tool that could cover both of these assessment modalitiesto a greater extent.We also compared the online assessment results with the traditional in-person examination results given a year ago to determine the difference between the outcomes of these two different assessment methods. The rationale of the study was to find out the feasibility of online assessment in our set-up and to find out if it can be used as an alternative in the coming years. METHODS The study was conducted after obtaining its approval from the Ethical Review Board of Allama Iqbal Medical College, Jinnah Hospital, Lahore. We compared two groups of fourth-year MBBS students. One group consisted of 336 students who underwent online examination, and the other group comprised 326 students who attempted on-campus paper-based test. We prepared a question paper for the online examination, which consisted of 50 MCQs, and the candidates were instructed to choose one best option from a given list of five options. We highlighted the correct answer in the question bank to enable automatic grading of the responses. Care was taken to turn off the option for automatic release of grades and visibility of correct answers at the end of response submission by the examinee. AIT-expert invigilator was assigned to be available to troubleshoot any IT problem during the actual examination. To avoid any technological hiccup during the real exam, we did not install any external plug-in in the Google Form and resorted to manually turning off response acceptance after the end of the designated examination time. In this way, no student could submit any response once the time for the exam was over. Secondly, we turned on the email collection of responses and did not allow multiple submissions from the same candidate. After devising the form, we test run the assessment form and administered it to the examiners to see the outlook of the document and its contents and improve its utility before the final examination.We reduced the time of this onlineexam, and the candidates had to answer 50 questions in 45 minutes instead of one hour, which was required for an in- person on-campus examination using paper. This step also ensured that the candidates focused on their exam and did not resort to unfair means due to a perceived shortage of time for attempting the examination. There were 50 questions in the online send-up examination with categorization into C1: C2: C3 (C1 = recall of knowledge, C2 = interpretation, C3 = application of knowledge) levels of cognition, which were in the ratio of 50: 30: 20 (Table 1). The Google Forms self- analyzed the responses and gave us the mean score of candidates and individual breakdown of correct answers to each question. We could individually assess each student's performance and provide focused feedback depending on his responses. All the students were eventually informed about their final results with focused feedback on their performance through email. The online examination experience was very smooth, and no student faced any connectivity issue or inability to submit the response before finishing time. The data recorded online could be transcribed into Microsoft Excel and various other statistical software for subsequent research and publications. The conventional examination consisted of 50 MCQs with categorization into C1, C2 and C3. The questions were shuffled in four different sets of paper. Each set was rotated in a set pattern to avoid repeating the same paper to the students sitting next to each other. The examination was held under direct invigilation, and the students were seated with ample spacing between the two adjacent students. All the data were entered and analyzed usingStatistical Package for Social Sciences (SPSS version 25.0, IBM Statistics Inc, Chicago, IL, USA). Numerical variables werecalculated in the form of Online Assessment of Ophthalmology Students and Its Comparison with Traditional Assessment Pak J Ophthalmol. 2021, Vol. 37 (4): 394-398 396 mean ± SD, whereas descriptive variables were evaluated in the form of frequencies and percentages. The results of the two examinations were compared using a student's t-test with a p-value < 0.05 considered significant. RESULTS A total of 336 students submitted their online responses for evaluation. The average score of the online exam was found to be 36.28 marks out of 50 ( 73.0%) with a range of 7 – 45 marks. Most of the students (95.3%) correctly answered C1 questions, whereas C2 and C3 questions were correctly attempted by 45.1 and 11.0% of the students, respectively. A total of 326 students appeared in the conventional examination. Question distribution into C1: C2: C3 levels was in the ratio of 34:36:30 respectively (Table 1). The detailed marks of the candidates in both formats of the examinationare given in Table 1. Table 1: Distribution of Questions and Scores Obtained by Students in Online and in-Person Examination Variable In-Person Exam (n,%) Online Exam (n,%) p-value Cognition Levels of Questions C1 17, 34.0 25, 50.0 0.075 C2 18, 36.0 15, 30.0 0.542 C3 15, 30.0 10, 20.0 0.613 Scores Obtained < 50% 54, 16.6 5, 1.5 0.0412 51-60% 129, 39.6 24, 7.1 0.051 61-70% 106, 32.5 101, 30.1 0.656 71-80% 35, 10.7 154, 45.8 0.0012 >80% 2, 0.60 52, 15.5 0.0042 Mean Score 56.0% 73.0% 0.001 The online exam yielded more mean scores than the in-person exam, possibly owing to a higher percentage of simple recall C1 questions (50.0% versus 34.0%, respectively). Similarly, marks distribution of both the exams revealed mode values as following: maximum number of students (129, 39.6%) scored marks in the range of 51 – 60% in face-to-face exam whereas the majority of the students (154, 45.8%) scored marks between 71 – 80% in the online format. Compared with the online format, the percentage of students who scored more than 70% in the in-person exam was significantly less than those taking an online exam, 11.30% versus 61.3% (p-value: 0.001). Figures 1 & 2 show the grading of individual questions and overall analysis of the responses obtained by Google Forms. Lastly, the rate of failure with scores less than 50% marks was significantly higher in students appearing in conventional exam versus online exam (16.6% versus 1.5% respectively, p-value: 0.0412). Figure 1: Test Question Analyzed by Google Forms with Chart Depiction of Correct Responses by Candidates. Figure 2: Automatic Analysis and Grading of Responses by Google Forms. DISCUSSION We successfully met the study's primary objectives to investigate an easily accessible software for administering the online examination and compare the results with an in-person paper-based test. We found Google Forms to be a potentially handy tool in giving assessments in a resource-deficient setting where it is challenging to purchase other expensive examination software. Since the coronavirus pandemic, medical colleges were faced with an urgent need to modify their teaching and assessment, which also received significant attention in the recent medical literature. 10,11 This led to innovations in various education and assessment tools for administering examinations, including shifting to online lectures, online examination using different virtual assessment Muhammad Hammad Ayub, et al 397 Pak J Ophthalmol. 2021, Vol. 37 (4): 394-398 platforms, and online invigilation using multiple software. 12-15 Imperial College London pioneered these online examinations and administered home-based online tests. 16 They also introduced the idea of 'open book examination'. The candidate was given access to normal reference values for various parameters but was given questions that required higher levels of cognition and application of knowledge at multiple steps to reach a final correct answer. 17 Secondly, they kept these exams time-pressured and gave 60 seconds to solve a clinical scenario. 17 These problematic questions could not be answered by simple google search and could only be attempted correctly if the candidate had a thorough understanding of the tested topic. We also employed a similar technique in our examination in which we administered C3 level questions. These questions were primarily clinical scenarios and tested the in-depth knowledge of the students. However, since the students had only attended online lectures, the proportion of C3 questions was relatively low (20.0%). This also explains the trend of a higher score in the online examination because of a higher percentage of simple recall C1 questions (50.0%). We administered more C1 questions intentionally in the online test due to the students' lesser clinical experience during the pandemic. The online examination consisted of various pictures of different clinical conditions. We believe that ability to incorporate multiple types of questions like pictures, videos and audios of various clinical signs in the examination, which can help the examiners assess the students from different aspects, shows one of the most significant advantages and strengths of the online system. This functionality of online display of various visual content can help administer objectively structured clinical or practical examinations (OSCE or OSPE) for undergraduate medical students. This could yet be another possible reason for a higher score in the online version, as the visual content might have been more straightforward for the students to recall. Furthermore, we could not rule out the possibility of students using unethical means or working in groups to solve the paper mutually and eventually scoring higher scores in the online examination. This deficiency of online examination can be resolved by incorporating invigilation through various methods, including asking the students to solve the questions in front of a camera by taking the help of ZOOM or any other similar application. 18,19 Nevertheless, invigilating a class of 300 medical students poses another challenge and requires significant capital and human resource. A recent study has reported the results of using ZOOM application for video monitoring and recording of students during online assessment. 20 They divided their class into multiple groups of nine students, each under the invigilation of one proctor. We also propose that the students be dividedinto a cohort of at least ten studentsobserved under direct invigilation by teams of multiple proctors. Needless to say that it will require a lot of trained IT workforce and capital resources. Unfortunately, most public-sector medical colleges still lack fully equipped IT departments. Clinical specialty departmentsare left with no choice but to manage all the IT-related planning and execution under the meager resources available. At least 30 invigilators with thoroughly modern computer systems must invigilate a class of 300 students subdivided into groups of 10 students. Only then can we improve the reliability of students' scores in the online examination. Although it was possible to assessthe type of responses to various test items and individualized feedback to the students could be provided using this digital platform, there are certain limitations of this research. Firstly, we could not assess the use of unfair means by the students during examinations. Secondly, the difficulty level of the questions could not be evaluated. It was a single center study. Lastly, online assessment poses multiple challenges. CONCLUSION We conclude that online assessment is possible through free online Google Forms assessment tools. The software automatically analyzes the results and helps in rapid paper marking and result compilation. Human resource training for online invigilation and allocation of funds to establish fully equipped IT departments still remain the most critical limitations for public sector medical colleges. Ethical Approval The study was approved by the Institutional review board/Ethical review board (136/16/09/2021/S2 ERB.) Conflict of Interest Authors declared no conflict of interest. Online Assessment of Ophthalmology Students and Its Comparison with Traditional Assessment Pak J Ophthalmol. 2021, Vol. 37 (4): 394-398 398 REFERENCES 1. Ahmed H, Allaf M, Elghazaly H. COVID-19, and medical education. Lancet Infect Dis. 2020; 20 (7): 777–778. 2. Arja SB, Wilson L, Fatteh S, Kottathveetil P, Fateh A, Arja SB. Medical education during COVID-19: Response at one medical school. J Adv Med Educ Prof. 2021; 9 (3): 176. 3. Metreveli L, Gabunia L, Akhvlediani L, Baramidze L, Gumbaridze L. Distance Learning in Medical Education: COVID-19 Pandemic-Contingency Plan or a Paradigm Shift? Int J Progress Sci Technol. 2021; 27 (1): 76–82. 4. Succar T, Beaver HA, Lee AG. Impact of COVID-19 pandemic on ophthalmology medical student teaching: educational innovations, challenges, and future directions [published online ahead of print, 2021 Apr 7]. Surv Ophthalmol. 2021; S0039-6257 (21): 00098-9. Doi:10.1016/j.survophthal.2021.03.011. 5. Mishra D, Nair AG, Gandhi RA, Gogate PJ, Mathur S, Bhushan P, et al. The impact of COVID-19 related lockdown on ophthalmology training programs in India - Outcomes of a survey. Indian J Ophthalmol. 2020; 68 (6): 999-1004. Doi: 10.4103/ijo.IJO_1067_20. 6. Rad FA, Otaki F, Baqain Z, Zary N, Al-Halabi M. Rapid transition to distance learning due to COVID-19: Perceptions of postgraduate dental learners and instructors. PLoS One, 2021; 16 (2): e0246584. 7. Elzainy A, El-Sadik A, Al-Abdulmonem W. Experience of e-learning and online assessment during the COVID-19 pandemic at the College of Medicine, Qassim University. J Taibah Univ Med Sci. 2020; 15 (6): 456–462. 8. Magee LA, Pels A, Helewa M, Rey E, von Dadelszen P. Diagnosis, evaluation, and management of the hypertensive disorders of pregnancy. Pregnancy Hypertens An Int J Women's Cardiovasc Heal. 2014; 4 (2): 105–145. 9. Elsalem L, Al-Azzam N, Jum'ah AA, Obeidat N. Remote E-exams during Covid-19 pandemic: A cross- sectional study of students' preferences and academic dishonesty in faculties of medical sciences. Ann Med Surg. 2021; 62: 326–333. 10. Alqudah NM, Jammal HM, Saleh O, Khader Y, Obeidat N, Alqudah J. Perception and experience of academic Jordanian ophthalmologists with E-Learning for undergraduate course during the COVID-19 pandemic. Ann Med Surg. 2020; 59: 44–47. 11. Chatziralli I, Ventura C V, Touhami S, Reynolds R, Nassisi M, Weinberg T, et al. Transforming ophthalmic education into virtual learning during COVID-19 pandemic: a global perspective. Eye, 2021; 35 (5): 1459–1466. 12. Sahi PK, Mishra D, Singh T. Medical education amid the COVID-19 pandemic. Indian Pediatr. 2020; 57 (7): 652–657. 13. Sabzwari S. Rethinking assessment in medical education in the time of COVID-19. Med Ed Publish, 2020; 9 (1): 80. https://doi.org/10.15694/mep.2020.000080.1 14. Farooq F, Rathore FA, Mansoor SN. Challenges of online medical education in Pakistan during COVID-19 pandemic. J Coll Physicians Surg Pak. 2020; 30 (6): 67–69. 15. Rajab MH, Gazal AM, Alkattan K. Challenges to online medical education during the COVID-19 pandemic. Cureus, 2020; 12 (7): e8966. doi:10.7759/cureus.8966 16. Mian A, Khan S. Medical education during pandemics: a UK perspective. BMC Med. 2020; 18 (1): 100. Doi: 10.1186/s12916-020-01577-y. 17. Monaghan AM. Medical Teaching and Assessment in the Era of COVID-19. J Med Educ Curric Dev. 2020; 7: 2382120520965255. 18. Shih KC, Chan JC, Chen JY, Lai JS. Ophthalmic clinical skills teaching in the time of COVID-19: a crisis and opportunity. Med Educ. 2020; 54 (7): 663– 664. 19. Kaup S, Jain R, Shivalli S, Pandey S, Kaup S. Sustaining academics during COVID-19 pandemic: the role of online teaching-learning. Indian J Ophthalmol. 2020; 68 (6): 1220. 20. Fatima SS, Idrees R, Jabeen K, Sabzwari S, Khan S. Online assessment in undergraduate medical education: Challenges and solutions from a LMIC university. Pakistan J Med Sci. 2021; 37 (4): 945. Authors’ Designation and Contribution Muhammad Hammad Ayub; Associate Professor of Ophthalmology: Concept, Data Analysis, Manuscript Writing, Final Critical Review. Muhammad Hassaan Ali; Senior Registrar Ophthalmology: Data Collection, Data Analysis, Manuscript Writing. Uzma Hamza; Assistant Professor of Ophthalmology: Concept, Data Analysis, Final Critical Review. Kashif Jahangir; Assistant Professor of Ophthalmology: Data Analysis, Literature Review, Final Critical Review. .…  ….