1 |JISAE (Journal of Indonesian Student Assessment and Evaluation) |Volume 9 Number 1 https://doi.org/10.21009/JISAE JISAE (Journal of Indonesian Student Assessment and Evaluation) ISSN : P-ISSN: 2442-4919│E-ISSN: 2597-8934 Vol 9 No 1 (2023) Website : http://journal.unj.ac.id/unj/index.php/jisae COMPUTER-BASED STANDARDIZED TESTING (CBT) SUMMATIVE ASSESSMENT: ONE INSTITUTION EXPERIENCE Naila Rajiha1, UIN Salatiga Setia Rini2 UIN Salatiga ABSTRACT One tool for measuring the result of learning is the summative assessment using Computer-Based Standardized Testing (CBT). The assessment process and the issues that arise in holding are fascinating to be discussed. This research was focused on: How the CBT final examination test is being implemented in a school in Pati Regency. This study employs descriptive case studies as one of the methods of qualitative research. Interviews, observation, and documentation are the data collection techniques for this study's primary and secondary sources. The researchers used a triangulation data source and triangulation technique to check the data integrity. The authors draw the following conclusions: Students' learning evaluation processes are through various activity procedures, such as simulating, implementing, overseeing evaluations, processing data, and using evaluation results. Moreover, observing student needs and readiness in all forms is an obligation applied in conducting CBT. Keywords: Summative Assessment; CBT Address for Correspondence: 1nailarajiha@gmail.com 2setiarini.setia@gmail.com INTRODUCTION It is very crucial to understand why teachers do evaluate their students. Because they must make decisions regarding these students, teachers frequently assess language learners. They can make wise decisions based on the evidence they gather. The unanimity of knowledge, skills, and attitudes manifested in a set of intelligent and responsible actions that a teacher has to assume the position of teacher as a professional can be interpreted as the teacher's competence (Afriadi & Dudung, 2021). For instance, teachers might need to determine whether every student has understood the material they just taught by them. Based on Green (2013), When evaluating the students in their classes, teachers do so for two main reasons. By ensuring that students are making progress, one goal is to enhance learning. This allows them to choose whether to offer more assistance, try a different explanation, or use different materials when students find something tricky or offer more difficult activities when they are prepared for them. The other goal is to assess how well students have learned a course's material so that parents, administrators, or other educational authorities can be informed. Usually, this entails selecting grades or scores. Formative assessment, also referred to as assessment for learning, is the first of these purposes. Summative evaluation, also known as learning assessment, is the second. Summative assessment measures learning, whereas formative assessment provides feedback to enhance learning (Ismail et al., 2022) The phenomenon of rapidly advancing technology in the field of education served as the driving force behind this research. Students generally welcome using technology in the learning process but not in establishing exams (Karibyan & Sabnis, http://journal.unj.ac.id/unj/index.php/jisae mailto:nailarajiha@gmail.com mailto:setiarini.setia@gmail.com 2 2021). Some higher education institutions and their faculty are urged to adopt new technologies as they become available to provide students of the digital age—often referred to as "digital natives"—with more learning opportunities. Green computing strategies such as computer-based testing (CBT) are used to reduce paper consumption. This approach would make it easier to collect individual and class assessment data, meeting the needs of students, the institution, and accreditors. For example, a search revealed that students preferred CBT over PPT and were more motivated to perform another CBT than another PPT. (Piaw Chua, 2012). Another example, the majority of courses in the veterinary professional curriculum are switching from traditional paper-based exams to electronic-based exams. In order to determine the effect of testing methodology on exam performance in a veterinary physiology course, a controlled trial was created and carried out (Washburn et al., 2017). CBT has spread to be implemented also among some higher schools. According to the findings of some studies (Boevé et al., 2015; Mason & Andrews, 2014; Tseng et al., 2023) CBT benefits include cost-effective administration, ease of administration, increased accuracy, immediate scoring and reporting, and flexible test scheduling and location. Furthermore, computer technology has persisted into the twenty-first century as an important and versatile tool for communication and education. Students who do not have a positive attitude toward computer-assisted English learning will be at a distinct disadvantage (Hosseini et al., 2014). There was lack evidence that using computer-based testing improved student exam performance (Pawasauskas et al., 2014). Bloom et al. (2018) found in their study that an examination of exam grades revealed that there was no consistent difference in performance between the paper and computer-based formats. Faculty and students quickly adapted to computer-based testing. There was limited evidence that using computer-based testing improved student exam performance. The inconsistency of the present studies about CBT encouraged the researchers to do related research in a school around Pati regency. Here the researchers limit the study to focus on scrutinizing the implementation of CBT summative assessment. Besides, most studies in the domain of assessment examined the effects of the summative and formative CBT assessments on the main skills (reading, speaking, writing, and listening) and they did not pay much attention to the reality and challenge in application; therefore, to cover the existing gap this research posed a question How is the CBT Final Examination Test being implemented in a school in Pati Regency? that including three parts discussion; describing the process of assessing CBT-based learning, identifying the advantages and disadvantages of CBT-based learning assessment, and identifying the recommendation of Implementation standard of CBT-based learning assessment in the research site. METHOD This study uses Descriptive Case Studies. The goal set by the researchers is to describe the data as they occur (Zainal, 2007). Observations here are to know the infrastructure, school environment, and learning activities. Documentations include syllabus, lesson plan, question paper, question cards, assessment/report and photos of teaching and learning activities at the research site. Interview guidelines are intended to a curriculum staff, ICT Team or the committee and two students. . 3 |JISAE (Journal of Indonesian Student Assessment and Evaluation) |Volume 9 Number 1 RESULTS AND DISCUSSION ICT Team View Final assessment of the odd semester 2022 2023 in this research site implements a semi-online CBT mode. It is where the data bank starts, with student data, subject data, and question data stored on the madrasah's local server. Considerations for using a semi-online mode and not being fully online include saving on the server (hosting) rental costs, which will be very high considering the large number of examinees, and minimizing server downtime, which will certainly disrupt the comfort of the exam, especially for the examinees. With semi-online student quotas, it's also safe because it doesn't require an internet quota. After all, the exam route uses Wi-Fi access, which directly leads to a local server. Standard Operating Procedure (SOP) for preparing for CBT by a proctor including 1) Ensuring local network infrastructure to access the school server with zero errors 2) Ensuring the data bank is complete (students, subjects, schedules, and questions) 3) The subject by teacher sends questions and worksheets to the administration, which then sends them to the proctor for entry into the application. Moreover, SOP during CBT running include 1) Proctor make sure the network connection is stable 2) The proctor monitors the participant during the exam 3) Release tokens for exam participants (each subject is locked using a token, tokens change every 5 minutes, and if there are students who are found to be cheating such as opening an application other than the exam page, participant browsing will be removed from the exam page and requires a token to re-enter) 4) After the exam subject ends, the proctor downloads the results of student work in the form of a list of numerical and graphic values as well as an analysis of student achievement items. 5) Facilitate follow-up for students who are unable to attend scheduled exams. The recommendation for the next CBT as follows: 1) There were several subjects collected on the day of the exam, so the proctor had to enter the questions on that day as well, resulting in a delay in the exam for students 2) Some of exam room is not representative, the light is too dim and the room temperature is too hot 3) It is necessary to upgrade/upgrading the school server considering the very large data circulation, for example, 1 exam day consumes at least 400,000 data records 4) There are so many subjects being tested that it takes many days to complete the test, and students look very bored in the last days (hopefully next year the students will utilize Kurikulum Merdeka so that the written test can be replaced by a project). Curriculum staff View Along with the progress of information technology and the demands of the times, the school needs to improve the quality of the learning process, including conducting assessments using the CBT model. Implementing the CBT learning system is certainly not as easy as turning the palm because this school started pioneering this system starting in 2018, starting with the preparation of human resources for educators and education staff, infrastructure, and mindsets for changing a system, so that by going through several trials to get maximum results, this school have started implementing this CBT-based learning assessment officially in 2020, even though it cannot yet cover all class levels. The advantages and disadvantages of CBT-based learning assessment; in implementing this CBT-based assessment, there are clear deficiencies. Here are some of the advantages. For school, educators, and education staff; 1) Archiving question files, question papers, scoring rubrics, and other devices is more orderly and easier when needed in various ways, such as when used in accreditation, evaluating 4 the performance of the head of the school, or making it easier for teachers to find assessment files 2) Facilitate and speed up the process of processing grades for daily assessments, midterm assessments, or end-of-semester assessments so that deadlines for inputting into report cards are easier and more efficient 3) Teachers are used to using assessment applications so that the expectations of school in the teaching and learning process are also expected to leave the classic learning model which is only centralized from the teachers. Secondly, for students; 1) Learners more easily and quickly access the results of the assessment activities carried out by the teacher 2) Equip students to get used to taking advantage of technological advances, so that when they proceed to a higher level, they become accustomed to it. There are some deficiencies; For school, educators, and education staff as follows; 1) Some educators and education staff at school with various backgrounds certainly cannot run and adapt to the application instantly, this is because there is a the minority teachers are incompetence at using computers or laptops 2) In the context of preparing question data should be in file form so that it can be sent more quickly to administration, of course, there are also some obstacles, such as some teachers who do not have laptop or computer facilities at their homes 3) Some of the teachers at this school are old enough, so it is rather difficult to change the existing habits, namely the method from the paper model to the CBT model. However, for students; Most of the students are Islamic boarding school students, where one of the rules is not allowed to carry cell phones, so it slows down the CBT-based assessment succession program of learning. Identifying the recommended standardization of the application of CBT-based learning assessment in the school include; 1) The standards that have been formulated when compiling the school work program, one of the targets of which is CBT-based assessment learning, begin with the preparation of existing facilities and infrastructure, both in the use of computers and laptops in the computer lab and in strengthening the internet network in each class. The active role of the teachers is indeed a determining factor in the application of this system; the more disciplined they are in preparing questions and sending grade-processing questions, the faster and more successful this program is. Apart from the teaching staff factor, another factor that makes it vital is the addition of ICT teachers who understand the software and hardware supporting this CBT. 2) Considering that this school is already in its second year of using digital report cards (RDM), then it is already a realistic target that all teachers who teach in tenth grade and eleventh grade must be proficient and accustomed to using the application. Students View 01 There are three statements; 1) the CBT exam process that was held last semester helped students work on the final semester in modern, but it didn't go well enough. However, the website page that was originally agreed to be used as the CBT exam subject to change. Second, the strengths of CBT: it is able to help the students in doing the final examination using technology, it also can reduce the waste of paper, and can prevent cheating by students because when students want to open a new tab, it will automatically exit the question work page. In fact, there are some disadvantages: Lots of errors, especially in loading an image. Sometimes there is experience about delays in loading questions, so the students need to refresh the page many times. 5 |JISAE (Journal of Indonesian Student Assessment and Evaluation) |Volume 9 Number 1 Thirdly, the application of CBT in last final exam was quite well and helped students to answer the questions efficiently, but the network in unpredictable. Hopefully, next midterm using CBT test there is a better signal or network. Students View 02 The first description is last CBT exam process went smoothly, but on the first and second day, there were several problems, and in the end the exam was postponed an hour. However, on the third day and next day, the exam was comfortable. However, sometimes there is a trouble in logging in. The advantage of this exam is that it can make students more time-disciplined because the exam page will automatically finish when the specified amount of time is met. In addition, this also reduces the culture of cheating, which is often done by students because the order of questions has been randomized between students. The drawback is that the test page often comes out if the cell-phone screen accidentally turns off. This put us in a bit of a rut, having to go back and forth asking for tokens from the supervisor. In addition, a network that is not good enough often makes questions unable to be shifted, and spends a lot of time on just a few questions. Finally, pictures often don't appear, and that takes up a lot of students' time because they have to wait for the pictures to appear, especially in arithmetic subjects (Mathematics, Chemistry, and Physics). The last, the implementation of CBT in class is quite good. But in the future, please try to improve the picture quality so that images can appear in each question properly and there are not some mistakes in writing questions and answer options. In addition, hopefully, in the next moment, the network can be better so that students are more comfortable answering the questions. Based on the CBT team view, although the complexity of CBT implementation, the preparedness of the team was extremely great. From the school view, the CBT program was planned some years to get the best implementation. The students view seems they are exiting and engaged to join the next CBT. The students need to complete the exam appropriately hence the risk of technical failure should be anticipated regarding to the necessitates of good protocols. Thus, there are some surprising views from students that they support about the school discipline such as the time management, the honesty, and no opportunity in cheating on the test. Thus, this result modifies the existing studies that CBT has some benefits (Boevé et al., 2015; Mason & Andrews, 2014; Tseng et al., 2023) even though there are some following notes. CONCLUSION From the result above can be concluded that the summative assessment using CBT processes through a variety of activity procedures, such as simulating, implementing, and overseeing evaluations, as well as processing data and using evaluation results. The school should be adapted and be anticipated because CBT have some advantages and a few of disadvantages. The most important point is the school should pay attention to student needs and readiness in all forms is a technique used in learning evaluation using CBT. Recommendation For the other school which want to do CBT test this research indicates that more training and sufficient opportunity practice should be increase both for teachers and 6 students. For further studies the researchers recommend to observe the psychological aspect of the implementation CBT among students such as motivation and anxiety. Acknowledgment The researchers would like to gratefully acknowledge the Rector, Director and the Vice Director of Post Graduate Program, Head of English Department Magister Program, all lecturers of Salatiga State University, the research site, and the participants for the support and grant gave in conducting this research. REFERENCE Afriadi, B., & Dudung, A. (2021). Evaluation of Teaching Skill Practice Programs, in The State University Education Development Institution Jakarta. 7(2). https://doi.org/10.21009/JISAE Bloom, T. J., Rich, W. D., Olson, S. M., & Adams, M. L. (2018). Perceptions and performance using computer-based testing: One institution's experience. Currents in Pharmacy Teaching and Learning, 10(2), 235–242. https://doi.org/10.1016/j.cptl.2017.10.015 Boevé, A. J., Meijer, R. R., Albers, C. J., Beetsma, Y., & Bosker, R. J. (2015). Introducing computer-based testing in high-stakes exams in higher education: Results of a field experiment. PLoS ONE, 10(12). https://doi.org/10.1371/journal.pone.0143616 Green, A. (2013). Washback in language assessment. International Journal of English Studies, 13(2), 39–51. https://doi.org/10.6018/ijes.13.2.185891 Hosseini, M., Abidin, M. J. Z., & Baghdarnia, M. (2014). Comparability of Test Results of Computer based Tests (CBT) and Paper and Pencil Tests (PPT) among English Language Learners in Iran. Procedia - Social and Behavioral Sciences, 98, 659–667. https://doi.org/10.1016/j.sbspro.2014.03.465 Ismail, S. M., Rahul, D. R., Patra, I., & Rezvani, E. (2022). Formative vs. summative assessment: impacts on academic motivation, attitude toward learning, test anxiety, and self-regulation skill. Language Testing in Asia, 12(1). https://doi.org/10.1186/s40468-022-00191-4 Karibyan, A., & Sabnis, G. (2021). Students' perceptions of computer-based testing using ExamSoft. Currents in Pharmacy Teaching and Learning, 13(8), 935–944. https://doi.org/10.1016/j.cptl.2021.06.018 Mason, E. C., & Andrews, G. (2014). The use of automated assessments in internet- based CBT: The computer will be with you shortly. Internet Interventions, 1(4), 216–224. https://doi.org/10.1016/j.invent.2014.10.003 Pawasauskas, J., Matson, K. L., & Youssef, R. (2014). Transitioning to computer- based testing. Currents in Pharmacy Teaching and Learning, 6(2), 289–297. https://doi.org/10.1016/j.cptl.2013.11.016 Piaw Chua, Y. (2012). Effects of computer-based testing on test performance and testing motivation. Computers in Human Behavior, 28(5), 1580–1586. https://doi.org/10.1016/j.chb.2012.03.020 7 |JISAE (Journal of Indonesian Student Assessment and Evaluation) |Volume 9 Number 1 Tseng, W. Y., Hsiang-Hua Lai, E., Wang, T. M., Wang, C. Y., Lin, T. F., Lin, L. D., & Tsai, Y. L. (2023). The multi-factor analysis of computer-based test scores in objective structured clinical examination of clerkship in school of dentistry, National Taiwan University. Journal of Dental Sciences. https://doi.org/10.1016/j.jds.2022.12.012 Washburn, S., Herman, J., & Stewart, R. (2017). Evaluation of performance and perceptions of electronic vs. paper multiple-choice exams. Adv Physiol Educ, 41, 548–555. https://doi.org/10.1152/advan.00138.2016.-In Zainal, Z. (2007). Case study as a research method.