Available online at www.HighTechJournal.org HighTech and Innovation Journal Vol. 3, No. 4, December, 2022 394 ISSN: 2723-9535 Evaluation of Hybrid Learning in the University: A Case Study Approach Yohannes Kurniawan 1* , Cinthia Sabrina Yulanthari Karuh 1 , Michelle Kirsten Ampow 1 , Mutiara Prahastuti 1 , Norizan Anwar 2 , Diego Cabezas 3 1 Information Systems Department, School of Information Systems, Bina Nusantara University, Jakarta, 11480, Indonesia. 2 School of Information Science, College of Computing, Informatics and Media, Universiti Teknologi MARA, Selangor, Malaysia. 3 Interdepartamental Center Embedded Systems of Automation and Computing, Peter the Great St. Petersburg Polytechnic University St. Petersburg, Russia. Received 18 September 2022; Revised 21 November 2022; Accepted 27 November 2022; Published 01 December 2022 Abstract Hybrid learning is a type of educational system in which some students attend class in person while others engage virtually from home. Hybrid learning has been applied in all educational institutions since the pandemic. BINUS University used a Learning Management System called Binusmaya to support teaching and learning activities for both hybrid and fully online learning. The objective of this study is to assess the effectiveness of hybrid learning at BINUS University based on student performance academically and non-academically. It focuses on students from the 2019–2020 academic year. The survey obtained 200 replies using a quantitative technique and a Google form as the media questionnaire. Previous research has shown that hybrid learning provides effective results in terms of academic achievement. Compared to this study, few criteria are used to measure the effectiveness of hybrid learning, such as environment, knowledge, and skills. In addition, we have added the comparison between hybrid learning and the full online method at BINUS University. The results indicate that hybrid learning is ineffective compared to fully online learning. Furthermore, hybrid students were dissatisfied with their overall results, and students pursuing full-time online courses outperformed hybrid students. This study also provides some approaches to increasing the effectiveness of hybrid learning at BINUS University. Keywords: Hybrid Learning; Full Online Mmethod; Learning Management System; Evaluation. 1. Introduction The COVID-19 pandemic that has swept the world, including Indonesia, has prompted the implementation of a stay- at-home policy to prevent the disease's spread. This stay-at-home order increases the mobility of residents inside the house while reducing mobility in public places [1]. Universities in Indonesia have been encouraged to conduct distance learning, also known as online education, since the Ministry of Education and Culture's Secretary General issued Letter No. 36603/A.A5/OT/2020 concerning the Prevention of Corona Virus Disease (COVID-19) [2]. The government made policies requiring that learning methods be changed to online, allowing students to continue learning while remaining safe at home. This online class method was successful, indicating that reactions were met, learning was improved, behaviour was maintained, and learning outcomes increased [3]. * Corresponding author: ykurniawan@binus.edu http://dx.doi.org/10.28991/HIJ-2022-03-04-03  This is an open access article under the CC-BY license (https://creativecommons.org/licenses/by/4.0/). © Authors retain all copyrights. https://creativecommons.org/licenses/by/4.0/ https://orcid.org/0000-0001-8876-3472 https://orcid.org/0000-0003-1971-3070 https://orcid.org/0000-0001-5120-2487 https://orcid.org/0000-0001-7437-9154 https://orcid.org/0000-0003-1104-1724 https://orcid.org/0000-0002-7966-5624 HighTech and Innovation Journal Vol. 3, No. 4, December, 2022 395 The term "hybrid learning" refers to a teaching approach in which some students attend classes in person while others engage virtually from their homes. Universities utilize video conferencing equipment and software to teach students remotely and in person. Hybrid classes include asynchronous learning elements such as online exercises and pre-recorded instruction videos to complement face-to-face classroom sessions [4]. The use of hybrid learning strategies has a significant effect on student learning outcomes [3]. Hybrid learning has many benefits, such as increasing student participation, increasing learning flexibility, increasing access to education, and improving teaching. Hybrid learning motivates learning for many students [5]. Motivation is one thing that needs to be improved because it is very influential on student participation. This hybrid learning model is widely preferred because it can improve student learning performance [4]. Hybrid learning has been implemented to support student learning activities on campus, including at Bina Nusantara University (BINUS University), to maximize learning in the new normal era. This hybrid learning is done online and onsite, where teachers simultaneously teach students directly and remotely. BINUS University is suitable to be the object of this research because BINUS University is the first campus in Indonesia to implement the Learning Management System (LMS). Since 2001, BINUS University has implemented a multi-channel learning system with an integrated LMS called Binusmaya to maximize the learning process for BINUS University students. Binusmaya is equipped with course materials in digital format, discussion forums, and virtual interactions between students and all lecturers so that students can learn independently and optimally. Over time, BINUS University has improved the features and quality of the LMS System. After the COVID-19 pandemic emerged, BINUS University added features that support hybrid and fully online learning. Among them is the development of a modular system, online exams, an onsite protocol, and entry passes that are useful in implementing hybrid learning at BINUS University. The onsite method at BINUS University is applied when the lecturer teaches students directly in class, while at the same time, online methods are used to teach other students online. The online method used by BINUS University is through video conferences using the Zoom Meetings platform. Students can also access Binusmaya to check the class schedule, download materials, conduct forum discussions, and submit assignments. When they want to enter Zoom meetings, students can also click the link in Binusmaya. Unlike in online classes, when students want to study face-to-face, Binusmaya can still be accessed and used. The Binusmaya platform supports students in carrying out hybrid learning. Binusmaya can be accessed in two ways: the web and the application. Through the Binusmaya application, onsite and online students can easily access any task and discuss it through the forum feature on the home page. So, both onsite and online students can easily participate in learning activities, whether from home or campus. Thus, students can easily access each learning activity when hybrid learning is taking place. As shown in Figure 1, the current implementation of the hybrid learning method at BINUS University has been implemented in all existing courses. The hybrid learning method at BINUS University is divided into two groups according to a predetermined schedule based on irregular attendance and even absenteeism. Students will take turns attending lectures online or onsite. The learning session is carried out in which the lecturer will teach in class with onsite students, and then students who get an online turn will join in learning with lecturers and other friends through video conference (Zoom). The learning session consists of the presentation of material by lecturers, and students can have discussions with other friends through platforms determined by each group. Onsite students can have discussions with students who also participate in onsite learning. In contrast, online students can conduct discussions through breakout rooms or other platforms such as Miro, Padlet, and Kahoot. Furthermore, if there are group assignments during the lecture session, the collection of assignments will be carried out in the lecture session through a form posted by group representatives. To support the learning system at BINUS University, all students, both onsite and online, can access the Binusmaya Learning Management System (LMS). Through Binusmaya, students can view dashboard activities that contain the performance and progress of student learning that is taking place in the current semester. This activity can be accessed through one of the Binusmaya features, the Dashboard feature. In addition, through the Courses feature, students can view and access any existing material based on the courses that take place in that semester. Then, there is also a Forum feature where students can have discussions with lecturers and other friends using this feature. The forum feature also supports students in collecting every assignment the lecturer gives. So, students frequently access this feature because students can communicate with lecturers and other friends. In addition, there are also informal sessions in which students and lecturers communicate using several platforms, such as Line, WhatsApp, and Email. Students can have discussions with lecturers via chat using these various platforms. Students can also request additional classes or discussions if needed. After that, if there is approval from the lecturer, then the additional class can be carried out. Previous research has shown that hybrid learning provides effective results in terms of academic achievement. Compared to this study, more criteria are used to measure the effectiveness of hybrid learning, such as environment, knowledge, and skills. In addition, this study also compares the implementation of learning methods between hybrid and fully online learning methods. Based on literature studies conducted by researchers from various sources, additional studies will be conducted regarding hybrid learning method matters relating to analyzing the advantages, disadvantages, and effectiveness of hybrid learning as a learning method for BINUS University students. HighTech and Innovation Journal Vol. 3, No. 4, December, 2022 396 Figure 1. Rich picture of hybrid learning implementation in BINUS University 2. Literature Review Table 1 shows the previous studies about hybrid and fully online learning. The researchers have included the approach used and an overview of each study. It discussed the subjects’ viewpoints, advantages, disadvantages, and how hybrid works in every targeted object of the study. Table 1. Previous studies’ journals Title Year Methods Summary Hybrid learning on problem-solving abilities in physics learning: A literature review [6] 2021 Various Scientific Publications (Books and scientific articles from reputable journals) This journal explores the role that hybrid learning can help in physics problem- solving. Therefore, in today's educational environment, hybrid learning needs to be fostered, and training for instructors to implement learning models is needed. The effect of motivation and self-efficacy against mathematics learning achievement in hybrid learning [7] 2022 Survey method (Questionnaires and tests) The results of this study revealed that student achievement was impacted by motivation in hybrid learning. Due to these issues, hybrid learning must be considered a solution for online mathematics instruction in primary schools. Human activity recognition based on hybrid learning algorithm for wearable sensor data [8] 2022 Deep learning (DL)-based methods The results of this study demonstrate that the hybrid learning method, which has a more sophisticated architectural model than the prior deep learning strategy, is quicker and more effective. Adaptation of AL-TST active learning model in hybrid classroom: Findings from teaching during COVID-19 pandemic in Egypt [9] 2022 Quasi-experimental research design In light of the COVID-19 pandemic in Egypt, this journal discusses how students can adjust to the AL-TST active learning approach in hybrid classes. Additionally, students can enjoy learning using this paradigm thanks to the hybrid learner. Enhancing Hybrid Learning using Open- Source GIS-Based Maps Archiving System [10] 2022 GIS and GIS-based maps archiving system This publication reports that students are satisfied with the Open-Source GIS- Based Map Archiving System to enhance hybrid learning. Through this hybrid learning, the amount of student participation in lectures is increased, and the essential learning resources are effectively delivered. Hybrid learning effectiveness in Learning Management during the Covid-19 pandemic [11] 2022 Quantitative Approach Based on the findings of the research reported in this article, the hybrid learning approach was quite effectively used to manage learning during the Covid-19 pandemic. Based on a hybrid learning approach that blends traditional and online learning, this is what is being discussed. The outcomes of distributing questionnaires, which had favourable and substantial results, demonstrate this. Evaluation of the Effectiveness of Hybrid Learning Activities Based on a Learning Community Network [12] 2022 Evaluation method The effectiveness of community-based hybrid learning activities is examined in this research based on the analysis. The success of utilizing DEA to assess the efficacy of hybrid learning activities is confirmed by the findings of this comparison experiment that was designed in this study. It also compares the assessment outcomes of other evaluation methodologies. Changes in Online Distance Learning Behaviour of University Students during the Coronavirus Disease 2019 Outbreak, and development of the Model of Forced Distance Online Learning Preferences [13] 2021 Qualitative Research (Case Studies) and Quantitative Method (Questionnaires) This study demonstrates that students' satisfaction with online education has a statistically significant impact on their preference for online education going forward. The pleasure with online learning, however, was not statistically significantly impacted by students' attitudes toward it. HighTech and Innovation Journal Vol. 3, No. 4, December, 2022 397 Students’ Perceptions of Hybrid Classes in the Context of Gulf University: An Analytical Study [14] 2021 Data Analysis The findings of this study indicate that the hybrid class is efficient and that the detected characteristics must be enhanced and altered to make it even more efficient. Students' Perceptions of a Blended Learning Environment to Promote Critical Thinking [15] 2021 Quantitative and Qualitative Research Methods The findings indicated that, on the whole, students were happy with the blended learning environment's design and that they thought it promoted critical thinking. Combining the Best of Online and Face-to- Face Learning: Hybrid and Blended Learning Approach for COVID-19, Post Vaccine, & Post-Pandemic World [4] 2021 Descriptive Method To deliver effective and engaging learning experiences to students, instructors and academic administrators should prioritize the development of adequate infrastructure so that teachers, administrators, and students can quickly adjust to changes beyond their control. Effect of Hybrid Learning Strategy and Self- Efficacy on Learning Outcomes [3] 2021 Quantitative, and Multivariate Analysis of Variance (MANOVA) for the data technique analysis. This journal explains that students who study with Hybrid learning strategies get higher learning outcomes than those who use traditional learning strategies. They used the quantitative method and Multivariate Analysis of Variance (MANOVA) for the data technique analysis. Evidence of Scientific Literacy Through Hybrid and Online Biology Inquiry-Based Learning Activities [16] 2021 Descriptive Qualitative method According to the findings of this study, science and technology cannot be separated in the modern world. Students can engage in epistemic practice through inquiry learning activities in an online context. This research can contribute to science, technology, and society's education, popularization, and democratization. Alternatives that promote core competencies are required for a more critical and democratic society. Hybrid Learning - The New Normal [1] 2022 Field Experiment (involving students) Hybrid learning offers both advantages and downsides. However, the benefits outweigh the drawbacks. While schools and instructors will remain crucial in the future of education, hybrid learning will not. Instead of simply providing content, they will become facilitators and motivators. The school will remain crucial because it will provide children with practical and social skills that will lead to a whole hybrid learning experience. Hybrid learning for the digital natives: Impacts on academic performance and learning approaches [17] 2022 Quantitative Method The study's findings have wide implications for teaching and learning. With proper teacher training, planning, integration of multiple techniques, student- centered assessments, and support from university administrators, hybrid learning may successfully equip learners with the talents demanded of 21st-century graduates. Hybrid Learning Here to Stay! [18] 2021 Descriptive Method (journals, reports, scholarly articles) This paper explains that hybrid learning is a way of simultaneously delivering lessons through face-to-face and online learning, which has many benefits, such as increasing student participation and learning flexibility. However, this hybrid learning is also noted to have several weaknesses. Maximizing the available positives can help to reduce any downsides. Hybrid Learning Model in Learning English (Effectiveness & Advantages) [19] 2022 Qualitative Method (Observation and Documentation) This study investigated the benefits and drawbacks of applying hybrid learning. Therefore, the importance of pleasant connections between students and teachers determines the efficiency of hybrid learning, frequent communication fosters student engagement, and the use of technology has a significant impact on schools. Hybrid Learning or Virtual Learning? Effects on Students' Essay Writing and Digital Literacy [20] 2022 Quantitative Quasi- experimental Design The results show the advantages that students prefer in the hybrid learning model. Several things have become an important role between hybrid learning and virtual learning of essay writing skills for high school students, one of which is digital literacy skills. Investigation of the Effectiveness of Hybrid Learning on Academic Achievement: A Meta- Analysis Study [21] 2022 Meta-analysis Statistical Method Hybrid learning is one of the major influences on the level of student achievement. Therefore, the necessary infrastructure and facilities must be encouraged to increase the use of hybrid learning effectively. Learners’ Satisfaction and Commitment Towards Online Learning During COVID-19: A Concept Paper [22] 2021 Descriptive resources (journals and articles) During the COVID-19 epidemic, this research demonstrated how online learning efficiency affects student satisfaction and commitment. Academic issues, accessibility issues, students' technological abilities, mental health, and lecturer dedication all influence students' pleasure and commitment to online learning. The Challenges of Application of The Hybrid Learning Model in Geography Learning During the Covid-19 Pandemic [23] 2021 Qualitative Method such as interviews, observations, and Focus Group Discussions (FGD) Based on the application of synchronous and asynchronous techniques, students can participate actively in online learning. The synchronous technique is applied to lecturers and students with standby on the internet at the agreed time. While the asynchronous technique, lecturers and students can access the Learning Management System portal anytime. The effectiveness of hybrid learning as instructional media amid the COVID-19 pandemic [24] 2021 The method used in this research is phenomenology by involving junior high school teachers This study concludes that although online learning provides convenience in accessing the internet, students still use technology not to find learning resources. Besides that, they also need help understanding the courses that should be done in practice. Therefore, this research implies that technology that continues to develop must be accompanied by user understanding. The Effectiveness of Hybrid Learning in Improving of Teacher-Student Relationship in Terms of Learning Motivation [5] 2021 The research used a quasi- experimental design and MANOVA for data technique analysis They remark in this publication that the hybrid learning model is an innovative learning paradigm. Because hybrid learning allows potential student instructors to express and grasp abstract mathematical concepts in learning, it is recognized that hybrid learning may construct and develop sophisticated mathematical thinking. The Effectiveness of the Hybrid Learning Materials with the Application of Problem Based Learning Model (Hybrid-PBL) to Improve Learning Outcomes during the COVID-19 Pandemic [25] 2022 Quasi-Experimental Method Using the Based Learning Method, this study also discovered that hybrid learning with problem-based learning models increased student learning independence and creativity compared to regular classes. Combining hybrid learning with problem- based methods can give good learning outcomes. HighTech and Innovation Journal Vol. 3, No. 4, December, 2022 398 The Effects of Online Learning on EFL Students’ Academic Achievement during Coronavirus Disease Pandemic [26] 2021 Qualitative, Quantitative, and Hypothesis Testing Online learning does not show a significant difference in learning scores in the EFL class. Overall, online learning is the best alternative to the education system for students preparing for the English language during the COVID-19 pandemic. The results were seen from attitudes, preferences, learning motivation, and self- confidence. The Pattern of Hybrid Learning to Maintain Learning Effectiveness at the Higher Education Level Post-COVID-19 Pandemic [27] 2022 Qualitative Descriptive Analysis and Quantitative Method The results prove that the hybrid learning model is the most appropriate for students after the COVID-19 pandemic because it runs effectively for students at the doctoral level. The application of hybrid learning must be adapted to the characteristics, direction, educational orientation, ability, readiness, and independence of students at every level. Undergraduate Students' Perception of Hybrid Learning: Voices from English Language Education Students in Pandemic Era [28] 2022 Quantitative Method According to the findings of this study, most students had a good opinion of hybrid learning during the epidemic. They believe lecturers supply a wealth of material for online learning. Furthermore, students believe that the learning objectives determine hybrid learning resources in each course. Understanding Research Trends in HyFlex (hybrid flexible) Instruction Model: A Scientometric Approach [29] 2022 Scientometric Approach and OSviewer and Bibliometrix R software to analyze data According to thematic analysis, terms like blended learning, hybrid learning, and e-Learning are closely related to HyFlex, as evidenced by their high relevance and demand. Online learning, distant learning, and assessment are terms that are growing toward progress. Curriculum, self-teaching, and first-year undergraduate are all declining in document usage. Using active learning in hybrid learning environments [30] 2021 Quantitative Method In this paper, using the flipped class and the active learning techniques draws a deeper understanding of the students. The student-centered approach increases the understanding of the students. However, it requires many technology tools, time, and effort. The Effects of Online Learning on EFL Students’ Academic Achievement during Coronavirus Disease Pandemic [26, 31] 2021 Qualitative, Quantitative, and Hypothesis Testing Online learning does not show a significant difference in learning scores in the EFL class. Overall, online learning is the best alternative to the education system for students preparing for the English language during the COVID-19 pandemic. The results were seen from attitudes, preferences, learning motivation, and self- confidence. There have been several studies conducted on hybrid and fully online learning. Previous studies examined how students perceived hybrid learning and fully online learning. The study found that students show positive perceptions regarding hybrid learning. However, drawbacks arise in the technical aspects, such as students’ concentration [2]. Moreover, Thamrin et al. (2022), who examined hybrid learning using Problem-based learning (PBL), found that it improved students’ outcomes and was more effective than the control class [25]. To measure the effectiveness of hybrid learning, there should be a learning management system that a university uses to support the learning activities. In contrast to earlier studies, BINUS University has implemented a multi-channel learning system with a self-built Learning Management System called Binusmaya, which differs from the previous studies. Hence, this current study applied the quantitative method and used a Learning Management System (LMS) that supports learning and teaching activities to know the advantages and disadvantages and to evaluate the effectiveness of hybrid learning in BINUS. This study evaluates the current situation and gives the best practice based on students’ perspectives. 3. Research Methods This study uses quantitative research methods using a descriptive statistical approach. This method uses an online questionnaire (Google form) targeted at all BINUS University students in all majors who start lectures from 2019-2022. The questionnaire was distributed through group chats and on-site respondents with a target of 200 respondents. The distribution of this questionnaire is carried out to determine the effectiveness of hybrid learning at BINUS University. As shown in Table 2, the quantitative data were gathered from the students’ college years, age, and gender. Table 2. Students’ Respondent Demography Year of College Age Gender Male Female 2019 <20 years old 0 1 20-22 years old 19 30 >22 years old 1 0 2020 <20 years old 7 13 20-22 years old 22 53 >22 years old 1 1 2021 <20 years old 5 12 20-22 years old 1 4 >22 years old 3 0 2022 <20 years old 12 12 20-22 years old 2 0 >22 years old 1 0 Total 200 HighTech and Innovation Journal Vol. 3, No. 4, December, 2022 399 To get accurate research results, we carry out a series of processes, from finding problems and determining target respondents to collecting data and visualizing data results to conclude learning evaluations at BINUS University which are carried out in hybrid learning and fully online (shown in Figure 2). Figure 2. Flowchart of Methodology Process To deeper the perception of students about hybrid and fully online learning methods, the researchers use indicators that can help build a proper answer (shown in Table 3). Table 3. Questionnaire Instrument Aspect Dimension Indicator Identity of Respondents Personal Gender Age Major Year of college Surroundings and Skills Environment The place where they stay Analyzing where the student lives, some work from home/learning from home or not. Knowledge The scale of student understanding of all ongoing courses. Analyze whether they are accustomed to reading lecture material before attending class. Student satisfaction with the value of learning outcomes in the current semester. Student score performance from last semester (odd) to this semester. Analyze what types of students are involved in doing assignments. Skills Student confidence in delivering presentations. The overall soft skill development scale experienced by students in the last two semesters. Soft skills of students that need to be improved. Attitude The responsibility of students to attend class on time. The concern of students for their friends who are experiencing difficulties. Learning Method Hybrid Learning Method The reason for choosing the Hybrid Learning method. What can be improved from the Hybrid Learning Method at BINUS University? Tools used to access Binusmaya. The Binusmaya feature that best supports student learning activities. Full Online Method The reason for choosing the Full Online method. What can be improved from the Online Learning Method at BINUS University? Tools used to access Binusmaya. The Binusmaya feature that best supports student learning activities. As stated in Table 3, in the first section, respondents were asked to fill out their personal information to help us analyze their answers. To understand more about our respondents, we asked them to describe their study environment and whether they were performing hybrid learning or online learning. Following that, responders must provide insight by selecting the best options that were given in the form during their learning. This result covers how they studied, what HighTech and Innovation Journal Vol. 3, No. 4, December, 2022 400 skills they obtained and which skills are most needed, and how they behaved in class. As a result, we may analyze hybrid learning at BINUS University and continue to improve the aspects still lacking in implementing Hybrid Learning. A media questionnaire (Google Form) study can give reliable data and reach more respondents, resulting in accurate research outcomes. 4. Results and Discussion 4.1. Result Based on the results of online questionnaires (Google Form), the researchers have collected 200 responses. The characteristics of the respondents used in this research were classified based on the respondent's gender, age, major, and year of entering college. The respondents were asked which learning method they were using at the time. To measure the effectiveness of their chosen learning method, we asked them about their academic performance during the odd semester. Figure 3 reveals that students taking fully online courses (39%) outperform those taking hybrid courses (30%). Meanwhile, most hybrid students' grades are consistent. Furthermore, around 17% of students attend hybrid classes, and their grades tend to drop. It is arranged further based on several factors in the next figure to determine the factors that influence the evaluation of the effectiveness of the learning method. Figure 3. Student’s Score Performance in the Odd Semester The first factor that measures the effectiveness of the learning method used is the level of student satisfaction based on the grades they get and their learning performance. Figure 4 shows students' satisfaction and dissatisfaction levels depending on their academic achievement results. As indicated above, around 38% of online students are satisfied with their performance outcomes. In the meantime, just 28% of hybrid students were pleased with their results. About 13% of hybrid students were dissatisfied owing to a drop in their grades. Regarding the results, the students also stated the reasons for their chosen learning method. Figure 4. Level of Satisfaction with Value Results vs. Learning Performance 0% 10% 20% 30% 40% 50% 60% Increase Stable Decrease Student's Score Performance in the Last Semester Hybrid Full Online 0.00% 8.57% 5.71% 38.57% 41.43% 5.71% 1.54% 15.38% 13.85% 28.46% 36.92% 3.85% 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% Increase Stable Decrease Increase Stable Decrease Not Satisfied Satisfied Level of Satisfaction with Value Results vs Learning Performance Full Online Hybrid HighTech and Innovation Journal Vol. 3, No. 4, December, 2022 401 The next factor is the student's reasons for choosing learning methods. According to Figure 5, they prefer full online learning because of the flexibility of their students (39%). Students can study from anywhere they want as long as they can access the internet and gadgets. Followed by other supporting reasons, such as technology that supports learning from home and a more efficient teaching process, another 4% feel they prefer the fully online learning method because there is still the Covid-19 virus. Some students are not allowed by their parents to attend face-to-face classes. Figure 5. Reasons for choosing Full Online Learning Method Meanwhile, according to Figure 6, 39% of students who choose the hybrid learning method feel that direct interaction is the biggest reason for choosing it. Many students feel that face-to-face learning makes it easier to understand the material. Besides that, the other 2% feel that the hybrid learning method can make them able to have more friends than when taking a fully online class. Figure 6. Reasons for choosing Hybrid Learning Method The third factor is the student's understanding of the chosen learning method. Table 4 shows the correlation of achievement scores with students' understanding of hybrid learning methods. Based on the data in the table above, the performance of student scores with students' understanding of the hybrid learning method is stable and tends to increase, with a stable percentage of 53% and an increase of 30%. In addition, the average level of students’ understanding of learning material is in the range of 7 and 8 out of 10, with a total percentage of 72% of all students who follow the hybrid learning method. None of the students felt that their understanding was at level 1 (the lowest), and only 18% felt that their understanding of their scores had decreased when they followed the hybrid learning method. 38% 19% 39% 4% Reasons for Choosing the Full Online Learning Method Efficient Technology Tools Support Flexibility Others 39% 32% 27% 2% Reasons for Choosing the Hybrid Learning Method Live Interaction Clearer Explanation Some subject are easier to understand face-to-face Others HighTech and Innovation Journal Vol. 3, No. 4, December, 2022 402 Table 4. The Correlation of Performance Values with Students' Understanding of the Hybrid Learning Method The Correlation of Performance Values with Students' Understanding of the Hybrid Learning Method Performance / Number of Scale 1 2 4 5 6 7 8 9 10 Grand Total Increase 0% 1% 0% 0% 0% 6% 15% 6% 2% 30% Stable 0% 1% 2% 3% 8% 18% 18% 1% 2% 52% Decrease 0% 1% 0% 1% 2% 11% 4% 0% 0% 18% Grand Total 0% 2% 2% 4% 9% 35% 37% 7% 4% 100% Table 5 shows the correlation between achievement scores and students' understanding of the fully online learning method. Based on the data in the table above, it can be seen that the performance of student scores with students' understanding of the Full Online Learning method is stable and increases with a proportion scale of 50% and 39%. Furthermore, students understanding of learning material is dominated at levels 7 and 8, with a total percentage of 75% to 100%. Only 11% of students who took part in fully online learning felt that their grades had decreased, and no students felt that their understanding could have been higher (understanding levels 1 and 2). This result proves that learning conducted using the full online method is quite effective at BINUS University. Table 5. The Correlation of Performance Values with Students' Understanding of the Full Online Learning Method The Correlation of Performance Values with Students' Understanding of the Full Online Learning Method Performance / Number of Scale 1 2 3 5 6 7 8 9 10 Grand Total Increase 0% 0% 0% 0% 3% 9% 19% 7% 1% 39% Stable 0% 0% 1% 3% 7% 9% 27% 3% 0% 50% Decrease 0% 0% 0% 1% 1% 3% 6% 0% 0% 11% Grand Total 0% 0% 1% 4% 11% 20% 51% 10% 1% 100% When compared to the understanding of students who follow the hybrid and full online methods, the level of effectiveness is more dominated by the fully online learning method with stable and increasing grades with a total of 89% and an understanding level of 82% (based on understanding level > 6). The next factor is the environment. The environment can affect student motivation in learning and support whether or not the student environment is conducive to learning at home. Table 6 shows the correlation between environment and grade performance of students who take lectures using the hybrid learning method. The results show that students who live with their families tend to have higher achievement scores than students who live alone or with friends. It can be seen in table 6 that the proportion of performance scores of students who live with their families is 66%. Most BINUS students have a home environment that is conducive to learning, supporting the learning system even though it is done at home. Students living alone or with friends tend to have lower grade performance than those living at home with their families. It can happen because students who live alone tend to experience decreased motivation and have more distractions to play with friends than study. Table 6. The Correlation of students' environmental situation with their grade performance (Hybrid Learning) The Correlation of students' environmental situation with their grade performance (Hybrid Learning) 2 4 5 6 7 8 9 10 Grand Total Live with friend 0% 0% 0% 1% 3% 1% 0% 0% 5% no one works from home/study from home 0% 0% 0% 0% 2% 0% 0% 0% 2% there are those who work from home/learning from home 0% 0% 0% 1% 1% 1% 0% 0% 2% Live Alone 0% 2% 2% 4% 9% 8% 4% 1% 29% no one works from home/study from home 0% 2% 2% 2% 5% 2% 3% 1% 16% there are those who work from home/learning from home 0% 0% 0% 2% 4% 6% 1% 0% 13% Live with Family 2% 1% 2% 5% 22% 28% 3% 3% 66% no one works from home/study from home 2% 0% 2% 3% 9% 5% 2% 0% 23% there are those who work from home/learning from home 1% 1% 1% 2% 13% 22% 1% 3% 43% Grand Total 2% 2% 4% 9% 35% 37% 7% 4% 100% As shown in Table 7, students living with their families tend to have higher performance scores than those living alone or with friends. We can see that the proportion of achievement scores of students living with their families is 70%. HighTech and Innovation Journal Vol. 3, No. 4, December, 2022 403 It shows that the home is an ideal place to study for students who follow the fully online learning method because it is quite conducive and increases their motivation in learning. Meanwhile, students who live alone or with friends tend to experience a decreased motivation to learn due to distractions from friends or demotivation when alone. Table 7. The Correlation of students' environmental situation with their grade performance (Full Online Learning) The Correlation of students' environmental situation with their grade performance (Full Online) Students' Residence / Number of Scale 3 5 6 7 8 9 10 Grand Total Live with friend 0% 0% 0% 0% 3% 1% 0% 4% No one works from home/study from home 0% 0% 0% 0% 3% 0% 0% 3% There are those who work from home/learning from home 0% 0% 0% 0% 0% 1% 0% 1% Live Alone 1% 0% 6% 6% 10% 3% 0% 26% No one works from home/study from home 1% 0% 1% 4% 9% 3% 0% 19% There are those who work from home/learning from home 0% 0% 4% 1% 1% 0% 0% 7% Live with Family 0% 4% 6% 14% 39% 6% 1% 70% No one works from home/study from home 0% 1% 3% 6% 19% 1% 0% 30% There are those who work from home/learning from home 0% 3% 3% 9% 20% 4% 1% 40% Grand Total 1% 4% 11% 20% 51% 10% 1% 100% The next factor is the development of soft skills when undergoing the chosen learning method. In Figure 7, on a scale of 1-10, students who take the fully online and hybrid learning methods experience an average increase at levels 7 and 8. For the full online method, most are at levels 7 and 8 with a balanced proportion of 30% in each level, and for students who take part in the hybrid learning method, the most increase in soft skills at level 8 with a proportion of 38%. Furthermore, for the whole, the increase in soft skills is measured by the total proportion of levels 7 to 10 in each learning method. The results show that the increase in soft skills is dominated by the fully online learning method, with a percentage of 83%, while the hybrid learning method is 75%. Students with the fully online learning method can improve their soft skills because, with a flexible class schedule, they can participate in other self-development activities more freely. Figure 7. Correlation of the increase in soft skills obtained with learning method implemented If we look at it in more detail regarding soft skills, the two learning methods still require improvement in the soft skills area. Figure 8 demonstrates the skills needed to be improved by the students. Hybrid students still need to work on their time management skills. It happens because hybrid students must divide their schedules wisely since they have online and offline campus activities. Conversely, fully online students need more communication skills because they only communicate through screens. In the figure above, both students can work together well regarding teamwork skills, and it does not show any concerns. To conclude, hybrid students must improve their time management skills, while full online students must focus on communication skills. 0% 10% 20% 30% 40% 50% 60% 70% 80% 1 3 4 5 6 7 8 9 10 Correlation of the increase in soft skills obtained with the learning method implemented Full Online Hybrid HighTech and Innovation Journal Vol. 3, No. 4, December, 2022 404 Figure 8. Soft skills that must be improved To assess the system of BINUS University, respondents were questioned about the most frequent methods they use to access Binusmaya. As shown in Figure 9, around 70% of students find it useful to access both websites and mobile applications. It means that students find it easy and helpful to use the Binusmaya, whether through an application or website. The easy access to the system can make daily learning activities easier for the students, such as downloading materials, checking on class schedules, and doing forum tasks. Furthermore, Binusmaya plays an important role in students daily learning activities. Figure 9. Binusmaya Access Media The next factor is the most useful features in Binusmaya. Based on Figure 10, the most beneficial aspect for students is the schedule feature, with a percentage of 28%. With one click away from the feature, students can check their class schedule easily, and it does not take too much time. They can see when the class will start and what subject they will have. Meanwhile, around 9% of students choose the announcement feature as the least useful feature in Binusmaya. It can happen because sometimes students only sometimes check their announcements, and the feature is not used daily. BINUS University could improve the announcement to make greater use of the feature and increase student attention. 0% 5% 10% 15% 20% 25% 30% 35% 40% Communication Critical Thinking Leadership Teamwork Time Management Soft Skills That Need to be Improved Hybrid Full Online 0% 10% 20% 30% 40% 50% 60% 70% 80% Both of them Mobile Apps Website Binusmaya Access Media Hybrid Full Online HighTech and Innovation Journal Vol. 3, No. 4, December, 2022 405 Figure 10. The Most Useful Features of Binusmaya Figure 11 illustrates some suggestions for improving hybrid learning at BINUS University. More emphasis should be placed on enhancing the quality of supporting technologies in the classroom, such as speaker usage. It can help the students communicate with the lecturers easily because sometimes they cannot hear what the teachers say. Along with it, teachers should balance the engagement between onsite and at-home students so that no one feels left out. It will help the students to be more attentive in learning their lessons. The teachers should be given training in using the technology, such as how to use the online platform. So, the learning process will be more effective and efficient. Some teachers found it difficult to share their screens because they do not know how to operate them. That is why proper training can help them improve and save more time. Figure 11. Improvement Needed in Hybrid Learning Not only for hybrid learning, but there is also some improvement needed for online learning classes. Figure 12 shows the enhancements required for online learning. The first one is to upgrade Binusmaya's server since it might go down unexpectedly, and students find it hard and time-consuming to access the system, especially when submitting their assignments. As a result, it slows down students' activities. Next, it would be nice to improve the teamwork skills. As for the online class, it is hard for students to improve their teamwork skills due to some screen boundaries. BINUS University also can provide some better features in Binusmaya to support the full online method. 14% 27% 22% 28% 9% The most useful features in Binusmaya Dashboard Courses Forum Schedule Announcement 38% 38% 24% Improvement Needed in Hybrid Learning Improving the quality of supporting devices Interaction balance Improving the understanding of technology use HighTech and Innovation Journal Vol. 3, No. 4, December, 2022 406 Figure 12. Improvement Needed in Full Online Learning 4.2. Discussion In this study, we have measured the effectiveness of each hybrid and fully online learning method. Four measuring factors are used: environment, skills, students’ understanding of each learning method, and academic achievement. Based on the overall analysis results, table 8 describes the core results of the factors that measure the effectiveness of the learning methods carried out at BINUS University. As shown in Table 8, those who take full online learning methods have a better understanding of each lesson in the class. As a result, students' grades increased from last semester. Based on this result, fully online learning has a better academic achievement than the hybrid learning students, as we have stated in Figure 3 above. As for improving skills, both hybrid and fully online learning students have different skills that need improvement. Hybrid students still need to improve in time management skills. They found it hard since they have to divide their time between online and offline classes and their other activities. Table 8. Measuring factors of the learning method Measuring Factors Superior Learning Methods Understanding of Learning Method Full Online Learning Academic Achievement Full Online Learning Skills Improvement Hybrid & Full Online Learning Environment Hybrid & Full Online Learning Meanwhile, fully online students need to improve their communication skills due to the barrier of the screen. Both students are good at teamwork skills and show the good result of it. Several studies have found that hybrid learning increases student learning independence and creativity compared to regular classes [25]. Another important factor is the environment. The respondents were asked whether to stay alone or with friends and family. The results show that those who live with family have higher performance scores than those who live alone or with friends. Those who live alone or with friends are less motivated and tend to have more distractions. As a result, it can downgrade their academic performance, and their grades tend to go lower. To compare with the previous study about evaluated the effect of hybrid learning on academic achievement from 13 different applied disciplines. The previous study found that hybrid learning was effective based on academic achievement in the applied meta-analysis. The results showed an increase in students' academic achievement with the hybrid learning method. For future use, they encourage using a hybrid learning model in educational backgrounds and provide better infrastructure and facilities [21]. In this study, we have compared hybrid learning and the full online methods. It showed that the implementation of hybrid learning does not show a significant improvement from last year's semester's grade. Regarding that, we have suggestions to improve the effectiveness of the hybrid learning method. We also have developed some components to improve the hybrid learning method. As shown in Figure 13, some components are categorized as necessary and unnecessary. 44% 24% 32% Improvement Needed in Full Online Learning Binusmaya server upgrade Improving Teamwork Improving the quality of Binusmaya features HighTech and Innovation Journal Vol. 3, No. 4, December, 2022 407 Figure 13. The Best Practice Components on Hybrid Learning From the components seen in Figure 13, the foundation of those components is the Network. A reliable network is necessary to support each of these elements so that learning can proceed without interruption. Additionally, there are several necessary components, such as Camera, Video Conference Tools, and Speakers. The teaching and learning processes between students and teachers can be enhanced by having and enhancing these elements. It will improve communication in both ways and increase learning engagements. Meanwhile, some components are optional such as Whiteboard, Headset, and Microphone. These are categorized as supporting tools. It can help the hybrid learning method to be more effective for teaching and learning activities. Based on the result we have gathered, there are still some improvements needed for hybrid learning at BINUS University. Figure 14 shows the improvement needed using the use case. Figure 14. Use Case of Improvement of Hybrid Learning The use cases highlighted in blue above are things that BINUS University must maintain. When it comes to onsite learning, the course description is straightforward. Also, students and lecturers communicate well. However, the ones marked in light red need to be addressed. Students found it difficult to hear the teachers when they were learning from home for the hybrid. Also, teachers sometimes pay more attention to onsite students than at home when it comes to teaching. Finally, more understanding of how to use technology would be beneficial. Training for teachers and students HighTech and Innovation Journal Vol. 3, No. 4, December, 2022 408 on how to use the online platform's capabilities and technical usage of technology for hybrid learning, such as moving the camera, properly displaying the screen, and reducing and maximizing the voice so students from home can hear properly. 5. Conclusions Based on research conducted on BINUS University students regarding the evaluation of the hybrid learning method, we can conclude that the hybrid learning method is less effective than the fully online learning method. Using academic achievements, skills improvement, environment, and understanding of the learning method as the criteria, students who take the fully online learning method perform better than students who take hybrid learning. The results show that the grades for hybrid learning are less stable since the odd semester than for students who take the fully online learning method. Furthermore, they are not very satisfied with the results obtained. Students also need help balancing interactive engagement with the teacher due to contact conflicts between on-site and home students. In contrast, fully online students showed good results from last semester's academic performance. By participating in fully online learning, they can take classes from anywhere as long as the internet is active and can be flexible in arranging a time to be more active in non- academic activities to improve their soft skills. Therefore, to improve the quality of implementing hybrid learning, BINUS University can make several improvements in campus management:  Re-evaluate the ability to use technology among lecturers and provide directions regarding technological knowledge that supports hybrid learning;  Facilitate more technological devices to support the continuity of hybrid learning;  In the face-to-face learning scheme, there should be more interaction between lecturers and students, both in class and at home;  It is necessary to improve server quality at Binusmaya so all students can access all learning materials easily. 6. Declarations 6.1. Author Contributions Conceptualization, Y.K., C.S.Y.K., M.K.A., and M.P.; methodology, Y.K., C.S.Y.K., M.K.A., and M.P.; software, Y.K., C.S.Y.K., M.K.A., and M.P.; validation, Y.K., C.S.Y.K., M.K.A., and M.P.; formal analysis, Y.K., C.S.Y.K., M.K.A., and M.P.; investigation, Y.K., C.S.Y.K., M.K.A., and M.P.; resources, Y.K., C.S.Y.K., M.K.A., and M.P.; data curation, N.A. and D.C.; writing—original draft preparation, Y.K., C.S.Y.K., M.K.A., and M.P.; writing—review and editing, N.A. and D.C.; visualization, C.S.Y.K., M.K.A., and M.P.; supervision, Y.K.; project administration, Y.K.; funding acquisition, Y.K. All authors have read and agreed to the published version of the manuscript. 6.2. Data Availability Statement The data presented in this study are available on request from the corresponding author. 6.3. Funding The authors received no financial support for the research, authorship, and/or publication of this article. 6.4. Declaration of Competing Interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. 7. References [1] Idrizi, E., Filiposka, S., & Trajkovijk, V. (2021). Analysis of success indicators in online learning. International Review of Research in Open and Distributed Learning, 22(2), 205-223. doi:10.19173/irrodl.v22i2.5243. [2] Daniel Hermawan. (2021). The Rise of E-Learning in COVID-19 Pandemic in Private University: Challenges and Opportunities. IJORER : International Journal of Recent Educational Research, 2(1), 86–95. doi:10.46245/ijorer.v2i1.77. [3] Rorimpandey, W. H. ., & Midun, H. (2021). Effect of Hybrid Learning Strategy and Self-Efficacy on Learning Outcomes. Journal of Hunan University Natural Sciences, 48(8), 181–189. [4] Aristika, A., Darhim, Juandi, D., & Kusnandi. (2021). The effectiveness of hybrid learning in improving of teacher-student relationship in terms of learning motivation. Emerging Science Journal, 5(4), 443–456. doi:10.28991/esj-2021-01288. [5] Singh, J., Steele, K., & Singh, L. (2021). Combining the Best of Online and Face-to-Face Learning: Hybrid and Blended Learning Approach for COVID-19, Post Vaccine, & Post-Pandemic World. Journal of Educational Technology Systems, 50(2), 140–171. doi:10.1177/00472395211047865. HighTech and Innovation Journal Vol. 3, No. 4, December, 2022 409 [6] Lestari, Syafril, S., Latifah, S., Engkizar, E., Damri, D., Asril, Z., & Yaumas, N. E. (2021). Hybrid learning on problem-solving abiities in physics learning: A literature review. Journal of Physics: Conference Series, 1796(1), 012021. doi:10.1088/1742- 6596/1796/1/01202. [7] Rahmawati, R. D., Sugiman, S., Nur Wangid, M., & Eko Atmojo, S. (2022). The effect of motivation and self-efficacy against mathematics learning achievement in hybrid learning. World Journal on Educational Technology: Current Issues, 14(6), 1978– 1990. doi:10.18844/wjet.v14i6.7795. [8] Athota, R. K., & Sumathi, D. (2022). Human activity recognition based on hybrid learning algorithm for wearable sensor data. Measurement: Sensors, 24, 1–14,. doi:10.1016/j.measen.2022.100512. [9] Hasnine, M. N., Ueda, H., & Ahmed, M. M. H. (2022). Adaptation of AL-TST active learning model in hybrid classroom: Findings from teaching during COVID-19 pandemic in Egypt. Procedia Computer Science, 207, 3226–3233. doi:10.1016/j.procs.2022.09.380. [10] Bedair, S., Sayed, S. A., & AlMetwaly, W. M. (2022). Enhancing Hybrid Learning using Open Source GIS-Based Maps Archiving System. Egyptian Journal of Remote Sensing and Space Science, 25(3), 779–793. doi:10.1016/j.ejrs.2022.07.003. [11] Dakir, D., & Fauzi, A. (2022). Hybrid learning effectiveness in Learning Management during the Covid-19 pandemic. Cypriot Journal of Educational Sciences, 17(11), 3924–3936. doi:10.18844/cjes.v17i11.7700. [12] Xie, G. (2022). Evaluation of the Effectiveness of Hybrid Learning Activities Based on a Learning Community Network. International Journal of Emerging Technologies in Learning (IJET), 17(21), 246–260. doi:10.3991/ijet.v17i21.35111. [13] Ploj-Virtič, M., Dolenc, K., & Šorgo, A. (2021). Changes in Online Distance Learning Behaviour of University Students during the Coronavirus Disease 2019 Outbreak, and development of the Model of Forced Distance Online Learning Preferences. European Journal of Educational Research, 10(1), 393–411. doi:10.12973/eu-jer.10.1.393. [14] Alfiras, M., Nagi, M., Bojiah, J., & Sherwani, M. (2021). Students’ perceptions of hybrid classes in the context of Gulf University: An analytical study. Journal of Hunan University Natural Sciences, 48(5), 180-188. [15] Lu, D. (2021). Students’ Perceptions of a Blended Learning Environment to Promote Critical Thinking. Frontiers in Psychology, 12, 1–9,. doi:10.3389/fpsyg.2021.696845. [16] Natale, C. C., Mello, P. S., Trivelato, S. L. F., Marzin-Janvier, P., & Manzoni-De-Almeida, D. (2021). Evidence of scientific literacy through hybrid and online biology inquiry-based learning activities. Higher Learning Research Communications, 11, 33–49. doi:10.18870/HLRC.V11I0.1199. [17] Rodrigo, R. T., & Platon, L. H. (2022). Hybrid learning for the digital natives: Impacts on academic performance and learning approaches. Kasetsart Journal of Social Sciences, 43(1), 201–208. doi:10.34044/j.kjss.2022.43.1.27. [18] Abi Raad, M. E., & Odhabi, Hamad. (2021). Hybrid Learning Here to Stay! Frontiers in Education Technology, 4(2), p121. doi:10.22158/fet.v4n2p121. [19] Umiyati, M. (2022). Hybrid Learning Model in Learning English (Effectiveness & Advantages). International Linguistics and TESOL Journal, 1(1), 5-9. [20] Alsowat, H. H. (2022). Hybrid Learning or Virtual Learning? Effects on Students’ Essay Writing and Digital Literacy. Journal of Language Teaching and Research, 13(4), 872–883. doi:10.17507/jltr.1304.20. [21] Kazu, I. Y., & Yalçin, C. K. (2022). Investigation of the Effectiveness of Hybrid Learning on Academic Achievement: A Meta- Analysis Study. International Journal of Progressive Education, 18(1), 249-265. [22] Ranadewa, D. U. N., Gregory, T. Y., Boralugoda, D. N., Silva, J. A. H. T., & Jayasuriya, N. A. (2021). Learners’ Satisfaction and Commitment towards Online Learning during COVID-19: A Concept Paper. Vision: The Journal of Business Perspective, 097226292110567. doi:10.1177/09722629211056705. [23] Prihadi, S., Sajidan, S., Siswandari, S., & Sugiyanto, S. (2021). The Challenges of Application of the Hybrid Learning Model in Geography Learning During the Covid-19 Pandemic. GeoEco, 8(1), 1. doi:10.20961/ge.v8i1.52205. [24] Sumandiyar, A., Husain, M. N., Sumule G, M., Nanda, I., & Fachruddin, S. (2021). The effectiveness of hybrid learning as instructional media amid the COVID-19 pandemic. Jurnal Studi Komunikasi (Indonesian Journal of Communications Studies), 5(3), 651–664. doi:10.25139/jsk.v5i3.3850. [25] Thamrin, T., Saidun Hutasuhut, Aditia, R., & Putri, F. R. (2022). The Effectiveness of the Hybrid Learning Materials with the Application of Problem Based Learning Model (Hybryd-PBL) to Improve Learning Outcomes during the COVID-19 Pandemic. IJORER : International Journal of Recent Educational Research, 3(1), 124–134. doi:10.46245/ijorer.v3i1.178. [26] Hidalgo-Camacho, C., Isabel, G., Villacís, W., & Varela, K. (2021). The Effects of Online Learning on EFL Students’ Academic Achievement during Coronavirus Disease Pandemic. European Journal of Educational Research, 10(4), 1867–1879. doi:10.12973/eu-jer.10.4.1867. HighTech and Innovation Journal Vol. 3, No. 4, December, 2022 410 [27] Sukiman, Haningsih, S., & Rohmi, P. (2022). The pattern of hybrid learning to maintain learning effectiveness at the higher education level post-COVID-19 pandemic. European Journal of Educational Research, 11(1), 243–257. doi:10.12973/eu- jer.11.1.243. [28] Erliza, S., & Septianingsih, P. A. (2022). Undergraduate Students’ Perception of Hybrid Learning: Voices from English Language Education Students in Pandemic Era. Journal of English Language Teaching and Linguistics, 7(1), 231. doi:10.21462/jeltl.v7i1.782. [29] Eduljee, N. B., Chakravarty, R., Croteau, K., & Murphy, L. (2022). Understanding Research Trends in HyFlex (hybrid flexible) Instruction Model: A Scientometric Approach. International Journal of Instruction, 15(4), 935–954. doi:10.29333/iji.2022.15450a. [30] Demazière, C. (2021). Using Active Learning in Hybrid Learning Environments. EPJ Web of Conferences, 247, 14001. doi:10.1051/epjconf/202124714001. [31] Adeyeye, B., Ojih, S. E., Bello, D., Adesina, E., Yartey, D., Ben-Enukora, C., & Adeyeye, Q. (2022). Online learning platforms and Covenant University students’ academic performance in practical related courses during COVID-19 pandemic. Sustainability, 14(2), 878. doi:10.3390/su14020878.