30 | IJET| Volume. 11, Issue 1. Juli 2022 Copyright 2022 Siti Nurul Hidayah, is licensed under Creative Commons Attribution-ShareAlike 4.0 International License. Evaluative Analysis On Mooc With Lms Based To Improve English Teachers’ Teaching Competences Dr. Siti Nurul Hidayah, M.Ed., sitinurulhiday@gmail.com, STIT Sunan Giri Trenggalek, East Java, Indonesia Abstract. In the 5th Industrial Revolution era, English teaching methods blended with technology and information-based skills will be more noteworthy to improve English teachers' teaching aptitudes. Here, the role of the Non-Formal Training Institute (LKP) with its new online teaching framework is crucial to increase English teachers' teaching competencies in this industrial revolution era, improve their students' attainment and advance their professional knowledge. To optimize this role, this study was conducted using Kirkpatrick's 4-level evaluation model to assess the implementation of Massive Open Online Course (MOOC) with Learning Management System (LMS) based in non-formal educational contexts for many English teachers in Indonesia. This study aimed to determine the benefits and constraints of implementing this online course using evaluative techniques with quantitative and qualitative research methodologies. Data collection techniques were interviews, document observation and questionnaires. The results showed that at the reaction level, the research participants revealed a positive response to the online course implementation; the participants also succeeded in increasing their teaching knowledge (learning level), changing their behavior in teaching English (behavior level), and improving their students' achievement (level result). The most significant benefits felt by participants included valuable teaching topics, video materials, and video conferencing frames which had a direct impact on their teaching professionalism. The obstacles faced by the participants were some problems with internet access, video teleconference access, and assignments. Keywords: English Teacher, MOOC, LMS, Kirkpatrick's 4-level Evaluation Model, Language Teaching, Speaking. 1. INTRODUCTION In succeeding online learning framework in Indonesia, Lembaga Ketrampilan dan Pelatihan (LKP) as a non-formal educational institution or a vocational school must be able to prioritize learners' valuable and adaptive learning knowledge, skills and characters in its digital learning platform for better competition and adjustment in this digital and technological learning environment, in the fifth industrial revolution era. At present, learners will face many new challenges as it is estimated that in 2045, 35% of many general jobs will be lost, and the other 65% will be transformed into new jobs related to technology and information. Here, LKP should always pay attention to the needs of the domestic labor market by constantly increasing their students' capability with many new digital and technological learning plans and designs. This is for enriching the students' adaptive skills in the global competition and competing professionally in Indonesia and international setting. By optimizing the potential of the Millennial Generation (Z-Gen) students or Digital Natives learners who are always active in using social media and fast learners, LKP will play a big role in helping these students develop their learning achievement and tackle negative learning effects. mailto:sitinurulhiday@gmail.com mailto:sitinurulhiday@gmail.com 31 | IJET| Volume. 11, Issue 1. Juli 2022 Copyright 2022 Siti Nurul Hidayah, is licensed under Creative Commons Attribution-ShareAlike 4.0 International License. Responding to these digital and globalization challenges, LKP Gazebo English Course, located in Trenggalek, East Java, Indonesia, is committed to developing the community's learning and working skills by providing new educational services, especially in the field of English language skills. This is to support the government educational programs that aim to strengthen learners' learning characters for a better Indonesia. One of the crucial educational programs designed by LKP is an English online learning course in the form of a Massive Open Online Course (MOOC) with a Learning Management System (LMS) based. This is a start-up program that only focuses on teaching English to English teachers, especially for teaching speaking skills. This online course is significant as teachers need a new well-designed distance learning program, where many teachers with limited exposure to learning across Indonesia gain more benefits from new teaching and technological skills in this course. Generally, this online course is more flexible and open for many teachers in Indonesia. In this study, the MOOC proposes a new online learning platform and the latest innovation in English teaching and learning methodology in Indonesia. As an adaptive and flexible online learning platform, MOOC gives a new face to Indonesian online learning practice where many teachers can gather and have face-to-face learning situations in their places with more accessible learning resources and broader internet access. In a study on MOOC in English language skills, MOOC was considered one of the technological innovations that could support teachers' professionalism, generate more dynamic, motivating, collaborative and successful learning processes (Gonçalves & Goncalves, 2019). Massive Open Online Course (MOOC) MOOC, an innovation in distance learning and education, was first introduced by George Siemens, Stephen Downes and Dave Cormier at the University of Manitoba, Canada in 2008 and emerged as a popular distance learning mode in 2012 (Joseph, 2020; Siemens et al., 2020; Kim et al., 2015; Teixeira & Mota, 2014). MOOC is an interactive, open, and systematized learning method to achieve certain target skills and knowledge with 24-hour access to information, consisting of learning from videos and teleconferences, with an integrated online learning system, using the help of instructors from outside the scope of the participant, with the benefit of cost-effectiveness. MOOCs can reach large groups of participants online and allow interaction among learners of different ages, cultures and nationalities. In this regard, currently, MOOC has received a lot of attention from the media and has received significant interest from higher educational institutions. In line with the discussion on MOOC by Siemens, et.al (2020), Baturay (2015) mentioned the characteristics of MOOC learning: 1) open, where participation in MOOC is free and open to anyone with access to the internet. A person may take more than one course and all contents are open to all participants; 2) participatory, where learning in MOOC is enhanced by participation both in the creation and sharing of personal contributions and in interaction with other people's contributions. However, participation is voluntary, and 3) can be distributed, where MOOC is based on a connectivity approach. Therefore, any knowledge in this learning mode must be distributed throughout the network of participants. Most of the course activities occur in a social learning environment, where participants interact with their instructors and peers to discuss and comprehend various learning materials. Furthermore, Baturay (2015) stated that the MOOC teaching mechanism had wide-ranging formats, including lecture videos, assessments, discussion forums, books, video conferencing, social media and additional video sources. 32 | IJET| Volume. 11, Issue 1. Juli 2022 Copyright 2022 Siti Nurul Hidayah, is licensed under Creative Commons Attribution-ShareAlike 4.0 International License. Many researchers have discussed the crucial role of online learning or MOOC in education. A well-designed and systematic MOOC has a positive role in improving students' learning achievement. There have been many discussions about the significance of MOOC in improving student competence. Joseph (2020) stated that MOOC was an online learning system that was flexible, open and cost-effective to improve the quality of learning, while Kovanović, et al. (2019) argued that the prominent role of conducting MOOC was clear, it had a big influence on students' positive learning outcomes. The implementation of well-designed and varied strategies of MOOC affected more effective training results (Rafiqa, et al., 2019). Furthermore, there was a strong influence between students' independent learning in MOOC and the achievement of learning targets (Handoko et al., 2019). However, there is also a large number of online learning participants who typically did not complete the MOOC course due to some factors. Some significant factors that influenced the students' drop-out rate in MOOC learning were personal, the MOOC platform and design (Aydin & Yazici, 2020). The students who dropped out of the course were busy with other working responsibilities. Poor MOOC platform and design discouraged the students from continuing learning. Similarly, in his mixed research methods, Santillán-Rosas & Heredia- Escorza (2019) reported that the effective learning results were highly interrelated to well- designed learning strategies and MOOC design. The MOOC program's better learning strategies and digital interventions affected more sustainable learners' professional development. In the MOOC study of English teaching, Gao (2019) reported that a well-designed MOOC and blended learning mode in the English language would increase classroom efficiency and improve students' learning enthusiasm and awareness. This also assisted them in having independent learning and helped them improve their language skills. Furthermore, Gonçalves & Goncalves (2019) affirmed in their study that systematic MOOC learning as a technology-based learning could increase teachers' professionalism and create a more fruitful teaching and learning process in English language classes. This research focuses on the implementation of MOOC in the language teaching context, especially teaching speaking, that will be discussed in the following section. Teaching Speaking Teaching is the activity of presenting and assisting someone in learning how to do something (Brown, 1987). In language teaching, teaching is about presenting the language elements to the learners and helping them understand the language that is being studied. Ur (1991, p.11) defines the teaching process as "a process of presenting of new words or grammatical structures and illustrating clear input from spoken and written texts as well as other instructions and discussions (assignments)". Meanwhile, Brown (2004, p. 140) outlines speaking as the result of a linguistic arrangement that is built creatively by selecting lexicons, and structures and deals with arranging words and phrases in a set of structures to deliver a conversational discourse. Nation & Newton (2009, p. 152) assert that speaking is about "how to develop aspects of fluency and accuracy (vocabulary, grammar, pronunciation) ."Meanwhile, Weir (1998, pp. 147 - 148) adds that several aspects are analyzed in oral competencies such as suitability, vocabulary, grammar, pronunciation, fluency, and content. 33 | IJET| Volume. 11, Issue 1. Juli 2022 Copyright 2022 Siti Nurul Hidayah, is licensed under Creative Commons Attribution-ShareAlike 4.0 International License. From the above definitions, it can be concluded that the success indicators in speaking English are Grammar, Vocabulary, Pronunciation, Fluency, Appropriateness, and Content. The six indicators will be used in testing the participants’ speaking skills in this online course study, which will be evaluated with Kirkpatrick’s 4-Level Evaluation Model. Kirkpatrick's 4-Level Evaluation Model Kirkpatrick's 4-Level Evaluation Model is an evaluation model that allows researchers to make a series of training evaluations, both in simple and complex training, in 4 (four) stages of evaluation (Bates, 2004). This evaluation model is used to determine the effectiveness of the training program and achieve some targeted goals. This evaluation model aims to gather information about the training programs' effectiveness, participants, facilitators/teachers, training designs/methods, resources and facilities. Kirkpatrick's evaluation model has four stages of evaluation which include reaction level, learning level, behavior level and result level. The four levels of the Kirkpatrick evaluation model are carried out in two ways of evaluation, namely formative and summative. Formative evaluation is intended to determine the success during the program, while summative evaluation is to determine the success at the end of the program. Purpose of the Study The purpose of this study is to determine the benefits and constraints of implementation of MOOC entitled “Fun Teaching for Fluent Speaking” with a Learning Management System (LMS) based to improve teachers' English teaching competence. To assess the success of the implementation of this online course, an evaluative analysis was used to analyze the data. The evaluation method used is the Kirkpatrick 4-level Evaluation Model. Kirkpatrick's 4-level evaluation model is an evaluation approach that includes the reaction level, learning level, behavior level and result level. The reaction level helps researchers evaluate the learners' reactions to the implementation of online courses. This includes participants' reactions to courses, materials, instructors, learning methods and tools. At the learning level, the level of learners' learning acquisition was identified. The behavior level is used to determine the extent to which participants can apply the knowledge they attain from the online course in their teaching workplace. And in the end, at the result level, it is to evaluate the extent to which the online course has generally influenced the teachers' professionalism. 2. METHOD MOOC Fun Teaching for Fluent Speaking is an online program organized by LKP Gazebo English Course Trenggalek in collaboration with SEAMOLEC and the Ministry of Education and Culture. This online course is held regularly, where the first batch of online courses has been organized since December 2018. Some following courses were organized with more participants until 2021. In this study, 126 participants participated from formal and non-formal educators and other private sectors around Indonesia. This online course was conducted online, shown on the website page of kursusdaring.kemdikbud.go.id which the participants can access for 24 hours. The curriculum, syllabus, lesson plans and evaluation in this online course are prepared and implemented by the instructor with reference to the English curriculum issued by the vocational directorate. After 34 | IJET| Volume. 11, Issue 1. Juli 2022 Copyright 2022 Siti Nurul Hidayah, is licensed under Creative Commons Attribution-ShareAlike 4.0 International License. the lesson plans were compiled, the design team then wrote a video script, recorded the video and then presented the videos on the Indonesian online course web page. Procedures of MOOC Implementation The implementation of the MOOC Fun Teaching for Fluent Speaking was conducted with some procedures. Firstly, the participants registered on the website, then systematically accessed the video material, modules, and assignments online. They accessed the website page of kursusdaring.kemdikbud.go.id. for more than a month. Participants in the MOOC had full access to online learning, such as: the pre-test, 4 (four) topics in 8 video materials in 4 (four) weeks, video conferencing 2 (twice a week) and 3 (three) assignments and final assignments. The course activities were completed by providing certificates to the participants. The MOOC online course included eight video materials with the theme of teaching speaking that emphasized the participants' understanding through video teleconferences and were held four times with the research team as the instructors using the Webex. The first video teleconference started with a technical introduction to the online courses through the Webex application. The learning topic in the first video teleconference was "The Introduction to the Knowledge of Speaking Teaching for Interactive Classrooms ."During the implementation, participants experienced several problems, such as log-in problems, website access, and signal problems that were difficult to access, so participants could not access them straightforwardly. The second video teleconference topic was "Designing Topics for Speaking Practice for Fun Teaching ."In this video teleconference, participants had better internet access. The participants could interact and access the learning topics easier, although there were still signal and audio problems. The third video teleconference focused on discussing "The Techniques of Teaching Speaking," which discussed some techniques in teaching speaking. In this learning phase, signal and audio problems still occurred. The fourth video teleconference topic was "Classroom Management for A Successful Class ."In this activity, participants did not experience any internet access problems because they were familiar with the online learning application. However, the participants still experienced some unstable signal and audio problems. Evaluative Research in Language Teaching Evaluative research in language education aims to determine the success, quality, and effectiveness of the learning implementation in terms of various contexts, inputs, processes, and products. This method is a part of applied research. This research uses evaluative research, which aims to reveal the benefits and constraints of the implementation of the MOOC Fun Teaching for Fluent Speaking. This research discusses the evaluation of the implementation of the MOOC Fun Teaching for Fluent Speaking to improve the teachers' English teaching competence in Indonesia. The research approach uses both qualitative and quantitative research approaches to analyze data with numbers. Qualitative research is research that uses little numbers in its data and is prepared to discuss the problem details related to human understanding and interaction (Silverman and Marvasti, 2008). Meanwhile, quantitative research includes numerical analysis with a few words. In this study, these two approaches were employed simultaneously to analyze the collected data comprehensively. 35 | IJET| Volume. 11, Issue 1. Juli 2022 Copyright 2022 Siti Nurul Hidayah, is licensed under Creative Commons Attribution-ShareAlike 4.0 International License. Kervin et al. (2016) describe five steps in collecting data: identifying research focus, collecting data according to the topic, selecting data collection techniques, selecting places and participants, and developing appropriate instruments for data collection. In this study, the research team used research instruments validated by experts. The data collection techniques include observation, interviews, documents and questionnaires. Kirkpatrick’s Four Stages Evaluation Method Particularly, this study used four stages of evaluation from Kirkpatrick to evaluate the effectiveness and the constraints of the MOOC implementation. As mentioned earlier, the four stages of Kirkpatrick’s evaluation were reaction level, learning level, behavior level and result level. The Kirkpatrick evaluation model offered two diverse evaluation frameworks that were formative and summative. A formative evaluation was intended to analyze the success during the program, while summative evaluation was aimed to assess the success at the end of the program. The four stages of Kirkpatrick's evaluation level will be illustrated in the following paragraphs. The reaction level aims to obtain data and information about the level of participants’ satisfaction in the online course, which includes the participants' feelings, thoughts, and desires during the learning process, sources and the environment. This level observes the participant's satisfaction, perception, activeness and level of willingness to apply the learning results in their workplace. In this study, the assessments were made on the learning content, teaching material, environment, teaching delivery by the instructor and participants’ expectation or passion to join a similar program, schedule, facilities, instructor quality, learning media, and instructor friendliness. The techniques used were questionnaires, happy sheets, interviews, and observations. Evaluation of the level of reaction was carried out immediately after the training program was completed (Clementz, 2002) The learning level is used to evaluate / determine the participants' success level in the learning process (transfer of learning), including the mastery of knowledge, skill levels and attitudes before and after the training program. The learning level is used to measure the skills, knowledge, attitudes, self-confidence and commitment of participants to the teaching materials presented in the training. At this stage, evaluation is aimed at extracting information about what the participants learned and did not learn, certain knowledge, and changes in attitudes. In this study, the evaluation of the implementation used pre-test and post-test techniques, observation by instructors or peers, interviews (before and after), or performance records (assignments). The learning stage was evaluated before, during and after the training program (Clementz, 2002). At the behavior level, the evaluation is carried out to determine whether participants can apply the learning results in their workplace, by measuring the distribution and retention of knowledge in the workplace, processes, and systems in improving performance. The evaluation process is achieved based on participants' behavior changes in their workplace, new knowledge and skills used, opinions and behaviors shown in their work environment, and changes in the environment. The techniques used in this study include interviews (by phone, face to face), direct observation, document analysis, performance appraisals, questionnaires, 360-degree feedback, or self-assessment. Evaluation of this stage was performed after the program was undertaken (Clementz, 2002) The last level is the resulting level and the final result of the program evaluation to determine the impact of the training program on the institution/scope of work of the participants. The purpose of this stage is to measure the impact and productivity of the training 36 | IJET| Volume. 11, Issue 1. Juli 2022 Copyright 2022 Siti Nurul Hidayah, is licensed under Creative Commons Attribution-ShareAlike 4.0 International License. for the participants' better learning environment. In this process, there is an increase in quality, a more efficient process, effectiveness, and more measurable interaction between participants. This evaluation technique in this study used observation techniques from participants' work/product results, interviews and questionnaires to their students or school principals. Evaluation of the result stage could be attained in the short or long term (Clementz, 2002). 3. RESULT Diverse Participants’ Profiles The number of participants who registered for the online course "Fun Teaching for Fluent Speaking" in the second batch was 126, coming from various elements of society, both preservice and prospective teachers. Based on questionnaire data, it showed that 22.7% of participants were teachers from formal and non-formal institutions, 36% of them were undergraduate students, 17.3% of participants were from private sectors and 24% of participants were from participants with unemployment and other various jobs coming from various places in Indonesia. These various participants' jobs seemed to influence other findings in this study. = Graph 1. Participants' Job in MOOC Participants' ages ranged from 18 to 40 years. Most participants had less than 1 year of teaching experience with a percentage of 80%. 18.7% of the participants had between 2 and 4 years of teaching experience, and the rest had teaching experience of more than five years. Graph 2. Participants Age in MOOC As many as 80% of participants accessed the internet through their cell phones with independent internet data packages, while the remaining 20% used WIFI to access the internet. 37 | IJET| Volume. 11, Issue 1. Juli 2022 Copyright 2022 Siti Nurul Hidayah, is licensed under Creative Commons Attribution-ShareAlike 4.0 International License. Graph 3. Participants’ Internet Access in MOOC According to the respondents (75 respondents), the quality of the internet used is classified as normal, where 60% of the participants marked with the score of 3 out of 5 as seen in the graph below. Meanwhile, 26.7% of respondents stated that they had good internet access and only 1.3% of participants experienced excellent internet access. Internet access constraints were a challenge in implementing this online course. Graph 4. Participants’ Internet Quality in MOOC Result of 4-Level Kirkpatrick Evaluation Model The evaluation of the MOOC implementation using Kirkpatrick 4-level Evaluation Model will be discussed in the following sections. From the reaction level, overall, the participants' reactions to the instructor's teaching were positive. They showed a high level of confidence toward the instructor's ability, acquisition of teaching materials and appropriate learning media. With 69 participants who responded to the questionnaire, the finding revealed that the participants' motivation increased 38 | IJET| Volume. 11, Issue 1. Juli 2022 Copyright 2022 Siti Nurul Hidayah, is licensed under Creative Commons Attribution-ShareAlike 4.0 International License. and the teaching methodology was effective and interesting. The participants felt that they could improve their teaching, as shown in the graphic below. In addition, 78.2% of online course participants highly agreed with their instructors' teaching effectiveness. It can be concluded that implementing the Fun Teaching for Fluent Speaking course has been successful in providing more teaching benefits for most participants. Graph 5. Participants' reactions towards the learning process Furthermore, as shown in the graph below, the participants felt satisfied participating in the online course, mainly because of the teaching teams' excellent responses and services. 72.4% of participants agreed they were satisfied and enjoyed the learning process. The majority of participants showed their satisfaction with the course, while 24.6% felt the course was on average level and only 2.9% of the participants felt dissatisfied with the course. Graph 6. The Level of the Participants’ satisfaction in the online course The next result is the learning level as shown in the following table. Generally, in assessment data from pre-test and post-test at the learning level, there was an increase in the participants’ learning level. The progress was around 13% where the participants’ learning score increased from the pre-test average score of 69.36 to the post-test average score of 78.68. This was a great progress as the participants' grammar skills were still poor. In a relatively short course, 4 weeks online course, it was categorized as successful as the participants were able to attain the basic English knowledge and skills. However, the further teaching training is essential to improve their teaching competency into higher levels. Table 1. The Learning Progress 39 | IJET| Volume. 11, Issue 1. Juli 2022 Copyright 2022 Siti Nurul Hidayah, is licensed under Creative Commons Attribution-ShareAlike 4.0 International License. Skor Pre-test Skor Post-Test Progress 69,36 78,68 13% At the behavior level, in this 2nd batch of Fun Teaching for Fluent Speaking online course, the participants conducted self-assessments before and after the online course. The results of the participants' self-assessment on their teaching confidence are presented in the table below. The data revealed that the teachers’ teaching confidence was enhanced. The most prominent increase was 63%, where the participants stated that they were confident to solve teaching difficulties. These results indicated that the Fun Teaching for Fluent Speaking course was successful in providing the appropriate solutions to the teaching problems in the classroom. Table 2. The Participants’ Confidence Level after Joining the Online Course Indicator Level of Agreement before the course (%) Level of Agreement after the course (%) Progress (%) Participants’ confidence in teaching in the classroom 47,3 58.8 24 Participants master the teaching material 52,8 54.4 3 Participants can solve all classroom problems 45,1 73.5 63 Participants master English both in written and spoken 47,3 51,5 8 On the other hand, there was a slight increase (3% - 8%) in the participants’ language acquisition and understanding. Additionally, from the survey data, 72% of the participants were students and workers in the private sector. Only 27.9% of the participants were pure English teachers. From this, it can be concluded that there was a possibility that the participants’ language background was varied. Majority of them had low English language skills, while others were already advanced in their language skills. From the level of participants' self-esteem, regardless of their job profile, there was a 24% increase in their teaching self-confidence. This figure shows that the online course has successfully developed the participants' self-confidence and motivation to improve their skills as English teachers. The following table will illustrate the participants' self-assessment. The biggest change in the teaching practice shown in the data was that the participants could update their teaching method, with a progress of 19%. Then, another change was the participants' teaching commitment with an 18% increase. The data showed that after participating in the course, the participants had higher commitment in improving the teaching quality and valued the 40 | IJET| Volume. 11, Issue 1. Juli 2022 Copyright 2022 Siti Nurul Hidayah, is licensed under Creative Commons Attribution-ShareAlike 4.0 International License. importance of updating their teaching knowledge and skills to achieve more effective teaching outcomes. Table 3. Participants’ self-assessment on their teaching Indicator Level of Agreement before the course (%) Level of Agreement after the course (%) Progress (%) Participants update their teaching methods 62,7 75 19 Participants include culture in teaching speaking 63,8 70,6 10 Participants create positive habits in the classroom 71,5 75 5 Participants help the students improve their knowledge in teaching 64,9 76,5 18 Participants use summative and formative assessment 63,8 73,6 15 Furthermore, to assess the participants' behavior level, they were interviewed randomly to evaluate their behavior after learning teaching in the MOOC. Majority of the participants revealed their satisfaction in the MOOC learning process. For example, Sita said about her satisfaction: "After participating in this online course, I get new and better knowledge. I also learn some teaching soft skills and understand that teachers should deliver their teaching with humor". Another participant also felt satisfied that the teaching strategies were applicable to teach another lesson with similar context. As confirmed by Tina: "The instructors are so entertaining and impressive in teaching. Moreover, I can apply the method in another lesson, for example in science class". Additionally, using new teaching approaches in the MOOC, Fara stated that her students were more enthusiastic and attained more understanding of the lesson contents. As a novice teacher, another participant exposed another benefit of joining this program. As stated by Andi," After participating in this online course, I feel I can teach better with many new teaching strategies and I am more confident in teaching English''. In summary, the participants proposed that online learning improved their teaching competences and made them more confident in their teaching. At the Result level, the questionnaire's data showed that the online course has developed the participants’ teaching knowledge and skills. Based on their student’s opinion, the participants who played a role as teachers in their classroom, evidently demonstrated outstanding progress on their teaching quality after joining the online course. The biggest progress was the participants’ involvement in the interactive classroom activities, with 94,74% of the students’ agreement. Mastery of teaching material was also considered exceptional with a percentage of 97.37%. Similarly, the participants’ teaching knowledge and confidence was 41 | IJET| Volume. 11, Issue 1. Juli 2022 Copyright 2022 Siti Nurul Hidayah, is licensed under Creative Commons Attribution-ShareAlike 4.0 International License. also remarkable. In terms of media used in the online course, the majority of the participants also applied the media in their teaching practices with 89.47% score. Table 4. Students’ opinion on the participants role as teachers after joining the course The Statement The percentage of the Students’ Agreement Participants master the material 97,37 Participants use the audio-video media in teaching 89,47 Participants teach in a fun way 84,21 Participants deliver the lesson effectively 89,47 Participants are responsive towards the students’ questions 73,68 Participants ask the students to ask and get involved in the classroom activities 94,74 Participants use varied teaching methods 89,47 The final result level focuses on the students’ learning environment in the classroom after their teachers (participants) joining the course. From the data, it can be identified that the students showed more active participation in every English-speaking activity. The participants’ discipline was also getting better where the students were more well-organized and on time in submitting their assignments. The students engaged more in the discussion sessions and they experienced a more positive classroom atmosphere. Also, the participants were also energetic in using media and technology in their teaching activities. In teaching and learning activities, the students also paid more attention to the teacher's explanation. This situation created a more productive teaching and learning process. Table 5. Students learning condition in the classroom Indicator Students’ agreement Percentage Students were active in every activity in speaking classroom 67,6 Students did all the assignments on time 72,1 42 | IJET| Volume. 11, Issue 1. Juli 2022 Copyright 2022 Siti Nurul Hidayah, is licensed under Creative Commons Attribution-ShareAlike 4.0 International License. Students were active asking questions about the lesson 63,3 Students were active in using the media and technology in learning in the classroom 70,6 Students paid attention when the participants were teaching 75 The Benefits and Problems in MOOC Implementation Overall, the MOOC offered high benefits for the participants' teaching improvement. Most participants (69 participants) responded to the questionnaire, with the majority of participants showing their positive responses on the benefit of the MOOC implementation. 81,2 % of the participants shared their high score of agreement, between 4 and 5 score. Only fewer participants (18.8%) affirmed their average score for their agreement of the MOOC benefits for their teaching development. Graph 7. The importance of MOOC for the participants Through a series of evaluation processes, the data showed some benefits of the MOOC implementation to the participants' better teaching knowledge and skills. The participants reported that they gained more benefits from various learning aspects in the MOOC, such as the teaching videos, teaching topics, video teleconferences, WhatsApp group forum for discussion, teaching module, and the assignments. Various teaching topics and videos were two principal aspects that were essential for the participants' teaching aptitude, with the highest 43 | IJET| Volume. 11, Issue 1. Juli 2022 Copyright 2022 Siti Nurul Hidayah, is licensed under Creative Commons Attribution-ShareAlike 4.0 International License. percentage, 40,6% for teaching topics and 34,8% for the teaching videos. Less participants or 8,7% participants also loved the video teleconference. Graph 8. Some learning aspects that provide some benefits for the participants Meanwhile, during the online course implementation process, the participants faced some constraints during participating in the MOOC, including limited internet access and video teleconference access, challenging assignments, and difficulty of mastering the teaching materials and the registration process. The internet access presented as the most problematic in the MOOC application, with more than 72,5% participants' affirmation. The last challenge was registration with only 11, 6% of participants experiencing it. It was reasonable as they did not understand the registration process in the first time joining the program. Graph 9. Some constraints of MOOC Implementation 4. CONCLUSION Through a series of evaluation processes, it can be concluded that the benefits of the implementation of the MOOC Fun Teaching for Fluent Speaking include the participants' excellent teaching and skills achievement. Specifically, they were satisfied with various teaching topics and videos designed in the online course. They also considered the video teleconferences as prominent learning parts where they could directly interact with their instructors and peers within interactive and reflective discussions. However, they also faced some difficulties in the MOOC. The internet access and audio video problems in the video 44 | IJET| Volume. 11, Issue 1. Juli 2022 Copyright 2022 Siti Nurul Hidayah, is licensed under Creative Commons Attribution-ShareAlike 4.0 International License. teleconference process were the most problematic for developing their teaching competency. Completing the assignments of teaching speaking was also challenging in the online courses. From all the study findings, several recommendations are offered to develop better MOOC design and implementation in the future. The majority of the research participants were preservice teachers; conducting a similar study to diverse novice and professional teachers in Indonesia will be essential to reach diverse, more effective and valuable research outcomes. In addition, more MOOC courses with more interactive and entertaining teaching topics and pre- recording videos are essential in supporting the teachers to gain more teaching credentials and competencies. Finally, along with both formal and informal educational institutions, the government should provide MOOC support and facilities for schools, teachers and students and collaborate with many other MOOC providers to broadcast wider and more intensive online learning courses for teachers’ professional development in Indonesia. REFERENCES Aydin, I. E., & Yazici, M. (2020). Drop-Out in MOOCs. Turkish Online Journal of Educational Technology-TOJET, 19(3), 9-17. Bates, R. (2004). A critical analysis of evaluation practice: the Kirkpatrick model and the principle of beneficence. Evaluation and program planning, 27(3), 341-347. Baturay, M. H. (2015). An overview of the world of MOOCs. Procedia-Social and Behavioral Sciences, 174, 427-433. Brown, H.D. (1987). Principles of language learning and teaching (2nd ed.). Prentice-Hall. Brown, H.D. (2004). Language Assessment: Principles and Classroom Practices. Pearson Longman. Clementz, A. R. (2002). Program level evaluation: Using Kirkpatrick's four levels of evaluation to conduct systemic evaluation of undergraduate college programs. Unpublished paper. Retrieved from http://web. Bryant. edu/~ assess/Program_Level_Evaluation. doc. Gao, B. (2019). Highly Efficient English MOOC Teaching Model Based on Frontline Education Analysis. International Journal of Emerging Technologies in Learning (iJET), 14(06), 138-146. Gonçalves, B. M. F., & Goncalves, V. B. (2019, June). Professional development in MOOC: Teacher’s motivation. In 2019 14th Iberian Conference on Information Systems and Technologies (CISTI) (pp. 1-6). IEEE. Handoko, E., Gronseth, S. L., McNeil, S. G., Bonk, C. J., & Robin, B. R. (2019). Goal setting and MOOC completion: A study on the role of self-regulated learning in student performance in massive open online courses. International Review of Research in Open and Distributed Learning, 20(3), 40-58. Joseph, M. R. (2020). Role of Moocs In Modern Education. Journal of Applied Science and Research, 8(2), 13-17. Kervin, Vialle, W., Okely, T., Howard, S. J., & Herrington, J. (2016). Research for Educators (Second edition.). Cengage Learning Australia. Kim, B., Ying, W., Pushpanadham, K., Yamada, T., Lee, T., Fadzil, M., & Gil-Jaurena, I. (2015). MOOCs and educational challenges around Asia and Europe.. DOI:ISBN 978- 45 | IJET| Volume. 11, Issue 1. Juli 2022 Copyright 2022 Siti Nurul Hidayah, is licensed under Creative Commons Attribution-ShareAlike 4.0 International License. 89-20-01809- 1(93370)IAL (Institute for Adult Learning) Singapore Sg. (n.d.). Retrieved from https://www.udemy.com/user/ialsg/ Kovanović, V., Joksimović, S., Poquet, O., Hennis, T., de Vries, P., Hatala, M., ... & Gašević, D. (2019). Examining communities of inquiry in Massive Open Online Courses: The role of study strategies. The Internet and Higher Education, 40, 20-43. Nation, ISP. & Newton, J. (2009), Teaching ESL/EFL Listening and Speaking. Routledge New York. Rafiqa, K. R. M., Hashimb, H., & Yunusc, M. M. (2019). MOOC for training: A review of the variations of MOOC. International Journal of Innovation, Creativity and Change, 5(6), 90-98. Santillán-Rosas, I. M., & Heredia-Escorza, Y. (2019, October). Learning strategies and digital interventions: An analysis in the context of education for sustainable development. In Proceedings of the Seventh International Conference on Technological Ecosystems for Enhancing Multiculturality (pp. 976-980). Siemens, G., Rudolph, J., & Tan, S. (2020). “As human beings, we cannot not learn”. An interview with Professor George Siemens on connectivism, MOOCs and learning analytics. Journal of Applied Learning and Teaching, 3(1). Silverman, D. & Marvasti, A. (2008). Doing Qualitative Research: A Comprehensive Guide. California: Sage Publications. Teixeira, A. M., & Mota, J. (2014). 3 A Proposal for the Methodological Design of Collaborative Language MOOCs. In Language MOOCs (pp. 33-47). Science Migration Ur, P. (1991). A Course in Language Teaching, Practice and Theory. Cambridge University Press, Cambridge. Weir, C.J. (1998). Communicative language testing. Prentice Hall New York. https://www.udemy.com/user/ialsg/