İpek, Ö. & Mutlu, H.T. (2022). Investigation of the New Paradigm in EFL Using Systems Approach: A Mixed Method Distance Education Program Evaluation Study. International Online Journal of Education and Teaching (IOJET), 9(2). 690-713. Received : 22.12.2021 Revised version received : 28.02.2022 Accepted :01.03.2022 INVESTIGATION OF THE NEW PARADIGM IN EFL USING SYSTEMS APPROACH: A MIXED METHOD DISTANCE EDUCATION PROGRAM EVALUATION STUDY Research article Corresponding Author: Ömer Faruk İPEK ORCID: 0000-0003-1921-3332 Bolu Abant İzzet Baysal Unıversıty, Turkey theipekk@gmail.com, Hakan Tahiri MUTLU ORCID: 0000-0002-8964-2696 Bolu Abant Izzet Baysal UniversityTurkey tahirimutlu@ibu.edu.tr Biodatas Ömer Faruk İPEK is an English language instructor at Bolu Abant Izzet Baysal University, School of Foreign Languages, Preparatory Department in Bolu, Turkey Hakan Tahiri MUTLU is an academic at Bolu Abant Izzet Baysal University, Faculty of Economics and Administrative Sciences, Department of Numerical Methods in Bolu, Turkey. Copyright © 2014 by International Online Journal of Education and Teaching (IOJET). ISSN: 2148-225X. Material published and so copyrighted may not be published elsewhere without written permission of IOJET. mailto:tahirimutlu@ibu.edu.tr http://orcid.org/xxxx http://orcid.org/xxxx International Online Journal of Education and Teaching (IOJET) 2022, 9(2), 690-713. 691 INVESTIGATION OF THE NEW PARADIGM IN EFL USING SYSTEMS APPROACH: A MIXED METHOD DISTANCE EDUCATION PROGRAM EVALUATION STUDY Ömer Faruk İpek theipekk@gmail.com Hakan Tahiri Mutlu tahirimutlu@ibu.edu.tr Abstract Although some education institutions have implemented open or online education in line with their own needs so far, distance education did not have national recognition until the pandemic that swept the world at the beginning of 2020. After this date, many institutions underwent transformation from conventional face-to-face education to distance or online education, and they had to change their programs accordingly. For this reason, this simultaneous explanatory mixed method study was conducted to examine the effectiveness and efficiency of distance education program of an English language institution situated in the northern part of Turkey. Five English language instructors and 403 English preparatory department students, five of whom were also interviewed, participated in this study. ‘Distance Education System Evaluation Scale (DESES)’, semi-structured interviews, observational journals and institutional documents were used as data collection tools. The results of quantitative data showed that students were mostly satisfied with instructors and least satisfied with the content of the program. Moreover, it is found out that students with higher grades gave high scores to DESES. On the other hand, from the results of qualitative data, while it is understood that the program was criticized negatively in terms of socialization, technical problems, teacher’s mood, pacing schedule, registration, attendance, feedback, assessment, language skills and distance education pedagogy; positive themes emerged as time and expenses, technical infrastructure, instructors and psychological factors. The results show that there are many issues that need to be considered for distance education implementations and pedagogy. Keywords: Distance education, EFL, curriculum, program evaluation, systems approach 1. Introduction Distance education has gained a lot of popularity in recent years (Aparicio et al., 2016) and this idea has come from the open education whose ratio is to enable learners study anywhere and anytime they wanted (Irawan et al., 2021). The reasons for the expansion of distance education can be listed as the development of technology, people's desire to save time and more interest in individuality. Keegan (1995) defines distance education as the separation of counterparts with the help of technology that gives freedom to both student and teacher to be in different settings and at different times. A wider definition is given by Saba (2003) that by using several technological tools in order to give the instruction, distance education is the separation of the teacher and the student. In the definition, it is also pointed out that distance learning has the convenience to reach the opportunity to study courses that learner was not able to take due to social, family, economic or other reasons. From this point of view, it can be concluded that there is not a socio-economic superiority among learners that stands as one of the irrevocable matters in education (Amiri & Maftoon, 2010). Definitions are necessary in order to comprehend the terminology, and key words may help everyone systemize their own definition for distance learning. Therefore, by using several key words, we can conclude that distance education is the system of teaching / learning process that enables learners to study and learn, and teachers to guide their learners at any time and any mailto:theipekk@gmail.com mailto:tahirimutlu@ibu.edu.tr İpek & Mutlu 692 place with the help of technology without any socio-economic superiority in equal circumstances. When we look at it from this perspective, we may list the advantages of distance education over traditional face-to-face education in many respects. For instance, Vaiz et al. (2021) state that distance education enables a more student-centered learning environment that permits learners reach knowledge themselves with or without any help of a teacher and it is not limited to time or place. Burr (2006) also explained that distance education can be more advantageous than face-to-face education. While face-to-face education is a process that must be done and completed within a certain time, it has been suggested that in distance learning process, learning can continue after one-to-one lesson is over. It has been emphasized that students can take active role in online discussion forums after the lesson, continue to practice with different mobile or online applications. Also, since individual learning speed and form will gain importance, learning will take place faster which motivates students even more. On the other hand, although many benefits can be listed, there are also several drawbacks of distance education mentioned in the literature. For instance, students may feel lonely because they are physically distant from the classroom environment (Littleton & Whitelock, 2005), which may lead to loss of motivation. There may also be a lack of instant pair support for students which can block the way students learn from each other (Johnson et al., 2001). Another problem is that there are problems such as not attending classes during long-lasting distance education. Finally, both for students and teachers, it can be mentioned that a lot of time should be spent for preparing new material. In addition, since many teachers received pre- service training before the spread of distance education, it may take a long time for teachers to adapt themselves to distance education processes (Hall & Knox, 2009; Talebi & Javidi, 2022). As the post-covid-19 pandemic requirement for distance education is new to many educational institutions, teachers, researchers, program designers and policy-makers, English program evaluation studies have not yet been extensively conducted or published. For this reason, it is necessary to examine the systems of English language teaching institutions that enable the transition from face-to-face education to distance education. 2. Conceptual Framework: Systems Approach There are many different approaches to program evaluation. Some of them are used quite popularly today in program evaluation research; for instance, Kirkpatrick's (1996) four-level program evalaution model, CIPP (context/input/process/product) model by Stufflebeam (1971), and utilization-focused model by Patton (1996). All these program evaluation models can be preferred by the researcher according to the focus points to be evaluated. However, in the current study, the system approach is used, where the focus of the evaluation process is on the connection of the education subsystems and components with each other and how these connections worked together to reach the program objectives. General systems approach was proposed by Bertalanffy (1968) and explained under two components: open system and closed system. In the closed system, the limits have been determined and it was suggested that the system does not have any interaction with other assets, that the system cannot change and will eventually disappear (Bertalanffy, 1972). This process is also called equilibrium in closed system. On the other hand, in the open system, it is explained that the sub-sections of the system are in constant interaction with each other and with the whole system and can change by interacting with other systems. Open systems, which interacts with each other, changes and evolves, is explained with the complexity approach. Complexity theory tells us that educational programs are not inanimate and inert entities, but a living mechanism that evolves and changes (Frye & Hemmer, 2012). In each entity that changes, it is indicated that the subsystems interact with each other and with other open systems; thus, it is claimed that education programs are complex and open-structured systems International Online Journal of Education and Teaching (IOJET) 2022, 9(2), 690-713. 693 (Mennin, 2010). In summary, in complex systems theory, the connection, interaction and communication of these open systems with each other is as important as how successful these subsystems are on their own separately. For this reason, in this study, as well as evaluating the success of sub-systems such as materials, technology used in distance education, weekly schedule and instruction, the connection of these sub-systems with each other is also examined. 3. Literature Review While distance education has become so widespread, its content and quality should be investigated by students, teachers, administrators, curriculum designers or researchers. In the literature, a great number of studies on face-to-face English as a foreign language (EFL) program evaluation study can be found (Chan, 2001; Coskun & Daloglu, 2010; Tunc, 2010; Tom-Lawyer,2014; Hsu, 2014; Efeoglu et al., 2018; Karabıyık & Mirici, 2018; Khorunnisa, 2018; Le & Tran, 2021). Moreover, although there are some studies for distance education implementations before the pandemic (Boehler, 1999; Pina, 2008, Tuzun et al., 2011), there are not many published studies on the EFL distance education programs. When we go through the literature, it can be seen that many different aspects of face-to-face EFL program evaluation are researched. For instance, Karatas and Fer (2009) studied the components of an English language course given at a university in Turkey that concluded it as context, input, process and product. Coskun and Daloglu (2010) investigated 4th year English Langauge Teching (ELT) teacher education program using Peacok’s (2009) model. Also, Salihoglu (2012) conducted research on pre-service ELT students and their instructors at a Turkish university. Moreover, Karakas (2012) studied an English language education program in order to reveal the weaknesses and the strengths of the program. Furthermore, Ilerten and Efeoglu (2021) investigated pros, cons and aspects to be improved in the intensive EFL program. On the other hand, Bilgic and Tuzun (2021) investigated the core issues and challenges of distance education programs at universities. Debes (2021) also looked into the advantages and disadvantages of distance learning at a university context. Sener et al. (2020) and Pastor (2020) also studied the teacher beliefs on online teaching experiences. Moreover, Tynan et al. (2015) emphasized the workload of teachers during online education. Also, Gur & van Schaak (2019), Ferretti et al. (2021) and Yoruk (2021) studied the assessment dimension of program evaluation of distance education. While there are many studies on face-to-face English language program evaluation studies, not many studies have been published on EFL distance education programs. Therefore, comprehensive empirical research on EFL distance education program evaluation is required. In line with the issues mentioned, the following research questions are investigated in the current study: 1) How is the program content of EFL distance education and its implementation? 2) What are the students’ thoughts on EFL distance education program implemented in the department? 2a) What are the statistical results of DESES regarding EFL distance education program evaluation? 2b) How do students view EFL distance education program according to the interview results? 3) How do EFL instructors working at the department evaluate EFL distance education program? 4. Method İpek & Mutlu 694 This study was designed as a mixed-method research. Mixed method is defined as combining the contents of both qualitative and quantitative research designs that is using both verbal and statistical results in order for the in-depth understanding of the specific case (Johnson et al, 2007). In this research, the scale designed to evaluate EFL distance education program called DESES (Ipek & Mutlu, 2022) consisting of 30 questions, semi-structured interviews both with students and instructors, institutional documents and observational journals are used in order to collect data. 4.1. Setting This study was carried out in the English preparatory department of a medium-sized state university in the northern part of Turkey. The school has been offering intensive English language education for nearly 30 years. Although the school provided face-to-face education until March 2020, it made a compulsory transition to distance education after this date. There are approximately 900 students in the school. These students start their education in September and study English until June. English education is given for two semesters. Every week, students are given 20 hours of English instruction which means approximately 600 hours of English education is taught in a year. Students who get 60 out of 100 from the proficiency exam at the end of academic year is considered successful and continues their education in their major departments. If they fail, they repeat English class one more year. 4.2. Sampling and Participants In this mixed method research, two types of sampling strategies are used. For quantitative part of this research, purposive sampling in which the researcher selects the participants for the appropriate representation for the whole group (Denzin & Lincoln, 2000) was used. For the qualitative part, random sampling where each member of the whole population has the equal chance to be selected is used to determine the students and instructors to be interviewed (Sharma, 2017). The participants consisted of 403 students (see table 1 below) who received distance education in the English preparatory department in 2020-2021 academic year. These students filled DESES and five of them were interviewed. The age range of the participants varies between 18-22 and students from 30 different departments participated. As it can be seen in table 1, optional means the students who study at English preparatory department voluntarily. In this group, students have the right to pass their majors even if they fail in English department. On the other hand, compulsory means that these students have to be successful in the English department in order to pass their major departments since 30 % or more of their courses in their major departments are given in English. GPA is the average scores that students obtained during their English education. Instructional section means whether the students are day or evening group students. While day courses are between 9 am and 3 pm, evening courses start at 5 pm till 10 pm. Lastly, level group indicates the level of the students. The levels are decided via the placement test at the beginning of the academic year. If a student gets a score between 0-50, they are placed in A groups. If the score is between 50-100, they are placed in B groups. C level accepts students only from English language teaching (ELT) and Translation and Interpretation department. Table 1. Demographics of the students participated to the scale International Online Journal of Education and Teaching (IOJET) 2022, 9(2), 690-713. 695 Variables N % Gender Female 259 64,3 Male 144 35,7 Total 403 100,0 Type of enrolment Optional 130 32,3 Compulsory 273 67,7 Total 403 100,0 GPA 0-39 10 2,5 40-59 57 14,1 60-79 160 39,7 80-100 176 43,7 Total 403 100,0 Section Day group 364 90,3 Evening group 39 9,7 Total 403 100,0 Level A 304 75,4 B 46 11,4 C 53 13,2 Total 403 100,0 Of the students participating in the study, 259 (64.3%) were female and 144 (35.7%) were male. While 364 (90.3%), of students were day students, 39 (9.7%) were in evening groups. Moreover, 130 (32.3%) were optional and 273 (67.7%) were compulsory. Considering the GPAs of the participants, 10 students (2.5%) were with a GPA between 0-39, 57 students were with a GPA between 40-59 (14.1%), 160 students with a GPA between 60-79 (39.7%) and 176 students (43.7%) were with a GPA between 80-100. Finally, when the level groups of the students participating in the research are examined, 304 students (75.4%) study at A level, 46 students (11.4%) at B level, and 53 (13.2%) at C level. Also, five instructors that taught English via distance education were interviewed. The demographic information of instructors can be seen in table 2 below. For ethical purposes, instructor participants are named as IP, and student participants are names as SP. Table 2. Demographics of instructor participants in the semi-structured interviews Participant Age Year of experience Major IP-1 47 22 BA- English Language and Literature IP-2 35 13 BA, ELT MA, Program Evaluation IP-3 32 6 BA, ELT IP-4 31 5 BA, ELT IP-5 30 5 BA, ELT MA, ELT Among five instructors participated in the semi-structured interviews, four of them were ELT majored and one of them held English language and literature degree. They all had more than five years of experience. While three of them held BA degree, two of them had MA degrees. Besides instructors, students were also interviewed. Below (table 3), the demographic information of the student participants can be seen: Table 3. Demographics of student participants in the semi-structured interviews Participant Age Major Department Level İpek & Mutlu 696 SP-1 23 22 A SP-2 21 13 B SP-3 19 6 C SP-4 18 5 A SP-5 18 5 A Five students participated in the semi-structured interviews in the end of the academic year. There are three A level, one B and one C level students. As the 75% of the whole sample in the quantitative section of the research consist of A level, A level participants in the interviews were more than B and C level students. 4.3. Data collection tools and procedure Four data collection instruments were used in order to collect the data (table 4). These are DESES, semi-structured interviews, documents, and observational journals. An observation journal was kept for 30 weeks from the beginning of the academic year. In addition, documents related to the program were collected at the same time. At the end of the academic year, DESES was applied to the students, and semi-structured interviews were conducted with both students and lecturers. Table 4. Data collection resources Data collection instrument Data Resources N / Period Purpose DESES Students 403 Decide the interview questions Interpret the thoughts on the program Semi-structured interviews Instructors Students 5 5 Interpret the thoughts on program Documents Website, E-mails Course materials Weekly schedule Exams 30 weeks Interpret the transition Observation Observational journal 30 weeks Interpret the transition As for data collection procedure, this study is designed as a simultaneous explanatory research in which quantitative and qualitative data is collected parallelly, and the interpretation of the results are given in the end by examining the two parties (Steckler et al., 1992). Here, the priority is usually given equally to the quantitative and qualitative data (Creswell et al. 2003). The sequence of data collection can be seen in figure 1 below: (September) ------------------------------------------------------ DESES (May) Interviews (June) International Online Journal of Education and Teaching (IOJET) 2022, 9(2), 690-713. 697 Figure 1. Data collection procedure At the beginning of academic year in September, when distance education was first implemented, observation and document collection processes started. These two data collection tools continued for thirty weeks until the end of June. At the end of the academic year, DESES was applied to 403 students and interview questions were prepared according to the scale results, and interviews were conducted with both students and faculty members. 4.3.1. The Scale: DESES DESES is a scale prepared by the authors (see Ipek & Mutlu, 2022) specifically to evaluate EFL distance education systems. In the scale, after five demographic questions, there are 30 five-point Likert type questions. In the demographic section, students' gender, attendance, GPA, level group and enrollment types were asked. In the second section, 30 statements are grouped under ‘Language Skills’, ‘Communication’, ‘Content Evaluation’, ‘Instructors’ and ‘Assessment’. In the scale, students are asked to choose 5 for strongly agree and 1 for strongly disagree to the statements given. The average scores obtained from DESES and its sub- dimensions were analyzed according to factors and the demographic characteristics of the students. In addition, the relationship of students' GPA with gender, type of enrollment and level group was examined. 4.3.2. Semi-structured interviews Thirteen semi-structured interview questions were prepared, and a pilot study was conducted with five students. The number of questions changed instantly during the interviews according to the answers given by the student. The duration of the interview for each participant varied between 15-20 minutes. Students were asked open-ended questions that they could easily comment on. After student interviews, instructors’ interviews started. Five instructors participated, and similar open-ended questions were asked to each of them. The list of interview questions is given below: 1. How did you feel when you learned that the courses will be given via distance education? 2. How did the transition to distance education begin and continue? 3. Did you have any training or experience on distance education? 4. Has there been any orientation or training regarding distance education? 5. How much did you know about distance education technologies before starting the classes? 6. Do you think the program helped you teach English well enough? 7. What areas of the curriculum did you see as effective or need improvement? 8. For which language skills is distance education suitable for EFL? 9. What do you think about the effectiveness of assessment? 10. Do you think the results of the exams are reliable? 11. How was communication established with both the administration and students in distance education? Document analysis & Observation Interpretation of the results İpek & Mutlu 698 12. In which areas of distance education did you have problems? 13. Are you satisfied with the results of distance education? 4.3.3. Documents Documents are any written papers or records which do not have to be the official or policy documents (Simons, 2009). The documents used in this study are the weekly schedule of the department, exams, e-mails sent by the department administration, the website that the department uses for communication purposes and course materials. An example document from the department can be seen in illustration 1 below. SCHOOL OF FOREIGN LANGUAGES ENGLISH PREPARATORY DEPARTMENT 2020-2021 ACADEMIC YEAR-SPRING TERM A GROUPS PACING SCHEDULE (MORNING & EVENING GROUPS) Feel free to make the possible changes in the week, like teaching the Writing Folder units. Week 25: 3rd-7th MAY DAY 1 DAY 2 DAY 3 DAY 4 SPEAKING 1 Inter. Unit 2 Inter. Unit 2 WF 12 Inter. Unit 3 Intermediate Unit 1 2 Inter. Unit 2 Inter. Unit 2 WF 12 Inter. Unit 3 3 Inter. Unit 2 Inter. Unit 2 WF 12 Inter. Unit 3 4 Inter. Unit 2 Inter. Unit 2 WF 12 Inter. Unit 3 Omissions: Unit 2: You may skip ‘key phrases’ and ‘write back’ part on page 29. Unit 3: Part 5 A-B-C on Page 33 Writing (Page 34) Reading (Page 35) could be assigned as homework. Part 3 C-D (page 36), Speaking (Pg. 36), Vocabulary Plus-Idioms (Pg. 37) Pages 38-39 ALL (Vocabulary, Function, Learn To, Speaking Parts) Write Back (pg. 41) *Extra time is scheduled for Unit 3 on the following week since there is Speaking Quiz 1 on 07.05.2021 / 13:30 Illustration 1. Example weekly schedule 4.3.4. Observation Journal According to Creswell (2009), observations need to be systemic and under discipline. It was also added that in order to capture the natural picture of the observed phenomenon, it is better for the observer to get involved in the process. Therefore, in this study, the role of the researcher is complete participant (Creswell (2013) that the observer fully become a part of the phenomenon. While being a complete participant, the researcher kept observation journals for data collection. 4.4. Data Analysis In this study, two different data analyses procedures were used for quantitative and qualitative data. For the qualitative data analysis, Creswell’s (2013) spiral data analysis procedures are used. Figure 2 below shows the steps and procedures. International Online Journal of Education and Teaching (IOJET) 2022, 9(2), 690-713. 699 Figure 2. Qualitative data analysis procedure (Creswell, 2013) For quantitative data, the data collection process was completed by applying the scale to 403 students with the online questionnaire method. First, the average scores obtained from the demographic section of DESES were examined. Then, answers given to 30 items in DESES were anaylzed. The data were analyzed with the help of SPSS 26.0 package program. First of all, with the help of the Skewness-Kurtosis coefficients, it was examined whether the data provided the normality assumption, and it was seen that the Skewness-Kurtosis coefficient for each item was in the range of ±2. The fact that the Kurtosis and Skewness coefficients are between +2 and -2 emphasizes the normal distribution of the data (Pallant, 2001). For this reason, parametric techniques were used in the analysis. Moreover, independent sample t-test was used to examine the differences in the mean scores obtained from the general and sub- dimensions of the scale according to gender, enrollment type and section; ANOVA was also used to examine the difference according to GPA and level. In the interpretation of the results, the level of statistical significance was accepted as 0.05. 5. Results 5.1. The content of the program and its implementation The researcher's observation journals, and institutional documents were used for the analysis of the transition to EFL distance education program. In order to understand the transition process and content of distance education better, face-to-face education implementations are also given comparatively. In March 2020, the central government suspended all face-to-face education implementations for three weeks, then announced that the education would be conveyed online in all levels. Thus, face-to-face education was stopped at the school, and online education started. This study examined the program of the school that started distance education in September 2020 and ended in June 2021. First of all, for communication, an orientation meeting for students was held at the beginning of the academic year, which lasted for an hour and a half via YouTube, where the students create and organize data read the text make notes in the margins form initial codesdescribe the context establish initial themes direct interpretation generaliztation present the in-depth picture İpek & Mutlu 700 were given information about the courses and the department. Students were also given the opportunity to ask questions. In the previous system, a second orientation meeting used to be held at the beginning of the second semester. However, it was not held in distance education. For instructional technical infrastructure, Microsoft Teams application was used, and classes were created here for all students and teachers. For communication throughout the year, e- mails and WhatsApp application were used. For feedback, instead of the evaluation meetings at the end of the semester with the teaching staff held in previous years, an assessment form was sent to the teaching staff and feedback was collected in this way. For the attendance requirement, which had been 80% in the previous system, it was abolished in online education. It was announced that students could also watch the recorded courses later. For assessment, different assessment tools were in distance education, and these include online quizzes, online midterms, speaking video projects, online writing exams, completing e-workbooks, online finals and end-of-year proficiency exams. Presentations applied in face-to-face education excluded in distance education. In terms of content, class hours were reduced to 20 in distance edcuation. A total of 600 lessons were held in 30 weeks. This had been 24 lessons per-week in face-to-face education, and a total of 720 lessons. E-book formats of the materials were used during distance education. For group A, British-originated beginner, elementary, pre-intermediate, and intermediate level textbook and their e-workbooks were used as the main sources. Also, a supplementary textbook prepared by the school, which contains grammar topics, exercises and paragraph writing rules and practices that were not included in the coursebooks were used. Listening and note-taking and academic reading books which had been used in face-to-face education in the previous year were removed from the program. Lastly, extracurricular speaking clubs which had been conveyed in face-to-face education were not held in distance education. 5.2a. Students’ thoughts on EFL distance education regarding DESES When the average values of the sub-dimensions of DESES, in which the participants received a minimum of 1 and a maximum of 5 points, are examined (table 5), it is seen that the Instructors (I) dimension has the highest average value. Content Evaluation (CE) was found to be the dimension with the lowest mean of the scale, where the overall mean was 3.67±0.66. When the distribution of the answers is made according to the average, it can be said that the CE dimension was generally answered as "I am undecided" and the other dimensions were answered as "I agree". The average scores given to DESES can be seen in table 5 below: International Online Journal of Education and Teaching (IOJET) 2022, 9(2), 690-713. 701 Table 5. Average and total values for DESES sub-dimensions and overall scale N Minimum Maximum Mean Std. Deviation 𝒙 Language Skill (LS) 403 1,00 5,00 3,7242 ,73916 Communication (C) 403 1,00 5,00 3,8238 ,79457 Content Evaluation (CE) 403 1,00 5,00 3,3815 ,89242 Instructors (I) 403 1,00 5,00 4,1902 ,70745 Assessment (A) 403 1,00 5,00 3,5151 ,87032 General 403 1,00 5,00 3,6678 ,66375 ∑𝒙 Language Skill (LS) 403 7,00 35,00 26,0695 5,17410 Communication (C) 403 7,00 35,00 26,7667 5,56197 Content Evaluation (CE) 403 8,00 40,00 27,0521 7,13932 Instructors (I) 403 3,00 15,00 12,5707 2,12234 Assessment (A) 403 5,00 25,00 17,5757 4,35160 General 403 30,00 150,00 110,0347 19,91246 When the total points that the students participating in the research can get from DESES are examined, it is seen that they can get a minimum of 30 and a maximum of 150 points for the whole scale. According to the answers given, the average total score obtained by the participants from the overall scale was 110.03±19.91. When the total points they can get from the sub-dimensions are examined; the minimum score that can be obtained from the LS dimension is 7 and the maximum score is 35; the mean total score of the participants was found to be 26.07±5.17. The minimum score that can be obtained from the C dimension is 7 and the maximum score is 35; the mean total score of the participants was found to be 26.77±5.56. The minimum score that can be obtained from the CE dimension is 8 and the maximum score is 40; the mean total score of the participants was 27.05±7.14. The minimum score that can be obtained from the I dimension is 3 and the maximum score is 15; the mean total score of the participants was 12.57±2.12. The minimum score that can be obtained from the A dimension is 5 and the maximum score is 25; the mean total score of the participants was found to be 17.58±4.35. It is also decided to present the results of DESES according to the demographic information of the participants which can be found in the following tables (Tables 6-9): Table 6. Examination of participants' DESES average scores according to demographic characteristics LS C CE I A Total Gender Female (N=259) 3,76 3,85 3,42 4,19 3,53 3,69 Male (N=144) 3,67 3,77 3,32 4,19 3,49 3,62 Total (N=403) 3,72 3,82 3,38 4,19 3,52 3,67 Sig. 0,261 0,292 0,285 0,915 0,722 0,290 Type of enrolment Optional (N=130) 3,69 3,84 3,34 4,19 3,60 3,67 Compulsory (N=273) 3,74 3,82 3,40 4,19 3,47 3,67 Total (N=403) 3,72 3,82 3,38 4,19 3,52 3,67 Sig. 0,484 0,784 0,563 0,992 0,155 0,998 GPA 0-39 (N=10) 3,53 4,01 3,04 4,03 4,08 3,65 40-59 (N=57) 3,27 3,56 3,09 3,93 3,26 3,35 60-79 (N=160) 3,56 3,77 3,27 4,11 3,39 3,56 80-100 (N=176) 4,03 3,95 3,60 4,36 3,68 3,87 Total (N=403) 3,72 3,82 3,38 4,19 3,52 3,67 Sig. 0,000 0,007 0,000 0,000 0,000 0,000 Section Day group (N=364) 3,73 3,84 3,38 4,20 3,51 3,68 Evening Group (N=39) 3,64 3,67 3,35 4,11 3,54 3,60 Total (N=403) 3,72 3,82 3,38 4,19 3,52 3,67 Sig. 0,441 0,205 0,832 0,463 0,860 0,503 Level A (N=304) 3,61 3,78 3,33 4,16 3,40 3,59 B (N=46) 3,77 3,86 3,41 4,30 3,80 3,75 C (N=53) 4,33 4,05 3,67 4,25 3,95 4,02 Total (N=403) 3,72 3,82 3,38 4,19 3,52 3,67 Sig. 0,000 0,069 0,036 0,413 0,000 0,000 İpek & Mutlu 702 It is seen that (table 6) the average scores of the females (N=259) were higher than the average scores of males (N=144). Since the values were greater than 0.05, it was seen that this difference was not statistically significant. It was revealed that similar results were seen in the type of enrollment and section. However, all of the sig. values obtained as a result of the ANOVA (F test), which was conducted to examine whether the average scores of the participants from DESES sub- dimensions and the general average differ according to the GPA of the students, were found to be less than 0.05. Therefore, it was seen that the average scores obtained from the individual sub-dimensions of DESES showed a statistically significant difference compared to the GPA. A post-hoc multiple comparison test was conducted to determine which GPAs caused this difference and this can be seen in table 7 below. Table 7. Multiple Comparisons (Tukey HSD) Dependent Variable (I) GPA (J) GPA Mean Difference (I-J) Sig. LS 40-59 60-79 -,28825 ,033 80-100 40-59 ,76179 ,000 60-79 ,47354 ,000 C 80-100 40-59 ,38577 ,007 CE 80-100 40-59 ,51242 ,001 60-79 ,33295 ,003 I 80-100 40-59 ,42813 ,000 60-79 ,24962 ,006 A 0-39 40-59 ,82386 ,026 80-100 40-59 ,42341 ,007 60-79 ,28830 ,011 General 80-100 40-59 ,51779 ,000 60-79 ,31513 ,000 As a result of the multiple comparison test; for the Language Skill (LS) variable, the average scores of students with a GPA between 80-100 were higher than the scores of students with a GPA between 40-59 and 60-79. Likewise, it was seen that LS scores of students with a GPA of 60-79 were higher than those of students with a GPA of 40-59. For the Communication (C) variable, the average scores of students with a GPA between 80-100 were higher than the scores of students with a GPA of 40-59. For the Content Evaluation (CE) variable, the average scores of students with a GPA of 80-100 were higher than those of students with a GPA of 40-59 and 60-79. For the Instructors (I) variable, the average scores of students with a GPA of 80-100 were higher than the scores of students with a GPA of 40-59 and 60-79. For the Assessment (A) variable, the average scores of the students with a GPA between 80-100 were higher than the scores of students with a GPA between 40-59 and 60-79. Likewise, it was seen that the scores of the students with a GPA of 0-39 were higher than those of students with a GPA of 40-59. For DESES in general, the average scores of students with a GPA of 80-100 were higher than those of students with a GPA of 40-59 and 60-79. Finally, the sig. values were less than 0.05 for the LS, CE, A variables and DESES scale in general. Therefore, it was seen that the average scores obtained from the LS, CE and A sub- dimensions of DESES and all of them showed a statistically significant difference according to the level groups. A post-hoc multiple comparison test (table 8) was performed in order to determine which level groups this difference was originated from. International Online Journal of Education and Teaching (IOJET) 2022, 9(2), 690-713. 703 Table 8. Multiple Comparisons (Tukey HSD) Dependent Variable (I) Level (J) Level Mean Difference (I-J) Sig. LS C A ,71430 ,000 B ,55285 ,000 CE C A ,34015 ,028 A A B -,39960 ,008 C -,55866 ,000 General A C -,42153 ,000 As a result of the multiple comparison test, for the Language Skill (LS) variable, the average scores of students in C level were higher than the scores of students in A and B levels. For the Content Evaluation (CE) variable, the average scores of students in C level were higher than those of students in A level. For the Assessment (A) variable, the average scores of students in B and C groups were higher than the scores of students in A level. For DESES in general, the average scores of students in C level were higher than those of students at level A. In addition, a cross-tabulation (chi-square analysis) was conducted (table 9) to determine whether the GPAs of the students were related to demographic characteristics. Table 9. Relation of participants' demographics to their grade point averages (cross table) GPA Total Sig. (p-value) 0-39 40-59 60-79 80-100 Gender Female Count 8 28 101 122 259 0,032 % within gender 3,1% 10,8% 39,0% 47,1% 100,0% Male Count 2 29 59 54 144 % within gender 1,4% 20,1% 41,0% 37,5% 100,0% Total Count 10 57 160 176 403 % within gender 2,5% 14,1% 39,7% 43,7% 100,0% Type of enrolment Optional Count 4 19 59 48 130 0,276 % within Type of enrolment 3,1% 14,6% 45,4% 36,9% 100,0% Compulsory Count 6 38 101 128 273 % within Type of enrolment 2,2% 13,9% 37,0% 46,9% 100,0% Total Count 10 57 160 176 403 % within Type of enrolment 2,5% 14,1% 39,7% 43,7% 100,0% Section Day group Count 9 44 145 166 364 0,002 % within Section 2,5% 12,1% 39,8% 45,6% 100,0% Evening group Count 1 13 15 10 39 % within Section 2,6% 33,3% 38,5% 25,6% 100,0% Total Count 10 57 160 176 403 % within Section 2,5% 14,1% 39,7% 43,7% 100,0% Level A Count 10 47 136 111 304 0,000 % within Level 3,3% 15,5% 44,7% 36,5% 100,0% B Count 0 8 15 23 46 % within Level 0,0% 17,4% 32,6% 50,0% 100,0% C Count 0 2 9 42 53 % within Level 0,0% 3,8% 17,0% 79,2% 100,0% Total Count 10 57 160 176 403 % within Level 2,5% 14,1% 39,7% 43,7% 100,0% When the p-values obtained as a result of the cross-table (chi-square analysis) are examined, it can be said that there is a statistically significant relationship in terms of variables less than 0.05. In this context, it can be said that there is a statistically significant relationship between the students' GPA and gender, enrolment type and level. When the results are examined, it is İpek & Mutlu 704 also seen that the GPA of females is higher than males. When the relationship between the GPA of the students and the section is examined, it was revealed that the day students’ GPA is higher than evening students. Finally, when the GPA of the students in different levels are examined, it was seen that students in C group have high GPAs, followed by B group. It was understood that most of the students in group A had a GPA of 60-79. In conclusion, DESES was applied to the students studying in the English preparatory program of a university, which was conducted to measure the effects of the distance education received by the students studying in the English preparatory program. The average scores obtained from the overall scale and its sub-dimensions were analyzed according to demographic variables; no difference was observed according to gender, enrollment type and education type. On the other hand, it was found that there was a significant difference according to the GPA. The average scores of the students whose GPA is between 80-100 were higher in general. In addition, it was observed that the average scores obtained from DESES in general and LS, CE and A sub-dimensions showed a significant difference according to the level group. It was revealed that the average scores of the students studying in the C level group are higher than those of the other level groups. 5.2b. Students’ views on EFL distance education program regarding interviews The themes emerged after analyzing interview results are given under two themes as deficiencies and satisfactions. When we examine figure 3 below, it can be seen that the satisfactions are about the secondary components of the program which are not directly related to the program content. However, the components of deficiencies are directly related to the content and the implementation of the distance education as communication, language skills, assessment, content evaluation and instructors which are also the factors of DESES. Figure 3. Satisfactions and Deficiencies For the satisfactions, students are happy to have more free time as they stated that distance education gave them opportunity working at a job, and they reported that they would not have found time and opportunity in face-to-face education. In addition, they pointed out that distance education is very advantageous for costs. They do not have expenses for transportation, accommodation, or food. Moreover, students stated that they did not get physically tired during opportunity to work expenses technical infrastructure being physically fit accomodation communication with instructors stress-free e-workbook socialization technical problems teacher's mood pacing schedule registration period lack of communcation attendance feedback assessment language skills International Online Journal of Education and Teaching (IOJET) 2022, 9(2), 690-713. 705 distance education and that was an advantage for them not to spend time traveling to school. For instructor’s category, students stated that they benefited more from the announcements made by the academic staff rather than the announcements made by the department administration or the university. They added that their communication with the teaching staff was very good. Finally, the students said that there is a stress-free environment in distance education courses, and this has a positive effect on their performance. On the other hand, students stated that they could not socialize during their distance education, which negatively affected the language learning process. They stated that they would learn more efficiently by socializing in real classroom environment. In another issue, they stated that the emotions of the teacher were hidden in online education, and they could not understand the emotional states such as anger, anger, disappointment, happiness, joy, or appreciation. When the interview results were examined, they explained that the students were very happy with the books, but they had a problem with the intermediate book which they had in the program in the second semester but were not able to study properly due to time constraints. As for the communication category, it is understood that the students have a lot of problems in this regard. Interview exerpts of the problems experienced by different students in this category are given in the table 10 below. Table 10. Excerpts from student participants Participant Excerpt SP-1 “During registration period, they asked me for a wet signed petition. it was supposed to be made easier during the pandemic period anyway.” “Also, the orientation meeting at the beginning of the semester was not very productive either. could have been more detailed.” SP-2 “My record did not appear for one month at the university. Also, I had to go to the university personally because I had a problem with the registration system.” SP-3 “When I called the school, no one answered the phones, I could not reach anyone, so, I called all the people one by one, starting with the letter A in the phone book of the university until I reached the person concerned.” SP-4 “The website was also inefficient. It could be used more effectively. Another issue is announcements. Sometimes I did not receive the notification emails.” SP-5 “Samples of the exams should have been posted on the website. We did not know what kind of exam we were going to take.” Another issue that the students perceived as a deficiency was about attendance. The students stated that they were not told anything openly about the attendance requirement. Another issue was related to the lack of feedback. The students stated that since the exams were held online, they could not see their exams for feedback. In the language skills category, the students emphasized that several technical problems occured during listening. Moreover, their speaking skills did not develop because they could not do pair and group work. For assessment, students pointed out that there were some students who cheated on exams, and that reduced their motivation towards assessment. 5.3. EFL instructors’ views on EFL distance education program Six categories emerged after analyzing the interview results of the instructor participants (Figure 4). These categories are listed as pedagogy, communication, attendance and motivation, language skills, pacing schedule, and assessment. İpek & Mutlu 706 Figure 4. Categories of instructor participants’ interview results For the pedagogy, two instructors stated that they were upset when they heard about distance education, because they had not had any knowledge or experience. They stated that they were not given any formal education related to distance education pedagogy, but rather they carried out the distance education processes with their own efforts. One of the excerpts can be seen below: “I preferred to adapt to distance education personally, thinking about how the system can be run better, I took actions accordingly. For example, I experienced the features of the online platform and tried to learn it by myself. (IP-4)” On the other hand, one of the participants (IP-5) stated that he was very happy that they started distance education and that distance education would add a lot to them. However, he stated that these thoughts that he had felt at the beginning turned into disappointment after some time. Moreover, it was pointed out that they started to feel fear and anxiety since distance education has a unique pedagogy and that they could not achieve this as a department. One of the participants stated this as follows: “We did the traditional face-to-face training on the computer. Distance education has a different methodology, we had the book in the lesson, but in distance education we showed it on the computer. While there were many different tools for distance education, we did not use them. If we had used them, maybe the level of students would have increased, and it would have been more effective. However, our inexperience in distance education did not make this possible. (IP-5)” In terms of communication, the participants stated that the communication was more one- way. They established a WhatsApp group where only the department administration could write a message. They could not write anything there, so the interaction was missing. Also, at the end of the term, an online form was sent to them instead of the face-to-face or online meeting. They stated that the opportunities for meeting and discussion for feedback disappeared. pedagogy communication attendance and motivation language skills pacing schedule assessment International Online Journal of Education and Teaching (IOJET) 2022, 9(2), 690-713. 707 The participants also touched upon the issue of the low motivation of the students. They stated that the students were highly motivated at the beginning of the academic year, but lost their motivation over time, so attendance rates decreased considerably. Factors such as lack of attendance requirement and watching the recorded courses later were stated as the reason for this deficiency. However, it was pointed out that e-workbooks used in the program, which are suitable for distance education pedagogy, had a positive effect on student motivation. It was stated that students who were particularly prone to technology were more willing to do that type of homework rather than pen-paper homework (IP-1). Another positive opinion about the e-workbook is that checking homework did not take too much teacher-time, since it was done automatically by the e-workbook system (IP-2). As for language skills, it was stated that receptive skills were improved, and unfortunately, no progress could be made based on production. One of the participants (IP-3) stated that for example, more appropriate learning outcomes could have been obtained by using the suitable technological applications where speaking environments were formed in small chat-rooms, where visual and auditory materials, more authentic tools and technology were used more effectively. Another participant (IP-4) stated that distance education conducted in this way prevented the development of students' basic four-language skills. Another participant (IP-3) stated that although attendance in classes started to decrease in the middle of the academic year, from the very beginning of the academic year, production skills such as speaking or writing due to the crowded classrooms could not be achieved. Also, the participants stated that the pacing schedule was tight, so the reading, listening and note-taking books given in face-to- face training could not be taught, and they had to remove many units in the intermediate book due to time constraints. IP-4 stated that time and language development did not progress in direct proportion. That is, while the pacing schedule progressed, the students remained in the same place. One of the participants explained this situation as follows: “Our pacing had been prepared for face-to-face education. Subsequent changes were not healthy. We had to remove some of the books we used before. Lesson hours in the program were reduced. However, this was not ensured in pacing. (IP-3)” As the last category, assessment emerged. All the participants stated that the assessment could not be done accurately, reliably, and effectively. IP-5 stated that exams were prepared in accordance with face-to-face education, these exams were only transferred to the computer, thus cheating and helping occured in the exams. It is also asserted that most of the grades in their GPAs were not real scores. It had been concluded that it would be very difficult for students when they passed their majors in the future. 6. Discussion and Conclusion This study was carried out to evaluate a university English preparatory program, where distance education had been introduced for the first time. The system approach was used as the conceptual framework. In order to obtain effective, reliable and valid results, data were collected from both faculty members and students, who were all stakeholders in the EFL distance education system. Data were collected from students via DESES, which consists of 30 five-point Likert type items and semi-structured interview. Also, semi-structured interview with faculty members were conducted. Moreover, document analysis and observation journals were used to make more detailed evaluation and analysis. At the end of this study, the following findings were obtained: 1) It was understood that female students had scored higher on DESES than male students, but the difference was not a statistically significant. 2) According to the GPA in DESES, which is an independent variable, the students in the upper-levels gave higher scores to the language skills, communication, instructors, İpek & Mutlu 708 assessment and content evaluation factor than the students at the lower-level. This revealed that as the level of English increased, students were satisfied with the program. 3) According to student level in DESES, statistically significant difference was found that the scores given by the students to the Language Ability, Content Evaluation and Assessment factors were higher in upper-level students than the students in lower-level groups. 4) According to the DESES results, it was determined that the GPAs of female students were higher than that of males. Moreover, day students’ grades were higher than evening students, and the grades of C group students were higher than the lower-level students. 5) When the factors of DESES were examined, it was determined that Instructors had the highest average, followed by Communication, Language Skills and Assessment, respectively. The factor with the lowest average was Content Evaluation. Interview results of students also supported DESES results. Results showed that the triangulation was quite effective. According to the interview results of the students, the following results were found: 6) According to the interview results of the students, the categories in the outer circle according to the system approach such as opportunity to work, expenses, technical infrastructure, being physically fit, accommodation, communication with instructors, and stress-free environment emerged as satisfied. 7) Socialization, technical problems, teacher’s mood, pacing schedule, registration period, lack of communication, attendance, feedback, assessment, language skills were categorized as deficiency. The opinions of the instructors, who are another stakeholder, were also taken, and it was revealed that they were not satisfied with the program. The views of the teaching staff are presented in the following items: 8) Teaching staff stated that distance education has a different pedagogy. This pedagogy is different from face-to-face education; therefore, it is necessary to master distance education pedagogy in order to be successful. 9) They stated that while the communication was irregular and there should have been frequent negotiations especially during the distance education experienced for the first time, mostly one-way communication channels were open, and this caused a lot of problems in the process. 10) It was concluded that because of the lack of attendance requirement and the fact that students can watch the lessons from the recording later, the students show indifference towards the lesson. 11) It was revealed that the e-workbook is an effective element for both students and teachers. 12) It was emphasized that productive skills such as speaking, and writing could not be taught well due to the crowded classrooms in the distance education system at the beginning of the academic year and the infrastructure used. 13) The fact that the pacing schedule was prepared in accordance with face-to-face education, it caused many topics to be taken out, especially in the second semester. Moreovoer, lack of reading, listening and note-taking books caused deficiency in teaching language skills. 14) Due to problems in the pacing schedule, it was stated that although the program continued, the student level did not progress at the same rate and pace. International Online Journal of Education and Teaching (IOJET) 2022, 9(2), 690-713. 709 15) It was pointed out that assessment methods were designed according to face-to-face education; therefore, cheating or copying assignments were faced in the exams, which adversely affected the exam results. As revealed in this study, Kara (2020) also pointed out that female students are more satisfied with distance education. As for the other demographic information, Aljumah (2020) asserted that there is a significant difference between the level group and satisfaction with the online learning as it is found in this study. About the technology use in distance learning, in a study conducted by Bozkurt et al (2020), it was stated that if distance education is applied correctly, it will have many benefits, but if it is applied incorrectly, it can face many problems. Another conclusion that this study came is time and expense. According to Karataş and Tuncer (2020), it was revealed that students have a comfortable atmosphere in distance education, have more free time than face-to-face education, and save time and expense. This result is in parallel with the findings of the current study. Moreover, Amiryousefi and Geld (2021) concluded that teachers' feedback is very effective on teacher motivation during distance education. This result supports the conclusion that the one-way communication mentioned in this study has a negative impact on teaching staff. In a study conducted by Ramirez (2020), it was stated that students are left alone too much in distance education, the classroom environment is not fully formed, and these negatively affect students. This situation parallels the issue of socialization that emerged in this study. As a result of this study, according to both student and the instructor interview results, it was revealed that speaking ability of the students did not improve as planned. A similar result was revealed in the study of Derakhsan (2021) who conveyed research on students in Iran, and it was found that speaking was the language skill that students had the most trouble with. Moreover, according to Gaquit (2020), it was revealed that students' individual speaking has improved through distance education. However, in our study, it was concluded that speaking skills did not develop since there were no individual studies. In the same study, social interaction and assessment issues emerged as problematic elements in distance education which goes parallel with the findings of the current study. For the language skills, Mahyoob (2021) also found similar results with our study. It is pointed out that students are not satisfied with the content of the program. In the study, it was understood that technical problems disrupt the course flow, and cause difficulties in impoving language skills. A similar result emerged in Sevik and Yucedag’s (2021) study. It was concluded that technical problems such as lack of internet and lack of technical knowledge affect language teaching negatively. As mentioned in the results section, technical problems emerged as the biggest source of disruption in distance education. Similarly, the study by Nguyen and Duong (2021) revealed that the deficiencies in the technological infrastructure used have a significant impact on student behavior. For pedagigy, in Sevik and Yucedag’s study (2021), it was emphasized that it is important to organize educational institutions for online education. This result is directly related to the lack of distance education pedagogy of the instructors in the current study. Likewise, Eela (2020) concluded that teachers are unprepared for distance education. As for assessment, in this study, it was concluded that different methods should be used in distance education, otherwise copying would cause many problems. According to Saleh and Meccawy (2021), it is stated that some students copy each other during the exam, while some students copy-paste other sources they find on the internet to their screens during the exam. It can be concluded from the results of this research that we need to understand that distance education and face-to-face education have very different pedagogies. While the distance education curriculum is being prepared, exams, books, communication skills, language skills training, in-service training for teachers, admission and registration procedures, pass-fail states, technological infrastructure should be arranged according to the distance education pedagogy. İpek & Mutlu 710 References Aljumah, F. H. (2020). The Impact of the EFL Undergraduate Students' Integration in Distance Learning with its Various Platforms. The Asian ESP Journal, 16(5.1), 298-317. Amiryousefi, M., & Geld, R. (2021). The role of redressing teachers’ instructional feedback interventions in EFL learners’ motivation and achievement in distance education. Innovation in Language Learning and Teaching, 15(1), 13-25. Amiri, M., & Maftoon, P. (2010). A critical look at distance education in Elt. in Edulearn 10 Proceedings (pp. 6792-6799). IATED. Aparicio, M., Bacao, F., & Oliveira, T. (2016). Cultural impacts on e-learning systems’ success. Internet and Higher Education, 31, 58-70. Boehler, T. (1999). A design plan for online distance learning program delivery. Doctoral Dissertation. Pepperdine University. Bozkurt, A.; Sharma, R.C. Emergency remote teaching in a time of global crisis due to Corona Virus pandemic. Asian Journak of Distance Education. 2020, 15, i–vi. Available online: https://www.asianjde.org/ojs/index.php/AsianJDE/article/view/447 (accessed on 15 June 2020). Burr, V. (2003). Social constructionism (2nd ed.). London: Routledge. Chan, V. (2001). Determining students’ language needs in a tertiary setting. English Teaching Forum July, 39(3), 16-27. Coskun, A., & Daloglu, A. (2010). Evaluating an English language teacher education program through Peacock's model. Australian Journal of Teacher Education (Online), 35(6), 24- 42. Creswell, J. W., Plano Clark, V. L., Gutmann, M. L., & Hanson, W. E. (2003). Advanced mixed methods research designs. Handbook of mixed methods in social and behavioral research, 209(240), 209-240. Creswell, J. W. (2009). Research design: Qualitative, quantitative, and mixed methods approaches. SAGE Publications, Incorporated. Creswell, J. W. (2013). Research design: Qualitative, quantitative, and mixed methods approaches. Sage publications. Debes, G. (2021). Distance Learning in Higher Education during the COVID-19 Pandemic: Advantages and Disadvantages. International Journal of Curriculum and Instruction, 13(2), 1109-1118. Derakhshan, A. (2021). Emergence Distance Education (EDE) role in the Learning of English Language Skills during COVID-19 Pandemic. Journal of Teaching Language Skills, 40(3), 41-82. Denzin, N. K., & Lincoln, Y. S. (2000). Handbook of qualitative research. London: Sage Publications. Do Le, T., & Tran, T. N. T. (2021). English Language Training Curriculum: An Evaluation From Learner’s Perceptions. Open Journal of Social Sciences, 9(3), 40-57. Nguyen, H. U. N., & Duong, L. N. T. (2021). The Challenges of E-learning Through Microsoft Teams for EFL Students at Van Lang University in COVID-19. AsiaCALL Online Journal, 12(4), 18-29. International Online Journal of Education and Teaching (IOJET) 2022, 9(2), 690-713. 711 Eela, L. (2020). Distance foreign language education in Finland: A comparison of conceptions and attitudes toward the use of distance education tools among foreign language teacher students and distance EFL teachers. Unpublished masters thesis, University of Jyväskylä, Finland. Efeoglu, G., Ilerten, F., & Basal, A. (2018). A Utilization Focused Evaluation of the Preparatory School of an ELT Program. International Online Journal of Educational Sciences, 10(4), 149-163 Ferretti, F., Santi, G. R. P., Del Zozzo, A., Garzetti, M., & Bolondi, G. (2021). Assessment Practices and Beliefs: Teachers’ Perspectives on Assessment during Long Distance Learning. Education Sciences, 11(6), 264-281. Frye, A. W., & Hemmer, P. A. (2012). Program evaluation models and related theories: AMEE guide no. 67. Medical teacher, 34(5), 288-299. Hall, D. R., & Knox, J. S. (2009). Language teacher education by distance. Cambridge guide to second language teacher education, 218-229. Ilerten, F., & Efeoglu, G. (2021). Utilization focused evaluation of an ELT prep program: A longitudinal approach. Pegem Journal of Education and Instruction, 11(2), 27-31. İpek, Ö. F., & Mutlu, H. T. Distance education system evaluation of English language teaching: A scale development study. RumeliDE Dil ve Edebiyat Araştırmaları Dergisi, (26), 978-994. Irawan, N., Retnaningdyah, P., & Mustofa, A. (2021, March). Distance learning of ELT for Chinese students in ICUTK: digital literacy practices. In Journal of Physics: Conference Series (Vol. 1842, No. 1, p. 012017). IOP Publishing. Gaquit, A. (2020). Teaching by distance: A qualitative investigation into upper secondary EFL teachers' perspectives on distance education in the Swedish context. Unpublished Master’s Thesis. Mälardalen University, Sweden Gur, B. S., & Van Schaack, A. J. (2004). Approaches to assessment of online learning: conceptual challenges. Loughborough University Johnson, E.M., Bishop, A., Holt, A., Stirling, J.A., & Zane, J. (2001). Reflections in cyberspace: Web conferencing for language teacher education. Australian Journal of Educational Technology, 17(2), 169–186. Johnson, R. B., Onwuegbuzie, A. J., & Turner, L. A. (2007). Toward a definition of mixed methods research. Journal of mixed methods research, 1(2), 112-133. Kara, M. (2021). Transactional distance and learner outcomes in an online EFL context. Open Learning: The Journal of Open, Distance and e-Learning, 36(1), 45-60. Karabıyık, C., Mirici, İ.H. (2018). Development and Validation of the Foreign Language Learning Effort Scale for Turkish Tertiary-Level Students. Educational Sciences: Theory & Practice, 18(2), 373–395. DOI: 10.12738/estp.2018.2.0010 Karataş, T. Ö., & Tuncer, H. (2020). Sustaining language skills development of pre-service EFL teachers despite the COVID-19 interruption: A case of emergency distance education. Sustainability, 12(19), 8188. Keegan, Desmond. Foundations of Distance Education. Routledge, 1996. Khoirunnisa, Suprano, & Supriyadi, S. (2018). ESP teacher’s and student’s perceptions on teaching speaking for a tourism program. Dinamika Ilmu, 18(1), 37-48. İpek & Mutlu 712 Kirkpatrick, D. (1996). Revisiting Kirkpatrick's four-level model. Training & Development, 50(1), 54-57. Littleton, K., & Whitelock, D. (2005). The negotiation and co‐construction of meaning and understanding within a postgraduate online learning community. Learning, Media and Technology, 30(2), 147-164. Mahyoob, M. (2020). Challenges of e-Learning during the COVID-19 Pandemic Experienced by EFL Learners. Arab World English Journal (AWEJ), 11(4), 351-362. Mennin, S. (2010). Self‐organisation, integration and curriculum in the complex world of medical education. Medical education, 44(1), 20-30. Patton, M. Q. (1986). Utilization focused evaluation (2nd ed.). Beverly Hills, CA: Sage. Pastor, C. K. L. (2020). Sentiment Analysis on Synchronous Online Delivery of Instruction due to Extreme Community Quarantine in the Philippines caused by COVID-19 Pandemic. Asian Journal of Multidisciplinary Studies, 3(1), 1-6. Peacock, M. (2009). The evaluation of foreign-language-teacher education programmes. Language Teaching Research, 13(3), 259-78. Pina, A. A. (2008). How institutionalized is distance learning? A study of institutional role, locale and academic level. Online Journal of Distance Learning Administration, 11(1). Suárez Ramírez, I. R. (2020). Exploring students’ beliefs about their digital literacy experiences in an efl distance education course. Unpublished MAsters Thesis, Unıversıdad pedagógıca nacıonal. Bologna, Colombia. Saba, F. (2003). Distance Education Theory, Methodology, and Epistemology: Pragmatic Paradigm. Handbook of distance education. Saleh, A. M., & Meccawy, Z. (2021). EFL Female Students' Perceptions towards Cheating in Distance Learning Programmes. English Language Teaching, 14(1), 29-36. Salihoglu, U. M. (2012). Pre-service teachers’ and their instructors’ beliefs on the effectiveness of an English language teacher education program. Procedia-Social and Behavioral Sciences, 46, 3440-3444. Şener, B., Ertem, İ. S., & Ahmet, M. E. Ç. (2020). Online teaching experiences of ELT instructors. Journal of Educational Technology and Online Learning, 3(3), 340-362. Şevik, M., & Yucedag, Z. An evaluation of distance education during the Coronavirus 19 pandemic: the views of Turkish EFL teachers. Mehmet Akif Ersoy Üniversitesi Eğitim Bilimleri Enstitüsü Dergisi, 9(11), 171-190. Sharma, G. (2017). Pros and cons of different sampling techniques. International journal of applied research, 3(7), 749-752. Simons, H. (2009). Listen, look, document: methods in case study research. In Case study research in practice (pp. 43-66). SAGE Publications, Ltd, https://www.doi.org/10.4135/9781446268322 Steckler, A., McLeroy, K. R., Goodman, R. M., Bird, S. T., & McCormick, L. (1992). Toward integrating qualitative and quantitative methods: An introduction. Health Education Quarterly, 19(1), 1–8. Stufflebeam, D. L. (1971). The relevance of the CIPP evaluation model for educational accountability. Journal of Research and Development in Education. 1-30. International Online Journal of Education and Teaching (IOJET) 2022, 9(2), 690-713. 713 Talebi, Z. & Javidi, S. A. (2022). The effect of task rehearsal, unguided strategic planning, and pressured online planning on writing accuracy of Iranian intermediate EFL learners. International Journal of Education, Technology and Science, 2(1), 058–094. Tom-Lawyer, O. (2014). An Evaluation of the English Language Curriculum of the Nigeria Certificate in Education: A Case Study of a College of Education. Open Journal of Social Sciences, 2, 69-79. Tunç, F. (2010). Evaluation of an ELT Program at a Public University Using CIPP Model. Saarbrucken: Lambert Academic Publishing. Tuzun, H., Bilgic, H.B., Kalayci, E., Cinar, M., Akinci, A., Yildirim, D., & Yuksel, Y. (2011). Yerlesik bir dersin web-tabanli uzaktan egitim icin yeniden tasarimi [Redesign of a residential course into web-based distance education form]. In B. B. Demirci, G. T. Yamamoto and U. Demiray (Eds). Turkiye’de e-ogrenme: Gelismeler ve uygulamalar II [E-learning in Turkey: Developments and applications] (pp. 175-199). Eskisehir, Turkiye: Anadolu Universitesi Yayinlari. Vaiz, O., Bahçelerli, N. M., Minalay, H., Türe, A., Ülgener, P., & Yaşar, H. (2021). A review of measurement and assessment in distance education. The Online Journal of New Horizons in Education, 11(2). Yörük, T. (2021). Practitioners' Opinions on Student Assessment Process in Distance Education in the Context of Educational Technologies. Shanlax International Journal of Education, 9(2), 12-17.