Balbay, S. & Kilis, S. (2018). Educational analytics on an opencourseware. International Online Journal of Education and Teaching (IOJET), 5(3). 673-685. http://iojet.org/index.php/IOJET/article/view/456/261 Received: 20.05.2018 Received in revised form: 05.07.2018 Accepted: 09.07.2018 EDUCATIONAL ANALYTICS ON AN OPENCOURSEWARE Seher Balbay Middle East Technical University seherb@metu.edu.tr (correspondence) Selcan Kilis Giresun University k1selcan@gmail.com Seher Balbay has been an instructor at Middle East Technical University since 1997. She received her MA from the same university in 2000. She’s working towards her PhD degree in ELT at Hacettepe University. Selcan Kilis is an assistant professor in the Division of Curriculum and Instruction in Giresun University. She received her PhD degree from the Department of Computer Education and Instructional Technology at METU in 2016. Copyright by Informascope. Material published and so copyrighted may not be published elsewhere without the written permission of IOJET. http://iojet.org/index.php/IOJET/article/view/456/261 https://orcid.org/0000-0001-6276-053X mailto:seherb@metu.edu.tr https://orcid.org/0000-0001-5751-2363 mailto:k1selcan@gmail.com International Online Journal of Education and Teaching (IOJET) 2018, 5(3), 673-685. 673 EDUCATIONAL ANALYTICS ON AN OPENCOURSEWARE Seher Balbay seherb@metu.edu.tr Selcan Kilis k1selcan@gmail.com Abstract Analytics as one of the recent fields in technology-based learning offers many benefits to educators, instructors, and administrators to improve the efficiency and quality of alternative educational materials, and learning experience through tracking and storing students’ log data on web platforms over an extended period of time. This mixed-method study investigates students’ log data retrieved from the opencourseware (OCW) specifically launched for a required academic English speaking skills course offered at Middle East Technical University in Turkey with the aim of enhancing the quality and efficiency of the materials available for the course. By understanding the reasons behind students’ behaviors via the interviews conducted with 50 students on this online courseware, this study also aims to provide useful practical hints to the instructors and guide them to act on future decisions. The analyzed data is based on learner behavior with a specific emphasis on average view duration, likes and dislikes, and comments. This study can serve as a starting point to guide and provide the instructors and administrators about the future of the aforementioned course which is also offered in a rotational hybrid learning format where the effectiveness of online materials gain even more importance. Keywords: learning analytics, online video-based learning, log data, academic speaking skills, opencourseware 1. Introduction 1.1. Why Track Learning Analytics? One of the major developments in data collection on educational statistics is the availability of ‘big data’ using the data visualization techniques online course platforms lets educators access to. The interest in this robust data draws attention to tracking the activities of students providing teachers, especially those in decision-making administrative positions with ‘navigation or behavior-focused statistics’ (Bull & Kay, 2016, p. 311). Although, publications on the interpretation of such data is unprecedented in English language teaching programs, it is directive value is undeniable. That is the learner analytics of log data can be guiding for future online tools and materials in similar teaching settings. A related field is educational data mining. EDM is a more generic term which encompasses both LA and Academic analytics. Although the investigators, methods and findings of Learning Analytics (LA) and Educational Data Mining (EDM) overlap to a great extent some researchers differentiate between the two by claiming that data mining encompasses both learning analytics and academic analytics. Academic analytics is basically defined as ‘the process of evaluating and analyzing organizational data received from university systems for reporting and decision making reasons’ (Campbell & Oblinger, 2007). mailto:seherb@metu.edu.tr mailto:k1selcan@gmail.com Balbay & Kilis 674 ‘Learning Analytics’ (LA) is the ‘measurement, collection, analysis and reporting of such big data about learners’ behaviors with the intention of understanding and optimizing learning and the environments in which it occurs’ (Sclatter, Peasegood & Mullan, 2018). LA is a new area of interest that bears a crucial importance in the era of technology in education. Simply put, learning analytics traces the learning process-related online data and reveals systematic measurement of the frequencies the online educational tools are used. The learner analytics field is data-driven and atheoretical. Online learning management systems today are becoming common in all levels and fields of education today, and the data retrieved from their use can be enlightening about which sort of materials are on demand by learners. This quantitative data available can shape the future of blended learning platforms, as it can be a predictor for detecting student preferences of course-related resources. It must be made clear that this statistical data does not predict the achievement of learning outcomes, but has the potential to impact the success (Arnold & Pistilli, 2012; Barber & Sharkey 2012; Gibson & de Freitas 2015; Mah, 2016). Yet, it is a valuable source to receive feedback on the learning process that takes place on the internet. Bienkowski, Feng and Means (2012) state that learning analytics provides institutions with ample opportunities to support the student learning process, and to enable personalized learning. That is; with the spread of online learning tools, personalized learning options will be developed based on the reports of learner preferences of online learning tools and materials. This easy to retrieve data is also easy to interpret thanks to the skillfully designed visualizations of analytics engines. Hence, it makes it possible for educational institutions to use the experience of the past and plan the future investments on education. Although many educational institutions are not ready to exploit learner analytics today, it is an undeniable fact that the future of online learning tools depend on it. Learner analytics use data generated from learner activities which can sometimes be not only watch time, but simply clicks, participation in online forums, or computer assisted testing (Tempelaar, Rienties, Giesbers, 2014). Shum and Crick (2012, p.3) state that learning activity generated data is also proof of an overview of ‘students’ values and attitudes which are fed back to students and teachers through visual analytics’. Data extracted from online institutional learning management systems by measuring learner preferences can also be used for motivation and engagement-related research in an era of intense online learners when making plans concerning longitudinal learning infrastructure. By the same token, Crede and Niehorster (2012) suggest that learner analytics data can be studied as an indicator of academic performance emotional and social factors. It is of significance at this point to iterate that learner analytics data alone would not be a direct and explicit proof of any correlations without a demographic overview of the participants in a study. At this point, the background information on the context of studies based on learner analytics bears undeniable importance. Knowledge of course design, instructor intentions and student, institution and the design of the Moodle, Web 2.0 tool or the open courseware that the data is extracted from is of utmost importance when analyzing learner analytics data since these are the major factors that determine ‘which variables can meaningfully represent student endeavor and engagement’ in activities provided online (Macfadyen & Dawson, 2010, p. 597). The goals of learner analytics are ‘predicting learner performance, suggesting new learner resources, increasing reflection and awareness, enhancing social learning environments and detecting undesirable learner behaviors’ as listed by Verbert, Manauselis, Drachsler and Duval (2012, p.138). Learner analytics provides self-evident data for online educational material which is less preferred by students, which can enlighten material designers as to the International Online Journal of Education and Teaching (IOJET) 2018, 5(3), 673-685. 675 type of content and means to opt for knowing which components of sources best serve the targeted population and objectives. Hence, bearing in mind the directive feature of the objective data learner analytics provides, it is apparent that this new field deserves further and longitudinal research. 1.2. Learning Analytics, the Research Field of 21 st Century Classrooms Learning analytics literature has recently been published citing the many benefits of both making course sources and interaction online through a variety of platforms and also by referring to making the input material for courses accessible online. It has been observed that students use social networks extensively as a learning tool (Agudo-Peregrina, Iglesias-Pradas, Conde-Gonzales, Hernandez-Garcia, 2014). One study observing the effects of social networks on performance was conducted on 300 medicine faculty students. The study concluded that the use of social networks has a predictive influence on the academic performance of the students. Another study conducted by Rienties, Hernandez Nanclares, Hommes and Veermans (2014) found out that 30-80% of learning occurred outside formal settings of education. Personal preferences of using online course material bears importance. Researchers have been vocal in expressing how individual learning styles are considered when preparing sources that students can access in online course material sharing platforms studying the handouts, videos or slides at their own pace. Personalization of teaching is one of the most significant benefits of interfaces that present course material (Bull & Kay, 2016). Online learning management systems prompt self-regulated learning and metacognitive skills too since such tools can on particular occasions play the role of the teacher. School leaders and policy makers are also stakeholders of learner analytics, which is emphasized by Long and Siemens (2011). Actually, the policy makers who allocate the personnel and funding of educational institutions need to collect and understand the learner analytics data to be able to make sound decisions about the efficiency of their future plans and investments. Institutional use of such ‘big data’ is the starting point of learner analytics. The increasing interest in collecting and interpreting this big data provided by learner analytics will keep the educational and governmental institutions, such as the university decision making boards, administrators, the higher education councils, and the Ministry of National Education as its beneficiaries. Apparently, ‘Educational Data Mining’ is expected to lend itself to be used for policy making at all levels. Another benefit of online course input is the undeniable fact that it is cost-effective (Bull & Kay, 2016). Cloud systems are widely popular today, yet opencourseware platforms of educational institutions present an alternative to the limited or costly expanses of cloud systems. Long term store is also possible thanks to these opencourseware interfaces. Bull and Kay (2016) emphasize the fact that learning technology has been exponentially pervasive in the past few years and hence shifted the perception of the means of learning in general making use of online resources outside the time and location constraints of the traditional classrooms in their research on SMILI☺ (Student Models that Invite the Learner In), an open learner framework created to provide a framework of other open learner models. Although this research paper focuses on the frequencies in the use of the Middle East Technical University open courseware for the Academic Oral Presentation Skills course, which is an evident sign of the change in the educational tools that we use, the current study still refers to the nature of learning tools today directly, since the aim of this study is to analyze the frequency of the use of online course materials and interpret this frequency Balbay & Kilis 676 analysis via the interviews conducted with students. Therefore, being one of the pioneers of such learning analytics studies in its field, it is going to be a forerunner of the use of effective computer-based or computer-dependent teaching systems in the English Language Teaching field. The analysis of the popularity of the sources and investigating the reasons behind this popularity will also be revealing in determining the student choices, hence will shed light on the type of future materials. The study intends to make sense of student preferences. Drawing on the statistical analysis of student behaviors, the broad knowledge LA provides on the use of different course materials will be data-driven. Bearing the above benefits of LA in mind, this research study explores the answers to the following research questions (RQ): RQ1: Which type of supplementary online course materials were used most by the students taking the Oral Presentation Skills Course? RQ2: What are students’ reasons for preferring certain types of materials over others? 2. Method This mixed-method study aims to investigate which type of supplementary online course materials were used most by the students on the OCW platform for the Oral Presentation Skills Course in addition to determine students’ reasons for preferring certain types of materials over others. With this aim, the current research analyzed the frequency of clicks on the materials made available to students on the open courseware of the required Academic Oral Presentation Skills course offered to an average of 1000 students each semester. 2.1. Research Context The research context of the study is the Middle East Technical University, one of the most prominent English-medium instruction universities of Turkey where the Academic Oral Presentation Skills course is a required freshman English course to students from all departments. The course is offered every semester including the summer school. During the fall and spring semesters, about 1000 students are offered the course. The number of students that can take the course during the summer school is about half that number, i.e. 500 students on average. In the course of the time that the open courseware was operational, an estimate of 5000 students were offered the course. While some instructors teaching the course referred their students to the open courseware, some instructors preferred to refer their students to their own supplementary materials. Still, as the name suggests, the ‘open’ courseware was available to all students. 2.2. Participant (Subject) Characteristics The blueprints of an average of 1000 students during five semesters (a total of 5000) students’ on the OCW platform were collected. The students taking the Academic Oral Presentation Skills Course who made use of the materials on the opencourseware were from different faculties (Faculty of Architecture, Arts and Sciences, Economic and Administrative Sciences, Education, and Engineering). The course is offered to all students who have passed the two prerequisite Academic English courses offered in their first year at their departments. So students were from 2nd to 4th year students, and their ages therefore ranged from 21 to 25 mostly. With regard to the qualitative part, the most clicked sources were identified and 50 students were chosen according to convenience sampling method to conduct interviews with, with the aim of investigating the reasons behind the popularity of the particular sources that stand out in the numerical learning analysis. International Online Journal of Education and Teaching (IOJET) 2018, 5(3), 673-685. 677 The students studying at the Middle East Technical university are digital technology natives, very efficiently making the best use of the fastest internet in Turkey (10 Gigabit Ethernet technology), available at all locations on the university campus ("Campus Backbone Network METU-NET | Computer Center", n.d.). The campus is well-equipped with computer laboratories at both department buildings and dormitories. Smartphones are an indispensable part of these students’ lives. A former unpublished research, Blended Learning in a Speaking Skills Course reports that 99% of students have smartphones and are very used to using them for both educational and entertainment purposes in addition to communication (Balbay & Kilis, 2017). 2.3. Research Design This mixed-method study applied explanatory sequential design which is also called two- phase model (Cresswell & Plano Clark, 2011). Its rationale is that findings retrieved from the quantitative data provide a general picture of the research problem, and then the qualitative data is analyzed to refine, extend, or explain the general picture. Therefore, this research design type basically consists of first collecting quantitative data to get a general picture of the problem investigated, and then collecting qualitative data to refine the results from the quantitative data (Cresswell, 2012). The process of explanatory research design is presented in Figure 1. Figure 1. Explanatory sequential design (adapted from Cresswell, 2012) Quantitative data is used for the first research question whereas qualitative data is for the second research question. 2.4. Data Collection and Instruments Quantitative data is composed of students’ log data on the open courseware platform used in the Academic Oral Presentation Skills course. The open courseware was launched in the fall semester of 2016 and the period that the analysis covers is the fall, spring and summer semesters of the academic year 2016-2017, and the fall and spring semesters of 2017-2018 academic year. Students’ log data was used to determine which type of supplementary online course materials were used most by the students taking the Oral Presentation Skills Course. Moreover, qualitative data was collected through a semi-structured interview protocol to learn about students’ reasons for preferring certain types of materials over others. The semi- structured interview questions were simply 'Which online sources did you benefit most from and why?'. 2.5. Data Analysis Quantitative data was analyzed using descriptive statistics. Qualitative data was analyzed with deductive content analysis. Interview data was categorized under the most common reasons and coded by the two researchers of this study for inter-rater reliability. The coding agreement by two raters was found to be at about 90% percentage. 3. Results Balbay & Kilis 678 The first research question ‘Which type of supplementary online course materials were used most by the students taking the Oral Presentation Skills Course?’ was analyzed with students’ log data on the OCW platform. First of all, course material in each unit on the OCW platform were analyzed by descriptive statistics. Table 1 and figure 2, below display the distribution of the use of supplementary course material according to the units of the course book. Table 1. Descriptive statistics about course units on the OCW Units Frequency Percentage Welcome 4083 7.98 Unit 1 8834 17.28 Unit 2 11417 22.33 Unit 3 13127 25.67 Unit 4 11735 22.95 Other 1939 3.79 Total logs = 51135 According to Table 1, students clicked mostly the materials on Unit 3 (n=13127, ~26%), followed by Unit 4 (n=11735, ~23%) and Unit 2 (n=11417, ~22%), and then Unit 1 (n=8834, ~17%). They also visited the 'Welcome' section on the OCW at about 8% (n=4083). Lastly, other logs of a total of 51135 logs were at about 4% percent (n=1939) as seen in Figure 1. Overall, the findings indicated that students mostly visited each unit almost about the same frequency. Figure 2. Distribution of students’ blueprints based on course units Findings about document types that students clicked on the OCW platform are presented in Table 2. Table 2. Descriptive statistics about the types of supplementary materials on the OCW Type of Document Frequency Percentage File 36139 70.67 Url 14802 28.95 Forum 194 .38 Total logs = 51135 International Online Journal of Education and Teaching (IOJET) 2018, 5(3), 673-685. 679 Students mostly used file type document (n=36139, ~71%) on the OCW platform. they clicked and benefitted from URL type document at the ratio of one third of file type document. They barely visited forum on the OCW platform. Findings about types of course content students clicked on the OCW platform with descriptive are presented in Table 3. Table 3. Descriptive statistics about type of course content on the OCW Content Frequency Percentage Custom-made Input Videos for 211 3943 7.71 Course-related Word Files (Guidelines & Rubrics) 15755 30.81 Sample Presentations 2461 4.81 Practice Materials 4960 9.70 Listening Practice Materials 5461 10.68 Relevant YouTube Videos 1623 3.17 Input Slides 10784 21.09 Games 1562 3.05 Other Logs 4586 8.97 Total Logs 51135 100.00 As can be seen in Table 3, students used mostly course-related word files such as guidelines and rubrics (n=15755, ~31%), followed by input slides (n=10784, ~21%). They benefited from practice materials (n=4960, ~10%) at a ratio of one-tenth of a total log of 51135 logs. After practice materials, the highest percentage belongs to the other logs at a ratio of about nine percent of total logs. Students benefited from custom-made input videos for the course that the current study was conducted at a ratio of about eight percent of total logs. Their logs ratio for sample presentations was about five percent (n=2461, ~5%), relevant YouTube videos was (n=1623, 3%), and lastly games was (n=1562, 3%). Finally, findings about the most and the least clicked documents on the OCW platform with descriptive are presented in Table 4. Table 4. The most and the least clicked documents or URL The Most and The Least Clicked Documents or URL Frequency of Clicks The Five Highest Clicked Documents or URL Marketing Presentation File 1555 Science And Technology Presentation File 1463 Course Outline File 1278 Weekly Schedule File 1212 Mini Presentation Rubric File 1003 The Five Least Clicked Documents or URL Final Presentations Playlist URL 62 Listening Material - Eternal Sunshine File 88 Video Sources - Your Food Is Shrinking URL 121 Balbay & Kilis 680 Video Sources - Life After Death By PowerPoint URL 133 Rubric - Debate Jury-sheet File 150 As can be seen in Table 4, students mostly clicked and used the file Marketing Presentation file n=1555), followed by Science and Technology Presentation file (n=1463), course outline file (n=1278), weekly schedule file (n=1212), and then Mini Presentation Rubric file (n=1003). On the other hand, the five least clicked documents or URL that students used or clicked were Rubric - Debate Jury-sheet file (n=150), Video Sources - Life After Death By PowerPoint URL (n=133), Video Sources - Your Food Is Shrinking URL (n=121), Listening Material - Eternal Sunshine File (n=88), and finally Final Presentations Playlist URL (n=62). The second research question ‘What are students’ reasons for preferring certain types of materials over the others?’ was analyzed with qualitative data obtained through semi- structured interviews with 50 students. The two researchers coded the recurring themes for students’ reasons for preferring certain type of materials. The coding agreement by two raters was found to be at about 90% percentage. To start with, when students were asked why they referred to the OCW most for Unit 3 materials, they repeatedly stated that exercises on visual representation of numerical data were to be asked in the exam and they referred to those materials for exam preparation practice. Secondly, most students said that the first major presentation they gave was in unit 3, so they needed the guidelines, slides and examples on the OCW for the Marketing Presentation. Apparently, in the two most recurring answers collected during the interviews assessment plays a major role. When asked why they though the first section on the OCW was not clicked frequently, students revealed that the outline and the weekly schedule posted on the Welcome session were two main handouts that they never needed to refer to throughout the semester. The almost equal distribution of the clicks among the units were supported with an answer from the students that focuses on assessment again. Students repeatedly revealed that they checked the guidelines, informative videos and slides for the tasks that were going to be evaluated and graded and since there are graded tasks in each unit, they checked the materials for each unit almost on an equal frequency. A majority of the students reported that they preferred handouts to slides and videos because checking information on handouts is what they are familiar with. Handouts also come in handy, they said. Some students stated that they can take a screenshot of the handouts on their phones to refer to them again when they need to. 'The Urls for videos were fun to watch but not practical when we are doing self-study' one student claimed. When students missed class they went back to what was covered during that session, they said during the interviews. That is why the percentage of clicks of the slides available on the OCW is among the highest. Students, without any exception stated that they needed to practice listening because they did not feel confident about their listening skills. Hence, the listening practice materials were the most clicked materials on the OCW. Although numerical data does not support the argument that the course-related games made available on the OCW were popular, students preferred to refer to them a lot during the interviews. The games on the OCW were mostly group games that the whole class played together. That is why the individual click rate is not high. However, the majority of the students mentioned them when they were asked which materials on the OCW they used most and why. They said that being an anonymous player in especially the Kahoot games, relieved the stress on their shoulders and the students who played in pairs were even more relieved International Online Journal of Education and Teaching (IOJET) 2018, 5(3), 673-685. 681 because they had the chance to ask for their peer's approval before they hit one of the multiple choice alternatives on their phones. 3. Discussion and Conclusion In this empirical study into student behavior different categories of data sources were examined to generate a knowledge of student preferences of online material available on the Middle East Technical University opencourseware. The data collected was from the last two years. It comprises of an input of student behavior from a relatively long period of time during which the Academic Oral Presentation Skills course was offered. During the course of time studied, the course was offered for five semesters including the summer school to an average of five thousand students. Due to this long course of observation period, the data represents a semi-static period of the use of supplementary online course material, it therefore dynamic in nature. This paper has reflected on the use of the open courseware materials of the Academic Oral Presentation Skills course offered as a required course to students from all the departments at the university. The study was based on objective, numerical data based on clicks of students, later discussed in group interviews with students. Although the qualitative data gathered during discussions provides reasons for students to prefer to ‘click’ certain materials provided online, the clicks themselves may not be a sign of to make assumptions on the reasons behind them, that is to say, as Macfayden and Dawson (2010, p 597) put it ‘simple clicking behavior in a learning management system is at best a poor proxy for actual user behavior of students’. While an account of the clicks of students cannot be considered as a sign of their learning, simple clicks alone can be attributed to intention and motivation to study, practice and explore course-related material, hence, can still be considered as a salient proof of interest, which embraces the growth of even more technology integration in education. First generation technology-driven learning analytics provides any stakeholder involved in the education process, including teachers, instructors, policy makers, and curriculum and material designers, educational Web 2.0 tools designers with insights from data-driven educational research as to the most efficient online learning tools. Shum and Crick (2012, p. 2) dwell on the invaluable consequences of the collection and interpretation of the growing evidence provided by learner analytics within educational research. It is claimed that ‘learners’ orientation towards learning their own learning dispositions significantly influence the nature of their engagement with new learning opportunities’. There are some potential limitations of this study. Firstly, there are limitations to the generalizability of the findings. The more and detailed analytics of users’ behaviors on online platform can be recorded and gathered for further and detailed analytics and explanation on their usage behaviors. Secondly, this study focused merely on users’ behaviors on the OCW platform. However, some guest users on the course page due to being an opencourseware page might affected the results. Finally, it would be better for learning analytics demonstrate change over time (Goggins, Galyens, Petakovic, Laffey, 2016), however the subjects of this current study during five semesters were different. Future research on the use of open courseware may make use of eye-trackers that provide data on focus and gaze (Kardan & Conati, 2012). Even more detailed inquiry is possible via emotion sensors which are promising in the education-related research (Arroyo, Cooper, Burleson, Woolf Muldner, Christopherson, 2009). There is no doubt that as technology becomes even more pervasive in education, the reflections of data-driven student behavior will gain even more importance in predicting the future of educational materials and Balbay & Kilis 682 preparing our mindset as educators accordingly to meet the growing demand of online learning contexts. International Online Journal of Education and Teaching (IOJET) 2018, 5(3), 673-685. 683 References Agudo-Peregrina, A. F., Iglesias-Padas, S., Conde-Gonzalez, M. A., & Hernandez-Garcia, A. (2014). Can we predict success from log data in VLEs? Classification of interactions for learning analytics and their relation with performance in VLE-supported F2F and online learning. Computers in Human Behavior, 31, 542-550. http://dx.doi.org/10.1016/j.chb.2013.05.031. American Psychological Association. (2010). Publication manual of the American Psychological Association (6th ed.). Washington, DC: American Psychological Association. Arnold, K. E., & Pistilli, M. D. (2012). Course signals at Purdue: Using learning analytics to increase student success. In LAK ‘12 Proceedings of the 2nd international conference on learning analytics and knowledge. New York: ACM. Arroyo, I., Cooper, D. G., Burleson, W., Woolf, B. P., Muldner, K., & Christopherson, R. (2009). Emotion sensors go to school. AIED, 17-24. Balbay, S., Kilis, S. (2017). Blended learning in a speaking skills course. In the proceedings of 3rd International Congress on Education, Distance Education and Educational Technology (ICDET), 24-25 November 2017, Antalya: Turkey. Barber, R., & Sharkey, M. (2012). Course correction: Using analytics to predict course success. In LAK ‘12 Proceedings of the 2nd international conference on learning analytics and knowledge (pp. 259–262). New York: ACM. Bienkowski, M., Feng, M., & Means, B. (2012). Enhancing teaching and learning through educational data mining and learning analytics: An issue brief. US Department of Education, Office of Educational Technology (pp. 1-57). Buckingham Shum, S., & Ferguson, R. (2012). Social learning analytics. Journal of Educational Technology & Society, 15(3). http://dx.doi.org/10.1145/2330601.2330616. Bull, S., & Kay, J. (2016). SMILI☺: A framework for interfaces to learning data in Open Learner Models, learning analytics and related fields. Artificial Intelligence in Education, 26(1), 293-331. Campbell, J.P., Oblinger, D.G. (2007). Academic Analytics, Educause. Campus Backbone Network METU-NET | Computer Center. Retrieved 22 May 2018, from https://bidb.metu.edu.tr/en/campus-backbone-network-metu-net Chapelle, C., & Douglas, D. (2006). Assessing language through computer technology. Cambridge, UK: Cambridge University Press. Crede, M., & Niehorster, S. (2012). Adjustment to college as measured by the student adaptation to college questionnaire: A quantitative review of its structure and relationships with correlates and consequences. Educational Psychology Review, 24(1), 133-165. http://dx.doi.org/10.1007/s10648-011-9185-5. Creswell, J. W. (2012). Educational research: Planning, conducting, and evaluating quantitative and qualitative research. Pearson Education, Inc. Creswell, J. W. & Plano Clark, V. L. (2011). Designing and conducting mixed methods research. SAGE Publications, Inc. Dörnyei, Z. (1998). Motivation in second and foreign language learning. Language Teaching, 31(3), 117-135. http://dx.doi.org/10.1017/S026144480001315X Balbay & Kilis 684 Dörnyei, Z. (2007). Research methods in applied linguistics: Quantitative, qualitative, and mixed methodologies. Oxford, UK: Oxford University Press. Dörnyei, Z., & Csizér, K. (2002). Some dynamics of language attitudes and motivation: Results of a longitudinal nationwide survey. Applied Linguistics, 23, 421–462. http://dx.doi.org/10.1093/applin/23.4.421 Dudeney, G., & Hockly, N. (2012). ICT in ELT: How did we get here and where are we going? ELT Journal, 66(4), 533-542. http://dx.doi.org/10.1093/elt/ccs050 Erten, I. H. (2014). The relationship between academic self-concept, attributions, and L2 achievement. System, 42, 391–401. http://dx.doi.org/10.1016/j.system.2014.01.006 Field, A. (2009). Discovering statistics for SPSS (3rd ed.). Los Angeles, CA: SAGE Publications. Ghonsooly, B., Khajavy, G. H., & Asadpour, S. F. (2012). Willingness to communicate in English among Iranian non–English major university students. Journal of Language and Social Psychology, 3, 197-211. http://doi.org/10.1177/0261927X12438538. Gibson, D., & de Freitas, S. (2015). Exploratory analysis in learning analytics. Technology, Knowledge and Learning, 21(1), 5–19. Goggins, S. P., Galyen, K. D., Petakovic, E., & Laffey, J. M. (2016). Connecting performance to social structure and pedagogy as a pathway to scaling learning analytics in MOOCs: an exploratory study. Journal of Computer Assisted Learning, 32(3), 244- 266. Hockly, N. (2013). Interactive whiteboards. ELT Journal, 67(3), 354-358. http://doi.org/10.1093/elt/cct021 Long, P., & Siemens, G. (2011). Penetrating the fog, analytics in learning and education. Educause Review, 46(5), 31-40. Macfadyen, L., P., & Dawson, S (2010). Mining LMS data to develop and “early warning system” for educators: A proof of concept. Computers & Education, 54(2), 588-599. http://dx.doi.org/10.1016/j.compedu.2009.09.008. Mah, D. K. (2016). Learning analytics and digital badges: potential impact on student retention in higher education. Technology, Knowledge and Learning, 21(3), 285-305. McCroskey, J. C., & Richmond, V. P. (1987). Willingness to communicate and interpersonal communication. In J. C. McCroskey & J. A. Daly (Eds.), Personality and interpersonal communication (129-156). Beverly Hills, CA: Sage. Peng, J. E. (2011). Towards an ecological understanding of willingness to communicate in EFL classrooms in China. System, 40, 203–213. http://dx.doi.org/10.1016/j.system.2012.02.002. Rienties, B., Hernandez Nanclares, N., Hommes, J., & Veermans, K. (2014). Understanding emerging knowledge spillovers in small-group learning settings: a networked learning perspective. In V. Hodgson, M. De Laat, D. McConnell, & T. Ryberg (Eds.). The Design, Experience and Practice of Networked Learning (Vol. 7, pp. 127-148). Dordrecht: Springer. Sclater, N., A. Peasgood, & J. Mullan (2016). Learning analytics in higher education: A review of UK and international practice. Technical Report April, Jisc, London. http://dx.doi.org/10.1093/applin/23.4.421 International Online Journal of Education and Teaching (IOJET) 2018, 5(3), 673-685. 685 Shum, B., S., & Crick, D., R. (2012). Learning Dispositions and Transferable Competencies: Pedagogy, Modelling and Learning Analytics. Proc. 2nd International Conference on Learning Analytics & Knowledge, (29 Apr-2 May, Vancouver, BC). ACM Press: New York Tempelaar, D. T., Rienties, B., & Giesbers, B. (2015). In search for the most informative data for feedback generation: Learning analytics in a data-rich context. Computers in Human Behavior, 47, 157-167. Verbert, K., Manouselis, N., Drachsler, H., & Duval, E. (2012). Dataset-driven research to support learning and knowledge analytics. Educational Technology & Society, 15(3), 133-148.