Irish Journal of Technology Enhanced Learning Vol 3, Issue 1, 2018 28 Using learning analytics to improve online formative quiz engagement Irene O’Dowd *1a a. Hibernia College (Received September 2017; final version received April 2018) Abstract This paper describes the findings of a small research study, conducted in a third-level online college, using learning analytics to examine the implementation of formative quizzes in a blended-learning post-primary teaching programme. Using historic data captured in a virtual learning environment (VLE) for a single cohort (n=126), patterns of use of formative knowledge check quizzes were analysed with particular regard to completion and retakes. Three hypotheses were tested using appropriate data correlation methods. Completion levels for quizzes were correlated with completion levels for other online tasks to see whether an increase in online task workload resulted in a decrease in quiz engagement. A second test compared levels of quiz re-attempts with completion levels for other online tasks, to see whether different patterns of quiz attempts were linked to different levels of online engagement. Finally, the data was analysed to ascertain the relationship, if any, between student gender and different patterns of quiz attempts, to see if gender might be a factor in quiz engagement. The findings of this study suggested that the decrease in engagement with quizzes was not significantly related to online task workload increase, and that there is a relationship between quiz re-attempts and higher module engagement. The findings are presented and discussed in the context of student engagement with online formative strategies in humanities-based subjects. Options are considered for enhancing engagement and formative value in this teaching and learning context; the potential of learning analytics in informing evidence-based improvements in digital learning design is also assessed. 1. Introduction The Professional Master of Education (PME) Post-Primary programme offered by the third- level online college in which this study was conducted is a two-year blended education programme. Students complete a range of ungraded tasks in the Moodle virtual learning environment (VLE) each week, including viewing presentations, contributing to discussion forums, completing quizzes, writing reflective content and participating in live webinars. * Corresponding author. Email: iodowd@hiberniacollege.net Irish Journal of Technology Enhanced Learning Ireland, 2018. © 2018 I O Dowd. The Irish Journal of Technology Enhanced Learning Ireland is the journal of the Irish Learning Technology Association, an Irish-based professional and scholarly society and membership organisation. (CRO# 520231) http://www.ilta.ie/ . This is an Open Access article distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. Irish Journal of Technology Enhanced Learning 29 However, academic staff have reported a consistent decrease in students' engagement with online activities throughout the programme lifecycle. As a starting point to find out more about this decrease in online engagement, this study will examine patterns of student engagement with formative quizzes at two different stages across the programme. These formative knowledge check quizzes enable students to monitor their own understanding of the course content and revisit and retake the quiz as needed. Unlike classical knowledge checks, which usually occur within the narrative or expository content (Lewis, 2010), these quizzes take place at the end of the session and are similar in format to a multiple-choice formative test (Simkin and Kuechler, 2005). The knowledge checks are ungraded but scored: that is, a student who completes the quiz receives a quiz result expressed as a percentage of questions answered correctly, but this result does not affect their overall grade for the course. A typical knowledge check contains between 6 and 10 questions, usually multiple-choice. Students can take a quiz multiple times until they feel they have understood the content and have received a mark they are happy with. The questions provide instant correct/incorrect feedback, specific to the options chosen, as well as general feedback that is given once the quiz is submitted. There is a substantial body of research validating the efficacy of using quizzes as a learning tool, as distinct from their use in summative assessment (see Salas-Morera et al., 2012; Cohen and Sasson, 2016; Tuttle, 2010; Nicol and Milligan, 2006; Kearns 2012). The theory behind formative quiz use emphasises their utility in student motivation, engagement, feedback and self-monitoring (Boud, 2000; Arend, 2007; Tuttle, 2010). These qualities are particularly important in online learning environments, where all or part of the instruction is engaged in by the student without a teacher or lecturer. The formative quiz is an iterative learning tool that requires both engagement and repeated use to realise its full potential. Recent research (Hoskins and Van Hooff, 2005; Richardson and Newby, 2006) highlights the importance of cognitive engagement – the engagement of students’ learning strategies and motivations – in the successful design of online instruction. Freasier et al. (2003) suggest that students are motivated to do formative quizzes if they perceive that the quizzes benefit their learning and if they can access them at any time; this is shown to be the case even when students understand that the formative quiz grades do not directly affect their course grade. Moreover, there is good evidence of positive correlations between students’ engagement with formative non-graded online quizzes and their final course grade. For example, Kibble (2011) finds that nonparticipation in formative quizzes is a consistent predictor of poor performance on summative examinations. The provision of instructive feedback is a key feature of formative quizzes, and most online quizzes are designed to provide this in some form. They provide an effective way of giving ongoing feedback on learning progress and enabling both tutors and students to identify gaps in knowledge of the content (Tuttle, 2010). The design of feedback is an important consideration in online formative quizzes: Lewis et al. (2010), from an examination of the literature on feedback from formative knowledge check quizzes, suggest inter alia that delayed feedback may be of greater benefit than immediate feedback, and that both can be provided if the content can be reviewed and the quizzes retaken. Cohen and Sasson (2016) inquire as to whether many attempts on an online quiz benefit the students’ learning, concluding that quiz designs that feature multiple permitted attempts (combined with randomisation) enable students to take responsibility for their own learning, reflect on their understanding, implement strategies to increase their understanding, and re-attempt the quiz to ascertain whether they have been successful. An important aim of online learning is to foster learning autonomy among its participants; Nicol and Milligan (2006) and others cite I. O’Dowd 30 feedback as a key tool for doing this. Interestingly, some research considers gender as a factor in online quiz engagement, in particular in the number of times a formative quiz is attempted. Gunn (2003) found that male students may have a tendency towards non- engagement with quizzes due to overconfidence in their abilities to meet course requirements, manifesting in a lower number of formative quiz attempts among other online behaviours. However, research in this area is far from unanimous: Hoskins and Van Hooff (2005) found little difference between the numbers of males and females making two or more attempts at self-assessment quizzes. Similarly, Fill and Brailsford (2005), in a large study of online formative quiz engagement, analysed possible gender distinctions in the number of quiz attempts made, and found no evidence of difference. The research landscape thus indicates that the examination of quiz use patterns may provide useful information regarding learner engagement and autonomy. Not only is quiz completion a signifier of engagement, but quiz re-attempts also serve as an indicator of whether the quizzes are being used formatively – that is, the feedback from the first attempt is encouraging learners to retake the quiz as a form of revision and knowledge consolidation. The user data available in Moodle for tasks such as quizzes is an obvious starting point for such an exploratory study, as it captures a wealth of information on engagement such as quiz reattempts. The inferential analysis of this data, following the “exploratory data analysis” approach that influences modern quantitative methods (Connolly, 2007), aims to build a picture of how and why engagement might decrease over time. The questions that this study aims to answer can be summarised as follows: • Does an increase in online task workload over the lifecycle of the programme result in a decrease in quiz engagement? • Are different patterns of quiz attempts linked to different levels of online engagement? • Is gender a factor in quiz engagement? 2. Methods The sample chosen for the study consisted of the entirety of a single programme cohort (n=126), which was selected on the basis that it has now completed the online component of the programme, including the two online modules selected for the study. One module, Foundations of Education 1, was selected from early in the students' first year; the other module, Foundations of Education 2, was selected from early in their second year. These modules were selected because they are foundational modules taken by all students in the cohort, and also to facilitate comparison between student activity rates at the start of the programme – when students' workload is light – and at the start of year 2 when workload is heavier. Two types of data were extracted from the college's Moodle VLE for the selected cohort: activity completion data for all tasks in the two selected modules, and usage logs for all knowledge check quizzes in those modules. In each case, the raw data was exported in Excel format and saved as Excel worksheets for cleaning and preparation. The data was anonymised by replacing student names with case numbers and removing all additional information on student identities such as e-mail addresses and IP addresses. Any unnecessary data was removed from the worksheets and codes were added to denote specific values. The final dataset was then imported into SPSS for descriptive and inferential analysis. Irish Journal of Technology Enhanced Learning 31 The study tested three hypotheses as follows: • Completion levels for quizzes were correlated with completion levels for other online tasks, to see whether an increase in online task workload over the lifecycle of the programme resulted in a decrease in quiz engagement. • Levels of quiz re-attempts were correlated with completion levels for other online tasks, to see whether different patterns of quiz attempts (that is, single attempts or multiple attempts) were linked to different levels of online engagement. • A correlation was conducted to ascertain the relationship, if any, between student gender and different patterns of quiz attempts, to see if gender might be a factor in quiz engagement. 2.1 Validity and reliability For the purposes of assessing the validity and reliability of this study, some observations should be made about the nature of user interaction data captured by the VLE. Dietrichson (2013) claims that current analytics methodology is as yet unable to conduct effective evaluation of most of the data captured by VLEs such as Moodle, particularly for the improvement of constructive pedagogical design. It is thus important to understand the limitations of data analytics, while at the same time appreciating its potential for providing insight into some key areas. For example, a simple and effective way to analyse student engagement with online content is to look at the activity data captured by the VLE. A study by Heinrich (2015) suggests that user access logs provide a good indicator of engagement with VLE content, while acknowledging that evidence of having accessed the content does not tell us much about the extent of engagement with the content once accessed. In Moodle, when an activity is logged as completed, this means that the action denoting activity completion, as defined in the activity settings, has been completed by the student. The nature and level of granularity available for this setting varies according to the type of activity. For example, a ‘passive’ activity such as a downloadable document for reading is assumed completed when the student is recorded by the VLE to have viewed the document. There is no way of verifying through data analytics alone whether the student actually read the document. The quiz activity, which we are concerned with here, enables the capture of somewhat more meaningful engagement data. Completion of a quiz can only be logged if the student both attempts the quiz and submits the attempt to obtain a score, so the activity data for a quiz is comparatively straightforward and unambiguous. We can tell from the data whether or not (and how many times) a student has answered the questions and submitted them, all of which enable a valid measurement of overall quiz engagement. Regarding reliability, the design of the study was aimed at ensuring a stable set of findings as the tests were conducted on data from two separate modules at different stages of the programme. The stability of the test results across both modules thus provides a strong indicator of reliability. 2.2 Ethical considerations No face-to-face or direct collection of participant data was involved in this study; the data used for the research was collated entirely from the logs of the college's Moodle VLE, a fully secured online learning environment that adheres to international data protection standards. The data, captured from previously completed online activity in specific modules, was subject to a stringent anonymisation process, ensuring that no personal information was retained or discernible. As this research has been undertaken with a view to improving the I. O’Dowd 32 design of online content, it falls under the college’s specification of “aggregate or anonymised information gathered [and] used/published to feed back into internal academic research [...] and overall programme enhancements including programme design, content, delivery and validation”, and thus participant consent has been obtained through the students’ signing of a data protection form at enrollment. 3. Results Table 1 shows that a maximum of 4 knowledge check quizzes was available for completion in Module 1 and a maximum of 9 for Module 2. There were 30 tasks available for completion in Module 1 and 70 in Module 2. Module 2 contained a greater number of sessions, hence the larger number of task components. Variable N Min. Max. Mean Standard Deviation Number of knowledge checks attempted in Module 1 126 0 4 2.27 1.627 Number of tasks attempted in Module 1 126 0 30 21.71 8.355 Number of knowledge checks attempted in Module 2 126 0 9 2.92 3.518 Number of tasks attempted in Module 2 126 0 70 33.92 23.947 Table 1: Average numbers of knowledge checks and tasks attempted in Modules 1 and 2 A series of histograms (Figures 1, 2, 3 and 4) shows the distribution of quiz and task completions across both modules. Irish Journal of Technology Enhanced Learning 33 Figure 1: Distribution of knowledge check quiz completions in Module 1 I. O’Dowd 34 Figure 2: Distribution of task completions in Module 1 Figure 3: Distribution of knowledge check quiz completions in Module 2 Irish Journal of Technology Enhanced Learning 35 Figure 4: Distribution of task completions in Module 2 It can be seen that the distributions for both sets of data are skewed, right for module 1 and left for module 2, and therefore the use of the mean and standard deviation in Table 1 do not adequately convey the average values of the data. The median and interquartile ranges in Table 2 give a more accurate indication of these. Variable Median IQR Number of knowledge checks attempted in Module 1 3 3 Number of tasks attempted in Module 1 24 10 Number of knowledge checks attempted in Module 2 1 6 Number of tasks attempted in Module 2 36.5 46 Table 2: Median and interquartile range for distribution of task and knowledge check attempts in Modules 1 and 2 As can be seen by comparing the median values between both modules, on average fewer knowledge check quizzes were attempted in Module 2 than in Module 1. The boxplot graphs in Figure 5 show the median and interquartile ranges in graph form and also show some outliers in the distribution of task completions in Module 1 that would otherwise have skewed the mean value. Figure 5: Boxplots showing distributions of quiz and task completions in both modules The pie charts in Figure 6 describe the percentages of the cohort who retook quizzes in both modules. The charts represent measurements of the same cohort at two different stages of I. O’Dowd 36 their programme: the Foundations 1 module at the start of year 1 and the Foundations 2 module at the start of year 2. As can be seen, a higher percentage of students (36.51%) retook quizzes in Module 1 than in Module 2 (17.6%). Figure 6: Percentages of cohort who retook quizzes in Modules 1 and 2 4. Findings Hypothesis 1: There is a correlation between the number of quizzes completed and the number of tasks completed in both modules. To test this hypothesis, a bivariate correlation test was conducted between the number of knowledge check quizzes completed in both modules and the number of tasks completed in both modules, with this relationship displayed in scatterplot graphs (Figures 7 and 8). The statistical significance and correlation coefficient between the two variables was calculated using a Pearson correlation test. A positive correlation was found in relation to the number of knowledge check quizzes and the number of tasks completed in Module 1, and the correlation was found to be statistically significant (p < 0.001, r = 0.812). Irish Journal of Technology Enhanced Learning 37 Figure 7: Scatterplot of correlation between task and quiz completion, Module 1 A positive correlation was also found in relation to the number of knowledge check quizzes and the number of tasks completed in Module 2, and the correlation was found to be statistically significant (p < 0.001, r = 0.810). Figure 8: Scatterplot of correlation between task and quiz completion, Module 2 I. O’Dowd 38 This finding rejects the null hypothesis and confirms that across both modules there is a statistically significant positive correlation between the number of quizzes completed and the number of tasks completed. Increases in knowledge check quiz attempts were positively correlated with increased numbers of attempts on module tasks. Hypothesis 2: There is a correlation between the incidence of re-taking of quizzes and the number of tasks completed in both modules. This analysis involved correlating whether or not quizzes are retaken (that is, taken more than once) with the number of tasks completed across both modules using an Independent Samples T-Test. The statistical significance, t-value, degree of freedom and equality of variance between the variables were calculated by comparing the means of the two nominal variable categories: the number of knowledge checks re-taken (>1 times) and the number of knowledge checks taken only once. For the first module, the difference between the mean number of tasks completed by those who retook quizzes and those who attempted quizzes only once was found to be statistically significant (p < 0.001, t = -8.077, df = 108.386). For the second module, the difference between the mean number of tasks completed by those who retook quizzes and those who attempted quizzes only once was found to be statistically significant (p < 0.001, t = -8.560, df = 67.042). The finding thus rejects the null hypothesis and confirms that, in both year 1 and year 2 modules, the mean level of engagement in module tasks was significantly greater among participants who attempted quizzes more than once than among participants who attempted the quiz only once. There is therefore a statistically significant correlation between high levels of engagement with module tasks and multiple quiz attempts. Hypothesis 3: There is a correlation between the re-taking of quizzes and the gender of the participants in both modules. This analysis involved correlating student gender with whether or not quizzes are retaken (that is, taken more than once) using a Chi-square test. No statistically significant evidence was found in either module as to difference between male students and female students, in terms of whether or not they took the quizzes more than once (Module 1: p = 0.05, Chi- square = 3.850, df = 1; Module 2: p = 0.1, Chi-square = 2.588, df = 1). The findings thus failed to reject the null hypothesis. 5. Discussion In online education, as in any other kind, predictors of student success vary greatly between institutions and learning situations. However, research does seem to suggest that many institutions find online engagement, even if this is measured by just logging on to the VLE, a key predictor of student success, particularly in the early weeks of the first year (Sclater and Mullan, 2017). An examination of the findings of this study suggests that online engagement with quizzes and other tasks decreases from year 1 to year 2 of the programme. The fact that fewer knowledge check quizzes were attempted in Module 2 than in Module 1 suggests that these quizzes might not be seen by many students as a formative learning tool to help them through the other tasks. On the other hand, the positive correlation between task completion and quiz completion across the modules examined in this study suggests that the problem of Irish Journal of Technology Enhanced Learning 39 engagement with quizzes might be a symptom of a problem with online engagement more broadly. The findings of this study do not suggest that quizzes are being ignored because of workload caused by other tasks. Indeed, the number of quizzes attempted by the cohort increases with the number of online tasks attempted, even in the module which takes place in year 2 when students’ workload would be heavier. However, as this study did not control for other variables such as offline workload and the nature of the tasks, it would be unwise to overstate the causality of this relationship. It is also worth mentioning that the study did not investigate the possibility of an upper cut-off point for number of tasks and quizzes for which the correlation remains true, although it would be reasonable to assume that such a cut-off point exists. The positive correlation between the incidence of students re-taking quizzes and the level of engagement across all tasks suggests that students who engage more with online content are also more likely to be using the quizzes in way they are intended – as formative tools for learning rather than as tests. The study failed to find any link between gender and likelihood of re-attempting quizzes, suggesting that if there are differences between how men and women interact with online content, these are not apparent in the formative use of quizzes online. 6. Conclusion The area of learning analytics is seeing rapid growth in institutions providing online education as the value of user data for informing evidence-based improvements in course design becomes apparent. This study was an attempt to begin to provide an evidence base from which to explore whether current methods of using and engaging with quizzes could be improved. The findings provide an evidential foundation for the acceptance or rejection of some assumptions regarding students' online engagement, such as the influence of task workload and gender on quiz engagement. The findings also showed that although online quizzes are intended and designed as formative assessment, in practice many students do not engage with them as a formative tool for iteratively improving their understanding. That the study suggested a strong relationship between formative use of quizzes and online engagement is well noted – although this connection needs to be interrogated further as it is unlikely to be a straightforward causal link. Future, possibly experimental studies might consider exploring different approaches in feedback and scaffolding, using alternatives to MCQs, using branching and pathways to encourage re-attempts, and looking at the frequency and distribution of quizzes across modules. I. O’Dowd 40 References Arend, B. (2007). Course Assessment Practices and Student Learning Strategies in Online Courses. Journal of Asynchronous Learning Networks, 11(4), 3-17. Boud, D. (2000). Sustainable Assessment: Rethinking assessment for the learning society. Studies in Continuing Education, 22(2), 151-167. doi: 10.1080/713695728 Cohen, D. and Sasson, I. (2016). Online quizzes in a virtual learning environment as a tool for formative assessment. Journal of Technology and Science Education, 6(3), 188-208. doi: 10.3926/jotse.217 Connolly, P. (2007). Quantitative Data Analysis in Education: A Critical Introduction Using SPSS, London: Routledge. Dietrichson, A. (2013). Beyond Clickometry: Analytics for Constructivist Pedagogies. International Journal on E-Learning, 12(4), 333-351. Fill, K. and Brailsford, S. (2005). Investigating gender bias in formative and summative CAA, in Proceedings of the 9th CAA Conference, Loughborough University, UK. Retrieved from https://dspace.lboro.ac.uk/dspace-jspui/bitstream/2134/1994/1/FillK_BrailsfordS.pdf [Accessed 30/03/2018] Freasier, B., Collins, G. and Newitt, P. (2003). A Web-Based Interactive Homework Quiz and Tutorial Package To Motivate Undergraduate Chemistry Students and Improve Learning. Journal of Chemical Education, 80(11), 1344-1364. doi: 10.1021/ed080p1344 Gunn, C. (2003). Dominant or Different? Gender Issues in Computer-Supported Learning. Journal of Asynchronous Learning Networks, 7(1), 14-30. Heinrich, E. (2015). Can learning analytics provide useful insights? An exploration on course level, in T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, and K. Clarke (eds), Globally connected, digitally enabled: proceedings of ascilite 2015, Perth, Australia, pp. 108- 112. Retrieved from http://research.moodle.net/81/ [Accessed 30/03/2018] Hoskins, S.L. and Van Hooff, J.C. (2005). Motivation and ability: which students use online learning and what influence does it have on their achievement?. British Journal of Educational Technology, 36(2), 177–192. doi: 10.1111/j.1467-8535.2005.00451.x Kearns, L. (2012). Student Assessment in Online Learning: Challenges and Effective Practices. MERLOT Journal of Online Learning and Teaching, 8(3). Retrieved from http://jolt.merlot.org/vol8no3/kearns_0912.htm [Accessed 30/03/2018] Kibble, J.D. (2011). Voluntary participation in online formative quizzes is a sensitive predictor of student success. Advances in Physiology Education, 35(1), 95-96. doi: 10.1152/advan.00053.2010 Lewis, D., Trail, T., Srinivasan, S., Lee, S. and Lopez, S. (2010). Knowledge check questions: Best practices for use of this instructional strategy, in Proceedings of the World https://dspace.lboro.ac.uk/dspace-jspui/bitstream/2134/1994/1/FillK_BrailsfordS.pdf http://research.moodle.net/81/ http://jolt.merlot.org/vol8no3/kearns_0912.htm Irish Journal of Technology Enhanced Learning 41 Conference on Educational Multimedia, Hypermedia and Telecommunications 2010, Chesapeake, VA, pp. 2783-2788. Retrieved from http://davidlewisphd.com/publications/2010-Lewisetal-EDMEDIAbestPracticesFinal.pdf [Accessed 30/03/2018] Nicol, D. J. and Milligan, C. (2006). Rethinking technology-supported assessment in terms of the seven principles of good feedback practice, in C. Bryan and K. Clegg (eds), Innovative Assessment in Higher Education. London: Taylor and Francis Group Ltd. Paquette, P. (2016). Instructing the Instructors: Training Instructors to Use Social Presence Cues in Online Courses. Journal of Educators Online, 13(1), 80-108. Richardson, J.C. and Newby, T. (2006). The Role of Students' Cognitive Engagement in Online Learning. American Journal of Distance Education, 20(1), 23-37. doi: 10.1207/s15389286ajde2001_3 Salas-Morera, L., Arauzo-Azofra, A. and García-Hernández, L. (2012). Analysis of online quizzes as a teaching and assessment tool. Journal of Technology and Science Education, 2(1), 39-45. doi: 10.3926/jotse.30 Sclater, N. and Mullan, J. (2017). Jisc briefing: Learning analytics and student success – assessing the evidence. JISC. Retrieved from http://repository.jisc.ac.uk/6560/1/learning- analytics_and_student_success.pdf [Accessed 30/03/2018] Simkin M.G. and Kuechler W.L. (2005). Multiple choice tests and student understanding: what is the connection?. Decision Sciences Journal of Innovative Education, 3, 73–97. doi: 10.1111/j.1540-4609.2005.00053.x Tuttle, H.G. (2010). 10 Reasons to Use Online Practice Quizzes (Formative Assessment). Education with Technology blog. Retrieved from https://eduwithtechn.wordpress.com/2011/04/04/10-reasons-to-use-online-practice-quizzes- formative-assessment/ [Accessed 30/03/2018] http://davidlewisphd.com/publications/2010-Lewisetal-EDMEDIAbestPracticesFinal.pdf http://repository.jisc.ac.uk/6560/1/learning-analytics_and_student_success.pdf http://repository.jisc.ac.uk/6560/1/learning-analytics_and_student_success.pdf https://eduwithtechn.wordpress.com/2011/04/04/10-reasons-to-use-online-practice-quizzes-formative-assessment/ https://eduwithtechn.wordpress.com/2011/04/04/10-reasons-to-use-online-practice-quizzes-formative-assessment/