williams.pdf Australasian Journal of Educational Technology 2012, 28(2), 199-213 The impact of online lecture recordings on student performance Andrew Williams, Elisa Birch and Phil Hancock The University of Western Australia The use of online lecture recordings as a supplement to physical lectures is an increasingly popular tool at many universities. This paper combines survey data with student record data for students in a Microeconomics Principles class to examine the relative effects of lecture attendance and online lecture recordings. The main finding is that students using the online lectures as a substitute for attending lectures are ultimately at a fairly severe disadvantage in terms of their final marks. Moreover, students attending few face to face lectures do not close this gap by viewing more lectures online. In contrast to this, students who attend the majority of lectures in person do receive a benefit from additional use of the lecture recordings. The results provide empirical evidence that, when used as a complementary tool, lecture recordings are a valuable supplement for students. However, when used as a substitute to attending lectures, lecture recordings provide no additional benefit. 1. Introduction and background The use of web-based learning technologies at universities has increased significantly over the past few years. This is particularly true of audio and/or visual recordings of face to face lectures, which are streamed via the web for students to view or download. A number of papers have attempted to examine the perceived benefits of using these online lecture recordings. For example, Preston, Phillips, Gosper, McNeill, Woo & Green (2010) surveyed both students and lecturers on their perceptions and use of web-based learning technologies (WBLT). Among other results, they found that a majority of the students surveyed (68%) felt they ‘could learn just as well using WBLT as face to face’. Students also appreciated the flexibility that the WBLT afforded them, a finding that others have also noted (for example, Taplin, Low & Brown, 2011; Bennett & Maniar, 2007 and others). Another perceived benefit is that lecture recordings provide a better ‘fit’ for the current generation of students (Skene, Cluett & Hogan, 2007), in terms of their extensive familiarity with web-based content, as well as their desire to have immediate and continuing access to course material. This may be particularly important for students who, whether by choice or necessity, are engaged in paid employment whilst concurrently studying for their degrees. What of the perceived negative effects of using web-based lecture recordings? Bennett and Maniar (2007) note two major issues with lecture recordings: (1) they make learning uninteresting, because the student loses the immediacy of the lecture, and the enthusiasm of the lecturer may not translate to the recording; (2) students may not develop as independent learners, as the recorded lecture content takes on a level of importance it probably should not have. A number of studies have incorporated the 200 Australasian Journal of Educational Technology, 2012, 28(2) qualitative comments of students (for example, Larkin, 2010; Skene et al, 2007; Preston et al, 2011, and others), and a common refrain has been that lecture recordings decrease lecture attendance (however, see Gysbers, Johnston, Hancock and Denyer, 2011 for a counter to that finding), thereby diminishing the ‘atmosphere’ of a lecture, and decreasing the degree of interaction during the lecture. Many of these papers, whilst adding to our understanding of this issue, have not subjected their data to any rigorous statistical analysis to support the use of these web- based lecture technologies, but instead have reported simple survey results, such as whether students ‘like’ having access to the online lecture recordings. In an interesting paper, Taplin et al (2011) try to address this issue by noting that, whilst the vast majority of students in their survey did indeed say that they valued having these recordings made available to them, far fewer were actually willing to pay (either directly or indirectly) for the right to do so. Where more formal and rigorous statistical methodology has been employed, the results have been somewhat mixed, in part because they often focus on slightly different scenarios. For example, Day and Foley (2006) ran a quasi-experiment, with half of one class attending traditional face to face lectures, whilst the other half used lecture recordings (along with other web-based learning materials). They found that the students using the lecture recordings performed better in their final grades than those students who were only attending lectures. Chiu, Lee and Yang (2006), using a similar methodology, noted a similar result. However, Figlio, Rush, and Yin (2010) found a modest, though significant, positive effect on grades for those students in their experiment who attended the ‘live’ lectures, as opposed to those who were only allowed access to the lecture recordings. McNulty, Hoyt, Chandrasekhar, Espiritu, Gruener, Price and Naheedy (2011) noted a negative correlation between medical students who accessed video recordings of lectures and their ultimate academic performance, though these were straight correlations, and did not account for other possible contributing factors. Others again (Brotherton & Abowd 2004; Bell, Cockburn, McKenzie & Vargo 2001) found no statistically significant difference in the grades obtained by the online versus face to face groups of students. Von Konsky, Ivins and Gribble (2009) provided some evidence that, whilst lecture attendance was roughly the same across all grades, passing students were more likely to use lecture recordings as a supplement to their lectures. Wieling and Hofman (2010) purported to find a strong substitution effect between face to face and online lecture recordings. Students who attended most or all of the lectures face to face received no additional benefit from viewing lectures online. Similarly, the positive effect on final grades of viewing online lectures was higher when few lectures were attended in person. This paper is an attempt to add to the literature on this issue by examining whether students who use lecture recordings extensively perform on average better or worse in terms of their final grade than students who predominantly attend face to face lectures, and what the interaction between the two modes of delivery might be. In other words, whether these online lecture recordings are more beneficial to students as a substitute for attending these lectures in person, or as a complement. In order to examine this issue, a survey of students was conducted, which was then married to student data gathered on the use of lecture recordings and other web-based learning materials. After controlling for a wide variety of student characteristics, our analysis shows that online lecture recordings are most effective as a complement to attending lectures, not as a substitute. Williams, Birch and Hancock 201 The paper is organised as follows. Section 2 discusses the methodology employed in the paper, including details of the survey, and other sources of data we use. Section 3 introduces a formal model of education achievement, based on a common education production function. In Section 4, we report and discuss the results of the formal regression analysis. Section 5 has some concluding comments. 2. Methodology The survey was conducted at The University of Western Australia, in classes for the first year Microeconomics Principles class, during tutorials in the final week of Semester 1 of 2010. The survey covered a range of issues that looked at what one might call the first year ‘experience’. Students were asked a number of questions relating to different aspects of their university experience, including questions on: (i) their characteristics (gender, birthplace, age, whether living at university colleges, or with parents, and so on); (ii) education information (prior education, including whether they had studied economics before, as well as information on parents’ education levels); (iii) employment information (whether they engaged in paid employment, how many hours per week, and their opinion on whether it helped or hindered their study); (iv) social networking (whether they belonged to university clubs, time spent at these clubs, as well as information on whether they had made friends since starting university); (v) experiences at university (a series of questions on their degree of satisfaction with their university life, what their biggest problems were in the transition from high school and so on); (vi) study habits (including hours per week of study, how many face to face lectures they attended, plus several questions on their use of lecture recordings). For the purposes of this paper, it is this last category that is of most interest, where students self-reported the number of lectures they attended (out of 26). There is an obvious problem here in that students may have over-reported the number of lectures they attended. Although we cannot guarantee students were being truthful, they were also asked a number of questions that were verifiable from other sources, such as student records. One question asked them to report their tertiary entrance score, which we then compared to their formal student records. Any student whose answer was more than 5 percentile points off their actual score was subsequently omitted from the analysis. To get as complete a picture as possible for the statistical analysis in the following section, we also took advantage of data from two other sources: (1) student record data, and (2) logging data from the university’s learning management system. Using this latter source, we were able to get information on how many times they visited the Lectopia web site and how many of the voluntary practice quizzes they attempted. Of the 866 students who ultimately received a mark for this unit in Semester 1, we were able to get complete data for 371 of them (43%). Table 1 gives some descriptive statistics of the variables used in the formal analysis in the following section, as well as highlighting whether the sample means differ from the overall population means. Almost all variables appear to be representative of the overall population of the unit, with the exception of the final mark, where the average within the sample is around four marks above the overall average for the unit. However, this is due largely to the fact that students with particularly low marks became disengaged from the unit during the semester, ceased to attend either lectures or tutorials, and thus did not take part in the survey. Therefore, the sample essentially represents those students still 202 Australasian Journal of Educational Technology, 2012, 28(2) actively engaged in the unit throughout the semester. The sample also over-represents students living with their parents, and students living at a University college (and therefore under-represents students living elsewhere). Nevertheless, despite this, we believe the sample is certainly representative of the overall population of this unit. In terms of individual usage of the recordings, the average number of hits across the 371 students for whom we obtained complete data for was 33.48 hits, with the smallest number being one hit and the largest 119. With respect to lecture attendance, around 65% of students attended at least 19 out of the 26 lectures, with students reporting across the entire range of attendance from zero through to 26 (there being two 45- minute lectures per week for 13 weeks in this semester). Figure 1 shows the average number of online recording hits by the number of lectures attended. From this, it appears that students who are not attending many lectures are at least (on average) viewing them more often online. For example, students only attending 0-6 lectures out of 26 have an average number of hits that is almost double the number of those attending either 22-24 or 25-26 lectures. In other words, many students appear to be using the online recordings often as a substitute for lectures (the pair-wise correlation between them is -0.28). The more pressing question, however, is whether the lecture recordings and face to face lectures are perfect substitutes – do students viewing fewer lecture recordings, but attending more lectures in person, ultimately receive the same final mark, ceteris paribus, as students who use lecture recordings extensively, but do not attend many face to face lectures? 0 5 10 15 20 25 30 35 40 45 50 0-6 7-13 14-18 19-21 22-24 25-26 Lectures attended Lecture recording hits Figure 1: Use of lecture recordings by attendance at lectures Williams, Birch and Hancock 203 3. Modelling student performance The majority of studies that estimate the determinants of academic performance (for example, Anderson, Benjamin & Fuss, 1994; Dobson & Skuja, 2005; Birch & Miller, 2007) are based on an education production function. In this model, a student’s tertiary academic performance (Api) is a function of a variety of student characteristics. In this analysis, we control for a large number of these characteristics: their prior academic achievements (Edui); their personal characteristics (Pci); characteristics of the secondary school attended (Ssi); whether they are repeating the unit (Rpi); a dummy for whether the student engaged in paid employment (Wi), plus an interaction term of this employment dummy multiplied by the number of hours of paid employment per week (HWi); whether they did certain selected subjects in high school (economics, lower level mathematics, English literature, and/or physics) (Pki); and their use of (voluntary) online quizzes, which here is used as a proxy for effort (Ei). Finally, we include our two main variables of interest: the number of online recording ‘hits’ during the semester (LRi), and the number of lectures attended (LECi). Api = F(Edui, Pci, Ssi, Rpi, Wi, HWi, Pki, Ei, LECi, LRi) . . . . . . . (1) It is common for studies to measure students’ academic performance by their final mark for their unit of study (usually measured as a mark out of one hundred) and estimate the production function using ordinary least squares (OLS). This procedure allows for the determinants of academic performance to be examined at the conditional mean of university marks, and is the approach we take here. We also ran these regressions using an ordered probit model (whereby students are grouped by their grade, rather than their percentage score), however, the results were quantitatively very similar, and so have not been included here. Furthermore, to take into consideration the existence of outliers that may bias the results, we removed observations whose leverage value indicated the presence of an outlier - generally this was either in the coefficient with hours worked, as well as a couple of students who had an exceptionally high number of lecture recording hits. The leverage value is a technique designed to identify observations that are far from the predicted values. Before looking at some of the specific results of this analysis, a couple of additional points are worth noting. We have included the online quizzes here as a proxy for ‘effort’, because we want to control for the fact that students not coming to lectures may simply be less engaged in the unit, and this results in a poor final mark. If we don’t control for this (lack of) effort, this could mean that a lack of lecture attendance would show up as positively related to student performance (that is, low attendance equals low final mark). But this is because they are not putting in any effort, not because they are not attending lectures per se. With respect to these quizzes, they were completely voluntary, could be attempted any number of times, and did not count towards their final grade. We believe in this instance that these quizzes therefore represent a better proxy for effort than, say, tutorial attendance, because tutorial attendance was worth 10% of the final grade, and hence there is an inherent bias in attendance with respect to ‘effort’. Because the quiz variable has a number of ‘zeros’, we can’t treat this as a continuous variable in our estimation, and so we have divided the quiz variable into dummy categories (0-1 quizzes attempted, 2-10 quizzes, 11-15 quizzes, 16-20 quizzes, 21-25 quizzes, and 26 and over quizzes). This issue also arises for our lecture attendance variable, and so we have also divided this into dummy variable categories (0-6, 7-13, 14-18, 19-21, 22-24, and 25-26 lectures). 204 Australasian Journal of Educational Technology, 2012, 28(2) Table 1, as well as reporting the summary statistics on each of the variables employed, also has information on the omitted variables for the various dummy variables employed. Statistically speaking, the coefficient on the included dummy variables reflects the mark advantage (or disadvantage) relative to the omitted group. Table 1: Descriptive statistics All studentsVariable/ Code Description Mean Std dev. Students’ marks Mark Continuous variable for students’ final mark in the first-year foundation economics unit at UWA. 64.590*** 12.874 Tertiary Ent- rance Score TER Continuous variable for students’ TER. It is ranked a mark out of one hundred. 90.027 7.307 Male Male students. 0.450 0.498Gender Female Omitted dummy variable for female students. Aust Students born in Australia. 0.612 0.488Country of birth Born overseas Omitted category - Dummy variable for students born overseas. Part Dummy variable for students who study at university on a part time basis defined as those whose aggregate ‘equivalent full time student unit’ for all units of study is less than 0.75. 0.040* 0.197Attendance type Full Omitted category – full-time students Parent Dummy variable for students who live with their parents. 0.749*** 0.434 College Dummy variable for students who live in one of the UWA colleges. 0.062** 0.241 Living arr- angements Other Omitted category – students who live on their own or with non-parents. Econs Dummy variable for students who studied economics in the final year of high school. 0.439 0.497 Noecon Omitted category. Discrete_ maths Dummy variable for students who studied discrete mathematics in final year high school. 0.485 0.500 Nodiscrete Omitted category. Englit Dummy variable for students who studied English literature in the final year of high school. 0.218 0.414 Noenglit Omitted category. Physics Dummy variable for students who studied physics in the final year of high school. 0.307 0.462 TEE subjects studied in high school Nophys Omitted category. Cath Dummy variable for students who attended a Catholic high school. 0.208 0.406 Indp Dummy variable for students who attended an Independent high school. 0.429 0.496 School type Govt Omitted category – students who attended a Government high school. Repeat Dummy variable for students who are repeating the first-year economics unit. 0.075 0.265Repeating the unit Norepeat Omitted category – students who are taking the first year economics unit for the first time. Work Dummy variable for students who engaged in paid employment. 0.631* 0.483Paid employment Nowork Omitted category – students who did not engage in paid employment. Williams, Birch and Hancock 205 Work x hours Interaction variable of the number of hours per week of paid employment multiplied by Work. 6.619 7.426 Lecture recording LR Continuous variable for the number of hits registered for lecture recordings (uncentred). 33.480 25.184 LEC 7-13 Dummy variable for students who attended between 7 and 13 lectures 0.105 0.307 LEC 14-18 Dummy variable for students who attended between 14 and 18 lectures 0.151 0.358 LEC 19-21 Dummy variable for students who attended between 19 and 21 lectures 0.229 0.421 LEC 22-24 Dummy variable for students who attended between 22 and 24 lectures 0.205 0.404 LEC 25-26 Dummy variable for students who attended between 25 and 26 lectures 0.210 0.408 Lectures attended LEC 0-6 Omitted variable - Dummy variable for students who attended between 0 and 6 lectures Quiz 0-1 Dummy variable for students who attempted between 0 and 1 quizzes. 0.108 0.311 Quiz 2-10 Dummy variable for students who attempted between 2 and 10 quizzes. 0.385 0.487 Quiz 11-15 Dummy variable for students who attempted between 11 and 15 quizzes. 0.272 0.446 Quiz 16-20 Dummy variable for students who attempted between 16 and 20 quizzes. 0.124 0.330 Quiz 21-25 Dummy variable for students who attempted between 21 and 25 quizzes. 0.062 0.241 Online quizzes attempted Quiz 26+ Omitted variable - Dummy variable for students who attempted more than 25 quizzes. Notes: Means and standard deviations based on the 371 student sample. *, **, *** indicates that the sample is statistically significantly different from the overall student population of the unit at the 10, 5, and 1% levels respectively. 4. Results Column 1 of Table 2 includes all of our control variables, but excludes the lecture recording and lecture attendance variables. Many of these results are similar to those presented in the existing literature (see Birch & Miller, 2004 for a review of studies). For example, the students’ TER (Tertiary Entrance Rank), which represents their prior academic achievement in the last year of high school, has a positive and statistically significant effect on student performance. Every 1 percentile increase in a student’s rank translates on average to a higher final mark of 0.84 percentage points. Of the Year 12 subjects studied, prior knowledge of economics and English literature resulted in a final mark that was 4.6 and 2.5 marks respectively higher than those who had not studied these subjects at secondary school, which is in line with previous research on the benefit of prior knowledge of economics (see for example, Birch & Williams, 2010). Students who took the lower level (discrete) mathematics, however, have a significantly lower mark than those who did not study discrete mathematics (by 6.8 marks). In Year 12 in Western Australia, students in 2009 and earlier could take discrete, applicable, or calculus mathematics (or no mathematics at all). Of these, applicable and calculus could be considered the ‘higher level’ mathematics courses in high school, with discrete being the lower level (and far more common) mathematics course. In terms of students’ previous schooling, attendance at either an independent or Catholic school had a negative effect on student performance, relative to those 206 Australasian Journal of Educational Technology, 2012, 28(2) attending a government school, of around 4 to 4.5 marks. This is similar to previous research, such as Birch and Miller (2007). Perhaps the most notable result in this initial regression relates to the voluntary quizzes attempted by students. Relative to students who attempted more than 25 quizzes, students who only attempted between 0 and 10 quizzes had final marks that were, on average, over 10 marks less. Given that there were only 10 separate quizzes put up for students to use, this suggests that students who attempted these quizzes multiple times performed much better than those who attempted each quiz only once or not at all. Table 2: Regressions results Dependent variable: Final mark (%) 1 2 3 0.839 0.843 0.861TEE Score 0.088 *** 0.086 *** 0.087 *** 0.743 -0.006 -0.250Male students 1.107 1.107 1.111 -0.379 -0.479 -0.664Born in Australia 1.195 1.184 1.187 -0.445 0.221 0.695Part-time student 2.788 2.753 2.778 -3.223 -3.567 -3.408Live with parents 1.579 ** 1.567 ** 1.569 ** 1.561 1.522 1.580Live at university colleges 2.457 2.472 2.496 4.574 4.920 4.787TEE Economics (1 = yes) 1.146 *** 1.129 *** 1.130 *** -6.783 -6.896 -6.899TEE Discrete Maths (1 = yes) 1.268 *** 1.261 *** 1.272 *** 2.512 2.989 2.746TEE English Literature (1 = yes) 1.368 * 1.355 ** 1.375 ** -1.133 -0.139 -0.227TEE Physics (1 = yes) 1.369 1.386 1.390 -4.181 -4.237 -4.510Independent school 1.301 *** 1.283 *** 1.288 *** -4.473 -4.059 -4.148Catholic school 1.539 *** 1.532 *** 1.531 *** 3.143 3.571 4.230Repeat students 2.143 2.139 * 2.159 * 0.555 0.906 0.930Work dummy 1.556 1.538 1.541 -0.148 -0.134 -0.128Work x workhours 0.103 0.102 0.102 -11.227 -10.345 -9.737Quizzes = 0-1 2.892 *** 2.882 *** 2.943 *** -10.559 -10.043 -9.241Quizzes = 2-10 2.538 *** 2.545 *** 2.615 *** -6.664 -6.261 -5.543Quizzes = 11-15 2.569 *** 2.574 ** 2.648 ** -4.754 -4.563 -4.027Quizzes = 16-20 2.814 * 2.801 2.839 -6.168 -6.490 -5.736Quizzes = 21-25 3.167 * 3.125 ** 3.160 * 0.075 -0.006Lecture recording hits (centred) 0.022 *** 0.056 Williams, Birch and Hancock 207 2.187 1.447Lectures = 7-13 2.333 2.371 4.009 3.232Lectures = 14-18 2.155 * 2.213 4.232 3.782Lectures = 19-21 2.048 ** 2.113 * 5.277 5.633Lectures = 22-24 2.115 ** 2.303 ** 7.429 7.779Lectures = 25-26 2.095 *** 2.222 *** 0.113Lectures = 7-13 x Lectopia hits 0.087 0.089Lectures = 14-18 x Lectopia hits 0.071 0.040Lectures = 19-21 x Lectopia hits 0.069 0.167Lectures = 22-24 x Lectopia hits 0.093 * 0.161Lectures 25-26 x Lectopia hits 0.082 ** 3.586 -1.729 -3.138Constant 7.931 7.949 7.998 Obs 371 371 371 Adjusted R2 0.41 0.43 0.44 Notes: Estimation uses ordinary least squares. *, **, and *** represent statistically significant coefficients at the 10, 5, and 1% level respectively. Standard errors are in italics. Turning now to Column 2 of Table 2, we introduce the lecture attendance groups, and the lecture recording ‘hits’ variable. We have centred the lecture recording variable to have mean of zero, by subtracting the sample mean (33.48 hits) from each observation. The simple reason for this is to make the coefficients in the following analysis read more intuitively. When evaluating the interaction effects in the following section, for example, each coefficient is interpreted relative to the lecture recording hits being zero. However, a more useful interpretation would be to ask what the effect is at the mean of the lecture recordings, not zero (as all students in this sample accessed the lecture recordings at least once). Hence, by centring the mean at zero, we are really evaluating these coefficients at the mean. With respect to the lecture attendance dummies, students who attend more lectures have higher marks relative to those who only attended 0-6 lectures (the omitted dummy). This differential is most pronounced for students attending either 25 or 26 lectures (7.8 marks), but is also a statistically significant factor for all students who attend more than half of the lectures (that is 14 or more). Although students who attend between 7-13 lectures do have a positive marks differential relative to those only attending 0-6 lectures, this difference is not a statistically significant one. These differences, particularly for those attending nearly all of the lectures in person, are substantial, and equates to almost a whole grade. In terms of the use of lecture recordings, the direct effect on student performance is positive, and also statistically significant. According to this result, a one standard deviation increase in the use of these lecture recordings (roughly 25 hits) translates into an increase in final marks of 2 percentage points. 208 Australasian Journal of Educational Technology, 2012, 28(2) However, whilst the direct effects of both lecture attendance and lecture recordings on student performance are positive, and significant, this is not the original hypothesis we want to test. What is of interest here is, if lectures and lecture recordings are substitutes, whether they are equivalent substitutes. In order to answer this, we need to introduce the interaction between lectures and lecture recordings (Column 3 of Table 2). We also looked at other transformations, such as whether the lecture recordings were quadratic, or by taking the log of lectures. However, as these did not prove significant (either practically or statistically) these were not pursued. The estimating equation for the interaction terms can be written as (for the sake of brevity, the additional control variables have been excluded from equation (2) below, whilst LRi and LECn stand for, as above, the lecture recordings and lecture attendance groups respectively): Marki = a + b1(LRi) + b2(LEC7-13) + b3(LEC14-18) + b4(LEC19-21) + b5(LEC22-24) + b6(LEC25-26) + b7(LRi* LEC7-13) + b8(LRi* LEC14-18) + b9(LRi* LEC19-21) + b10(LRi* LEC22-24) + b11(LRi* LEC25-26) + … + ei . . . . . . . (2) The addition of these interaction terms throws up some interesting results. Overall, the coefficients on each of the interaction terms (the coefficients b7 to b11 in equation (2) above) are positive – in each category of lecture attendance, higher lecture recording hits equated with better student performance. But this was only statistically significant for two groups – those attending 25-26 lectures, and those attending 22-24 lectures. In other words, the students deriving the greatest benefit from the lecture recordings were the students who also went to the most lectures. Another way to look at the effect of lecture recordings, conditional on the lectures attended, is to graph the effect on final marks for each of these groups as the number of lecture recording hits increases, according to the slopes obtained from Column 3 of Table 2 (see Figure 2). To illustrate: taking the 7-13 lecture attendance group, the calculation for the slope for this group is b1 + b7 ( -0.0056 + 0.1126 = 0.107), while the intercept at zero lecture recording hits (i.e. the centred mean of lecture recordings) is given by b2 (= 1.447). Therefore, this says that, if a student attending 0-6 lectures and a student attending 7-13 both had the mean number of lecture recording hits, then the mark advantage for the student attending 7-13 lectures was 1.45 marks. Every additional lecture recording hit for the 7-13 lecture group results in an additional mark of 0.107. The slope for the base case (0-6 lectures) is simply the coefficient on the lecture recording variable (b1 = -0.0056). Figure 2 summarises the effects for four lecture groups: 0-6 (our base category), 7-13, 22-24, and 25-26 lectures, as the number of lecture recording hits increases. The most striking aspect of this graph is the lack of impact that lecture recordings have on the group that is probably in most need of them – those who only attend 0-6 lectures. The slope here is essentially zero (-0.0056), which means that for those students going to few lectures, no amount of lecture recordings will allow them to ‘catch up’ to those students attending most or all lectures. As a test of sensitivity, we experimented with different lecture categories. However, they all gave qualitatively the same story, with the slope for the base category (the lowest category of lecture attendance) being at or around zero. For example, with the base category being 0-12 lectures, rather than 0-6, the coefficient on the lecture recording variable was marginally positive (0.011), but nowhere near statistically significant. Viewing the lecture recordings 100 times would still have resulted in a marks increase of less than 1 Williams, Birch and Hancock 209 percentage point. The sizeable differentials between the lowest and higher categories remained. Evaluated at the mean of lecture recordings, the benefit of attending 25-26 lectures is 7.78 marks over those attending only 0-6. However, a one standard deviation increase in lecture recording hits for both groups actually increases this differential to around 11.7 marks, and keeps getting wider as lecture recording hits increases beyond this. -5 0 5 10 15 20 25 -33 -13 7 27 47 67 87 Lecture recording hits (zero = mean of recording) Effect of additional lecture recording hits on final mark (percentage points) by lecture grouping Effect of lecture recording if attending 0-6 lectures Effect of lecture recording if attending 7-13 lectures Effect of lecture recording if attending 22-24 lectures Effect of lecture recording if attending 25-26 lectures Figure 2: Visual demonstration of the effect on final marks of lecture recording use, by lecture attendance groups Compare this with those attending only marginally fewer lectures (22-24 lectures). Here, although there is a marks differential of around -2.2 marks compared to those going to 25-26 lectures, evaluated at the mean of lecture recordings (5.63 - 7.78 = -2.15), the effect of lecture recordings is essentially the same for this group as it is for the group attending 25-26 lectures (that is, the slopes for the two lines are virtually identical), and so the only difference lies in the 2.2 marks additional benefit the students appear to have received from attending a couple of extra lectures. For lower levels of lecture attendance, the effect of lecture recordings is greater than the base (0-6 lecture) case, which shows that improvements in final performance can be made with greater use of these recordings. However, in each group the slope is flatter than the slope for students attending 25-26 lectures, and so again the marks differential between these groups gets wider as lecture recordings increase. In Figure 1, it was observed that lecture recordings hits were on average larger for those who attended fewer lectures. We can make use of this data to get a more detailed picture of the effect of lectures and recordings on student performance we observe in practice for each group (see Table 3). If evaluated at the group (not overall) means, even though students attending 25-26 lectures only viewed the lecture recordings on average 24 times, versus 45 times for the 0-6 group, the marks differential is still 6.7. In other words, the additional lecture recordings that students in the 0-6 group view is 210 Australasian Journal of Educational Technology, 2012, 28(2) not enough to counter the loss of marks from not attending the lectures in the first place. This differential does, however, get smaller for other groups. For example, students going to 19-21 lectures (which equates to missing around 3 weeks of lectures) did, on average, have quite a high number of lecture recordings (41). This reduced the differential between this group and those going to 25-26 lectures to around two marks, whereas when we were just evaluating at the overall mean the differential was almost four marks. Table 3: Effect of lecture recordings by lecture attendance groups, using group lecture recording means Lecture attendance Lecture recording, group means (uncentred) Mark advantage over 0-6 lecture group if view recordings at group mean 0-6 45.27 ... 7-13 36.95 1.818 14-18 38.59 3.779 19-21 40.99 4.585 22-24 23.76 4.594 25-26 23.77 6.740 5. Concluding comments The results arising from this analysis demonstrate that, for this cohort at least, greater attendance at lectures has had a positive and statistically significant effect on ultimate performance. Moreover, this effect is fairly linear in nature – the more lectures students went to, the higher their eventual marks were. As for lecture recordings, the effects were also positive, but conditional. If a student attempted to almost completely substitute face to face lectures with the online recordings, then no matter how often they viewed the recordings, they never made up the lost marks from not attending the lectures in person. This is not to say, however, that the lecture recordings were not beneficial to students. In each group other than the base 0-6 lecture grouping, additional lecture recording hits had a positive effect on their final mark. Even then, however, the groups deriving the greatest additional benefit from the recordings were those who attended the overwhelming majority (22-26) of lectures. In very broad terms then, there appear to be two groups using lecture recordings: one, using them as a substitute for lectures, the other, as a complement. The evidence presented here strongly suggests that the lecture recordings are most useful as a complement to attending lectures, rather than as a substitute. From a pedagogical perspective, if students want to receive lecture content via digital means rather than attending in person, they should be free to do so. However, if there is a demonstrable difference in outcomes in the two approaches, then students should at least be made aware of the consequences of the choices they are making. One should also be cognisant of the fact that these results come with a number of caveats. Probably the most important of these is that the number of online lecture recordings ‘hits’ used in this analysis does not necessarily equate with effective use. Secondly, the measure of lectures attended is self-reported. Although we have tried to take this into consideration, by removing those students who entered verifiably false information elsewhere in the survey, we cannot discount the fact that many students may have over-represented the number of lectures they attended. Despite these Williams, Birch and Hancock 211 caveats, however, we believe that these results are a useful step in providing some evidence-based analysis of the relative merits of attending lectures versus viewing them online. The question posed at the beginning of this paper asked whether online lecture recordings and physical attendance at lectures could be considered perfect substitutes. Based on the results presented here, the answer appears to be that they are not. In many respects, these results highlight the fact that these lecture recordings are most beneficial when used for their original intention – as a complement to lectures. For many (though not all) institutions, online lecture recordings were developed principally to help on campus students catch up on the occasional lecture missed due to illness or some other unavoidable event, as well as a useful study tool. They were never designed to completely circumvent attendance at lectures, even if this has become the choice of some students. Unfortunately, this study cannot answer the question as to whether these results translate across disciplines, or years of study. This could be a useful area of potential future research, because at the moment it is impossible to ascertain whether these same results might be found in, for example, a mathematics unit, or a sociology course. It is important therefore to not overstate the possible policy implications of this paper for university educators (or students). It does, however, point to the fact that educators cannot just assume that the online delivery of material is a perfect substitute for students attending lectures, and that more empirical work is needed to improve our knowledge in this area. Acknowledgments The authors wish to acknowledge the UWA Business School’s Research Development Scheme for funding, Paul Lloyd for data provision, and Tijana Mirkovic for research assistance. Opinions expressed in this paper are those of the authors and should not be attributed to The University of Western Australia. References Anderson, G., Benjamin, D. & Fuss, M. A. (1994). The determinants of success in university introductory economics courses. Journal of Economic Education, 25(2), 99-120. http://www.jstor.org/pss/1183277 Bell, T., Cockburn, A., McKenzie, B. & Vargo, J. (2001). Flexible delivery damaging to learning? Lessons from the Canterbury Digital Lectures Project. In Proceedings of the World Conference on Educational Multimedia, Hypermedia and Telecommunications - ED-MEDIA (pp. 117-122). AACE. [verified 18 Feb 2012] http://www.cosc.canterbury.ac.nz/andrew.cockburn/papers/edmedia.pdf; http://www.editlib.org/p/8347 Bennett, E. & Maniar, N. (2007). Are videoed lectures an effective teaching tool? [verified 18 Feb 2012] http://stream.port.ac.uk/papers/Are%20videoed%20lectures%20an%20effective %20teaching%20tool.pdf Birch, E. R. & Miller, P. W. (2004). The determinants of students’ tertiary academic success. In Productivity Commission Conference on Quantitative Tools for Microeconomic Policy Analysis: Conference Proceedings. Productivity Commission, Melbourne, Australia, pp. 45-80. [verified 18 Feb 2012] http://msc.uwa.edu.au/?f=148836 212 Australasian Journal of Educational Technology, 2012, 28(2) Birch, E. R. & Miller, P. W. (2007). The influence of type of high school attended on university performance. Australian Economic Papers, 46(1), 1-17. http://dx.doi.org/10.1111/j.1467- 8454.2007.00302.x Birch, E. R. & Williams, A. (2010). The importance of prior knowledge of the subject studied in explaining the academic outcomes of first-year commerce students. Quantitative Analysis of Teaching and Learning in Higher Education Forum, Melbourne, Australia, February 2010. Brotherton, J. A. & Abowd, G. D. (2004). Lessons learned from eClass: Assessing automated capture and access in the classroom. ACM Transactions on Computer-Human Interaction (TOCHI), 11(2), 121-155. ACM Press. http://dx.doi.org/10.1145/1005361.1005362 Chiu, C., Lee, G. & Yang, J. (2006). A comparative study of post-class lecture video viewing. In Proceedings of the 5th IASTED international Conference on Web-Based Education (pp. 126-130). ACTA Press. Day, J. & Foley, J. (2006). Evaluating web lectures: A case study from HCI. Paper presented at the Conference on Human Factors in Computing Systems (CHI EA '06), Montreal, Canada, 22-27 April. http://dx.doi.org/10.1145/1125451.1125493 Dobson, I. & Skuja, E. (2005). Secondary schooling, tertiary entrance ranks and university performance. People and Place, 13(1), 53-62. Figlio, D., Rush, M. & Yin, L. (2010). Is it live or is it Internet? Experimental estimates of the effects of online instruction on student learning. National Bureau of Economic Research Working Paper 16089, National Bureau of Economic Research, Cambridge, USA. http://www.nber.org/papers/w16089 Gysbers, V., Johnston, J., Hancock, D. & Denyer, G. (2011). Why do students still bother coming to lectures, when everything is available online? International Journal of Innovation in Science and Mathematics Education, 19(2), 20-36. http://escholarship.usyd.edu.au/journals/index.php/CAL/article/viewFile/4887/5979 Larkin, H. (2010). “But they won’t come to lectures …”: The impact of audio recorded lectures on student experience and attendance. Australasian Journal of Educational Technology, 26(2), 238- 249. http://www.ascilite.org.au/ajet/ajet26/larkin.html McNulty, J. A., Hoyt, A., Chandrasekhar, A. J., Espiritu, B., Gruener, G., Price, R. Jr. & Naheedy, R. (2011). A three-year study of lecture multimedia utilization in the medical curriculum: Associations with performances in the basic sciences. Journal of the International Association of Medical Science Educators, 21(1). http://www.iamse.org/artman/publish/article_586.shtml Preston, G., Phillips, R., Gosper, M., McNeill, M., Woo, K. & Green, D. (2010). Web-based lecture technologies: Highlighting the changing nature of teaching and learning. Australasian Journal of Educational Technology, 26(6), 717-728. http://www.ascilite.org.au/ajet/ajet26/preston.html Skene, J., Cluett, L. & Hogan, J. (2007). Engaging Gen Y students at university: What web tools do they have, how do they use them and what do they want? Proceedings First Year in Higher Education Conference, 2007. http://www.fyhe.com.au/past_papers/papers07/final_papers/pdfs/2b.pdf Taplin, R. H., Low, L. H. & Brown, A. (2011). Students’ satisfaction and valuation of web-based lecture recording technologies. Australasian Journal of Educational Technology, 27(2), 175-191. http://www.ascilite.org.au/ajet/ajet27/taplin.html Williams, Birch and Hancock 213 Von Konsky, B. R., Ivins, J. & Gribble, S. J. (2009). Lecture attendance and web based lecture technologies: A comparison of student perceptions and usage patterns. Australasian Journal of Educational Technology, 25(4), 581-595. http://www.ascilite.org.au/ajet/ajet25/vonkonsky.html Wieling, M. B. & Hofman, W.H.A. (2010). The impact of online video lecture recordings and automated feedback on student performance. Computers & Education, 54(4), 992-998. http://dx.doi.org/10.1016/j.compedu.2009.10.002 Authors: Associate Professor Andrew Williams, Economics Program, UWA Business School, The University of Western Australia, 35 Stirling Highway, Crawley WA 6009, Australia. Email: Andrew.Williams@uwa.edu.au Dr Elisa Birch, Economics Program, UWA Business School, The University of Western Australia. Email: Elisa.Birch@uwa.edu.au Professor Philip Hancock, Associate Dean (Teaching and Learning), UWA Business School, The University of Western Australia. Email phil.hancock@uwa.edu.au Please cite as: Williams, A., Birch, E. & Hancock, P. (2012). The impact of online lecture recordings on student performance. Australasian Journal of Educational Technology, 28(2), 199-213. http://www.ascilite.org.au/ajet/ajet28/williams.html