Microsoft Word - 581-6487-1-CE_MM.docx Australasian Journal of Educational Technology, 2014, 30(4). ascilite 375 Development of student and academic staff perceptions of the elements of an online learning environment over time Stuart Palmer, Dale Holt Deakin University, Australia Academic staff play a fundamental role in the use of online learning environments (OLEs) by students, yet studies investigating the perspectives of staff are limited, and the least common investigations are those comparing the relative perceptions of students and staff. Many OLE evaluations present a point-in-time snapshot of the system, but extended evaluation of OLE usage is required to reveal the detail in the evolution of system use. The data from a large, repeated cross-sectional and quantitative survey of university students and staff were analysed. Drawing on 4680 survey responses over the period 2004–2011, it was found that: • both student and staff mean ratings of importance and satisfaction suggest an increasingly positive engagement with the OLE over time; • virtually all mean ratings of satisfaction, and many of the mean ratings of importance, for students were higher than for staff, supporting the view in the literature that students have a more positive attitude than staff toward OLE use; and • the general trend was a reduction in the magnitude of the differences between students and staff between 2004 and 2011. More fine-grained analyses of the differences observed for individual elements of the OLE are also reported. Introduction Academic staff play a fundamental role in the use of online learning by students (Bolliger & Wasilik, 2009) – in a specific learning context students can only use those aspects of the OLE that staff make available to them. Yet, compared to the number of studies reporting investigations of student perspectives on online learning, studies investigating the perspectives of academic staff are much more limited in number, and those that exist are often limited in sample size (Jones & Jones, 2005). Quantitative investigations of staff perspectives are even rarer still (Woods, Baker & Hopper, 2004). Perhaps the least common investigations of all are those that compare the relative perceptions of students and academic staff using the same OLE (McGill & Hobbs, 2008). A review of existing investigations published in the literature that compared student and staff perceptions of OLEs was undertaken. Only two published studies were noted that incorporated comparatively large samples of students and staff (Jones & Jones, 2005; Weaver, Spratt, & Nair, 2008), though the latter did not employ the same survey instrument for students and staff. A commonly reported finding in the student versus staff comparative literature that does exist is that, while both students and staff often give generally positive ratings to different aspects of OLEs, significant differences between student and staff ratings are often observed. The importance of systematic organisational policies as a contributing factor supporting the adoption of online technologies in teaching and learning has been noted (Baek, Jung, & Kim, 2008; West, Waddoups, & Graham, 2007). However, it has also been found that academic staff attitudes to educational technology are a critical factor in the diffusion and use of such technologies (Albirini, 2006; Mahdizadeh, Biemans, & Mulder, 2008). Additionally, it has been found that the views and acceptance of educational technology vary widely across academic staff (Dutton, Cheong, & Park, 2004; Salinas, 2008). There is a significant body of research that indicates that academic staff initially value OLE systems primarily as a mechanism for efficient and accessible delivery of teaching and learning materials to students (Dutton et al., 2004; Jones & Jones, 2005; Mahdizadeh et al., 2008; Morgan, 2003; Sharpe, Benfield, & Francis, 2006; Wingard, 2004; Woods et al., 2004). While it might be tempting to dismiss this as a trivial use of the OLE, it would appear to be an important, perhaps essential, point of initial engagement for staff with the OLE (Dutton et al., 2004). It should not be forgotten that there is evidence that students also value highly and demand the online material delivery function of OLEs (Dutton et al., 2004; Palmer & Holt, 2010). Australasian Journal of Educational Technology, 2014, 30(4). 376 Published OLE evaluations often present a point-in-time snapshot of the system and do not capture the development of system use over time (Lonn & Teasley, 2009). Extended evaluation of OLE usage is required to reveal the detail in the evolution of system use (Browne, Jenkins, & Walker, 2006). There is a call in the literature for the application of extended/repeated evaluation to better understand the impact of OLEs and to optimize their use in online teaching and learning (Bates & Khasawneh, 2007; Davis & Wong, 2007; Drennan, Kennedy, & Pisarski, 2005; Mikropoulos & Natsis, 2011). Existing published extended evaluations of OLEs are often those compiled by industry bodies and are comprised of data drawn from across the sector (Browne et al., 2006; Smith & Caruso, 2010). Such evaluations are valuable for sector benchmarking comparisons, but potentially miss/subsume the nuances of the specific characteristics and context of individual institutions. Evaluation surveys that include consistent question items over a longer period of time permit more robust extended analysis of data trends (Browne et al., 2006; Smith & Caruso, 2010). In Australia, Deakin University is a provider of on-campus, distance and online education. Iterating through a number of commercial learning management systems (LMSs) the university eventually settled on a single system in 2003, branding it internally as Deakin Studies Online (DSO). The new LMS was trialled in 2003 and fully implemented in 2004. Concurrently, the university introduced policies requiring academic departments to migrate all online learning environment (OLE) activity to the centrally supported LMS – at that time the LMS officially became the institutional OLE. Another key initiative in the university’s strategy to expand its online and distance education profile was the requirement that all of its units of study have at least a basic online presence from 2004. Additionally, all students enrolled in Deakin University undergraduate courses had to undertake at least one unit wholly online from 2004. Deakin University selected a new LMS in 2010, and all online support for teaching and learning was provided via the OLE based on the new LMS by the first teaching period in 2012. Thus, the period 2004- 2011 represents the life of the original central LMS at Deakin University. Given the scope of Deakin University’s commitment (in terms of central infrastructure, policy development and roll-out of online components to all taught units) to online education, it was considered essential to evaluate the effectiveness of this investment. A pilot survey of students and staff using DSO was conducted in 2003 to establish perceptions of importance and satisfaction with various elements of the OLE. Following the full mainstreaming of DSO in 2004, the survey instrument was revised and the survey process was expanded to include all Deakin University students and staff, and repeated again in 2005. These surveys were administered using a university online survey tool and produced a large pool of data that provided insights into the initial engagement with the institutional OLE by students and staff (Palmer & Holt, 2009, 2010). However, this previous research covered only the first two years of the full-scale system roll out. A new DSO evaluation survey was developed and administered to all students and staff in 2011. Crucially, the set of question items relating to use and perception of the core elements of the OLE was largely common with the previous surveys run in 2004-05. We acknowledge that during the period included in this study the university LMS has gone through a series of upgrades that have altered the operation of some of the functions included in the surveys. However, these changes were largely points of finesse rather than fundamental changes to the operation of the LMS functions and the same core set of LMS functionality was available throughout the entire survey period. Here we present an investigation that employed a large, repeated, fine-grained, representative and quantitative survey of students and academic staff that included a common core of question items relating to perceptions of the elements of an OLE across the period 2004-2011. We respond to the call in the literature for more OLE evaluation research using long time period robust data, more research on the development of staff perceptions and use of OLEs over time, and specifically, more research on the differences between student and staff perceptions of elements of OLEs. Our aim is to answer the following research questions: 1. How have student perceptions of elements of OLEs developed over time? 2. How have academic staff perceptions of elements of OLEs developed over time? 3. How do student and staff perceptions of elements of OLEs differ? 4. How has any difference in student and staff perceptions of elements of an OLE developed over time? Australasian Journal of Educational Technology, 2014, 30(4). 377 Better understanding these questions will allow more informed policy and decision making regarding future developments in this area that has become central to teaching and learning and many universities. Methodology Details of the student and staff 2004 DSO evaluation surveys, their methodology, respondent samples and results have been presented previously (Palmer & Holt, 2009, 2010). Across the period 2004-2011, all versions of the DSO evaluation survey sought responses from students and staff relating to: • demographic and background information; • perception of importance and satisfaction with a range of OLE elements; • a number of overall OLE satisfaction measures; and • open-ended written comments about the OLE. We use the demographic information to test whether the sample respondent group was representative of the overall population of interest. All three surveys contained 13 common OLE elements for which respondents were asked to indicate both their rating of importance and their level of satisfaction using ordinal response scales. A response scale of 1 – 7 was used in 2004, and a response scale of 1 – 5 was used in 2011. The 13 common OLE elements were: 1. Accessing unit guide and other unit information; 2. Accessing unit lecture, tutorial or lab notes etc.; 3. Interacting with unit learning resources; 4. Using the unit calendar; 5. Contacting teachers via internal unit messaging; 6. Contacting students via internal unit messaging; 7. Reading contributions to online discussions; 8. Contributing to online discussions; 9. Completing online quizzes/tests; 10. Submitting assignments; 11. Receiving feedback on assignments; 12. Working collaboratively in a group; and 13. Reviewing unit progress Given the changes in the student and staff populations over the time period covered in this investigation, it was not possible to follow a cohort of specific respondents over time. Instead, we sought a representative sample of the student and staff population in each survey. We compare the ratings of importance and satisfaction for each of the 13 OLE elements and identify any statistically significant differences between: • students in 2004 and 2011; • staff in 2004 and 2011; • students and staff in 2004; • students and staff in 2011; and • the difference between students and staff in 2004, and, the difference between students and staff in 2011 – the difference of the differences. We also present a method for visualizing this importance-satisfaction data across all the surveys. While they form a valuable data set in their own right, the overall satisfaction and open-ended written comment data are not included here due to practical space limitations. A definitive indication of the significance of the differences between the mean ratings for an item between 2004 and 2011 for a particular respondent group (students or staff), or between students and staff for a particular year, is obtained from an analysis of variance (ANOVA) test. A requirement for the ANOVA test is that the variation of the mean rating be similar in all groups under test. Where Levene’s test of homogeneity of variance failed, a robust ANOVA test using the Welch test statistic was performed instead. Testing the significance of the difference of the differences of the means is a more complex Australasian Journal of Educational Technology, 2014, 30(4). 378 between-subject test design (of the form given in Equation 2), and requires a modification to the standard t test (Myers, Well, & Lorch, 2010). The modified t statistic (t') is computed from Equation 1, where the contrast of the means is given by Equation 2, the combined standard error is estimated by Equation 3, the combined degrees of freedom are estimated by Equation 4, and where sj, s 2 j and nj are the standard deviation, variance and sample size respectively for each of the four respondent groups included in Equation 2. 𝑡′ = Ψ 𝑠! (Equation 1) Ψ = 𝑠𝑡𝑢𝑑𝑒𝑛𝑡  𝑚𝑒𝑎𝑛!"## − 𝑠𝑡𝑎𝑓𝑓  𝑚𝑒𝑎𝑛!"## − 𝑠𝑡𝑢𝑑𝑒𝑛𝑡  𝑚𝑒𝑎𝑛!""# − 𝑠𝑡𝑎𝑓𝑓!""# (Equation 2) 𝑠! = !!! !! ! (Equation 3) 𝑑𝑓′ = !! ! !! ! !! ! !!!! ! (Equation 4) We acknowledge that the ratings provided by respondents are fundamentally ordinal in nature. The use of ordinal data in many parametric statistical procedures, while commonplace in the social sciences, is not universally accepted as valid. However, there is a significant body of research that has demonstrated the practical utility of analysis of ordinal data, based on the robustness of many parametric methods to significant departures from assumptions about the underlying data, including departures from normality and ‘intervalness’ that might be present in ordinal scale data (Jaccard & Wan, 1996; Norman, 2010). In all statistical analysis, a two-sided significance level of p < 0.01 was used. A discussion of the observed results is also presented. As required by Deakin University human research ethics procedures, all of the surveys were anonymous and voluntary. Results Sample and demographic information Table 1 presents a summary of the response rates obtained in the 2004 DSO evaluation surveys for students and staff. For the 2011 DSO evaluation surveys, a slightly different range of demographic information was available for the overall Deakin University student and staff populations, as well as collected as part of the surveys, including gender, mode of study, enrolled faculty, enrolled campus and duration of current enrolment for students, and gender and age range for staff. Table 2 provides a comparison between the respondent sample and overall population of students in 2011. Table 3 provides the same comparison for staff. Table 1 Response summary for 2004 student and staff DSO evaluation surveys Respondents Enrolled population Respondent sample Response rate Students 31641 2908 9.19% Staff 772 156 20.21% Australasian Journal of Educational Technology, 2014, 30(4). 379 Table 2 Response rate and demographic information for 2011 student DSO evaluation survey Sample Population No. of respondents 1353 22760 Response rate 5.95% — Gender Female 67.3% 59.5% Male 32.7% 40.5% Mode of study On-campus 68.7% 71.4% Off-campus 31.3% 28.6% Faculty Arts and Education 29.8% 30.4% Business and Law 28.8% 36.9% Health 22.8% 19.8% Science & Technology 17.0% 12.9% Other 1.6% — Campus Burwood 43.7% 50.3% Waurn Ponds 14.3% 12.1% Waterfront 7.4% 5.8% Warrnambool 3.3% 3.2% Off-campus 31.3% 28.6% Mean enrolment duration 2.42 years 2.24 years Table 3 Response rate and demographic information for 2011 staff DSO evaluation survey Sample Population No. of respondents 263 2126 Response rate 12.37% — Gender Female 55.9 % 59.8 % Male 44.1 % 40.2 % Age range <25 3.0% 2.8% 25-29 4.9% 9.1% 30-34 5.7% 14.5% 35-39 13.3% 13.3% 40-44 16.0% 14.8% 45-49 15.6% 14.4% 50-54 17.9% 14.3% 55-59 10.6% 10.0% 60-64 9.1% 6.0% >64 3.4% 0.9% Not indicated 0.4% — Australasian Journal of Educational Technology, 2014, 30(4). 380 Importance-satisfaction analysis In both 2004 and 2011, the DSO evaluation survey asked respondents to rate the importance of, and their satisfaction with, a range of elements of the OLE at Deakin University. In 2004, a rating scheme of 1 = low importance/satisfaction and 7 = high importance/satisfaction was used. In 2011 a rating scheme of 1 – 5 was employed. Research indicates that re-scaling of scale item data is possible (Dawes, 2008), and in the following analysis we have re-scaled the 2004 importance and satisfaction data to the range 1 – 5. For both importance and satisfaction a not applicable option was also provided to permit respondents not using a particular element to avoid having to provide a contrived rating. Table 4 provides a summary of the mean student responses for the importance and satisfaction ratings for both 2004 and 2011, with the percentages of ‘not applicable’ responses (which are the same for importance and satisfaction rating pairs) shown in parenthesis. Table 4 also shows the difference in mean student ratings between the two years (2011-2004) and the associated statistical significance result based on the appropriate ANOVA test. Table 5 shows the same data for the staff surveys. A method for visualizing the difference between student and staff importance-satisfaction mean ratings was developed. Using a two-dimensional grid, importance and satisfaction rating pairs for a survey item can be plotted as a point, with the importance rating as the vertical coordinate and the satisfaction rating as the horizontal coordinate. By using the corresponding student and staff importance-satisfaction rating pairs for a survey item as end points, a line segment can be plotted for each OLE element that visually represents the difference in mean importance-satisfaction rating between students and staff for OLE elements. Figure 1 presents the difference in mean importance-satisfaction ratings between students and staff for 2004. The student ratings for each survey item are represented by the circular end of the line and the staff ratings for the corresponding survey item are represented by the arrow end of the line. The numbering of the lines is the same as that given in Table 4, and is located adjacent to the student/circle end of the corresponding line. Figure 2 presents the same form of visualization for the difference in mean importance-satisfaction ratings between students and staff for 2011. Figures 1 and 2 visualize the differences in the student and staff mean importance and satisfaction ratings in 2004 and 2011 respectively by plotting the actual values for the mean ratings. This format allows the mean ratings for the various OLE elements for 2004 and 2011 to be compared to each other in absolute terms relatively easily. However, it is not as straightforward to appreciate how the comparative differences in student and staff mean importance and satisfaction ratings (student rating minus staff rating) vary over time between 2004 and 2011. This change in relative ratings of elements of the OLE between students and staff is an important part of understanding the differences in perceptions of students and staff. Table 6 provides the values of the differences in the student and staff mean importance and satisfaction ratings, along with the associated statistical significance result based on the appropriate ANOVA test, for both 2004 and 2011. Table 6 also shows the statistical significance test result of the ‘difference of the differences’ of the student and staff mean ratings between 2004 and 2011, based on the modified t test described in Equations 1 - 4. By using the corresponding 2004 and 2011 importance-satisfaction difference (student-staff) rating pairs for a survey item as the end points for a line, a two dimensional vector can be plotted for each OLE element that visually represents the comparative difference in student and staff mean importance and satisfaction ratings over time for OLE elements, as presented in Figure 3. The 2004 difference in ratings for each survey item are represented by the circular end of the line, and the 2011 difference in ratings for the corresponding survey item are represented by the arrow end of the line. The numbering of the vectors is the same as that given in Table 6, and, in this case, is located adjacent to the arrowhead/2011 end of the corresponding vector. Because Figure 3 shows comparative differences, most plotted points are near to the origin, and where the difference between student and staff mean ratings for an OLE element in a given year was negative (staff rating higher than the student rating), part or all of the element vector can be outside of the first (top right) quadrant of Figure 3. Australasian Journal of Educational Technology, 2014, 30(4). 381 Table 4 Mean Importance and Satisfaction ratings for students in 2004 and 2011 OLE element (Importance and Satisfaction) Mean rating (1-5) ( n/a % in brackets) Test statistic and significance value 2004 2011 Diff. 1.Accessing unit guide, etc. (I) 4.30(1.9) 4.72(0.2) 0.43* F(1,3881)=272.75 p < 0.001 1.Accessing unit guide, etc. (S) 3.42 4.13 0.71* F(1,3474)=475.54 p < 0.001 2.Accessing unit lecture notes etc. (I) 4.60(3.4) 4.81(2.0) 0.21* F(1,3746)=108.62 p < 0.001 2.Accessing unit lecture notes etc. (S) 3.31 3.90 0.59* F(1,3183)=272.90 p < 0.001 3.Interacting with learning resources (I) 4.05(7.7) 4.43(1.1) 0.39* F(1,3256)=172.08 p < 0.001 3.Interacting with learning resources (S) 3.12 3.73 0.61* F(1,3116)=305.41 p < 0.001 4.Using the unit calendar (I) 2.35(20.6) 2.91(15.2) 0.56* F(1,3532)=137.25 p < 0.001 4.Using the unit calendar (S) 2.65 3.18 0.53* F(1,2498)=164.93 p < 0.001 5.Contacting teachers via messaging (I) 4.03(7.4) 4.08(6.1) 0.05 F(1,2572)=1.82 p = 0.17 5.Contacting teachers via messaging (S) 2.99 3.59 0.60* F(1,2795)=220.14 p < 0.001 6.Contacting students via messaging (I) 3.39(11.7) 3.61(9.5) 0.22* F(1,3825)=27.41 p < 0.001 6.Contacting students via messaging (S) 2.94 3.56 0.62* F(1,2929)=253.02 p < 0.001 7.Reading online discussions (I) 3.92(9.0) 4.28(1.9) 0.36* F(1,3222)=124.25 p < 0.001 7.Reading online discussions (S) 3.30 3.82 0.52* F(1,3335)=206.75 p < 0.001 8.Contributing to online discussions (I) 3.62(10.3) 3.97(2.0) 0.35* F(1,3052)=94.74 p < 0.001 8.Contributing to online discussions (S) 3.10 3.75 0.65* F(1,3302)=336.69 p < 0.001 9.Completing online quizzes/tests (I) 3.60(32.1) 4.32(13.3) 0.72* F(1,3001)=315.33 p < 0.001 9.Completing online quizzes/tests (S) 2.93 3.76 0.83* F(1,2885)=390.69 p < 0.001 10.Submitting assignments (I) 4.45(21.3) 4.70(4.9) 0.26* F(1,3430)=79.19 p < 0.001 10.Submitting assignments (S) 2.95 3.70 0.75* F(1,3105)=276.15 p < 0.001 11.Receiving assignment feedback (I) 4.48(17.7) 4.63(7.0) 0.14* F(1,3058)=28.10 p < 0.001 11.Receiving assignment feedback (S) 2.53 3.29 0.76* F(1,2694)=279.42 p < 0.001 12.Working in a group (I) 3.41(29.4) 3.76(17.2) 0.35* F(1,2527)=60.61 p < 0.001 12.Working in a group (S) 2.67 3.09 0.42* F(1,2388)=89.27 p < 0.001 13.Reviewing unit progress (I) 4.21(15.1) 4.07(9.8) -0.14* F(1,3707)=16.77 p < 0.001 13.Reviewing unit progress (S) 2.70 3.24 0.54* F(1,2686)=171.96 p < 0.001 * = significant difference at the two-sided p < 0.01 level Australasian Journal of Educational Technology, 2014, 30(4). 382 Table 5 Mean Importance and Satisfaction ratings for staff in 2004 and 2011 OLE element (Importance and Satisfaction) Mean rating (1-5) ( n/a % in brackets) Test statistic and significance value 2004 2011 Diff. 1.Accessing unit guide, etc. (I) 4.29(2.6) 4.60(2.1) 0.31* F(1,256)=10.28 p < 0.01 1.Accessing unit guide, etc. (S) 3.36 4.01 0.65* F(1,264)=30.20 p < 0.001 2.Accessing unit lecture notes etc. (I) 4.31(5.8) 4.76(6.1) 0.45* F(1,200)=23.12 p < 0.001 2.Accessing unit lecture notes etc. (S) 3.28 4.02 0.74* F(1,242)=36.27 p < 0.001 3.Interacting with learning resources (I) 3.69(23.7) 4.48(4.8) 0.79* F(1,158)=32.70 p < 0.001 3.Interacting with learning resources (S) 2.73 3.63 0.90* F(1,194)=43.10 p < 0.001 4.Using the unit calendar (I) 1.84(34.6) 2.48(31.7) 0.64* F(1,297)=16.26 p < 0.001 4.Using the unit calendar (S) 2.36 2.98 0.62* F(1,165)=17.60 p < 0.001 5.Contacting teachers via messaging (I) 2.78(28.2) 3.10(37.5) 0.32 F(1,213)=2.35 p = 0.12 5.Contacting teachers via messaging (S) 2.17 3.12 0.95* F(1,211)=32.89 p < 0.001 6.Contacting students via messaging (I) 3.70(15.4) 3.83(25.7) 0.13 F(1,331)=0.61 p = 0.43 6.Contacting students via messaging (S) 2.48 3.17 0.69* F(1,320)=20.62 p < 0.001 7.Reading online discussions (I) 3.94(15.4) 4.41(5.1) 0.47* F(1,211)=14.97 p < 0.001 7.Reading online discussions (S) 2.92 3.64 0.72* F(1,380)=28.97 p < 0.001 8.Contributing to online discussions (I) 3.86(12.8) 4.41(6.3) 0.55* F(1,211)=20.62 p < 0.001 8.Contributing to online discussions (S) 2.79 3.65 0.86* F(1,238)=36.37 p < 0.001 9.Completing online quizzes/tests (I) 3.03(57.1) 3.70(31.9) 0.67* F(1,99)=9.14 p < 0.01 9.Completing online quizzes/tests (S) 2.76 3.19 0.43 F(1,98)=5.01 p = 0.02 10.Submitting assignments (I) 3.93(42.3) 4.29(22.6) 0.36 F(1,138)=4.18 p = 0.04 10.Submitting assignments (S) 2.17 3.39 1.22* F(1,288)=58.05 p < 0.001 11.Receiving assignment feedback (I) 3.58(46.8) 4.14(27.8) 0.56* F(1,129)=8.24 p < 0.01 11.Receiving assignment feedback (S) 2.20 3.27 1.07* F(1,264)=39.69 p < 0.001 12.Working in a group (I) 3.41(46.8) 3.80(35.0) 0.39 F(1,257)=4.74 p = 0.03 12.Working in a group (S) 2.28 3.14 0.86* F(1,247)=29.72 p < 0.001 13.Reviewing unit progress (I) 3.61(39.1) 3.73(27.6) 0.12 F(1,289)=0.57 p = 0.45 13.Reviewing unit progress (S) 2.55 3.22 0.67* F(1,278)=20.66 p < 0.001 * = significant difference at the two-sided p < 0.01 level Australasian Journal of Educational Technology, 2014, 30(4). 383 Figure 1. Difference in student and staff mean Importance and Satisfaction ratings in 2004. Figure 2. Difference in student and staff mean Importance and Satisfaction ratings in 2011. 1 2 3 4 5 6 7 89 10 11 12 13 1.75 2.00 2.25 2.50 2.75 3.00 3.25 3.50 3.75 4.00 4.25 4.50 4.75 5.00 2.00 2.25 2.50 2.75 3.00 3.25 3.50 3.75 4.00 4.25 Importance Satisfaction • Students ! Staff 1 2 3 4 5 6 7 8 9 1011 12 13 1.75 2.00 2.25 2.50 2.75 3.00 3.25 3.50 3.75 4.00 4.25 4.50 4.75 5.00 2.00 2.25 2.50 2.75 3.00 3.25 3.50 3.75 4.00 4.25 Importance Satisfaction • Students ! Staff Australasian Journal of Educational Technology, 2014, 30(4). 384 Table 6 Differences in mean Importance and Satisfaction Ratings for students and staff in 2004 and 2011 OLE element (Importance and Satisfaction) 2004 2011 Diff. ’11-’04 Diff. Test and significance Diff. Test and significance Test and significance 1.Accessing unit guide, etc. (I) 0.00 F(1,3004)=0.01 p = 0.90 0.12 F(1,319)=4.30 p = 0.03 t’(291)= -0.98 p = 0.32 1.Accessing unit guide, etc. (S) 0.06 F(1,3004)=0.34 p = 0.55 0.12 F(1,1607)=3.65 p = 0.05 t’(305)= -0.47 p = 0.63 2.Accessing unit lecture notes etc. (I) 0.29* F(1,155)=10.80 p < 0.01 0.05 F(1,1574)=1.90 p = 0.16 t’(220)= 2.47 p = 0.01 2.Accessing unit lecture notes etc. (S) 0.03 F(1,2995)=0.12 p = 0.73 -0.12 F(1,1571)=3.42 p = 0.06 t’(283)= 1.25 p = 0.21 3.Interacting with learning resources (I) 0.35* F(1,123)=7.68 p < 0.01 -0.05 F(1,1592)=0.81 p = 0.36 t’(172)= 2.89 p < 0.01* 3.Interacting with learning resources (S) 0.39* F(1,126)=10.43 p < 0.01 0.10 F(1,1587)=1.91 p = 0.16 t’(219)= 2.12 p = 0.03 4.Using the unit calendar (I) 0.51* F(1,2410)=15.08 p < 0.001 0.43* F(1,1419)=16.47 p < 0.001 t’(265)= 0.51 p = 0.60 4.Using the unit calendar (S) 0.29 F(1,2410)=5.08 p = 0.02 0.20 F(1,226)=5.96 p = 0.01 t’(192)= 0.61 p = 0.54 5.Contacting teachers via messaging (I) 1.25* F(1,115)=53.69 p < 0.001 0.98* F(1,199)=65.00 p < 0.001 t’(228)= 1.27 p = 0.20 5.Contacting teachers via messaging (S) 0.82* F(1,119)=35.60 p < 0.001 0.47* F(1,1405)=23.39 p < 0.001 t’(236)= 2.10 p = 0.03 6.Contacting students via messaging (I) -0.31 F(1,140)=5.10 p = 0.02 -0.22 F(1,1457)=5.26 p = 0.02 t’(285)= -0.57 p = 0.56 6.Contacting students via messaging (S) 0.46* F(1,2699)=15.68 p < 0.001 0.39* F(1,224)=14.13 p < 0.001 t’(318)= 0.48 p = 0.62 7.Reading online discussions (I) -0.02 F(1,2776)=0.03 p = 0.87 -0.13 F(1,1580)=4.56 p = 0.03 t’(241)= 0.91 p = 0.36 7.Reading online discussions (S) 0.38* F(1,2776)=11.18 p < 0.001 0.18 F(1,317)=5.15 p = 0.02 t’(264)= 1.38 p = 0.16 8.Contributing to online discussions (I) -0.24 F(1,2743)=5.15 p = 0.02 -0.44* F(1,1578)=41.04 p < 0.001 t’(248)= 1.62 p = 0.10 8.Contributing to online discussions (S) 0.31 F(1,146)=6.18 p = 0.01 0.10 F(1,310)=1.73 p = 0.18 t’(269)= 1.39 p = 0.16 9.Completing online quizzes/tests (I) 0.57* F(1,69)=7.90 p < 0.01 0.62* F(1,216)=36.48 p < 0.001 t’(106)= -0.21 p = 0.83 9.Completing online quizzes/tests (S) 0.17 F(1,2040)=1.02 p = 0.31 0.57* F(1,1338)=45.23 p < 0.001 t’(108)= -2.08 p = 0.04 10.Submitting assignments (I) 0.51* F(1,92)=10.39 p < 0.01 0.41* F(1,229)=23.13 p < 0.001 t’(145)= 0.63 p = 0.53 10.Submitting assignments (S) 0.78* F(1,2377)=24.82 p < 0.001 0.31* F(1,1483)=11.65 p < 0.001 t’(174)= 2.68 p < 0.01* 11.Receiving assignment feedback (I) 0.91* F(1,84)=27.27 p < 0.001 0.49* F(1,218)=28.85 p < 0.001 t’(134)= 2.08 p = 0.03 11.Receiving assignment feedback (S) 0.33 F(1,2475)=4.44 p = 0.03 0.02 F(1,1432)=0.06 p = 0.80 t’(192)= 1.73 p = 0.08 12.Working in a group (I) 0.00 F(1,2135)=0.01 p = 0.94 -0.04 F(1,1325)=0.10 p = 0.74 t’(153)= 0.11 p = 0.91 12.Working in a group (S) 0.39* F(1,2135)=7.99 p < 0.01 -0.05 F(1,1258)=0.28 p = 0.59 t’(190)= 2.70 p < 0.01* 13.Reviewing unit progress (I) 0.60* F(1,99)=20.72 p < 0.001 0.34* F(1,243)=14.62 p < 0.001 t’(194)= 1.62 p = 0.10 13.Reviewing unit progress (S) 0.15 F(1,2561)=1.19 p = 0.27 0.02 F(1,1388)=0.10 p = 0.74 t’(189)= 0.74 p = 0.45 * = significant difference at the two-sided p < 0.01 level Australasian Journal of Educational Technology, 2014, 30(4). 385 Figure 3. Difference of the differences of the student and staff mean Importance and Satisfaction ratings between 2004 and 2011. Discussion Sample and demographic information The demographic match between the sample and population for both groups in 2004 was generally good across the dimensions of gender, mode of study, level of study, enrolled faculty and enrolled campus for students, and gender, age range and home faculty for staff. The full comparison of the 2004 demographic data has been detailed elsewhere (Palmer & Holt, 2009). The comparison between the respondent samples and the overall populations of students and staff on their respective demographic dimensions, as presented in Tables 2 and 3, also shows a generally good match. Although the response rates obtained for both students and staff in both years were comparatively low, they were not unexpected for an online voluntary survey (Cook, Heath, & Thompson, 2000). The generally good match between the sample and population demographic characteristics in both groups in both years, and the relatively large numbers of respondents, suggests that we can have some confidence in drawing more general inferences from the respondent data for the Deakin University context. Importance-satisfaction analysis Table 4 shows that the mean student ratings of importance and satisfaction were, except for one case, all significantly different between 2004 and 2011, and, except for one case, all increased across the period. The same result is apparent in the movement of the student circles generally diagonally upward and to the right between Figures 1 and 2. This result suggests an increasingly overall positive engagement with the OLE by students over time, possibly due to increasing exposure, training, experience, student IT skills and confidence levels over time. It is reasonable to assume that as the institution has progressively embedded the OLE across campuses and programs of study that the perceived importance of the system might be viewed as higher, with increasing weight being given to use of the system by university 1 2 3 4 5 6 7 8 9 10 11 12 13 -­‐0.50 -­‐0.25 0.00 0.25 0.50 0.75 1.00 1.25 1.50 -­‐0.25 0.00 0.25 0.50 0.75 1.00 Difference  in Importance   rating (student  -­‐staff) Difference  in  Satisfaction  rating  (student  -­‐staff) • 2004 !2011 Australasian Journal of Educational Technology, 2014, 30(4). 386 management and teaching staff. One marked difference from the general positive trend was that only one item returned a significantly lower mean rating across the period 2004–2011; that was the importance rating for the element Reviewing unit progress. While this result suggests that there is an opportunity to better use the OLE as part of engaging students in self-managing their studies, it is noted that the absolute mean importance rating for this item remained relatively high, and Table 6 shows that the mean rating from students was significantly higher than from staff. Table 5 shows a generally similar trend for staff as for students – all mean staff ratings of importance and satisfaction were higher across the period 2004– 2011, and the majority were significantly higher. Again, the general diagonal migration of the staff arrowheads in Figures 1 and 2 confirm this, and suggest that the overall engagement with the OLE by staff has also been increasingly positive over time. Figures 1 and 2, with few exceptions, indicate students having higher mean ratings of OLE element satisfaction than staff in both 2004 and 2011. Many of the lines in both figures also indicate staff having lower mean ratings of OLE element importance than students, though the result is more mixed in 2011. Although Table 6 shows that not all of the observed differences between student and staff mean ratings are statistically significant, there is a measure of coherence in the alignment of the lines, especially in 2004 (Figure 1). McGill & Hobbs (2008), noting the dual (end user and designer) and more complex (compared to students) role played by academic staff when operating in an OLE, and also prior research indicating a negative link between task complexity and user satisfaction, hypothesized that staff would, and found that staff did, report a lower satisfaction than students when using an OLE. The reported satisfaction rating results presented here support their findings. McGill & Hobbs (2008) also noted the anecdotal evidence that students, being generally younger than staff, appear to have a greater comfort with technology, and hence they also hypothesized that students would, and also found that students did, report a more positive attitude towards OLE use. Generally, the reported importance rating results presented here also support their findings. Table 6 indicates a number of OLE elements for which the mean student and staff ratings were significantly different in 2004 and/or 2011. There are a number of items for which the significant difference was persistent in both 2004 and 2011, and in all these cases, the difference was positive, i.e., students’ mean ratings were significantly higher than staff. OLE elements for which student mean rating of importance was higher than for staff in both 2004 and 2011 suggest areas where staff could look to change their pedagogical designs and practices to make relative improvements to student engagement with the OLE. Items in this category included using the unit calendar, contacting teachers via messaging, online quizzes/tests, submitting assignments, receiving assignment feedback and reviewing unit progress. Given the contribution of the final four OLE elements here listed in the process of providing timely feedback to students about progress and clarifying expected and actual performance, so as to influence students to take an active role in their learning, and for students’ development as self-regulated learners (Nicol & Macfarlane-Dick, 2006; Yorke, 2003), these importance results should be of concern. They act as a flag for action that could have a positive impact on the contribution of the OLE to student learning and staff satisfaction. While a number of OLE elements that recorded significantly different mean ratings between students and staff in 2004 no longer do so in 2011, there are two OLE elements for which the reverse was the case – their difference in mean rating has become significant in 2011. The first is satisfaction with completing online quizzes – a significantly higher mean rating from students suggests that this assessment approach is valued by students and could be employed where appropriate. The second is perceived importance of contributing to online discussions – the only item for which students recorded a significantly lower mean rating than staff. There would be value in staff being more overt in explaining the relevance and value of student participation in online discussions as a learning activity. While the significant difference of the differences indicated in the final column of Table 6 are few (only three observed), some general observations can be made about Figure 3. With the exception of very short vector for item 1, and the obvious counter examples of items 8 and 9, the vectors for the remainder of the OLE elements are directed generally towards the origin. The origin in Figure 3 represents no difference in mean rating of importance and satisfaction between students and staff. So while the 2011 column of Table 6 shows that there still are a number of significantly different mean ratings, there has been a general trend between 2004 and 2011 of a reduction in the differences in perception of students and staff of the value of almost all of the elements of the OLE. Australasian Journal of Educational Technology, 2014, 30(4). 387 General It can be seen from Figures 1-3 that the OLE elements with both the highest absolute, and the smallest difference in, mean ratings for importance and satisfaction for both students and staff across 2004 and 2011 are items 1 and 2 – accessing unit guides and learning materials. While many academic staff report mainly pragmatic factors that influence their initial engagement with online teaching and learning (Morgan, 2003; Wingard, 2004), and that enhanced OLE functionality is the least important factor in adoption of online teaching and learning (Baek et al., 2008), there is also evidence in the literature that staff perceptions of online technology in teaching and learning develop in pedagogical sophistication over time (Morgan, 2003; Wingard, 2004; Woods et al., 2004). Here, Figures 1-3 suggest that staff perceptions of the value (as measured by mean ratings of importance and satisfaction) of elements of the OLE have generally risen over time, and have come more closely into alignment with the perceptions of students. Acknowledging that the statistically significant movements represented by the vectors in Figure 3 are few, it can be seen that the two ‘pedagogical limited' items (1 and 2) related to content download have moved only a relatively small amount adjacent to the origin between 2004 and 2011. However, a range of other OLE elements that offer potentially more sophisticated teaching and learning affordances have made larger movements toward the origin, indicating a relative increase in staff perceptions of the value of the fuller range of OLE features and/or a closing of the gap between student and staff perceptions/expectations. Previous investigations of changes in perceptions of the value of OLE elements over a single year period (2004-2005) for students (Palmer & Holt, 2010) and staff (Palmer & Holt, 2009) revealed few statistically significant differences. However, over the longer time frame 2004-2011 presented here, many significant increases in mean rating were observed. This confirms observations elsewhere (Smith & Caruso, 2010) and supports the proposition that extended/repeated evaluation of OLE usage is required to reveal the statistically significant details in the evolution of system use (Browne et al., 2006; Conrad, 2005). It also suggests that reliable proof of the ‘payoff’ from the investment in an institutional LMS/OLE, as measured by increased perceived value of the OLE by students and staff, is only likely to be observed in the longer- term. Conclusions The research project documented here provides a bookend measure of the development of student and staff perceptions of the value of the existing OLE across the seven year lifespan of the underpinning LMS between 2004 and 2011 at Deakin University. The data from a large, repeated cross-sectional and quantitative survey of university students and staff were analysed to investigate the development of student and staff perceptions of the elements of an OLE over time. Drawing on 4680 survey responses over the period 2004–2011, the following research questions were investigated. 1. How have student perceptions of elements of OLEs developed over time? Student mean ratings of importance and satisfaction were almost exclusively significantly higher in 2011 compared to 2004, suggesting an increasingly positive engagement with the OLE by students over time. 2. How have academic staff perceptions of elements of OLEs developed over time? Staff mean ratings of importance and satisfaction were predominantly significantly higher in 2011 compared to 2004, suggesting an increasingly positive engagement with the OLE by staff over time. 3. How do student and staff perceptions of elements of OLEs differ? In both 2004 and 2011 virtually all mean ratings of satisfaction, and many of the mean ratings of importance for students were higher than for staff, though not all of the observed differences were significant. This finding lends support to the view both theorized and observed in the literature that students have a more positive attitude than staff toward OLE use. 4. How has any difference in student and staff perceptions of elements of an OLE developed over time? While some significant differences between student and staff mean ratings of importance and satisfaction remain in 2011, the general trend was a reduction in the magnitude of the differences, with student and staff perceptions coming more into agreement between 2004 and 2011. Australasian Journal of Educational Technology, 2014, 30(4). 388 We have responded to the call in the literature for more OLE evaluation research using long time period robust data, more research on the development of staff perceptions and use of OLEs over time, and specifically more research on the differences between student and staff perceptions of elements of OLEs. We have also provided a methodology that others can employ to investigate the development of the perception of the value of elements of their OLE by students and staff over time. In answering our research questions, we note some implications for practice. Generally, student and staff perceptions of the OLE have become significantly more positive over time – system administrators may have to bide their time to see positive perception ratings for a new OLE. OLE functions that might be classified as transmissive continue to be perceived most positively by students and staff – while not viewed as strong value-adders for learning, OLEs need to do these basic functions well. Between 2004 and 2011, the gaps in perceived ratings between students and staff for some more pedagogically sophisticated LMS functions were seen to narrow significantly – OLE use appears to start simple and become more complex over time; OLE support systems for students and staff should be planned accordingly. A particular observation in this investigation was a persistent difference in mean rating of a number of OLE items related to assessment and feedback – with students rating these as more important than staff in both 2004 and 2011. Given the critical role of assessment and feedback in the learning process, this finding suggests an important area where staff could look to change their pedagogical designs and practices to make relative improvements to student engagement with the OLE. Our findings support the observation in the literature that evaluation over an extended time period is necessary to observe statistically significant changes in user perceptions of an OLE system. References Albirini, A. (2006). Teachers' attitudes toward information and communication technologies: The case of Syrian EFL teachers. Computers & Education, 47(4), 373–398. Baek, Y., Jung, J., & Kim, B. (2008). What makes teachers use technology in the classroom? Exploring the factors affecting facilitation of technology with a Korean sample. Computers & Education, 50(1), 224–234. Bates, R., & Khasawneh, S. (2007). Self-efficacy and college students' perceptions and use of online learning systems. Computers in Human Behavior, 23(1), 175-191. Bolliger, D. U., & Wasilik, O. (2009). Factors influencing faculty satisfaction with online teaching and learning in higher education. Distance Education, 30(1), 103-116. Browne, T., Jenkins, M., & Walker, R. (2006). A longitudinal perspective regarding the use of VLEs by higher education institutions in the United Kingdom. Interactive Learning Environments, 14(2), 177- 192. Conrad, D. (2005). Building and maintaining community in cohort-based online learning. Journal of Distance Education, 20(1), 1-20. Cook, C., Heath, F., & Thompson, R. L. (2000). A meta-analysis of response rates in web- or internet- based surveys. Educational and Psychological Measurement, 60(6), 821-836. Davis, R., & Wong, D. (2007). Conceptualizing and measuring the optimal experience of the elearning environment. Decision Sciences Journal of Innovative Education, 5(1), 97-126. Dawes, J. (2008). Do data characteristics change according to the number of scale points used? An experiment using 5-point, 7-point and 10-point scales. International Journal of Market Research, 50(1), 61-77. Drennan, J., Kennedy, J., & Pisarski, A. (2005). Factors affecting student attitudes toward flexible online learning in management education. The Journal of Educational Research, 98(6), 331-338. Dutton, W. H., Cheong, P. H., & Park, A. (2004). An ecology of constraints on e-learning in higher education: The case of a virtual learning environment. Prometheus, 22(2), 131–149. Jaccard, J., & Wan, C. K. (1996). LISREL approaches to interaction effects in multiple regression. Thousand Oaks, CA: Sage. Jones, G. H., & Jones, B. H. (2005). A comparison of teacher and student attitudes concerning use and effectiveness of web-based course management software. Educational Technology & Society, 8(2), 125–135. Lonn, S., & Teasley, S. D. (2009). Saving time or innovating practice: Investigating perceptions and uses of Learning Management Systems. Computers & Education, 53(3), 686-694. Mahdizadeh, H., Biemans, H., & Mulder, M. (2008). Determining factors of the use of e-learning environments by university teachers. Computers & Education, 51(1), 142–154. Australasian Journal of Educational Technology, 2014, 30(4). 389 McGill, T. J., & Hobbs, V. J. (2008). How students and instructors using a virtual learning environment perceive the fit between technology and task. Journal of Computer Assisted Learning, 24(3), 191-202. Mikropoulos, T. A., & Natsis, A. (2011). Educational virtual environments: A ten-year review of empirical research (1999-2009). Computers & Education, 56(3), 769-780. Morgan, G. (2003). Faculty use of course management systems - ERS0302. Boulder: EDUCAUSE. Myers, J. L., Well, A. D., & Lorch, R. F. (2010). Research design and statistical analysis (3rd ed.). New York: Routledge. Nicol, D. J., & Macfarlane-Dick, D. (2006). Formative assessment and self-regulated learning: A model and seven principles of good feedback practice. Studies in Higher Education, 31(2), 199-218. Norman, G. (2010). Likert scales, levels of measurement and the “laws” of statistics. Advances in Health Sciences Education, 15(5), 625-632. Palmer, S., & Holt, D. (2009). Staff and student perceptions of an online learning environment: Difference and development. Australasian Journal of Educational Technology, 25(3), 366-381. Palmer, S., & Holt, D. (2010). Students’ perceptions of the value of the elements of an online learning environment: Looking back in moving forward. Interactive Learning Environments, 18(2), 135-151. Salinas, M. F. (2008). From Dewey to Gates: A model to integrate psychoeducational principles in the selection and use of instructional technology. Computers & Education, 50(3), 652–660. Sharpe, R., Benfield, G., & Francis, R. (2006). Implementing a university e-learning strategy: Levers for change within academic schools. ALT-J, 14(2), 135-151. Smith, S. D., & Caruso, J. B. (2010). The ECAR study of undergraduate students and information technology, 2010 - ECAR Research Study 6. Boulder, Co.: EDUCAUSE. Weaver, D., Spratt, C., & Nair, C. S. (2008). Academic and student use of a learning management system: Implications for quality. Australasian Journal of Educational Technology, 24(1), 30-41. West, R., Waddoups, G., & Graham, C. (2007). Understanding the experiences of instructors as they adopt a course management system. Educational Technology Research and Development, 55(1), 1– 26. Wingard, R. G. (2004). Classroom teaching changes in web-enhanced courses: A multi-institutional study. EDUCAUSE Quarterly, 27(1), 26–35. Woods, R., Baker, J. D., & Hopper, D. (2004). Hybrid structures: Faculty use and perception of web- based courseware as a supplement to face-to-face instruction. The Internet and Higher Education, 7(4), 281–297. Yorke, M. (2003). Formative assessment in higher education: Moves towards theory and the enhancement of pedagogic practice. Higher Education, 45(4), 477-501. Corresponding author: Stuart Palmer, stuart.palmer@deakin.edu.au Australasian Journal of Educational Technology © 2014. Please cite as: Palmer, S., & Holt, D. (2014). Development of student and academic staff perceptions of the elements of an online learning environment over time. Australasian Journal of Educational Technology, 30(4), 375-389.