Microsoft Word - palmer.doc Australasian Journal of Educational Technology 2009, 25(3), 366-381 Staff and student perceptions of an online learning environment: Difference and development Stuart Palmer and Dale Holt Deakin University Academic staff play a fundamental role in the use of online learning by students. Yet, compared to studies reporting student perspectives on online learning, studies investigating the perspectives of academic staff are much more limited. Perhaps the least common investigations are those that compare the perceptions of academic staff and students using the same online learning environment (OLE). Much research indicates, at least initially, academic staff most value OLE systems as a mechanism for efficient delivery of learning materials to students. Following the mainstreaming of an OLE at Deakin University in 2004, the data from a large, repeated, representative and quantitative survey were analysed to investigate comparative staff and student evaluations of an OLE, and to explore the evidence for development in the use of an OLE by academic staff. Generally, students were found to give higher importance and satisfaction ratings to elements of the OLE than staff. Students were also more likely than staff to agree that the OLE enhanced their learning. A comparison of the mean ratings recorded for staff in 2004 and 2005 showed that both importance and satisfaction ratings of elements of the OLE were almost universally higher after a year of use of the OLE. Introduction Online learning environments (OLEs) are perhaps currently the most widely used and most expensive educational technology tool (Salinas, 2008; West, Waddoups & Graham, 2007). Like many other learning technology trends before them, OLEs (in some contexts often referred to as learning management systems, LMS) have been adopted by higher education institutions almost automatically and uncritically (Reynolds, Treharne & Tripp, 2003), and often without evaluation of their effectiveness (Mahdizadeh, Biemans & Mulder, 2008). Academic staff play a fundamental role in the use of online learning by students (Bolliger & Wasilik, 2009) – in a specific learning context, students can only ‘use’ those aspects of the OLE that staff make available to them. Yet, compared to the number of studies reporting investigations of student perspectives on online learning, studies investigating the perspectives of academic staff are much more limited in number, and those that exist are often limited in sample size (Jones & Jones, 2005). Quantitative investigations of staff perspectives are even rarer still (Woods, Baker & Hopper, 2004). Perhaps the least common investigations of all are those that compare the relative perceptions of academic staff and students using the same OLE (McGill & Hobbs, 2008). Our aims here are to add to the literature on comparative staff and student evaluations of an OLE, and to explore the evidence for development in the use of an OLE over time by academic staff, through an analysis of the results obtained from a large, repeated, representative and quantitative survey of academic staff and students that included a common core of question items relating to perceptions of the elements of an OLE. Palmer and Holt 367 Background A review of existing investigations published in the literature that compared staff and student perceptions of OLEs revealed several that employed different survey instruments and/or different research methodologies for collecting staff and student responses (Daugherty & Funke, 1998; Gupta, White & Walmsley, 2004; Marek & Sibbald, 2005; Tao, 2008; Tao & Rosa Yeh, 2008; Weaver, Spratt & Sid Nair, 2008), one that was qualitative only in method (Kumar, 2007), and several that were based on comparatively small and/or demographically constrained samples (Daugherty & Funke, 1998; Gupta et al., 2004; Kumar, 2007; McGill & Hobbs, 2008; Shuster, Birkholz & Petri, 2005; Tao, 2008; Tao & Rosa Yeh, 2008; Trinidad, Aldridge & Fraser, 2005). Only two published studies were noted that incorporated comparatively large samples of staff and students (Jones & Jones, 2005; Weaver et al., 2008) – though the latter did not employ the same survey instrument for staff and students. Investigations that do not employ the same instrument and research methodology for surveying staff and students provide a limited basis for direct comparison of the experiences of both groups. Investigations that use demographically constrained respondent samples are limited in the generalisability of their results. Investigations that collect qualitative data only are limited in the range of statistical analysis that can be applied to their results. Here we present an investigation that employed a large, repeated, representative and quantitative survey of academic staff and students that included a common core of question items relating to perceptions of the elements of an OLE. There is a significant body of research that indicates, at least initially, academic staff view, and most value, OLE systems primarily as a mechanism for efficient and accessible delivery of teaching and learning materials to students (Dutton, Cheong & Park, 2004; Jones & Jones, 2005; Mahdizadeh et al., 2008; Morgan, 2003; Sharpe, Benfield & Francis, 2006; Wingard, 2004; Woods et al., 2004). While it might be tempting to dismiss this as a ‘trivial’ use of the OLE, it would appear to be an important, perhaps essential, point of initial engagement for staff with the OLE (Dutton et al., 2004). It is often the case that academic staff taking on the task of online teaching of a course are doing so as either an additional mode of delivery of their existing teaching, or as an addition to their current teaching workload. The literature suggests that even where an online teaching task is a ‘replacement’ for an existing conventional class based teaching role, there will be additional preparation and delivery work required (Bolliger & Wasilik, 2009; de Vries et al., 2005; Rumble, 2001; Spector, 2005). For these staff, if there are not some ‘efficiency gains’ to be made in their initial use of an OLE, then the increased teaching workload burden may mean that they are never able to develop their online teaching beyond a basic transmissive model. It should not be forgotten that there is evidence that students also value highly and demand the online material delivery function of OLEs (Dutton et al., 2004; Palmer & Holt, 2009). In Australia, Deakin University is a major provider of distance and online education. In addition, it teaches on campus at four campuses located in three cities in the State of Victoria. OLEs have been a feature of the educational landscape at Deakin University since the early 1990s. Starting first with a range of different systems used in different academic departments of the university, and primarily used for particular courses, units of study or functions, the university gradually moved toward centralisation through the implementation of a corporately supported learning management system (LMS). Iterating through a number of commercial LMSs, the university eventually 368 Australasian Journal of Educational Technology, 2009, 25(3) settled on the WebCT LMS in 2003, branding it internally as Deakin Studies Online (DSO). The new LMS was trialled in 2003, and fully implemented in 2004. Concurrently, the university introduced policies requiring academic departments to migrate all OLE activity to the centrally supported LMS – at this time the LMS officially become the institutional OLE. Another key initiative in the university’s strategy to expand its online and distance education profile was to require that, from 2004, all its units of study have at least a basic online presence. Additionally, from 2004, all students enrolled in Deakin undergraduate courses had to undertake at least one unit wholly online, with few exemptions given. The period, therefore, between 2003 and 2005 represented an important time in which the university strategically repositioned itself with a demonstrably strengthened commitment to online education. It represents a significant historical context of investigation. Nationally, during this period, it was reported that the majority of Australian students accessed online course resources in their university studies (Krause, Hartley, James & McInnis, 2005). Given the scope of Deakin University’s commitment (in terms of central infrastructure, policy development and roll out of online elements to all taught units) to online education, it was considered essential to evaluate the effectiveness of this investment. In 2003, a pilot survey of staff and students using DSO was conducted to establish perceptions of importance and satisfaction with various elements of the OLE. Following the full mainstreaming of DSO in 2004, the survey instrument was revised, and the survey process was expanded to include all Deakin staff and students, and repeated again in 2005. The surveys were administered using a university online survey tool. These surveys produced a large pool of data, and some aspects of the student survey results have been reported previously (Challis, 2005; Palmer & Holt, 2009). Here we present a comparative analysis of staff and student evaluations of an OLE, based on a large, repeated, representative and quantitative survey of academic staff and students. We highlight that at the time of the study presented here, the institutional LMS represented the entirety of the OLE. Methodology During 2004 and 2005, all students and teaching staff at Deakin University were invited to complete the DSO evaluation survey. The DSO evaluation survey sought responses from students and staff relating to: • demographic and background information; • their perception of importance and satisfaction with a range of OLE elements; • a number of overall OLE satisfaction measures; and • open ended written comments about the OLE. Acknowledging the related but different roles that staff and students have in their use of OLEs (McGill & Hobbs, 2008), and reflecting the fact that 2005 was no longer the initial phase of the university wide roll out of DSO, there were some minor differences in the question items contained in the evaluation survey between staff and students, and between 2004 and 2005. In the analysis presented here, only those question items common to all surveys are reported, and the question item numbering refers to that used in the 2005 staff survey. As required by Deakin University human research ethics procedures, the surveys were anonymous and voluntary. The collected data were analysed and the following information was compiled: Palmer and Holt 369 • response rate and demographic comparison information; • importance-satisfaction analysis; • staff versus student importance-satisfaction perceptions; • staff versus student perceptions of OLE contribution to enhancing learning; and • development of staff perceptions of the OLE over the time period of the surveys. Nearly 1000 open ended written comments were received from students and nearly another 100 from staff – this rich qualitative data source is worthy of its own separate analysis, and is not included here. Findings and discussion Response rates and descriptive statistics Table 1 provides a summary of the response rate and demographic information for the overall staff population and staff survey respondents in 2004 and 2005. The effective response rate was 20.2% in 2004, and 14.9% in 2005. A range of demographic information was available for the overall Deakin University teaching staff population (Department of Education Employment and Workplace Relations, 2006), as well as collected as part of the survey, including gender, age range and home faculty. This permitted a comparison between the respondent sample and the overall staff population on these demographic dimensions, as presented in Table 1. Table 1: Response rate and demographic information for staff data 2004 2005 Sample Population Sample Population No. of respondents 156 772 120 805 Female 44.9% 46.0% 55.0% 46.7%Gender Male 55.1% 54.0% 45.0% 53.3% 20 – 39 years 41.7% 23.7% 39.2% 22.7% 40 – 59 years 53.8% 63.7% 57.5% 63.2% Age range 60+ years 4.5% 12.6% 3.3% 14.1% Arts 16.7% 32.0% 19.2% 30.5% Business and Law 35.2% 15.6% 36.7% 16.0% Education 7.7% 10.2% 10.0% 10.1% Health and Behavioural Sciences 16.7% 15.3% 20.8% 17.3% Home faculty Science and Technology 23.7% 26.9% 13.3% 26.1% Table 2 provides a summary of the response rate and demographic information for the overall enrolled student population and student survey respondents in 2004 and 2005. The effective response rate was 9.2% in 2004, and 7.8% in 2005. A range of demographic information was available for the overall enrolled student population (Deakin University, 2007), as well as collected as part of the survey, including gender, mode of study, level of study, enrolled faculty, and campus attended. This permitted a comparison between the respondent sample and the overall student population on these demographic dimensions, as presented in Table 2. Although the response rates obtained were comparatively low, they were not unexpected for an online voluntary survey (Cook, Heath & Thompson, 2000), and the generally good match between the sample and population demographic characteristics and the relatively large absolute numbers of respondents, for both staff and students, 370 Australasian Journal of Educational Technology, 2009, 25(3) in both years, suggests that we can have some confidence in drawing more general inferences from the respondent data. Table 2: Response rate and demographic information for student data 2004 2005 Sample Population Sample Population No. of Respondents 2908 31641 2526 32354 Female 58.9% 56.8% 61.5% 57.3%Gender Male 41.1% 43.2% 38.5% 42.7% On campus 62.3% 60.4% 61.8% 64.7%Mode of study Off campus 37.7% 39.6% 38.2% 35.3% Undergraduate 73.9% 73.4% 75.1% 73.7%Level of study Postgraduate 26.1% 26.6% 24.9% 26.3% Arts 14.0% 19.4% 16.0% 20.0% Business and Law 43.8% 37.1% 34.4% 36.9% Education 9.0% 13.1% 12.0% 13.7% Health and Behavioural Sciences 13.5% 13.9% 17.6% 14.2% Faculty Science and Technology 19.7% 16.5% 20.1% 15.2% Burwood 54.0% 56.8% 52.5% 58.3% Toorak 8.8% 5.2% 6.8% 5.5% Waurn Ponds 24.7% 20.6% 25.8% 19.6% Waterfront 4.3% 5.6% 7.5% 6.3% Warrnambool 4.9% 5.6% 4.7% 5.3% Campus* Offshore 3.3% 6.2% 2.7% 5.0% * In 2008, Deakin divested itself of the Toorak campus, with all Toorak operations moving to the Burwood campus. Importance-satisfaction data The DSO evaluation survey asked respondents to rate the importance of, and their satisfaction with, a range of elements of the OLE at Deakin University. A rating of 1 represented low importance, while a rating of 7 represented high importance. A rating of 1 represented low satisfaction, while a rating of 7 represented high satisfaction. Table 3: Mean importance and satisfaction ratings for staff from 2004 and 2005 2004 2005Element of the OLE Imp. Sat. Imp. Sat. 15. Accessing Unit Guides/unit information 6.00 4.71† 5.75 5.29 16. Accessing lecture notes/tutorial notes/lab notes 6.03 4.59† 5.94 5.25 17. Contacting teaching staff via internal unit messaging 3.89 3.04† 4.33 3.81 18. Contacting students via internal unit messaging 5.18 3.47† 5.59 4.19 19. Using calendar 2.57 3.30 2.70 3.45 20. Interacting with learning resources 5.16 3.82† 5.59 4.60 21. Contributing to discussions 5.40 3.91 5.90 4.53 22. Reading contributions to discussions 5.52 4.09 5.92 4.65 23. Using chat and/or whiteboard 3.42 2.87 3.76 3.27 24. Working collaboratively in a group 4.78 3.19 5.28 3.76 25. Completing quizzes/self tests 4.24 3.87 4.59 4.11 26. Submitting assignments 5.50 3.04 5.65 3.32 27. Receiving feedback on assignments 5.01 3.08 5.54 3.43 29. Reviewing unit progress 5.06 3.57 5.32 4.13 † 0.01 < p ≤ 0.001 Palmer and Holt 371 For both importance and satisfaction a ‘not applicable’ option was also provided to permit respondents not using a particular element to avoid having to provide a contrived rating. Table 3 provides a summary of the mean responses for the importance and satisfaction ratings for staff from 2004 and 2005. Based on a t-test of differences in mean ratings from staff between 2004 and 2005, accounting for inequality of variance, Table 3 also indicates where the corresponding ratings were significantly different between 2004 and 2005, and provides an indication of the level of significance. Table 4 provides a summary of the mean responses for the importance and satisfaction ratings for students from 2004 and 2005. Based on a t-test of differences in mean rating between staff and students, accounting for inequality of variance, Table 4 also indicates where the corresponding ratings were significantly different between staff and students, and provides an indication of the level of significance. For consistency, all responses are reported using the item numbering from the 2005 staff survey. Table 4: Mean importance and satisfaction ratings for students from 2004 and 2005 2004 2005Element of the OLE Imp. Sat. Imp. Sat. 15. Accessing Unit Guides/unit information 6.01 4.79 6.32‡ 5.19 16. Accessing lecture notes/tutorial notes/lab notes 6.44† 4.63 6.51! 5.01 17. Contacting teaching staff via internal unit messaging 5.64! 4.19! 5.63! 4.63! 18. Contacting students via internal unit messaging 4.75 4.11! 4.73! 4.60 19. Using calendar 3.29‡ 3.71 3.08 3.94 20. Interacting with learning resources 5.66† 4.37† 5.62 4.68 21. Contributing to discussions 5.07 4.34 5.08! 4.82 22. Reading contributions to discussions 5.49 4.62‡ 5.62 5.05 23. Using chat and/or whiteboard 3.81 3.39† 3.59 3.70 24. Working collaboratively in a group 4.77 3.74† 4.67† 4.00 25. Completing quizzes/self tests 5.04† 4.10 5.36† 4.68 26. Submitting assignments 6.22† 4.13! 6.30† 4.58! 27. Receiving feedback on assignments 6.28! 3.54 6.36‡ 3.86 29. Reviewing unit progress 5.90! 3.78 5.96† 4.17 † 0.01 < p ≤ 0.001 ‡ 0.001 < p ≤ 0.0001 ! p ≤ 0.0001 Staff versus student importance-satisfaction perceptions A method for visualising the difference between staff and student importance- satisfaction mean ratings was developed. Using a two-dimensional grid, importance and satisfaction rating pairs for a survey item can be plotted as a point, with the importance rating as the vertical coordinate and the satisfaction rating as the horizontal coordinate. By using the corresponding staff and student importance- satisfaction rating pairs for a survey item as the end points for a line, a two dimensional vector can be plotted for each survey item that visually represents the difference in mean importance-satisfaction rating between staff and students for OLE elements. Figure 1 presents the difference in mean importance-satisfaction ratings between staff and students for 2004. The student ratings for each survey item are represented by the circular end of the line and the staff ratings for the corresponding survey item are represented by the arrow end of the line. The numbering of survey items is based on the question numbers from the 2005 staff survey, and corresponds to the item numbers given in Tables 3 and 4. Figure 2 presents the same form of visualisation for the difference in mean importance-satisfaction ratings between staff and students for 2005. 372 Australasian Journal of Educational Technology, 2009, 25(3) Figure 1: Mean importance-satisfaction staff versus student 2004 Figure 1 and Figure 2 showed a marked similarity in the relative location and orientation of the survey item vectors for both years under consideration. Noting that the labelling of vectors with an arrow head representing staff importance-satisfaction data was an arbitrary decision, it was observed that many of the vectors in both years point down and to the left, indicating lower ratings for a survey item by staff than students for both importance and satisfaction. Where the vectors do not point downward, except for two items in 2005, they exclusively point to the left, indicating that staff almost universally recorded a lower mean satisfaction with OLE elements than students. McGill & Hobbs (2008), noting the dual (end user and designer) and more complex (compared to students) role played by academic staff when operating in an OLE, and also prior research indicating a negative link between task complexity and user satisfaction, hypothesised that staff would, and found that staff did, report a lower satisfaction than students when using an OLE. The reported satisfaction rating results presented here support their findings. McGill & Hobbs (2008) also noted the anecdotal evidence that students, being generally younger than staff, appear to have a greater comfort with technology, and hence they also hypothesised that students would, and also found that students did, report a more positive attitude towards OLE use. Generally, the reported importance rating results presented here also support their findings. Palmer and Holt 373 Figure 2: Mean importance-satisfaction staff versus student 2005 The indicators in Table 4 highlight where the reported mean ratings of importance and/or satisfaction are significantly different between staff and students. There were two OLE elements where the importance and satisfaction ratings for both 2004 and 2005 were significantly lower for academic staff compared to students; they were ‘Contacting teaching staff via internal unit messaging’ and ‘Submitting assignments’. The two elements represent key interfaces/interactions between staff and students operating in an OLE. The personal knowledge of the authors revealed that, during the time period in question, while it was university policy (as well as widespread general practice) that all study units had at least a basic presence in the OLE, not many academic staff used the messaging system internal to the OLE. This finding was not surprising given that the OLE messaging system was effectively a separate email system that could only be accessed when logged on to the OLE, and which did not integrate with any existing external email system. Likewise, during the time period in question, even though the OLE provided a facility for the online submission and return of student assignments, a majority of academic staff (often for valid reasons) maintained conventional methods for assignment management. The mean staff satisfaction ratings for this element suggest that, for those staff that did use the online assignment management features of the OLE, the 374 Australasian Journal of Educational Technology, 2009, 25(3) assignment management features were not easy/convenient to use. This second item actually forms part of a larger group of OLE elements where staff mean ratings of importance were significantly lower than for students. This group includes the OLE elements ‘Completing quizzes/self tests’, ‘Submitting assignments’, ‘Receiving feedback on assignments’ and ‘Reviewing unit progress’. Given the critical importance of timely formative/progressive feedback to students for delivering information about progress and clarifying expected and actual performance, so as to influence students to take a proactive role in their learning and for their development as self regulated learners (Nicol & Macfarlane-Dick, 2006; Yorke, 2003), these importance results should be of concern, and act as a flag for action that could have a positive impact on the contribution of the OLE to student learning and staff satisfaction. Staff versus student perceptions of contribution to enhancing learning The DSO evaluation survey asked staff respondents to rate their level of agreement with the statement ‘DSO enhances learning by my students’. A rating of 1 represented strong disagreement, while a rating of 5 represented strong agreement. The DSO evaluation survey asked student respondents to rate their level of agreement with the statement ‘The use of DSO enhanced my learning experience’. A rating of 1 represented strong disagreement, while a rating of 5 represented strong agreement. Table 5 provides a summary of mean levels of agreement recorded by staff and students in 2004 and 2005. Based on a t-test of differences in mean ratings between staff and students, accounting for inequality of variance, Table 5 also indicates the level of significance of the difference in mean ratings of staff and students in 2004 and 2005. Table 5: Staff and student perceptions of contribution to enhancing learning - 2004 and 2005 2004 2005 Staff – ‘DSO enhances learning by my students’ 2.79 3.16 Students – ‘The use of DSO enhanced my learning experience’ 3.23 3.67 Significance test t = -4.39p < 0.00002 t = -5.05 p < 5×10-7 In both 2004 and 2005, students recorded a significantly higher rating of agreement than staff that DSO enhanced their learning. This outcome again supports the findings of McGill & Hobbs (2008) that students generally hold a more positive attitude to OLE use than academic staff. Additionally, in a previous large comparative investigation of staff and student attitudes concerning the effectiveness of an OLE, a similar significant difference between staff and student agreement that the OLE has improved student learning was recorded – with student agreement being higher than that of academic staff (Jones & Jones, 2005). In that study, by way of explanation for the higher student rating of the OLE, it was posited that, anecdotally, many academic staff found their students “incautiously optimistic about their performance and their grade” (Jones & Jones, 2005, 132) – the implication being that at least some student ratings of OLE effectiveness were artificially inflated along with the student’s unrealistically optimistic assessment of their own general academic performance. The fact that both McGill and Hobbs (2008) and Jones and Jones (2005) include anecdotal observations in their explanations of the comparative difference in staff and student attitudes to the use of OLEs suggests that further research is required in this area. Palmer and Holt 375 An investigation of the factors contributing to staff and student perceptions of contribution of the OLE to enhancing learning was undertaken. For students, for both 2004 and 2005, a multivariate linear regression of all the DSO evaluation survey items was performed against the item ‘The use of DSO enhanced my learning experience’. The full description of this work is presented elsewhere (Palmer & Holt, 2009), but, in summary, in both 2004 and 2005, students felt that using DSO enhanced their learning experience when they were: • adequately supported by unit teachers and technical support services; • when they are able to find and use unit information in DSO; and, • when they are able to read the online discussion contributions of other unit members. Forthcoming work will document a similar analysis performed on the staff DSO evaluation survey data. In summary, in both 2004 and 2005, staff felt that DSO enhanced their students’ learning when: • they were satisfied that their students were able to access and use their learning materials; and • they were satisfied with the DSO professional development they have received / they were confident with their ability to teach with DSO. While the regression models developed were statistically significant, they did not explain all of the variation observed in the dependent variables, hence there exist other factors with a significant influence on student and staff satisfaction that were not included in the DSO evaluation survey. While a number of factors were observed to contribute to both student and staff satisfaction with DSO, one possible explanation for the observed difference in staff versus student perceptions of contribution to enhancing learning is that students were (or at least perceived that they were) better supported than staff in the adoption and use of the OLE. In the case study documented here, while a significant program of institutional staff professional development was undertaken during the roll out phase of the OLE implementation, this was primarily aimed at developing basic competency in the use of the OLE, and was delivered by a comparatively small (relative to the total number of academic staff) group of trainers. This professional development initiative relied significantly on staff self-learning, and on academic staff who had developed some knowledge of the system then acting as ‘local experts’ who could be consulted by their peers. As we will note below, the Deakin OLE has since expanded significantly in terms of component tools and functionality. Appropriate and adequate staff support and development to ensure staff satisfaction and the effective use of educational technologies remains a key issue. Development in staff perceptions of the OLE As noted previously, the literature suggests, at least initially, academic staff value OLE systems primarily as a mechanism for efficient and accessible delivery of teaching and learning materials to students, and some evidence for this view can be found in the data collected from academic staff here. Inspecting the results presented in Table 3 shows that the survey items with the highest initial (in 2004) importance ratings were item 15 ‘Accessing Unit Guides/unit information’ and item 16 ‘Accessing lecture notes/ tutorial notes/ lab notes’. In addition, the regression analysis reported previously found that academic staff felt that DSO enhanced their students’ learning 376 Australasian Journal of Educational Technology, 2009, 25(3) when they were satisfied that their students were able to access and use their learning materials. In the field of human resource management, Frederick Herzberg proposed a two- factor theory of human motivation (Herzberg, 1964). Herzberg believed that people have lower and higher level needs. He called the lower level needs ‘hygiene factors’, and the higher level needs ‘motivating factors’. Herzberg contended that the presence of hygiene factors does not motivate staff, but that their absence can quickly lead to dissatisfaction. In an analogy to Herzberg’s two-factor theory, while online transmission of learning materials may not be a great value adder for online teaching and learning, failure of an OLE to provide this ‘hygiene’ functionality is likely to lead to significant staff and student dissatisfaction. While many academic staff report mainly pragmatic factors that influence their initial engagement with online teaching and learning (Morgan, 2003; Wingard, 2004), and that enhanced OLE functionality is the least important factor in adoption of online teaching and learning (Baek, Jung, & Kim, 2008), there is also evidence that their perceptions and use of online technology in teaching and learning develop in pedagogical sophistication over time (Morgan, 2003; Wingard, 2004; Woods et al., 2004). For many academic staff, the starting point of a primarily transmissive conception of online teaching and learning may be a practical and/or developmental necessity for the eventual development of richer pedagogical conceptions of online teaching and learning. Using the same importance-satisfaction data visualisation method described above, Figure 3 presents the difference in mean staff importance-satisfaction ratings between 2004 and 2005. The 2004 ratings for each survey item are represented by the circular end of the line and the 2005 ratings for the corresponding survey item are represented by the arrow end of the line. The numbering of survey items is based on the question numbers from the 2005 staff survey, and corresponds to the item numbers given in Table 3. While noting that the differences in ratings indicated in Table 3 suggest that statistically significant differences between 2004 and 2005 were few, and that those that did exist were limited to a higher satisfaction rating for some OLE elements, a visual inspection of Figure 3 shows a notable coherence in the direction of the survey item vectors. The standard deviation of the angles of all the vectors was 20.75 (out of 360) degrees and the range of the angles of all the vectors was 79.88 degrees. Apart from vectors for two survey items that present a slight downward inclination, all vectors had a positive inclination in the first quadrant, averaging approximately 30 degrees, and the change in mean satisfaction ratings were without exception positive. There is some evidence here for a positive change in the staff perception of the OLE from the time of the initial institutional roll out of the system to the point of the second survey one year later. Likewise, while the statistical significance of the change in the staff perception item between 2004 and 2005 given in Table 5 is marginal (t = -2.50, p < 0.013), it is again positive. Overall, in 2005, academic staff perceived the functions of the OLE as more important, and were more satisfied with the OLE. Given that 2004 was earlier in the university-wide, compulsory roll out of the OLE to all units of study, it is not unreasonable to expect that, by 2005, following an extra year of experience with the OLE, that academic staff would be better placed to use and support the OLE, and hence, be more satisfied with it. Whether this also represents a development in the pedagogical approaches underpinning the use of the OLE by staff cannot be determined conclusively at this point. Palmer and Holt 377 Figure 3: Mean staff importance-satisfaction ratings 2004 versus 2005 Again, while noting that statistically significant differences between 2004 and 2005 were few, we observe that only two vectors exhibit a downward (to the right) slope, and that these are also the two most equivocal changes in importance rating (lowest absolute deviation from the horizontal). Interestingly, these items (15 and 16) relate to ‘Accessing Unit Guides/unit information’ and ‘Accessing lecture notes/tutorial notes/lab notes’, and, employing the Herzberg two-factor theory analogy again, they could be classified as ‘hygiene’ OLE functions. While acknowledging that it is somewhat speculative, it could be argued that these two ‘taken for granted’ OLE functions offer the least scope for improvement in importance rating. Whereas, some of the OLE functions identified in other survey items represent comparatively more sophisticated aspects of online teaching and learning, and that these might be expected to be rated as more important over time, as academic staff develop their online teaching and learning pedagogical approaches beyond the basics, and begin to engage with more complex conceptions of teaching and learning online. From the view of the present day, the large, quantitative and comparative analysis of staff and student perceptions of elements of an institutional OLE presented here provides an important but largely historical perspective on staff use of, and interaction with, the OLE. At Deakin University, since the time that the DSO evaluation survey reported here was conducted, DSO has expanded beyond being an internal tag for the 378 Australasian Journal of Educational Technology, 2009, 25(3) WebCT LMS. DSO is now the Deakin University ‘brand’ for a portfolio of e-learning technologies. All of these new e-learning technologies have been brought on stream in response to requests from academic teaching staff to expand and develop their repertoire as they adopt more sophisticated pedagogical approaches to online learning. The status of the LMS has evolved from being the entirety of the OLE to effectively having an underpinning infrastructure/gateway role, with its presence and features now being presumed and taken for granted, and providing a linking platform for the support of other, value adding e-learning technologies, including recorded lectures and ‘Web 2.0’ social software tools. The university’s new teaching and learning plan countenances the addition of extra e-learning technologies under the DSO banner. These developments suggest new and important questions; questions that can only be begun to be answered by updating the information that the university has, at an institutional level, about its staff usage of its OLE. These questions include: • In the intervening period since the surveys reported here, has the use of the OLE by academic staff changed and developed? • In what ways are academic staff engaging with this dramatically expanded palette of educational technologies at their disposal? • What combination of e-learning technologies, chosen from the available portfolio, creates the greatest potential educational value in a given teaching and learning context? The findings presented here suggest some guidance to policymakers responsible for online teaching and learning. Academic staff play a central role in the design, delivery and success (or otherwise) of OLEs. The initial period in which staff engage with OLEs is crucial, and their first experiences may influence the pace at which they develop more sophisticated pedagogical conceptions of online teaching and learning beyond automated learning resource delivery. The perceptions of academic staff about OLEs generally do develop more positively over time, so it is important to support and nurture staff beyond the initial start up of such systems. While research here and elsewhere suggests that academic staff generally indicate lower satisfaction with OLEs than students, one key group of OLE functions found to have significantly lower importance ratings by staff relates to the use of OLEs in the provision of feedback to students on their academic performance and progress in their studies. This is an area of fundamental importance for student learning, and suggests an important area for action and improvement. Educational technologies have continued to evolve over time and play an expanding role in teaching and learning. There is a need for on going and up to date, evidence based evaluation of the performance of these technologies, and the contribution they make to the endeavours of both staff and students in teaching and learning. Conclusions The data from a large, repeated, representative and quantitative survey of academic staff and students were analysed to investigate comparative staff and student evaluations of an OLE, and to explore the evidence for development in the use of an OLE by academic staff. In both 2004 and 2005, the mean ratings for importance and satisfaction with elements of the OLE were generally lower for staff than for students, providing support for previous comparative research that has found that students tend to rate aspects of online learning more highly than staff. Both academic staff and students were asked to indicate their level of agreement as to whether the OLE Palmer and Holt 379 enhanced student learning. Here the mean ratings were also significantly higher for students compared to staff, again providing support for previous research findings. A multivariate linear regression analysis of the factors contributing to staff and student ratings here identified perceived support for use of the OLE as a significant factor for both staff and students, and suggested that students may have perceived that they had better support than staff. There was evidence in the data collected from staff in the 2004 survey, at the time of the initial roll out of the OLE, that staff most highly valued the OLE as a mechanism for efficient and accessible delivery of teaching and learning materials to students. A comparison of the mean ratings recorded for staff in 2004 and 2005 showed that both importance and satisfaction ratings of elements of the OLE were almost universally higher after a year of widespread use of the OLE. Given the intervening period since the institutional surveying presented here, there is a pressing need to update this information to understand that ways in which academic staff are using the OLE, and whether this use has developed in pedagogical sophistication. References Baek, Y., Jung, J. & Kim, B. (2008). What makes teachers use technology in the classroom? Exploring the factors affecting facilitation of technology with a Korean sample. Computers & Education, 50(1), 224-234. [verified 3 Jul 2009] http://ciillibrary.org:8000/ciil/Fulltext/computer _and_education/vol_50_1_2008/Article_15.pdf Bolliger, D. U. & Wasilik, O. (2009). Factors influencing faculty satisfaction with online teaching and learning in higher education. Distance Education, 30(1), 103-116. Challis, D. (2005). Eroding distinctiveness: Blurring the boundaries between on- and off-campus students by the adoption of learning management systems. In Proceedings ODLAA Adelaide 2005. http://www.odlaa.org/events/2005conf/ref/ODLAA2005Challis.pdf Cook, C., Heath, F. & Thompson, R. L. (2000). A meta-analysis of response rates in web- or Internet-based surveys. Educational and Psychological Measurement, 60(6), 821-836. Daugherty, M. & Funke, B. L. (1998). University faculty and student perceptions of web-based instruction. Journal of Distance Education, 13(1), 21-39. [verified 3 Jul 2009] http://www.jofde.ca/index.php/jde/article/viewArticle/134/411 de Vries, F. J., Kester, L., Sloep, P., van Rosmalen, P., Pannekeet, K. & Koper, R. (2005). Identification of critical time-consuming student support activities in e-learning. ALT-J, 13(3), 219-229. [verified 3 Jul 2009] http://repository.alt.ac.uk/98/1/ALT_J_Vol13_No3_2005_ Identification%20of%20critical%20tim.pdf Deakin University (2007). 2007 Pocket Statistics. [viewed 3 Mar 2007] http://www.deakin.edu.au/planning-unit/pocket-stats/pocket-stats-2007.xls Department of Education Employment and Workplace Relations (2006). Higher Education Statistics Collection. [viewed 14 Apr 2008] http://www.dest.gov.au/sectors/higher_ education/publications_resources/statistics/higher_education_statistics_collection.htm Dutton, W. H., Cheong, P. H. & Park, A. (2004). An ecology of constraints on e-learning in higher education: The case of a virtual learning environment. Prometheus, 22(2), 131-149. Gupta, B., White, D. A. & Walmsley, A. D. (2004). The attitudes of undergraduate students and staff to the use of electronic learning. British Dental Journal, 196(8), 487-492. 380 Australasian Journal of Educational Technology, 2009, 25(3) Herzberg, F. (1964). The motivation-hygiene concept and problems of manpower. Personnel Administration, 27(1), 3-7. Jones, G. H. & Jones, B. H. (2005). A comparison of teacher and student attitudes concerning use and effectiveness of web-based course management software. Educational Technology & Society, 8(2), 125-135. http://www.ifets.info/journals/8_2/12.pdf Krause, K.-L., Hartley, R., James, R. & McInnis, C. (2005). The first year experience in Australian universities: Findings from a decade of national studies. http://www.griffith.edu.au/__data/assets/pdf_file/0006/37491/FYEReport05.pdf Kumar, S. (2007). Student and professor perceptions of course web site use in web-enhanced instruction. In C. Montgomerie & J. Seale (Eds.), Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2007 (pp. 4321-4326). Chesapeake, VA: AACE. Mahdizadeh, H., Biemans, H. & Mulder, M. (2008). Determining factors of the use of e-learning environments by university teachers. Computers & Education, 51(1), 142-154. Marek, S. & Sibbald, A. M. (2005). WebCT-assisted learning at Napier University: Student and staff perceptions. In Proceedings 4th European Conference on e-Learning, Amsterdam, 10-11 November. [verified 3 Jul 2009] http://books.google.com/books?hl=en&lr=&id=p20KP8Y7HCsC &oi=fnd&pg=PA237&dq=++%22WebCT-assisted+learning+at+Napier+University:+Student+ and+staff+perceptions%22&ots=F--T0rYQd5&sig=tHCm18J0McI427EFRjwX1s7nd1U McGill, T. J. & Hobbs, V. J. (2008). How students and instructors using a virtual learning environment perceive the fit between technology and task. Journal of Computer Assisted Learning, 24(3), 191-202. Morgan, G. (2003). Faculty use of course management systems. Boulder: EDUCAUSE. http://www.educause.net/ir/library/pdf/ecar_so/ers/ERS0302/ekf0302.pdf Nicol, D. J. & Macfarlane-Dick, D. (2006). Formative assessment and self-regulated learning: A model and seven principles of good feedback practice. Studies in Higher Education, 31(2), 199- 218. [verified 3 Jul 2009] http://tltt.strath.ac.uk/REAP/public/Resources/DN_SHE_Final.pdf Palmer, S. & Holt, D. (2009). Students’ perceptions of the value of the elements of an online learning environment: Looking back in moving forward. Interactive Learning Environments. http://www.informaworld.com/smpp/content~db=all?content=10.1080/09539960802364592 Reynolds, D., Treharne, D. & Tripp, H. (2003). ICT - the hopes and the reality. British Journal of Educational Technology, 34(2), 151-167. Rumble, G. (2001). The costs and costing of networked learning. Journal of Asynchronous Learning Networks, 5(2), 75-96. [verified 3 Jul 2009] http://www.sloan- c.org/publications/jaln/v5n2/pdf/v5n2_rumble.pdf Salinas, M. F. (2008). From Dewey to Gates: A model to integrate psychoeducational principles in the selection and use of instructional technology. Computers & Education, 50(3), 652-660. Sharpe, R., Benfield, G. & Francis, R. (2006). Implementing a university e-learning strategy: Levers for change within academic schools. ALT-J, 14(2), 135-151. http://repository.alt.ac.uk/112/ Shuster, G. F., Birkholz, G. & Petri, L. (2005). Faculty and student evaluations of a web based nursing program. Paper presented at the 21st Annual Conference on Distance Teaching and Learning, 3-5 August, Madison, WI. [verified 4 Jul 2009; abstract] http://apha.confex.com/apha/133am/techprogram/paper_102409.htm Palmer and Holt 381 Spector, M. J. (2005). Time demands in online instruction. Distance Education, 26(1), 5-27. Tao, Y.-H. (2008). Typology of college student perception on institutional e-learning issues: An extension study of a teacher's typology in Taiwan. Computers & Education, 50(4), 1495-1508. Tao, Y.-H. & Rosa Yeh, C.-C. (2008). Typology of teacher perception toward distance education issues: A study of college information department teachers in Taiwan. Computers & Education, 50(1), 23-36. Trinidad, S., Aldridge, J. & Fraser, B. (2005). Development, validation and use of the Online Learning Environment Survey. Australasian Journal of Educational Technology, 21(1), 60-81. http://www.ascilite.org.au/ajet/ajet21/trinidad.html Weaver, D., Spratt, C. & Sid Nair, C. (2008). Academic and student use of a learning management system: Implications for quality. Australasian Journal of Educational Technology, 24(1), 30-41. http://www.ascilite.org.au/ajet/ajet24/weaver.html West, R., Waddoups, G. & Graham, C. (2007). Understanding the experiences of instructors as they adopt a course management system. Educational Technology Research and Development, 55(1), 1-26. Wingard, R. G. (2004). Classroom teaching changes in web-enhanced courses: A multi- institutional study. EDUCAUSE Quarterly, 27(1), 26-35. http://www.educause.edu/EDUC AUSE+Quarterly/EDUCAUSEQuarterlyMagazineVolum/ClassroomTeachingChangesinWebE/157279 Woods, R., Baker, J. D. & Hopper, D. (2004). Hybrid structures: Faculty use and perception of web-based courseware as a supplement to face-to-face instruction. The Internet and Higher Education, 7(4), 281-297. Yorke, M. (2003). Formative assessment in higher education: Moves towards theory and the enhancement of pedagogic practice. Higher Education, 45(4), 477-501. Dr Stuart Palmer is a Senior Lecturer in the Institute of Teaching and Learning, Deakin University, Geelong, Victoria, 3217, Australia. Email: spalm@deakin.edu.au Web: http://www.deakin.edu.au/itl/contact/profiles/s-palmer.php Dr Dale Holt is Associate Director of the Institute of Teaching and Learning, Deakin University, Geelong, Victoria, 3217, Australia. Email: dholt@deakin.edu.au Web: http://www.deakin.edu.au/itl/contact/profiles/d-holt.php