Microsoft Word - benson.doc Australasian Journal of Educational Technology 2006, 22(4), 548-567 Introducing a new learning management system: An institutional case study Robyn Benson and Tom Palaskas Monash University The introduction of an online learning management system (LMS) raises a number of complex issues involving institutional responses at various levels to the adoption and diffusion of technological change. Issues include those related to governance, management and technical support, as well as to core learning and teaching matters associated with the professional development and teaching of academic staff, and the support of staff and students. This paper draws on two cycles of an evaluation conducted in one institution as WebCT Vista was introduced and piloted, highlighting the key issues that emerged from the evaluation. These issues are considered in the context of a selected model for examining the adoption and diffusion of information and communication technologies (ICTs) in higher education, with a view to analysing the outcomes of the initiative, and guiding future planning. Introduction The experience of introducing an enterprise learning management system (LMS) provides an opportunity for reviewing technology adoption and diffusion in relation to learning and teaching at an institutional level. It is possible that a case study of an institutional response to technological change will highlight some issues and outcomes that may be relevant to other institutions. This paper examines some issues raised in an evaluation of the pilot of a new LMS at Monash University. Results are considered in the context of a selected model for examining the adoption and diffusion of information and communication technologies (ICTs) in higher education, in order to determine some possible future directions for guiding institutional initiatives of this kind. The institutional context Monash University is a large, global, multi-campus institution with six campuses in Victoria (Australia), one in Malaysia and one in South Africa, and centres in London (United Kingdom) and Prato (Italy). It has over Benson and Palaskas 549 52,000 students (15.9% of them studying off campus, including students studying through partnership arrangements, particularly in Singapore and Hong Kong), and over 2,800 academic staff (Monash University, 2006). Outside the faculties, at the time of the pilot, central academic support services included Information Technology Services (ITS), the Centre for Learning & Teaching Support (CeLTS) and Monash University Library. WebCT Campus Edition (CE) was introduced as a University wide central service in 2002. By late 2003 the new WebCT enterprise product (WebCT Vista) was seen as having the potential to deliver substantial benefits to the University and address deficiencies encountered with the CE version. These deficiencies included: • a labour intensive and inflexible account creation process; • non-intuitive courseware creation and maintenance processes; • the lack of archival facilities and database management; and • the lack of open standards and of ease of integration with other corporate systems. Consequently, a pilot project was initiated in 2004 to trial aspects of the WebCT Vista LMS, with a view to University support of a full production service by Semester 1, 2005. The project included development of the following elements: • an integrated training and academic staff development program; • educational design support resources; • helpdesk support for staff and students; and • a robust and scalable hardware and software configuration, together with hardware and software management procedures, documentation and middleware to allow the WebCT Vista software to communicate with existing student and course based information systems and resources. Development of appropriate middleware by ITS would facilitate functions such as automatic and seamless enrolment of students into unit websites and provide for different faculty administrative processes. These processes ranged from centralised to more decentralised faculty control over site creation and management. Devolution of administrative procedures to appropriate faculty staff, based on individual faculty requirements, was a key organisational change to be implemented in the transition to the new system, given the centralised nature of the existing CE system. The pilot project was designed to support the University’s commitment to student centredness and flexibility in its learning and teaching programs, taking direction from its Global Development Framework which specified institutional use of opportunities presented through changing technology. 550 Australasian Journal of Educational Technology, 2006, 22(4) The project was sponsored by the Deputy Vice Chancellor (Academic and Planning), with steering committee members from the Office of the sponsor, ITS, CeLTS and the faculties. A reference group consisting of representatives appointed by Deans was responsible for faculty related decision making, and a project team represented key ITS and CeLTS service responsibilities. A theoretical framework for considering the adoption and implementation process The adoption and implementation of WebCT Vista at Monash University was not conceptualised on the basis of a single theoretical framework. The process was volatile and subject to evolution in response to the priorities of a number of different stakeholders, resulting in a highly pragmatic and dynamic adoption environment. Thus, the practical contingencies of the process precluded the use of a theoretical framework during the planning of the project. However, it was felt that analysis of the results could be guided by one or more existing adoption and implementation models to assist in analysing the outcomes and guiding future practice. Most models initially examined were relevant to the adoption and implementation of an LMS at an institutional level, but did not have the breadth of coverage to constitute an appropriate framework on their own. This was because they often focused on one or more aspects of the process rather than the whole (Burkman, 1987; Ely, 1990; Hall & Hord, 1987; Havelock & Zlotolow, 1995; Palaskas, 2002; Rogers, 2003; Sherry, Billig, Tavalin & Gibson, 2000; Stockdill & Morehouse, 1992; Tessmer, 1990; Zaltman & Duncan, 1977). These models can be categorised under three broad headings: those focusing on the characteristics of the adopters and users of technology; those that address concerns about the environment; and those that consider the change process itself and conditions that support or constrain it. In the first group, the seminal work on the diffusion of innovation by Rogers (2003) deals with various attributes of diffusion including the rate of adoption, adopter categories, innovation attributes, and the diffusion process itself. Burkman’s (1987) model focuses on the maintenance of positive perceptions towards the innovation by prospective adopters. Typical of the second category is the model by Tessmer (1990) which supports an environmental analysis, including physical considerations and patterns of use. Several models in the third category examine the change process in broad terms and facilitate the identification of conditions that support or impede its progress. For example, Stockdill and Morehouse (1992) identify critical factors that facilitate adoption; Havelock and Zlotolow (1995) focus on the various stages of planned change; and Benson and Palaskas 551 Zaltman and Duncan (1977) identify eighteen potential barriers to change. Palaskas (2002) supports ICT based innovation through a framework for the development of technology mediated teaching strategies. The model which appeared to be most useful for post-adoption analysis of an institutional innovation, with the potential for pre-adoption guidance of future practice, was the RIPPLES (Resources, Infrastructure, People, Policies, Learning, Evaluation and Support) model developed by Surry, Ensminger and Haab (2005). This model comprehensively covers a range of factors for consideration including: • the fiscal resources associated with innovation adoption; • the institution’s infrastructure namely, the hardware, software, facilities and network capabilities in support of teaching resources, production resources, communication resources, student resources and administrative resources; • the needs, hopes, values, skills and experiences of the people involved; • institutional policies and procedures; • the relationship between the technology and learning outcomes; • evaluation and review (both summative and ongoing), including the impact of the technology on learning goals; and • the support systems and scaffolding required to ensure successful implementation. Due to its breadth of coverage, the RIPPLES model incorporates considerations that are individually the focus of other adoption models, including some of those mentioned above. It also allows for the inclusion of key considerations that are not necessarily encompassed by other adoption models, such as the learning and teaching implications of the use of a learning management system (Coates, 2005; Coates, James & Baldwin, 2005). Thus, the RIPPLES model forms a broad theoretical ICT adoption and implementation framework that was seen as potentially useful for considering the evaluation of the WebCT Vista pilot project. The evaluation framework and the RIPPLES model Evaluation aims, strategies and responses The WebCT Vista pilot involved a small structured trial of the training, support, administrative and technical aspects of the service in Semester 1, 2004 (involving 15 units of study and 1,600 students, primarily studying on campus, at four of the University’s Victorian campuses), and an expanded trial in Semester 2 which involved 80 units across nine faculties and approximately 5,500 students across all campuses (including Malaysia and South Africa) who were studying on campus, off campus or offshore. The 552 Australasian Journal of Educational Technology, 2006, 22(4) evaluation of the pilot was undertaken by the project team during both semesters with a view to informing the transition to a full production service. It aimed to obtain user and service provider perspectives relating to training and professional development, pedagogical issues, support, administration and technical infrastructure, particularly highlighting issues which were likely to impact on wider institutional implementation (Benson, Palaskas, Van Dyk & Trahair, 2005). In the course of data collection in Semester 1, 2004 some issues about communication were also raised which related to the governance and management of the project and, more broadly, to technology adoption for learning and teaching across the University, so this information was also recorded as part of the evaluation results. Like the pilot itself, the evaluation was driven and framed by pragmatic issues, reflecting the Eclectic Mixed Methods Pragmatic Paradigm identified by Reeves and Hedberg (2003). Consequently, the evaluation consisted of a mix of qualitative and quantitative strategies, using reflective practice (Schön, 1983, 1987) to draw together the data from different sources and consider conclusions. Evaluation strategies in both semesters included analysis of project documentation, along with a range of approaches to gain information directly from respondents. In Semester 1 the evaluation focused on responses from members of staff (though some individual staff members implemented student surveys, and one a tutor survey, and provided their results for inclusion in the evaluation). In Semester 2 both staff and student responses were sought (Table 1). The ‘online communication space’ was a set of discussion topics on a WebCT (CE) site in Semester 1 (as WebCT Vista stability could not be guaranteed at that stage), and on WebCT Vista in Semester 2. The student survey in Semester 2 was administered by the University’s Centre for Higher Education Quality. Table 1: Evaluation strategies and responses Semester 1 Semester 2 Strategy Responses Strategy Responses Analysis of project documentation Analysis of project documentation Online communication space (staff) 215 Online communication space (staff) 149 Staff interviews 24 Staff focus groups (4) 19 End of semester staff questionnaire 5 Staff questionnaire 29 Student surveys administered by individual staff members: - Medicine, Nursing & Health Sci - Pharmacy - Information Technology 129 53 30 Student questionnaire 553/5500 Tutor survey administered by individual staff member 3 Student focus group 2 Benson and Palaskas 553 Applying the RIPPLES model In order to apply the RIPPLES model to the results obtained from the above strategies, relevant aspects of the model were mapped to each component of the evaluation framework (Table 2). In the following section the evaluation results are outlined and then considered in this context, with reference also to other theoretical perspectives where relevant. Note that the resources component of the RIPPLES model is not included because resources had already been secured and their availability was not addressed by the evaluation. Table 2: Applying the RIPPLES model to the evaluation framework Evaluation framework RIPPLES model Training and professional devt issues People; Learning; Support Pedagogical issues People; Learning [and teaching] Staff and student support issues People; Support Administrative issues Infrastructure; People; Support Technical issues Infrastructure, People; Support Communication issues People; Policies; Evaluation; Support Overall response People; Evaluation; Support Results Training and professional development Training and professional development for teaching staff in relation to the use of WebCT Vista consisted primarily of provision of training documentation (initially from sources outside the institution), together with a face to face workshop program. Institutional resources and a fully developed training program were available by second semester. Major issues raised by teaching staff during Semester 1 included: • training implications relating to the WebCT Vista hierarchical access levels; • the need for training and ongoing support in relation to WebCT Vista functionality; and • the need for training to occur within a pedagogical context. In Semester 1, responses indicated that the preferred training method for staff without experience of WebCT Campus Edition was face to face workshops, followed by one on one mentoring, printed resources and online learning. For staff with experience, the order was the same except that the first two preferences were reversed. In Semester 2, respondents placed considerable emphasis on the importance of informal, personalised assistance, especially after initial training. This could be provided either by faculty or central staff, although there was some preference for the former. 554 Australasian Journal of Educational Technology, 2006, 22(4) Some staff again emphasised the need for training of new staff to begin with pedagogical issues. Although there were positive comments about the central training workshops, some staff members indicated that they preferred not to train at these because of the wide range in the skill levels of participants. The need for tutor orientation was noted, related to the level of access being provided. Four components of support which are important to the successful introduction of learning technologies are identified in the RIPPLES model: training, technical support, pedagogical support, and administrative leadership (Surry, Ensminger & Haab, 2005). Thus, in this context, the above findings highlight the importance of training support, indicating a preference for informal training over formal training, as familiarity with the technology develops. There was also recognition of the need for pedagogical support in the way that the training was framed, which links to the learning component of the model, reflecting some recognition that the use of technology should be primarily driven by the fulfilment of learning needs. The concerns of teaching staff about training and other issues also underline the importance of individual people gaining the skills to use the technology appropriately, suggesting the relevance of stages of concern and levels of use (Hall & Hord, 1987) in adoption patterns. Pedagogical issues The concerns of people were also evident from responses on pedagogical issues. Early concerns in Semester 1 included: • problems with migrating course materials, particularly quizzes, which led to obtaining a University licence for Respondus (2000-2006) as initial experience from some early users indicated that this test creation tool facilitated easy migration of quizzes from WebCT Campus Edition to WebCT Vista; • concerns about whether mandating faculty templates at institutional level would reduce pedagogical flexibility; and • concerns about the pedagogical implications of an early decision to turn off the Mail and Chat functions, and the My Files area, because of their potential to overload the infrastructure in the event of a large number of concurrent users. However, staff interviews during Semester 1, for the most part, did not focus on pedagogical issues. Teaching staff tended to feel that they were still learning to use the system which limited their capacity to explore innovative teaching approaches, although there was evidence of some using the environment in creative ways. No comments in the online communication space on pedagogical issues were generated by teaching staff. Responses to the end of semester questionnaire in Semester 1 Benson and Palaskas 555 indicated that the use of W e b C T V i s t a predominantly involved announcements, quizzes, discussions, use of the grade book and the selective release function. There was some use of additional online components including animations, video, crosswords, a report writing multimedia tool, and images. Semester 1 student responses to the survey in a Pharmacy unit were positive about the use of the online environment for student discussion and for facilitating study efficiency. A few of the respondents suggested improvements, such as the use of diagrams, quizzes with answers, and more lecturer input. A small group of these students considered that face to face teaching suited their needs better. An effort was made in Semester 2 to broaden the scope of the staff evaluation to cover ways that WebCT Vista was used and to note pedagogically innovative and effective uses which had potential for sharing with other teaching staff. However, staff focus group responses indicated that its main use was to provide lecture notes or off campus materials online, although use also included social interaction and provision of assistance to students. Responses to the Semester 2 online student survey were consistent with this, indicating that the main reasons students used WebCT Vista were to access unit outlines and other unit information, and the calendar and discussion function. In some cases staff reported that they were responding to the demand from students to provide information online, noting the importance of a flexible learning environment for students who are employed, and the potential of providing extension materials for more capable students. Table 3 indicates the most frequent uses and intended uses of the LMS by Semester 2 staff questionnaire respondents. Table 3: Current and intended use of LMS as indicated by Semester 2 questionnaire responses Current Intended • Provision of a unit outline or unit information • Discussions open to the whole class • Calendar • Learning modules • Links to other web sites • My Grades tool • Quizzes (self-tests) • Quizzes (graded) • Surveys (e.g. unit evaluation) • Assignment drop box • Private discussions (e.g. tutorial groups) • Learning groups Responses on pedagogical issues relate most obviously to the learning aspect of the RIPPLES model, though they also reflect concerns and levels of use of p e o p l e, as indicated above, and the potential for the social engagement of people. Despite recognition of the importance of pedagogy in the use of the technology in responses on training issues, it did not appear to be the centre of attention for teaching staff. Use of the LMS for 556 Australasian Journal of Educational Technology, 2006, 22(4) learning, for the most part, seemed to involve fairly unsophisticated use of the tools available, and in some cases it was used primarily to provide access to information, rather than to engage students directly in an online learning environment. These uses can be compared with the findings of an earlier study of the use of WebCT CE at the University (Weaver, Nair & Spratt, 2005) which suggested that staff focused on the technical, administrative and workload aspects of using the LMS and that many students reported poorly designed sites, little or no feedback from staff, outdated information on sites and broken links. They also reflect the limitations in research and understandings about the pedagogical issues related to LMS use which are evident in the broader higher education community (Coates, 2005; Coates, James & Baldwin, 2005). Nevertheless, there was evidence in the pilot evaluation of some inclusion of online pedagogical components additional to LMS tools, and of intentions to explore the use of LMS tools not currently being implemented. Staff and student support issues Investigation of support issues referred particularly to staff and student support from the central WebCT Helpdesk (as seamless provision of this resource was a key project objective), but also to expectations of and requirements for faculty support. Staff perceived the service provided by the WebCT Helpdesk to be efficient, though many had not used it, and a preference for support at faculty level was expressed. There was some concern that the level of expertise was not sufficient to deal with staff problems. WebCT administrative staff from faculties were particularly dissatisfied about the arrangement for them to seek support via the Helpdesk, and considered that it was more appropriate as a source of student support. Just under half of the Semester 2 student survey respondents reported that they were aware that the WebCT Helpdesk was the first point of contact if they had problems, and about four in ten indicated that they knew how to contact the Helpdesk if they needed to. General satisfaction with the services was reported, though nearly two thirds indicated that they had not used or tested them. Of those who had contacted the Helpdesk, the two main purposes were to seek assistance with login problems and to access features or materials on WebCT Vista sites. As the title of this aspect of the evaluation suggests, responses on this issue relate centrally to the importance of support systems identified by the RIPPLES model, and to their appropriateness for the people using them. In this case, responses referred specifically to technical support, indicating qualified satisfaction with this, despite some clear directions for rethinking support arrangements at faculty level (for teaching and administrative Benson and Palaskas 557 staff), and evidence of limited ability to comment by students who had not used the services (which could indicate success in other aspects of implementation if this level of support had not been required). Administrative issues Devolved administration to staff within faculties was envisaged as a major project benefit, which would result in increased flexibility and reduced reliance on central support services. This devolution was not implemented until Semester 2, when faculties took over the WebCT administration role. During Semester 1, references to administrative issues primarily related to academic staff dealing with the administrative aspects of WebCT Vista functionality. In Semester 2, faculty administrators were included as respondents in the evaluation and one focus group was organised specifically to obtain their responses. They expected that the transition phase would be difficult but that problems would ease when everyone was using the LMS. They perceived that the 2005 rollout would result in a substantial increase in their workload and raised the following issues: • concerns about the administrative workload associated with the concurrent use of two platforms during the transition period (CE and Vista); • difficulties in administration created by different school structures within faculties; • problems resulting from lack of awareness of administrative requirements by faculties, schools and departments; • the need for staff computers to be set up for WebCT Vista before rollout; • the need for attention to student access issues before rollout; and • the need for the Flash plugin [http://www.adobe.com/products/flash/flashpro/] on all staff and student desktops (the absence of this plugin was noted as potentially problematic in the use of home computers). Faculty administrators had found the training provided for them useful but noted that its timing before the introduction of middleware applications to facilitate faculties’ use of the system had affected its relevance. Hence, it appeared that training of faculty administrators should have occurred after the development of software (middleware) which gave them control over LMS functions, so that this major aspect of their role was included in the training. One of their subsequent responses to the middleware was that it was useful for bulk uploads of student cohorts but cumbersome for single or small uploads. Manuals from other institutions that were provided at training were considered useful and the introduction of a local manual recommended. Overall, the administrative workload for faculties and the type of support available for administrative staff were their main concerns. 558 Australasian Journal of Educational Technology, 2006, 22(4) From the perspective of the RIPPLES model, administrative aspects of the evaluation referred particularly to the new roles required of some people (the faculty administrators) as devolution related to the innovation was implemented. Their concerns also raised issues about infrastructure, indicating a need to ensure that this was adequate at faculty level. Responses also raised again the importance of training support associated with the group administration role. Although, at the time the evaluation was implemented, there was evidence of some apprehension related to this new role, no major problems were indicated. Technical issues The major technical focus of the project was intended to be on the provision of hardware infrastructure, technical support services and procedures to run the LMS, and on determining, developing and testing the middleware applications mentioned above. However, evaluation responses indicated that the major focus of user concerns was on technical issues concerning LMS functionality. In Semester 1 the main infrastructure issues raised by staff users related to server downtimes (particularly unscheduled downtimes), the speed of the server, and the need to monitor server performance. As a consequence, timing of scheduled downtimes was discussed to find a time which caused minimum disruption to teaching. Disaster recovery planning was recognised as a key issue and addressed. A server administration and archiving policy was also discussed and accepted. Decisions about middleware functionality were made in consultation with staff users to provide for different models of faculty administration, leading to a targeted completion date in mid 2004, which was successfully achieved. A major concern of users was the balance between server performance and LMS functionality, resulting from the decision to turn off the Mail and Chat functions, and the My Files area, to improve performance. Functionality issues dominated discussion in project documentation, interviews and the project communication space, with major concerns including problems with quiz migration, bugs identified, and student login problems (including issues related to absence of the Java Virtual Machine which was needed to run the system, and to promote the browser tune up function to ensure correct browser configuration). A particular issue raised during the evaluation, with implications at faculty level relating to login and access, was the need for appropriately supported computer laboratories. By Semester 2, despite the escalation in service that had been required to accommodate the increased number of units, focus group respondents indicated that access was generally good, although there had been some Benson and Palaskas 559 login problems. However, questionnaire responses suggested that technical problems caused a substantial number of interruptions to preparation time and class time, and staff postings in the online communication space were predominantly concerned with technical issues (130 of 149 postings). As in Semester 1, the focus of most of these was on specific aspects of LMS functionality, rather than the infrastructure itself, though there were some comments about modifications to the service being made without notifying the user community, and there was some tension evident in postings late in the semester as performance problems arose during the assessment period. There was evidence of continuing access difficulties by some tutors. The availability of technical support on a 24/7 basis was requested. Technical issues formed the recurring theme in relation to Semester 1 staff information about student responses, particularly relating to student problems in logging into sites or accessing site features. In the Pharmacy survey this did not appear to be a major issue, but there were some vigorous complaints about the occasional server downtime. Access was more of a problem in a survey of two units (with 129 responses) in the Faculty of Medicine, Nursing & Health Sciences, with about four in ten students indicating login problems on or off campus, and some experiencing difficulty in accessing site features. Similar access problems were reported in a survey of 30 Information Technology students. In many cases students were not aware of, or had not undertaken, the browser tune up process. Nearly three quarters of the respondents to the Semester 2 student survey indicated no login problems, with only small percentages of those using various tools reporting problems in accessing them. Over four in five had not performed a browser tune up. Staff focus group respondents in Semester 2 also indicated that students did not have problems, though access and login problems experienced by off campus students were mentioned. However, only half of the staff questionnaire respondents considered that their students were able to access the LMS when required. A third of them suggested that students had reported that technical problems constantly interrupted their work time. Responses on technical issues relate to a major focus of the RIPPLES model, that of infrastructure, which was also intended as the major technical focus of the evaluation. While infrastructure issues were raised, the preoccupation of users (people) with the functionality of the system highlighted once again the importance of support, both training and technical support, as many of the problems experienced did not appear to be as major as they were perceived to be as a result of heightened anxiety levels. In this context, there was also some evidence that staff perceived students’ problems to be greater than the students did themselves. While 560 Australasian Journal of Educational Technology, 2006, 22(4) there were some problems as the infrastructure was established and fine tuned, from a technical service provider perspective these issues were not insurmountable, and the identification of them was a major purpose of the pilot. Communication issues During Semester 1, project communication occurred primarily through reference group meetings, pilot participant meetings, and participation in the online communication space. Concerns began to be raised about delays in circulating minutes of meetings, and the lack of scheduled meetings was seen as inhibiting the necessary flow of information. These problems were addressed, and it was recognised that communication would need to be more structured in Semester 2 because of the increased number of participants. Pilot participant discussion in Semester 2 was planned to take place on a WebCT Vista pilot site, while other communication would be by email and newsletters. A number of other communication problems were reported in Semester 1 evaluation interviews, both from support staff and teaching staff. Specific concerns, from a central support perspective, included: • the need for more effective communication by ITS with users (including the need to avoid perceptions that tools were turned off without warning); • failure to achieve sufficient faculty representation at reference group meetings because of inadequate attendance by appointed faculty representatives; • problems relating to communication between the ITS project team, ITS staff not involved in the project, and project stakeholders; • lack of information, or conflicting information, from ITS to other support services; • conflicting, insufficient and poorly documented information provided to ITS on problems relating to the LMS; and • delays in identifying pilot participants for Semester 2 which impacted on plans for training. From the point of view of teaching staff or faculty support staff, specific concerns included: • lack of response or inconsistent responses from ITS to information about bugs; • lack of clarity about the server policy in relation to deletion of course materials; • tensions resulting from the provision of information considered insufficient by ITS; Benson and Palaskas 561 • the need for regular meetings, scheduled in advance, in order to voice concerns that may be experienced with the development of sites; and • lack of clarity by participants about help procedures. These concerns raised implications for the management of the project, and ultimately its governance, and were also related to the complex structure of the institution, with central service providers being unable to ensure that adequate arrangements were made within faculties, and with communication gaps evident between the service providers themselves. In relation to the RIPPLES model, these issues continue to highlight the importance of the perceptions of people, related to their stages of concern and levels of use (Hall & Hord, 1987). They also reflect the importance of institutional policies and of support, in terms of the administrative leadership role identified by the model. Thus, these concerns address management issues involved in introducing an innovation in a large institution, with implications for the relative value of middle-out, top- down and bottom-up approaches as highlighted by Cummings, Phillips, Tilbrook and Lowe (2005), and also for the way that the strategic direction of the university is expressed and operationalised. Although the introduction of appropriate learning technologies supported the strategic direction of the University in relation to technology supported learning, its Learning and Teaching Plan (Monash University, 2003) did not specifically address implementation of an institutional LMS. As a consequence, the evaluation which was implemented did not provide input at this level. Overall response Overall, staff responses during Semester 1 reflected various levels of satisfaction with the LMS, subject to addressing teething problems. Tutors’ responses collected via an individual unit survey reflected a lower level of satisfaction, related to difficulties that they had had in accessing and marking assignments. Just over half of the responses to the Semester 2 staff survey indicated satisfaction with the LMS, while about a quarter were dissatisfied. All respondents to the Semester 1 staff survey indicated that their students were satisfied with the WebCT Vista learning environment. The Pharmacy student survey also indicated a general sense of satisfaction from students. About a third of the respondents to the student survey implemented in the Faculty of Medicine, Nursing & Health Sciences found it harder to use than the existing WebCT service, while about half the Information Technology students indicated satisfaction, with a staff member posting the following student comment in the online communication space: 562 Australasian Journal of Educational Technology, 2006, 22(4) I think Vista is absolutely fantastic! Especially since I am quite busy with work, it allows me to catch up on all of the discussions, lecture material, grades and notices in one location, at any computer. Also the ability to save our work to the site is of great importance. The lack of media such as paperwork and computer disks, has made it easier to concentrate on the contents of the subject, rather than ‘administration’ tasks. I personally would like to see Vista used for every subject I am doing at Monash, and hopefully the lecturers will be as great as [name deleted] has been in supporting the students through Vista. Only a third of the Semester 2 student survey respondents indicated high satisfaction with the system, nearly half expressing varying degrees of dissatisfaction and about one in five expressing neither satisfaction nor dissatisfaction. A third of staff questionnaire respondents agreed that their students had a positive attitude to WebCT Vista and the same percentage disagreed. It is acknowledged that questionnaire responses need to be considered in the context of the limited response rates achieved, though comparison of survey data with information derived from other strategies provided a means of triangulation and identification of common themes. A more limiting factor may have been the scope of the evaluation itself. Surry, Ensminger and Jones (2003), in an earlier paper, refer to four areas of evaluation which apply to the introduction of technology: evaluation in relation to learning goals; evaluation of the technology, including ongoing assessment of technology alternatives; evaluation of the integration plan to determine the factors that have facilitated or impeded the introduction of technology; and a benefit/cost evaluation. The evaluation reported here primarily focused on the third of these, though efforts were made to focus on the first as the major issue underpinning the innovation. Consequently, the overall responses outlined above summarise the responses of people (the users), reflecting more directly the adequacy of technical support, than the infrastructure and learning opportunities offered by the innovation. Discussion It is evident that, in terms of the RIPPLES model, the WEBCT Vista evaluation primarily highlighted issues related to people and support, as part of testing the innovation to prepare for a full production service. While this was consistent with the aim of gaining the perceptions of service providers and users in relation to aspects of the service, it indicates secondary levels of concern about learning and about infrastructure (except when it failed), which were also intended to be major foci of the evaluation. In relation to policies, while the existence of the pilot indicated institutional intent, some of the communication problems experienced during the pilot suggested a lack of alignment between top-down, bottom-up and middle-out Benson and Palaskas 563 approaches to innovation (Cummings et al, 2005), particularly in an environment where individual faculty structures are strong, and despite the fact that the decision to proceed with the project was undertaken with some buy in at individual and central service middle management levels. In this context, Zaltman and Duncan (1977), Ely (1990) and Stockdill and Morehouse (1992) provide tools for further analysis of the context which could guide future change. Similarly, although the existence of the evaluation process indicated acknowledgement of the need to monitor the introduction of the technology, the evaluation did not cover all of the areas suggested by the RIPPLES model, and was not adequately integrated into the project to inform decision making at an institutional level while the pilot progressed. A separate technical evaluation and decision making process was, in fact, driving much of the project, perhaps underpinning the limited emphasis on learning. While technical evaluation (including evaluation of technical alternatives) has continued since the project, this has not been complemented by an ongoing coordinated evaluation of other aspects of the service. Another notable absence in the evaluation described here is the lack of reference to resources. Surry, Ensminger and Jones (2003, p.14), in explaining the breadth of their model and the need to plan for adequate funding, comment that: It is interesting to note … that the adoption and diffusion literature includes very little discussion about the importance of financial resources to the change process. One reason for this may be that most adoption and diffusion models assume that funding has already been secured and an innovation is available for adoption. As indicated earlier, this latter situation was the case in relation to the WebCT Vista pilot at Monash University. Two related questions arise from considering the results of the evaluation in the context of the RIPPLES model: • Does use of the model assist in a post-adoption review of the pilot to highlight issues for guiding future practice? • Are there aspects of the model which could be extended or modified to make it more comprehensive or flexible? The answer to the first question is yes: the RIPPLES model has been useful in providing a structure to review the pilot and the evaluation outcomes. Although the evaluation did not cover all the aspects included in the RIPPLES model (for example, by excluding resources), there is potential for 564 Australasian Journal of Educational Technology, 2006, 22(4) analysing those aspects that are covered, and using the model to summarise areas that were adequately addressed and to highlight priorities for future action (Table 4). Table 4: Analysis of results using the RIPPLES model RIPPLES elements Review Priorities for action Resources Not applicable Not applicable Infrastructure Addressed by pilot Low: appropriate infrastructure in place People Concerns in a number of areas require addressing for transition to a pedagogically-effective production service High: to improve perceptions and use of the LMS Policies Not specifically articulated at institutional level and not operationalised through appropriate management structures which foreground use of the LMS to improve learning and teaching High: policies and strategies to link governance, management and administration needed, involving leadership by educators supported by appropriate technical infrastructure Learning Importance recognised but not the major focus of pilot participants High: to improve use of the LMS Evaluation Scope and potential for impact too limited for an institutional initiative High: needs to be ongoing and better integrated with strategic goals and implementation plans related to pedagogy. Responsiveness to learning and teaching needs identified by student evaluations would also be desirable. Support Addressed by pilot and evaluation Medium/low: ongoing monitoring needed As indicated, a review of evaluation results using the RIPPLES model suggests that the highest priorities for action relate to people, policies, learning and evaluation. This differs from the findings of Surry, Ensminger and Jones (2003, p.13) who state that according to their study ‘technology infrastructure is the single most important factor in integrating technology into the curriculum.’ While it could be argued that all subsequent elements are contingent on adequate infrastructure, the findings of this study suggest that these other elements have a high degree of importance, if improvements in learning and teaching are to be achieved. In response to the second question, use of the model for the purposes described in this paper has raised some issues that might suggest potential for its refinement. Firstly, although the model addresses the importance of Benson and Palaskas 565 using technology for improving learning, there is scope for increasing focus on the implications of this for teaching, since the quality of online teaching is a major factor in successful online learning. While the model acknowledges the need for pedagogical support, this appears to underplay the professional development that is frequently needed for teaching staff to adapt to teaching online (Epper & Bates, 2001). Secondly, there appears to be room for increased focus on management issues relating to the adoption of technology innovations, particularly to accommodate different institutional structures. Appropriate policies are vital but they require complementary implementation processes. The model assumes administrative leadership but does not suggest ways of achieving this. Thirdly, broadening the resources element of the model to include more than fiscal resources seems appropriate, given the resource commitments required at various levels to introduce an institutional technology innovation successfully. Conclusion The RIPPLES model appears to be a useful tool for analysing institutional innovations such as the one outlined in this paper. Given that it was designed for institutional initiatives of this kind, this is not surprising. Although there is potential for extending some aspects of the model, it covers major factors that need to be considered in the higher education environment and its breadth allows other narrower models to be incorporated where necessary, in order to focus in more detail on specific aspects of the change process. It thus avoids the limitation of a number of other macro models which do not simultaneously permit focus on specific pedagogical issues. In the case study described here, the model highlights issues relating to people, policies, learning and evaluation as the areas in most need of attention for monitoring the innovation. Assuming that resources are available, identifying and implementing institutional policies and procedures to facilitate continual improvement of the effective pedagogical use of the LMS, coupled with strategies to monitor these through ongoing evaluation, could provide a basis for guiding institutional improvement in the quality of online learning and teaching. Acknowledgement The 2004 WebCT Vista Pilot Project evaluation was completed under the project direction of Di James, Special Adviser to the Deputy Vice Chancellor (Academic and Planning). Marie Therese Van Dyk (CeLTS) and Paul Trahair (ITS) were project team members and co-authors of the evaluation report. Research assistance for the Semester 1 evaluation was provided by Fatma Ismail and for the Semester 2 evaluation by Gayle Nicholas and staff from the Facilitated Learning for Information 566 Australasian Journal of Educational Technology, 2006, 22(4) Technology Education (FLITE) Centre, Faculty of Information Technology. The Semester 2 student survey was implemented by Sid Nair of the Centre for Higher Education Quality and the Semester 2 staff focus groups were facilitated by Kate Duggan, Program Manager, Global Online Learning Development (GOLD) Program, CeLTS. The contributions of CeLTS and ITS in funding the evaluation, and of the staff and students who provided evaluation feedback, are especially acknowledged. Endnote In 2006 CeLTS was restructured into several separate units with some support functions no longer available as central services. The central provision of WebCT staff training is now undertaken by ITS. References Benson, R., Palaskas, T., Van Dyk, M.T. & Trahair, P. (2005). WebCT Vista pilot project 2004. Final evaluation report. [viewed 5 Apr 2006] http://www.monash.edu.au/muso/lmsproject/assets/pdf/WebCTVista%20e valuation_finalreport_finalrev260805.pdf Burkman, E. (1987). Factors affecting utilization. In R. M. Gagne (Ed), Instructional Technology: Foundations (pp. 429-455). New Jersey: Lawrence Erlbaum Associates. Coates, H. (2005). Leveraging LMSs to enhance campus-based student engagement. Educause Quarterly, 28(1). http://www.educause.edu/apps/eq/eqm05/eqm05110.asp?bhcp=1 Coates, H., James, R. & Baldwin, G. (2005). A critical examination of the effects of learning management systems on university teaching and learning. Tertiary Education and Management, 11, 19-36. Cummings, R., Phillips, R., Tilbrook, R. & Lowe, K. (2005), Middle-out approaches to reform of university teaching and learning: champions striding between the top-down and bottom-up approaches. International Review of Research in Open and Distance Learning, 6(1). http://www.irrodl.org/content/v6.1/cummings.html Ely, D. (1990). Conditions that facilitate the implementation of educational technology innovations. Journal of Research on Computing in Education, 23(2), 298- 305. Epper, R.M. & Bates, A.W. (Eds) (2001). Teaching faculty how to use technology. Best practices from leading institutions. Westport, USA: Oryx Press. Hall, G. & Hord, S. (1987). Change in schools: Facilitating the process. Albany, NY: State University of New York Press. Havelock, R., & Zlotolow, S. (1995). The change agent's guide (2nd ed.). Englewood Cliffs, NJ: Educational Technology Publications. Monash University (2003). Learning and Teaching Plan 2003-2005. Melbourne: Monash University. Benson and Palaskas 567 Monash University (2006). Monash statistics. [viewed 5 Apr 2006] http://www.monash.edu.au/about/stats.html Palaskas, C. (2002). A model for selecting technology mediated teaching strategies. Educational Technology Magazine, 42(6), 49-54. Reeves, T. C., & Hedberg, J. G. (2003). Interactive learning systems evaluation. Englewood Cliffs, NJ: Educational Technology Publications. Respondus (2000-2006). Assessment, survey and game applications for elearning. http://www.respondus.com/ Rogers, E. M. (2003). Diffusion of innovations (5th ed.). New York: The Free Press. Schön, D. A .(1983). The reflective practitioner: How professionals think in action. Aldershot: Ashgate. Schön, D. A. (1987). Educating the reflective practitioner. San Francisco: Jossey-Bass. Sherry, L., Billig, S., Tavalin, F. & Gibson, D. (2000). New insights on technology adoption in communities of learners. In C. Crawford et al. (Eds), Proceedings of Society for Information Technology and Teacher Education International Conference 2000 (pp. 2044-2049). Chesapeake, VA: AACE. Stockdill, S. H., & Morehouse, D. L. (1992). Critical factors in the successful adoption of technology: A checklist based on TCD findings. Educational Technology, 32(1), 57-58. Surry, D. W., Ensminger, D.C. & Haab, M. (2005). A model for integrating instructional technology into higher education. British Journal of Educational Technology, 36(2), 327-329. Surry, D. W., Ensminger, D.C. & Jones, M. (2003). A model for integrating instructional technology into higher education. [viewed 5 Apr 2006] http://www.iphase.org/papers/RIPPLES.rtf Tessmer, M. (1990). Environmental analysis: A neglected stage of instructional design. Educational Technology Research & Development, 38(1), 55-64. Weaver, D., Nair, S. & Spratt, C. (2005). Experiences of online teaching using WebCT: An institutional evaluation. Paper presented at the Fourth International Conference on Science, Mathematics and Technology Education, 25-28 August, Victoria, Canada. Zaltman, G., & Duncan, R. (1977). Strategies for planned change. New York, NY: John Wiley and Sons. Dr Robyn Benson, Centre for Medical & Health Sciences Education, Building 52, Clayton Campus, Monash University, Victoria, 3800. Email: Robyn.Benson@med.monash.edu.au Dr Tom Palaskas, Faculty of Business and Law, Footscray Campus (G4.11), PO Box 14428,Victoria University, Victoria, 8001. Email: Tom.Palaskas@vu.edu.au