Microsoft Word - JDL05maintext1WORD.doc Journal of Distance Learning, Vol 9, No 1, 2005 © Distance Education Association of New Zealand 58 Research Reports INTRODUCTION In mid-2004 the New Zealand Ministry of Education announced the funding of five research projects under the TeLRF Round One. The Ministry’s Request for Proposals indicated that the goal of the first round of research was to establish the current context and future impact of tertiary e-learning on learners, teachers, and providers in New Zealand. Shortly after the Ministry’s announcement, the Journal editors asked project leaders if they would be prepared to submit a brief description of the nature of their project and the methodology being used within the research. We are pleased to be able to present accounts from two of those projects in this issue. Determination of New Zealand Tertiary Institution E-learning Capability: An Application of an E-learning Maturity Model STEPHEN MARSHALL UNIVERSITY TEACHING DEVELOPMENT CENTRE, VICTORIA UNIVERSITY OF WELLINGTON WELLINGTON, NEW ZEALAND RATIONALE A significant challenge facing most tertiary institutions is identifying organisational strategic and operational priorities for investment in e-learning capability—in short, containing the rising cost of e-learning infrastructure while maximising the educational outcomes for students. This project is engaging in a detailed e-learning capability determination exercise over a number of New Zealand Tertiary Education Organisations (TEOs). This will assess their current strategic and operational capability for delivering and maintaining e-learning in an educationally effective but also organisationally effective manner. This process is intended to validate a framework for conducting such capability determinations and provide a documented standard methodology for determining e-learning strengths and weaknesses institutionally and by sector. Use of such a framework and the underlying e-learning Maturity Model represents a number of significant opportunities for New Zealand: 1. Firstly, an e-learning process improvement model could provide a road map for higher education institutions looking to improve their e-learning processes. It is clear that a series of signposts or a map that might guide institutional planners in areas of resource allocation and staff and student support has some merit. 2. An accepted framework might also provide academics with the necessary means to encourage greater institutional involvement. The advantage of the Journal of Distance Learning, Vol 9, No 1, 2005 © Distance Education Association of New Zealand 59 proposed model is that it presents a high enough overview that it can be usefully understood without a need to examine detailed reports and metrics or be familiar with an extensive pedagogical and technological research literature. 3. Support for institutional planning might be enhanced by an institution or unit’s ability to benchmark its current capability, to identify and prioritize necessary improvements in its current practices. The model allows for different technical platforms, organizational models, and pedagogical beliefs. Perhaps most importantly, like the software process improvement approaches on which the model is based, this project might form the basis for an ongoing discussion within the e-learning community with a view to identifying the key practices, heuristics, or activities necessary for achieving improvements in e-learning activities. CONTEXT Large sums of money are being invested in expensive systems such as those provided by Blackboard (http://www.blackboard.com) which can cost hundreds of thousands of dollars annually to license and deploy. Despite this investment, uncertainty remains over whether the investment is resulting in improved learning outcomes for students (Conole, Oliver, & Harvey, 2000; Taylor 2001). This problem was noted in the New Zealand Government–funded “Highways and Pathways” document: In its research to date, the Advisory Group has found very little evidence of evaluation of strategies that might inform New Zealand’s strategic vision [on e-learning]. (E-learning Advisory Group, 2002, p. 19) More fundamentally, concerns remain among many teachers about the validity and quality of learning possible through on-line provision. These concerns are reinforced by the lack of agreed standards for academic quality and resource development. (E-learning Advisory Group, 2002, p. 11) In an attempt to move from an area characterised by individual heroics and unsustainable projects driven by passionate teachers, there have been numerous attempts to document “best practice” such as Chickering’s Seven Principles (Chickering & Ehrmann, 1996), the work of the National Learning Infrastructure Initiative (Hagner, 2001), and the benchmarks created for the Institute for Higher Education Policy (2000). These heuristics have been complemented by a development of technical standards such as the Shareable Content Object Reference Model (Department of Defense, 2001) and the work of the IMS Global Learning Consortium (2003). A weakness with these approaches is the focus on the outcomes of individual initiatives rather than a deeper analysis of the institutional context. The problem is the need for a more holistic approach with a focus on best systems rather than on individual practices (Hagner, 2001). Much of the published e-learning literature has resulted in recommen- dations for institutions, but identifying limitations with current institutional e-learning practices in an efficient and detailed way remains challenging. Such an understanding is needed for institutions to incrementally improve organisational e-learning capability (Laurillard, 1997), particularly as institutions will usually need to choose Journal of Distance Learning, Vol 9, No 1, 2005 © Distance Education Association of New Zealand 60 from the range of possible areas needing improvement, those that result in the greatest benefits for students and for the institution as a whole. The challenge facing institutions engaged in e-learning is similar to that which faces organisations developing complex software systems and which resulted in the development of process improve- ment models such as the Capability Maturity Model (Paulk, Curtis, Chrissis, & Weber, 1993) and SPICE (Software Process Improvement and Capability dEtermination) (El Emam, Drouin, & Melo, 1998; SPICE, n.d.). The similarities have led us to apply the same approaches used in the development of the software process improvement models to the development of a potential e-learning process improvement model or eMM (Marshall & Mitchell, 2002; 2003; 2004). The development of the CMM and SPICE models resulted from extensive consultation and workshopping industry practitioners, to identify the practices that resulted in high-quality software devel- opment. These processes were validated and tested extensively through additional pilots and evaluations (El Emam, Drouin, & Melo, 1998). In the case of SPICE, this work resulted in a model identifying five main areas or process categories; these are used to organise a collection of processes and the practices that contri- bute to the effective performance of individual processes. Table 1 shows the development of related categories for e- learning, which are presented elsewhere (Marshall & Mitchell, 2002; 2003). Table 1 eMM process categories Process Category Brief Description Learning Processes that directly impact on pedagogical aspects of e-learning Development Processes surrounding the creation and maintenance of e-learning resources Coordination Processes surrounding the oversight and management of e-learning Evaluation Processes surrounding the evaluation and quality control of e-learning through its entire lifecycle Organisation Processes associated with institutional planning and management The model rates performance of each of the process categories at six levels similar to that used in SPICE (see Table 2 on the following page) and generates an overview of organizational e-learning performance that potentially can inform strategic and operational decision- making by management (Marshall & Mitchell, 2004). Journal of Distance Learning, Vol 9, No 1, 2005 © Distance Education Association of New Zealand 61 Table 2 Levels of process capability E-learning Maturity Model: Levels Level Focus 5: Optimising Continual improvement in all aspects of the e-learning process 4: Managed Ensuring the quality of both the e-learning resources and student learning outcomes 3: Defined Defined process for development and support of e-learning 2: Planned Clear and measurable objectives for e-learning projects 1: Initial Ad-hoc processes 0: Not performed Not done at all The expectation from the experience of software maturity models (SEI, 2004) is that most, if not all, organisations will perform poorly under this model over the majority of areas assessed. This is consistent with the widely held perception that much work remains to be done to improve the quality and effectiveness of e-learning both nationally and internationally. METHODOLOGY The project is intended to have the following outcomes: 1. A validated model and expanded set of process categories identified through examination of existing e-learning projects in the participating institutions which can form the basis of ongoing research, development, and examination of institutional e-learning capability. 2. A detailed evaluation of e-learning process capability for each participating institution which could be used to guide internal consideration of its e-learning capability. 3. A summary of e-learning capability across all participating institutions which would provide an initial overview of capability and relative performance on a sector-wide basis (all institutions would be anonymously reported on), suitable for publishing, peer review, and for informing strategic planning at both institutional and sector-wide levels. As we already have a theoretical model, this is being used as the basis of the work. It is important to emphasize that the individual institution reports are confidential and are only going to be reported publicly in a summary form. The intention is to apply the model methodology outlined above and determine whether the theoretical design is both practical and useful, thus validating the approach. The real output is the confidence that the outcomes will provide institutions choosing to use the model to inform their own strategic and operational management of e-learning. Journal of Distance Learning, Vol 9, No 1, 2005 © Distance Education Association of New Zealand 62 In order to achieve these outcomes in the limited time available, the project was broken down into three phases: Phase 1 (July 2004 to August 2004) • Distribution of e-learning process- capability model to interested institutions for consultation and final agreement to participate in project. • Development of detailed programme for evidence collection and visits to participating institutions in e-learning process-capability study. Phase 2 (September 2004 to January 2005) • Collection of evidence of e-learning process-capability from participants. • Evaluation of e-learning process- capability evidence collection process. • Redevelopment of e-learning process- capability model to reflect feedback and evidence collected. Phase 3 (February 2005 to March 2005) • Dissemination of validated e-learning process capability model and self- assessment methodology to all New Zealand TEOs. • Reporting project outcomes to Ministry. A total of twenty-one organisations have been approached to participate in the study. It is expected that detailed reports will be produced for between five and ten of these, covering a range of institutions from small rural polytechnics to wänanga and large urban universities. REFERENCES Chickering, A., & Ehrmann, S. (1996). Implementing the seven principles: Technology as lever. AAHE Bulletin, October, 3–6. Conole, G., Oliver, M., & Harvey, J. (2000). Toolkits as an approach to evaluating and using learning material. Proceedings of the seventeenth annual conference of the Australasian Society for Computers in Learning in Tertiary Education, Southern Cross University, Coffs Harbour, Australia. E-learning Advisory Group. (2002). Highways and pathways: Exploring New Zealand’s e-learning opportunities. Report of the E-learning Advisory Group, March 2002. El Emam, K., Drouin, J.-N., & Melo, W. (Eds.) (1998). SPICE: The theory and practice of software process improvement and capability determination. Los Alamitos, California: IEEE Computer Society Press. Hagner, P. R. (2001). Interesting practices and best systems in faculty engagement and support. Paper presented at the National Learning Infrastructure Initiative Focus Session, Seattle, Washington. IMS. (2003). IMS Global Learning Consortium. Retrieved August 20, 2003, from http://www.imsproject.org Institute for Higher Education Policy. (2000). Quality on the line: Benchmarks for success in Internet-based distance education. Retrieved August 20, 2003, from http://www.ihep. com/Pubs/PDF/Quality.pdf Laurillard, D. (1997). A systems model of individual learning applied to organisational learning and the HE system. Paper presented at the Applying Systems Thinking to Higher Education Conference, The Open University, Milton Keynes, United Kingdom, July 12, 1997. Marshall, S., & Mitchell, G. (2002). An e-learning maturity model? In A. Williamson, K. Gunn, A. Young, & T. Clear (Eds.), Proceedings of the nineteenth annual conference of the Australian Society for Computers in Learning in Tertiary Education. Auckland, New Zealand: Australian Society for Computers in Learning in Tertiary Education. Retrieved from http://docushare.unitec.ac.nz/ dscgi/ds.py/Get/File-231/173.pdf Marshall, S., & Mitchell, G. (2003). Potential indicators of e-learning process capability. In Proceedings of EDUCAUSE in Australasia 2003. Adelaide: EDUCAUSE. Marshall, S., & Mitchell, G. (2004). Applying SPICE to e-learning: An e-learning Journal of Distance Learning, Vol 9, No 1, 2005 © Distance Education Association of New Zealand 63 maturity model? Paper presented at the Sixth Australasian Computing Education Conference (ACE2004), Dunedin, New Zealand, January 2004. In R. Lister, & A. Young (Eds.), Conferences in research and practice in information technology, vol. 30. Paulk, M., Curtis, B. Chrissis, B., & Weber, C. V. (1993). Capability maturity model: Version 1.1. IEEE Software 10(4), 18–27. SEI. (2004). Process maturity profile: Software CMM 2004 mid-year update. Pittsburgh, Pennsylvania: Software Engineering Institute, Carnegie Mellon University. Retrieved November 19, 2004, from http://www.sei.cmu.edu/sema/pdf/SW- CMM/2004aug.pdf SPICE. (n.d.). Software process assessment version 1.00. Retrieved December 18, 2002, from http://www.sqi.gu.edu.au/spice/ Taylor, J. (n.d.). Twenty-nine steps to heaven? Strategies for transforming university teaching and learning using multimedia and educational technology. Milton Keynes, UK: The Open University Knowledge Network. Retrieved July 18, 2005, from http://kn.open.ac.uk/public/document. cfm?documentid=3027 U.S. Department of Defense. (2001). Sharable content object model: Reference model 1.1. Retrieved August 20, 2003, from http://www.adlnet.org/index.cfm? fuseaction=scormabt Stephen Marshall is the Project Director for the eCDF project “Determination of New Zealand Tertiary Institution on E-learning Capability: An Application of an E-learning Maturity Model.” Marshall is a Senior Lecturer in Educational Technologies and currently Acting Director of the University Teaching Development Centre at Victoria University of Wellington. Research in Progress: Learning from Adopters and Resisters of E-learning in New Zealand Polytechnics and Institutes of Technology HUGH BARR, STEPHEN BRIGHT, JOHN CLAYTON, STEPHEN COX, BEVERLEY GOWER, DAVID MITCHELL WAIKATO INSTITUTE OF TECHNOLOGY HAMILTON, NEW ZEALAND INTRODUCTION The exponential growth of e-learning practices in higher education has resulted in an increasing interest in the ways in which faculty members in tertiary institutions perceive e-learning and the ways in which they apply e-learning in their courses. Faculty who might be described as “innovators” or “early adopters” have generally embraced e-learning enthusiastically; other faculty remain disengaged or disinterested. Disengagement may be due to faculty concerns about access to technology, software, and networks or it may stem from concerns about time allocations and staff support. Disinterest may be due to a perceived lack of relevance of e-learning strategies to particular courses, or it may simply reflect faculty members’ dispositions to change, innovation, and adoption. The project briefly described in this paper seeks to determine the causes of disengagement and disinterest in the adoption of e-learning approaches among teaching faculty.