mcnaught.pdf Australian Journal of Educational Technology 2003, 19(1), 72-86 It takes more than metadata and stories of success: Understanding barriers to reuse of computer facilitated learning resources Carmel McNaught, Andrew Burd Chinese University of Hong Kong Kevin Whithear The University of Melbourne John Prescott University of Guelph Glenn Browning The University of Melbourne A national study in Australia in the late 1990s explored barriers to the adoption and reuse of computer facilitated learning (CFL) in Australian universities. These barriers will be summarised. One of these barriers is that it is hard to find information on courseware that is educationally sound; usually such courseware is expensive to produce and so reuse is especially desirable. However, even when information and access to electronic courseware exists, reuse may still not occur. Two cases will be described to illustrate the complexity of reuse. These cases are: 1) a collection of 169 plastic surgery websites; and 2) an international consortium of veterinary microbiology resources based on a well-evaluated case study design. Some strategies for improving reuse are suggested. Just what is this paper about? One of the conference themes is the ‘Navigators’ log’. The suggestions there for paper orientations are: positive learning experiences, teaching innovations that worked, case studies, successful use of the web and learning technologies, successful course creation and pedagogical dimensions, educational design tools for learning, and exemplary projects. This paper will be within this theme BUT will not have success as the main focus; it will concentrate on issues and concerns - more on what did NOT work than on what did. Successful projects are often reported in glowing terms and the issues and concerns that arise from them are not explored in McNaught, Burd, Whithear, Prescott and Browning 73 sufficient detail. In this paper we will examine our experiences in order throw some light on the complex issues associated with academics sharing and reusing existing electronic resources. Both examples of computer facilitated learning (CFL) that are discussed in this paper have expensive electronic media as a key focus. In one of the cases optimum reuse has not been achieved, despite fairly extensive evaluation and reporting about the projects in many forums. The other project is still in design phase and is struggling to find pragmatic ways to reuse a large pool of existing web resources. The intention in this paper is not to come up with defined solutions to the challenge of reuse; rather we wish to use our examples to clarify the nature of some of the challenges about reuse and indicate directions for future work. To provide the ‘ambience’ of questioning, we will use questions for headings. If it works with text books, why not with electronic media? Reuse of educational materials has occurred for many years in the print domain. The ‘Pearson phenomenon’ is a clear example, as evidenced by this description from their website: Pearson Education’s reach extends across the globe through its seventy regional Web sites and twenty-five publishing centers, developing educational products for children, schools, universities, adults and corporations in thirteen languages. Long renowned as the world’s market leader in English Language Training, Pearson Education maintains publishing operations in seven regions: the U.S., Europe, Middle East and Africa, Canada, Latin America, North Asia, South Asia, and Australia & New Zealand. Global imprints include Longman, Scott Foresman, Prentice Hall, Addison-Wesley, Prentice Hall-Financial Times, Markt & Technik, CampusPress, Direct English and Éditions du Renouveau Pédagogique. (http://www.pearsoned.com/globalPub.htm) Many of the Pearson texts have electronic companions as CDs or websites; both open sites, Blackboard and WebCT are used. But the text remains the primary market commodity and the electronic materials are just enhancements. In order to get similar models operating with purely electronic courseware, what is needed? Is it just a question of organising electronic resources better? Or are there deeper issues about the use of electronic media in education? We are now in the position that there has been a great deal of development of electronic educational resources in universities world wide. This has occurred together with substantial investment of IT systems and infrastructure. However, there is little evidence of dissemination of these electronic resources and practices. Greater collaboration and sharing of resources is becoming an increasingly urgent issue. There are several existing databases of computer facilitated learning 74 Australian Journal of Educational Technology, 2003, 19(1) (CFL) materials, but these databases do not appear to have significantly increased the take up of CFL materials and strategies. In order to make the most of the valuable resources which exist, a range of educational, technological and management factors needs to be addressed. Let’s look briefly at one study which examined adoption of CFL across Australian universities. What major issues facilitate or mitigate against adoption of CFL? A recent national study in Australia (McNaught, Phillips, Rossiter & Winn, 2000) examined this issue. The study used a multi-method approach, employing online surveys of institutional practice (28 Australian universities responded); a literature survey; and a case study of five universities at project, faculty and institutional levels. The data included survey results, interview transcripts, focus group transcripts, institutional documentation, and short descriptions or vignettes. This study found that the issues surrounding the adoption of CFL at university are complex, and no single factor will result in adoption. Instead, there is a range of factors, all of which must be addressed. Several universal factors in relation to widespread use of CFL were identified: • coherence of policy across all levels of institutional operations and specific policies which impact on CFL within each institution; • intellectual property, particularly the role of copyright in emerging online environments; • leadership and institutional culture; • staff issues and attitudes: namely, professional development and training, staff recognition and rewards, and motivation for individuals to use CFL; and • specific resourcing issues related to funding for maintenance or updating of CFL materials and approaches, staff time release and support staff. While it is clear that many universities are actively engaged in producing CFL resources to enhance the educational offerings they have, the information about existing CFL resources at Australian universities is patchy and incomplete. In particular, during the case study interviews, it was noted frequently that there is limited or no information across the sector in general about: • the educational design of the CFL resources being produced, • the incentives and support that exist for individuals to produce CFL resources, McNaught, Burd, Whithear, Prescott and Browning 75 • the technical design and access specifications for using these CFL resources, • the experience of using the CFL resources in actual teaching contexts, • evaluations carried out to determine how educationally effective these resources have been in practice, • intellectual property and copyright issues which might affect the use by others, and • how access can be obtained to these CFL resources from either colleagues in the same university or another institution. How much will access to information about electronic material assist adoption and reuse? The implication above is that good information is the key. The plethora of current work on metadata has a similar implicit message. No doubt it is important, but maybe it is in the category of ‘necessary but not sufficient’. The two cases chosen here illustrate two key issues about educational reuse. The first is a case where there are a large number of useful electronic resources but the issue is how well existing known resources might support educational purposes for a particular program or course. The second case examines how difficult it is for the momentum of a collaborative consortium, dedicated to sharing and reuse, to be maintained after the initial flush of enthusiasm fades. 1. How does one find, select and organise educationally appropriate websites? The Chinese University of Hong Kong is a bilingual University but the official medium for teaching in the Medical Faculty is English. In this example we explore the value of the Internet in assisting learning for final year Medical students during their attachment to the Division of Plastic and Reconstructive Surgery. The Internet is an educational resource but the volume of information available limits its value. Many of the websites relating to plastic surgery are of high visual quality (Van Heijningen, Mannaerts, Blondeel & Spauwen, 1998) but are these sites the most appropriate to integrate into a medical curriculum (Prater & Smith, 1989)? In this short study we aimed to see if we could provide some insight into how to optimise the educational value of the Internet for undergraduates studying plastic surgery, as this is an area of significant interest in medical education (Shazly, Mohamed & Maiwald, 2000). Commercial search engines were used to identify plastic surgery websites in July 2000. One year later all sites were revisited. Those no longer available were excluded and the remaining sites were scored according to content, complexity, technical difficulty and readability using the Flesch- 76 Australian Journal of Educational Technology, 2003, 19(1) Kincaid Grade Level Score (e.g. Johnson, 1998). This readability index computes readability based on the average number of syllables per word and the average number of words per sentence. The score in this case indicates a grade school level. For example, a score of 8 means that an eighth grader would understand the document. Standard writing approximately equates to the seventh to eighth grade level. The maximum score is 10. Two hundred and five sites were identified in July 2000 and one year later 169 were still online. Less than 10% of sites scored 8 and above, 60% scored 4 and below. There is clearly a wide range of material available on the Internet but the question remains how to identify the sites with the greatest educational value, especially for students for whom English is a second language? Is reading ease a key factor in a visually rich environment? Are the objectively high scoring sites the best? In order to begin to answer these questions, we asked 150 students to evaluate these 169 Internet sites according to their subjective perception of educational value, including language and content. Students related to the sites according to their own personal judgments of their educational interests and needs. They were not given specific criteria but ranked them on subjective impressions of their value to themselves at the time of ranking. The rankings from the objective (reading ease) and the subjective (personal interests and needs) scoring did not correlate. Clearly the reading ease of the text was not a key decisive factor in deciding students’ preferences. While this is not a rigorous study, it has highlighted the wide variety of student preference about the style of web material they prefer. We are left with a wealth of material but no clarity at this point about how to organise and frame this material so as to support individual student learning styles (Coates & Rowsell, 1998). How does one design the basic shell which points to these sites? How do we order and groups these sites? Certainly a list of 169 URLs is not likely be an educationally engaging site! Should detailed descriptions be added, pointing out a range of ways in which the material on each site might be used? Should we add questions which can be answered by searching a number of sites? Should we use these sites as a basis for case study questions? Some or all of the above? All we know at this stage is that an enormous amount of work will be needed before this potential wealth of the Internet can be realised for the benefit of medical students studying plastic surgery. And we are unclear about how to find the time and technical resources to do this work. This case shows clearly that good educational design is one important element in integrating existing electronic resources into a coherent learning resource for students. Knowing about resources and having access to them is important, but knowing what to do with them in order to facilitate learning provides another set of complex challenges. McNaught, Burd, Whithear, Prescott and Browning 77 2. If it’s too hard for one group to search, evaluate and design, can a consortium work better? This story begins with an innovation in the teaching of veterinary microbiology in the early 1990s at the University of Melbourne. The original VetBac project began with second year veterinary students (in their third year of university studies) studying Systematic Bacteriology and Mycology. The project set out to use computer technology to overcome some of the current problems of course presentation and integration in overcrowded veterinary curricula. It involved the development of interactive, multimedia databases of veterinary curriculum materials, linked to a revised set of educational objectives, a range of other learning situations and more appropriate assessment procedures. The expanded set of learning objectives required students to acquire information management and problem solving skills, as well as understanding bacterial and fungal diseases of animals. The development of group learning skills and communication skills was also explicitly built into the course. The curriculum rationale is explained in detail in Whithear, Browning, Brightling and McNaught (1994). The short term evaluation findings were positive: students developed including better information management skills, improved higher order cognitive skills, increased interest in the subject, better collaborative learning skills, better written and oral communication skills and better computer skills (McNaught, Whithear & Browning, 1994). International interest in this problem based learning (PBL) approach grew. In January 1996 an international consortium of over 30 veterinary bacteriology and mycology teachers from Australia, Canada and the USA was formed to provide new online materials for veterinary courses. One of the main focuses in the formation of this ‘consortium’ was the emphasis on learning microbiology as a tool for preparing professional veterinarians, rather than learning microbiology for its own sake. PBL was to be a foundation principle for all projects. As a pilot, a range of materials was produced to allow members of the international consortium to comment on issues such as content and style. In order to gauge student response to these initial VetSource materials, five students were interviewed in September 1997. Student response to the initial VetSource materials was very encouraging (McNaught, Whithear, Browning, Hart & Prescott, 1998). All this is very positive, but two years later the consortium had petered out. Only three universities actually provided materials and this is not enough to build the foundation for a true multi-institutional consortium. The funding that the consortium hoped to obtain through the US Department of Agriculture did not eventuate. The University of 78 Australian Journal of Educational Technology, 2003, 19(1) Melbourne was building a web based platform that would be highly suitable for the interactive nature of the VetSource modules (Goschnick, 1998). Despite a public launch of Creator (as the platform was called), funding for completion of the project was withdrawn. The lack of buy-in in terms of producing materials, the lack of a technical platform and the limited funding took its toll. The few VetSource modules were incorporated in the VetBac project at the University of Melbourne, which still continues, but these materials were not used by any of the other consortium members. The model of VetBac has influenced the development of other computer facilitated PBL veterinary work at the University of Melbourne (McNaught, Whithear & Browning, 1999) but the scale of operation has a much more limited scope than we hoped in the mid-1990s. In this story there were three barriers—funding, a technical platform and insufficient production of the modules. A conclusion we might draw about why there was insufficient production of the modules is that the work environments in the universities in the consortium did not support the time and energy these academics needed to devote to making the consortium a sustainable entity. The models of how to do PBL exist; the willingness to engage can be kindled; but the work environment of higher education needs to be more supportive towards the time needed for innovation and collaboration. So, what is needed to further the effective reuse of electronic media content? The two cases above highlight the need for: 1. effective educational design in combining a number of resources designed elsewhere, and 2. policies and processes which support the culture of collaborative work environments, and enable the recognition of this form of work. 1. What is meant by effective educational design in this context? Unpacking all the aspects of effective educational design is beyond the scope of this paper. Two aspects that might relate particularly to design with the idea of reuse in mind are: • improving the understanding of how electronic resources might work with other design elements of the student learning experience; and • the development of generic designs. McNaught, Burd, Whithear, Prescott and Browning 79 Table 1: Examples of online strategies to fulfill students’ learning needs (McNaught, Kennedy & Majoor, 2002) Student learning need (after Open University, 1998) Examples of appropriate use of online strategies Building and maintaining motivation • engaging scenarios • clear and current information*; • early diagnostic feedback. Negotiating choices • clear and current information*; • topic choice based on graduate capabilities; • assessment choice; • discussion threads for negotiation; • evaluation forms for two way feedback. Information handling skills • web searching for online resources; • use of databases; • online sharing of resources; • FAQ sites and/or glossary. Independent learning skills • negotiated assignments; • access to IT/info literacy & study skills resources; • use of personal online journals. Developing understanding • building websites; • collaborative tasks (document sharing, specialist chat groups); • problem solving exercises – from simple quizzes (for formative assessment) to the use of databases linked to case studies; • concept mapping software; • a collection of past/recent exams and sample tests (where appropriate). Linking theory to practice • embedded media and simulations; • online tutorials; • students as mentors in online groups; • samples of previous assignments/project work (with documented student permission). Practising discussion, argument, articulation of ideas • online tutorials; • online debates using a threaded discussion; • role playing using a threaded discussion; • sharing essays online. Rehearsing skills and procedures • regular and frequent online quizzes with feedback; • interactive activities using spreadsheets, multimedia objects, laboratory notes and guides. Practising teamwork • group projects progressively sharing resources and annotating peer work using collaboratory software. 80 Australian Journal of Educational Technology, 2003, 19(1) *Clear and current information includes items such as: • direct access to the related approved subject or course guide; • a current timetable/timeline related to outlining face to face tutorials, lectures, laboratory/ field work and online activities (with times, dates and location details); • a map showing online learning activities clearly described/linked to learning outcomes; • current contact details of lecturers, teachers and tutors; • lecture outlines; and • news announcements. Improving the understanding of how electronic resources might work with other design elements of the student learning experience Several people are working in this area. One model is given by McNaught, Kennedy & Majoor (2002). They describe a model of how different student learning needs can be supported by the various functions of and strategies for online learning. This model will hopefully further the understanding of how the various components of an online learning site, such as information areas, interactive tutorials, quizzes, and access to threaded discussions and chat can support learning for more students most effectively (Table 1). This may make it easier for teachers to see how a variety of electronic materials they have collected (such as the plastic surgery resources described earlier) might be used in designing a coherent student learning environment. The development of generic designs There is certainly a major interest in whether it is possible to develop generic designs. Generic designs can be described as a combination of clearly articulated learning designs combined with templates into which media elements can be inserted. The Australian University Teaching Committee (AUTC) has funded a project titled: ‘Information and Communication Technologies (ICTs) and their role in flexible learning’ The project is a two year endeavour and began in November 2000. The project’s aims as stated at the end of 2001 were: 1. The identification of a range of learning designs that have been demonstrated to contribute to high quality learning experiences and which can be applied generically; 2. The design and subsequent development of a series of re-usable software, templates and/or exemplars for the learning designs previously identified; and 3. The development of a series of guidelines for good practice in the use of or implementation of the software, templates and/or exemplars in new contexts. McNaught, Burd, Whithear, Prescott and Browning 81 Crucial to the success of this project is the development of an evaluation instrument with a twofold purpose: 1. To facilitate the identification of learning designs that foster high quality learning experiences; and 2. To provide a mechanism to determine whether such learning activity designs have the potential for re-development in a more generic form. (Harper, Oliver & Agostinho, 2001, p. 255) It is hoped that the templates produced will enable university teachers to embed existing electronic media content into a learning design that is appropriate to their own needs. At this stage the outcomes of the project are not known, nor is the response that university teachers might have to any generic designs produced. The range of generic designs needs to cater for a large range of diversity in higher education - in students, in the range of learning contexts they engage in, and in teachers’ approaches to and beliefs about teaching and learning. But even if this project enables some measurable reuse of electronic objects to occur in a variety of learning designs, a significant advance will have been made. 2. What policies and processes are needed to support the culture of collaborative work environments to foster adoption and reuse? This is a very difficult question and, again, beyond the scope of this short paper. As stated earlier (McNaught et al., 2000), work is needed on several interrelated aspects, two of which have been mentioned in this paper: • provision of access to information about CFL resources; and • policies and processes to enable collaborative work, both within and across institutions. There are obvious financial resourcing implications but we will not comment on these here. Access to information about CFL resources While we have said this is not the determining factor in the use and reuse of CFL, it is important, and the current work being done on metadata and interoperability standards needs to be followed so that Internet architecture is developed with learning in mind. A good site to keep on eye on is the Australian IMS project site, hosted by EdNA Online at [http://ims.edna.edu.au/] It provides information on metadata standards and projects in Australia and elsewhere. Though not recently updated (most of the pages are August 2000), it does provide contact points for further searching. 82 Australian Journal of Educational Technology, 2003, 19(1) AustLit, [http://www.austlit.edu.au/] (Australian Literature Gateway) and UniServe Science [http://science.uniserve.edu.au/] are examples of well-known information points and serve as continuing models of provision of access to relevant information and resources. In addition, UniServe Science organises events such as national symposiums in order to enact a role of fostering “a sense of community among tertiary teachers of science” (Johnston & Peat, 2002). A recent Australian survey by the Department of Education, Science and Training (DEST) (Bell, Bush, Nicholson, O’Brien & Tran, 2002) investigated the extent of the use of online technologies by Australian universities. Fifty-four percent of university units now include an online component. The report describes the nature of online courses, what disciplines they are in, and how they are supported by online services. This is useful information and may hopefully be the beginning of more detailed sharing of information about online resources being used in Australian universities. Policies and processes to enable collaborative work, both within and across institutions Just how much activity is there in this area? Just how high up on the priority list for our governments is this type of policy item? If it is seen as a having a high priority at government level, it is much more likely to get attention at institutional level. Here we will focus on the Australian government. In order to get a rough idea of the number of studies and reports in the area of collaborative policy in higher education commissioned by the Australian government in recent years the Australian Department of Education and Science (DEST) website was searched using InfoCat. The InfoCat (‘Information Catalogue’ http://infocat.dest.gov.au/IE/) covers most research and statistical information published by DEST since 1995, with some information for work done prior to 1995. EdNA Online (http://www.edna.edu.au/) was also searched looking for Australian sites. The search words and results are shown in Table 2. While this is hardly an exhaustive or rigorous search, it demonstrates two important points. 1. There is little development of policy which actively fosters collaboration in Australian universities at the present time. Without such policy a culture of sharing and reuse is unlikely to flourish. 2. Examples of collaboration can be found. It is important that where these educational collaborations exist, they are carefully evaluated to see whether and how such activities lead to increased sharing and reuse of any educational electronic resources. McNaught, Burd, Whithear, Prescott and Browning 83 Table 2: Results of searches on Australian DEST and EdNA document collections ‘All words’ search on 23 July 2002 Site and no. of results Comment Collaboration policy DEST 5 Four of these are research related, rather than related to teaching and learning policy; the fifth is McNaught et al. (2000), which is largely an analysis of existing issues. Collaboration process DEST 6 One is research related, one is a 1992 curriculum document, one is about academic writing support; one is an equity report, and the fifth is on youth issues. The final is Taylor & Richardson (2001), which is a proposal for peer review of ICT resources. If implemented (and there is no certainty about this), this proposal could support reuse. Collaboration policy EdNA Australian sites. 20 out of 382,331 in the collection Eight of these are focused more on research, industry or the general community. The other 12 do provide examples of organisations/ networks which illustrate collaborative policy. Collaboration process EdNA Australian sites. 19 out of 382,331 in the collection Eleven of these relate to teaching and learning in some way (the others relate more to research or industry); a couple of examples of actual collaborative processes exist here. One government initiative in Hong Kong is worth sharing because of its explicit recognition of the importance of inter-institution collaboration. In Hong Kong there are eight higher education institutions (see http://www.ugc.edu.hk/english/fund_inst.html [verified 5 Feb 2003]), each with a distinctive character. To date, there have been three rounds of Teaching Development Grants which are awarded on a competitive basis. These are substantial grants in the range $HK1-10M. The criteria on which grant proposals are judged are impact (60%), outcomes (20%), collaboration (10%) and alignment with institutional goals (10%). Within the collaboration category, the following scale is used, 5. The project involves two or more institutions, all of which have contributed to the project design and will have a significant and well- defined role in the project’s implementation and resulting benefit. 4. The project involves two or more institutions, all of which have contributed to the project design, have a significant and well-defined role in its implementation, although the benefit for involved institutions is not equivalent. 84 Australian Journal of Educational Technology, 2003, 19(1) 3. The project was designed by the submitting institution, but more than one institution will have a significant and well-defined role in its implementation, but benefit for all participants will not necessarily be equivalent. 2. Individual staff members from outside the submitting institution appear to have contributed to the project design, but only one institution has a significant role in its implementation and will in all likelihood be the prime beneficiary of input upon completion. 1. The project involves individual staff members from outside the submitting institution, but they appear to have only a pro forma involvement and thus benefit to their institution is questionable. 0. The project does not involve more than one institution and benefit is restricted to the submitting institution. While this might mean that some ‘surface’ collaborations are formed at grant writing time, this policy has resulted in several real collaborations where resources are produced and shared more widely. Summary Higher education faces a major challenge in improving the quality and cost effectiveness of online materials by reuse of electronic media content. The success to date has been very limited. In this paper we have tried to reflect on the experience of two projects in order to better understand the issues about reuse. Solutions to this problem will include addressing issues in diverse areas of higher education and here we have commented on two very different areas – specific issues of education design, and policy at sector and institution level. References Australian Department of Education, Science and Training (DEST). http://www.dest.gov.au/ [23 July 2002] Australian IMS project. http://ims.edna.edu.au/ [23 July 2002]. Australian Literature Gateway. http://www.austlit.edu.au/ [23 July 2002] Bell, M., Bush, D., Nicholson, P., O’Brien D. & Tran, T. (2002). Universities Online: A survey of online education and services in Australia. Department of Education, Science and Training Higher Education Group, Occasional Paper Series 02-A. http://www.dest.gov.au/highered/occpaper/02a/02_a.pdf [23 July 2002, verified 5 Feb 2003] Coates J., & Rowsell, A. (1998). Plastic surgical training and the Internet: How plastic surgeons learn. British Journal of Plastic Surgery, 51, 74-77. McNaught, Burd, Whithear, Prescott and Browning 85 EdNA Online (Education Network Australia). http://www.edna.edu.au/ [23 July 2002] Goschnick, S. G. (1998). Design and development of Melbourne IT Creator TM - A System for authoring and management of online education. http://www.solidsoftware.com.au/Information/Paper/Tools98/Creator.html [23 July 2002, verified 5 Feb 2003] Harper, B., Oliver, R., & Agostinho, S. (2001). Developing generic tools for use in flexible learning: A preliminary progress report. In G. Kennedy, M. Keppell, C. McNaught & T. Petrovic (Eds.), Meeting at the Crossroads. Proceedings of the 18th Annual Conference of the Australian Society for Computers in Learning in Tertiary Education. (pp. 253-262). Melbourne: Biomedical Multimedia Unit, The University of Melbourne. [23 July 2002, verified 5 Feb 2003] http://www.ascilite.org.au/conferences/melbourne01/pdf/papers/harperb.pdf InfoCat (DEST ‘Information Catalogue’). http://infocat.dest.gov.au/IE/ Johnson, K. (1998). Readability. [23 July 2002, verifed 5 Feb 2003] http://www.timetabler.com/readable.pdf Johnston, I., & Peat, M. (2002). Scholarly inquiry and flexibility. In Scholarly inquiry in flexible science teaching and learning, (pp. 1-2). Proceedings of Symposium, 5 April. UniServe Science: The University of Sydney. [verified 4 Feb 2003] http://science.uniserve.edu.au/pubs/procs/wshop7/schws001.pdf McNaught, C., Kennedy, D. & Majoor, J. (2002). Designing online learning sites to cater for learners’ needs. In Proceedings of the International Conference on Computers in Education (ICCE 2002). Auckland, New Zealand, 3-6 December. McNaught, C., Phillips, P., Rossiter, D. & Winn, J. (2000). Developing a framework for a usable and useful inventory of computer-facilitated learning and support materials in Australian universities. Evaluations and Investigations Program report 99/11. Canberra: Higher Education Division Department of Employment, Education, Training and Youth Affairs. [verified 5 Feb 2003] http://www.detya.gov.au/highered/eippubs1999.htm#99_11 McNaught, C., Whithear, K. & Browning, G. (1994). The role of evaluation in curriculum design and innovation: A case study of a computer-based approach to teaching veterinary systematic bacteriology and mycology. In K. Beattie, C. McNaught & S. Wills (Eds), Interactive multimedia in university education: Designing for change in teaching and learning, (pp. 295-308). Amsterdam: Elsevier. McNaught, C., Whithear, K. & Browning, G. (1999). Systems not projects: Focusing on evaluating the overall student experience, rather than isolated innovations. Higher Education Research & Development, 18(2), 247-259. McNaught, C., Whithear, K., Browning, G., Hart, G. & Prescott, J. (1998). The best of both worlds: Redeveloping a multimedia project for the web. In T. Ottmann & I. Tomek (Eds), Proceedings of Ed-Media & Ed-Telecom 98. 10th World Conference on Educational Multimedia and Hypermedia and World Conference on Educational Telecommunications, volume 1, (pp. 946-951), Freiburg, Germany, 20-25 June. Open University (1998). Technology Strategy for Academic Advantage. [23 July 2002, verified 5 Feb 2003] http://www2.open.ac.uk/ltto/tsaa/index.htm 86 Australian Journal of Educational Technology, 2003, 19(1) Pearson Education Global Publishing. http://www.pearsoned.com/globalPub.htm [23 July 2002] Prater, M. A. & Smith, D. J. (1989). Determining undergraduate curriculum content in plastic surgery. Plastic Reconstructive Surgery, 84, 529-533. Shazly, E. L., Mohamed, M., & Maiwald, G. (2000). The Internet: A new friend to plastic surgeons. Plastic Reconstructive Surgery, 106, 235. Taylor, P. G. & Richardson, A. S. (2001). Validating scholarship in university teaching. Constructing a national scheme for external peer review of ICT-based teaching and learning resources. Evaluations and Investigations Program report 01/3. Canberra: Higher Education Division Department of Employment, Education, Training and Youth Affairs. [23 July 2002, verified 5 Feb 2003] http://www.dest.gov.au/highered/eippubs/eip01_3/01_3.pdf UniServe Science. http://science.uniserve.edu.au/ [23 July 2002] Van Heijningen, R. I., Mannaerts, G. H. H., Blondeel, P. H. N. & Spauwen, P. H. M. (1998). PLink, Plastic surgery and the Internet. British Journal of Plastic Surgery, 51, 86-89. Whithear, K. G., Browning, G. F., Brightling, P. & McNaught, C. (1994). Veterinary education in the era of information technology. The Australian Veterinary Journal, 71, 1-3. This article was nominated for an Outstanding Paper Award at ASCILITE 2002, gaining the additional recognition of publication in AJET (with minor corrections). The reference for the Conference version is: McNaught, C., Burd, A., Whithear, K., Prescott, J. and Browning, G. (2002). It takes more than metadata and stories of success: Understanding barriers to reuse of computer facilitated learning resources. In A. Williamson, C. Gunn, A. Young and T. Clear (Eds), Winds of Change in the Sea of Learning: Proceedings of the 19th Annual Conference of the Australasian Society for Computers in Learning in Tertiary Education, pp451-460. Auckland, New Zealand: UNITEC Institute of Technology. Carmel McNaught, Centre for Learning Enhancement and Research, Chinese University of Hong Kong, Hong Kong. carmel.mcnaught@cuhk.edu.hk Andrew Burd, Department of Surgery, Chinese University of Hong Kong, Hong Kong. andrewburd@cuhk.edu.hk Kevin Whithear, Faculty of Veterinary Science, The University of Melbourne, Australia. k.whithear@vet.unimelb.edu.au John Prescott, Ontario Veterinary College, University of Guelph, Canada. prescott@ovcnet.uoguelph.ca Glenn Browning, Faculty of Veterinary Science, The University of Melbourne, Australia. g.browning@vet.unimelb.edu.au