Understanding the development of teaching and learning resources: a review Louise Plewes and Kim Issroff University College London (UCL) email: k.issroff@ucl.ac.uk, louise_plewes@hotmail.com This paper is a literature review of research concerned with the production of learning resources in higher education (HE). It forms part of a larger research project in progress. We identify and discuss six models which may be used to understand the development and reuse of learning resources. We focus specifically on the level at which the models operate, their purpose, structure and content and critically analyse the advantages and disadvantages associated with their application. Aspects which the models do not address, for instance actual staff practices and the cultural context of development, are then discussed. We focus on understanding academic staff practices and behaviour in learning resources production. Staff time is a major cost element in the production of learning resources. Despite recent international work on costing methodologies for HE and their application to networked learning resources (for example, Bacsich and Ash, 2000; DETYA, Ernst and Young, 2000; Flashlight, 2002; Jones, 2001; NBEET, 1994), there is little documentation of the use of staff time in the production of learning resources for face- to-face campus-based teaching. This paper is not concerned with costing methodologies, rather we are interested in the non-economic factors which influence the everyday practice of staff in the learning resources production process. We argue that the latter will enable us to understand better the needs of staff for support in the production of learning resources. A phenomenographic approach is needed to understand the everyday practices of staff and factors influencing the learning resources production process. Introduction In the next five to ten years there will be a sector-wide need in HE to produce a wider range of teaching and learning materials, for example, more resource-based learning (RBL) materials for self-directed study. The context for this is rapid technological advancement, 4 ALT-] Volume 10 Number 2 increasing numbers and diversity of students and a decline in the unit of resource (NCIHE, 1997). In particular, networked learning resources are increasingly presented as a cost-efficient means of maintaining teaching quality (NCIHE, 1997). However, the relationship between cost-effectiveness and quality is contested. This future scenario has significant implications for university staff, whose time represents the largest cost element in the production of learning resources (Chiddick, Laurillard, Quigley and Wolf, 1997). Some commentators (such as Noble, 1998) present a pessimistic view of the future status of university staff given the increasing automation of teaching and intrusion of commercial interests. The increase in student-centred learning and use of technology may also lead to a significant change in the role of the teacher, from the 'sage on the stage' to 'guide on the side' (Jones, 1999; Salmon, 2000). An extensive survey of the literature has identified few studies which specifically investigate the production of learning resources in campus-based higher education institutions (HEIs). We suggest that the production of teaching and learning resources has not been considered a legitimate research topic, since it is an implicit and routine aspect of academic staff duties. Therefore, there have been few in-depth studies which record everyday accounts of learning resources production and the factors affecting this. Firstly we introduce the background to the research project, of which this paper is one part. We then critically review six models of the learning resources production process and its organization. Finally, we suggest that a phenomenographic account of the everyday practices of learning resources production is needed. Background to the research project The review presented here is part of a research project funded by the Higher Education Funding Council for England (HEFCE)'s Teaching Quality Enhancement Fund (TQEF). The project aims to support the implementation of the institution's Learning and Teaching Strategy, which aims to promote effective student learning. For a national overview of Learning and Teaching Strategies see Gibbs, Habeshaw and Yorke (2000) and HEFCE (2001a; 2001b). The research project will develop an understanding of the needs of academic staff for support in the production of a variety of teaching and learning resources, including networked learning resources, for use in undergraduate teaching across all subject disciplines within the university (Plewes and Issroff, 2002). The project will make recommendations about possible future structures for learning resources production at the university, including changes to the support and training infrastructure. Literature review In this section we critically review six prominent models which document and cost the learning resources production process. We focus specifically on two aspects: firstly, the level at which the analysis operates - for example, institution, course, faculty or department, member of staff, or student; secondly, the levels at which the costs of production of course materials are experienced. We also consider the purpose, structure and content of the models, critically analyse the advantages and disadvantages of their applications and issues arising from this. We discuss: 5 Louise Plewes and Kim Issroff Understanding the development of teaching and learning resources: a review • Open University (OU) Course Materials Production Models (Rumble, 1976; Bates, 1994); . • Costs of Networked Learning Course Lifecycle Model (Bacsich, Ash, Boniwell, Kaplan, Mardell and Caven-Atack, 1999); • The Pedagogic Toolkit Model (Oliver and Conole, 1999); • Cost Structures of Teaching Methods Model (NCIHE / Chiddick et al., 1997); • Student Preferences / Consumption of Learning Resources Model (Hobbs and Boucher, 1997; Boucher, 1998), and • The Course Resource Appraisal Model (Laurillard, 1999). Table 1 presents summary information for the models reviewed. Model Level of analysis Level at which costs are experienced Process of application and variables analysed Open University (OU) Faculty and Course Materials course Production Models Institution (specifically distance learning) Course team, specialized division of labour: Staff time. Costs of Networked Learning Course Lifecycle Model Course over full lifecycle. Activity based- costing can be applied at multiple levels. PedagogicToolkit Model Multiple levels, Institution e.g., session, week, term, course Cost Structures of Course Institution Teaching Methods Model Student Preferences / Course Institution Consumption of Learning Resources Model Institution. Also identifies Accounting method activity- hidden costs absorbed by based costing, stages of course staff and students. lifecycle. Input teaching activities / media, allocate student time between media, look at staff time costs and resources needed. Based on Laurillard's (1993) Conversational model. Teaching methods (large and small group, external and in-house RBL), student time between these, staff time to prepare and present for each of these methods. Economic cost curve analysis (graphical). Amount of teaching delivered for fixed cost while incorporating student preferences. 6 ALT-J Volume 10 Number 2 Model Level of analysis Level at which Process of application and costs are experienced variables analysed Course Resource Course Institution Activities map on to media and Appraisal Model technologies, student time split between media, workload of three groups of O U staff to prepare and deliver teaching. Table I: Summary information for the models reviewed Open University (OU) Course Materials Production Models As described previously, there is a lack of published literature describing the course materials production process in campus-based HEIs. However, there is an established literature on the course material production process for distance learning, in particular the UK Open University (OU) (Rumble, 1976; 1992; 1997; Bates, 1994; 1995; 2000). Recent developments in online distance education and virtual or 'mega-universities' (Daniel, 1996) also build upon this OU model. We therefore use this model as a starting point for our review. However, since the OU is not representative of most HEIs, the conclusions may not be readily transferable to other contexts. OU course materials are generally developed by a course team of thirty people, who meet regularly. The team includes subject experts, educational technologists, a BBC producer, designers, editors and a course assistant. Since there can be several drafts, the development and approval of course materials is time-consuming (for example, 18-20 months; Rumble, 1976), and therefore also expensive (£450-500 for one hour of printed study materials in 1988; Bates, 1994). Such a course production system can only be justified by producing materials for large numbers of students (some OU science foundation courses have student numbers of 30,000; Bates, 2000), and/or by using the same materials for many years. Typical course life is 4-8 years (Rumble, 1976). Course materials are maintained by one to two members of the course team over the course life, at the end of which they are remade or replaced with new materials (Rumble, 1997). This type of mass, industrialized production with a highly specialized division of labour (Lewis, 1971a, 1971b; Rumble, 1992; Fames, 1993) is specific to the OU and is not common in campus-based HEIs where learning resources production is more of an ad hoc 'cottage industry' (Fames, 1993; Bates, 1997; Peters, 1998). Oliver, Bradley and Boyle (2001) have recently described the organizational and pedagogical difficulties in a collaborative, iterative approach to the distributed authoring of course materials for a virtual university. This builds upon the OU course materials production process described above, and again includes a highly specialized division of labour (Peters, 1998). Rumble (1997) has developed a weighting technique which expresses the staff time required to develop OU course materials. The unit of analysis is the faculty, but there is some inter- faculty variation which reflects disciplinary differences in the types of learning resources developed (e.g. Becher, 1989; Smeby, 1996; Neumann, 2001). Sufficient course materials to keep the average learner studying for 10-12 hours per week are termed 1 unit. The ratio of course development time to course maintenance time is 10:1. That is, staff could maintain 7 Louise Pfewes and Kim Issroff Understanding the development of teaching and learning resources: a review ten units of existing materials in the time taken to develop one new unit from scratch. Where staff act as consultants and contribute to the development of units by other people this is weighted at 0.5 of a new unit. Rumble (1976) and Wagner (1977) have also conducted macro-analyses of staff time taken to develop course materials at the level of the whole institution. Rumble and Wagner have produced formulae to calculate the total cost of course material production to the institution. This is a function of the number of courses in development and presentation, student numbers, average cost of a course in development and presentation, average delivery cost per student, institutional overhead costs (fixed) and average course life (Wagner, 1977). The purpose of this analysis was to inform an ongoing debate about the comparative cost of educating one student by the OU rather than a traditional HEI. Course Lifecycle Model The JlSC-funded 'Costs of Networked Learning' Project (Bacsich, Ash and Heginbotham, 2001; Bacsich et al, 1999) developed a three-stage cyclic course lifecycle model and advocated the use of activity-based costing (ABC) methods to investigate the costs of networked learning. The purpose of this model is to make bidden costs explicit and to promote the use of a standard costing methodology for the HE sector. The three stages of the course model are: planning and development; production and delivery; maintenance and evaluation. The model is based on course planning frameworks from the distance education sector (see previous section), designed to be equally applicable to both traditional and electronic learning resources. The model considers the course in the full context of pre-course (R & D) and post-course (evaluation) activities. ABC is an accounting method where the cost of a product or service is determined by the activities involved in its production, not by volume-related allocation of overheads such as staff hours. ABC methods may be applied at a number of different levels, such as department, faculty or institution (Bacsich et al., 2001); Application of the model involves use of a nested set of spreadsheets to examine costs at a variety of levels and to identify which of three stakeholders (the institution, staff or students) meets these costs. The student perspective is rarely considered, although the Hobbs and Boucher (1997) model considers student preferences (see later section). However, this model does not investigate the non-economic issues surrounding learning resources production. Also, specialist training and software are required before the ABC methods promoted may be used. Therefore, this approach may not be accessible to those without training in ABC methods. The results, however, should be comprehensible to all university staff and the standard costing methodology should enable comparisons to be made. The Pedagogic Toolkit Model This project produced an online course design toolkit (Oliver and Conole, 1999), designed to operate flexibly at a variety of levels - session, week, term and course. The toolkit enables teachers to input teaching activities and educational processes and to plan and cost the course development process. Although targeted at implementation of learning technology into courses, the toolkit can also be used for traditional teaching methods and resources. The toolkit has three elements. Firstly, 'media rater' looks at the teaching activities to be used. 8 ALT-J Volume 10 Number 2 Secondly, 'course modeller' allocates student time between these teaching activities and finally, 'media selector' considers the costs of staff time and resources required to support the course. Estimates of the time taken to develop various types of materials from scratch are presented, although there is no indication of how these data were derived. The toolkit is based on Laurillard's (1993) 'conversational framework' which also informs two other models discussed here (Chiddick et al, 1997; Laurillard, 1999). The toolkit is easy to use, encourages staff to reflect on the mix of media and teaching activities in their courses and to consider alternatives in terms of potential implications for their workload. Cost Structures of Teaching Methods Model NCIHE (1997) noted the recent shift in UK HEIs from a mixture of small-group teaching and lectures towards increased use of RBL materials for self-directed study. The implications of this scenario for staff workloads and costs to institutions were examined in Appendix 2 of NCIHE (1997) by Chiddick et al. (1997). Chiddick et al. (1997) analyse how a student's time might be distributed across three combinations of teaching methods each based on different ratios of small group teaching, lectures and RBL. Assumptions are made about staff time for preparation and presenta- tion required to produce one hour of student learning for each method (Table 2). However there is no indication of the source of these time estimates or how it has been established that these would produce 'one hour of student learning'. Teaching method Preparation time (hours) Presentation time (hours) Total time (hours) I 1.5 0 2 0 20 Table 2: Assumptions of preparation and presentation staff time required to produce one hour of student learning for four different teaching methods (Chiddick et al., 1997) The effect of increasing student numbers on staff time spent on learning resources production is analysed graphically as cost curves for each of the three scenarios. The report recommends the use of fixed-cost (for example, RBL) rather than variable-cost (for example, small-group) teaching methods. Fixed-cost teaching methods are insensitive to increased student numbers and the high production costs can be amortized over large numbers of students. Externally developed RBL material which can be adapted quickly is a more cost-effective option than developing RBL materials in-house, and a shift towards this is promoted (Table 3). Methods Traditional study hours Lectures (max. 100 students) Groups (max. 10 students) RBL (external) RBL (in-house) 3 0 2 20 Lectures Groups RBL (External) RBL (In-house) 30 50 15 5 Current study hours 60 5 15 20 Future study hours 10 30 50 10 Table 3: Alternative combinations of teaching methods (Chiddick et al., 1997,Table I) 9 Louise Plewes and Kim Issroff Understanding the development of teaching and learning resources: a review Chiddick et al. (1997) developed their model in the context of continuing rises in student numbers. However, recent data (HEFCE, 2001c) suggests that large increases in student numbers between 1988 and 1994 (a 67 per cent increase) were followed by a levelling off of demand for full-time undergraduate HE places between 1995 and 2001 (a 6 per cent increase). Also, in practice, the shift away from small-group teaching has not occurred in all HEIs. Some have continued to offer small-group teaching while reducing costs by employing lower unit cost staff (such as hourly-paid postgraduate students). Student Preferences / Consumption of Learning Resources Model Hobbs and Boucher (1997) suggest that student attitudes towards teaching methods should be considered, especially given the increasingly 'consumer-led' nature of HE. Therefore, they develop an economic analysis, similar to that of Chiddick et al, using cost curves but incorporating 'end user preferences', that is, the consumption of learning resources by students. They suggest that according to NCIHE (1997) students prefer traditional, non- technology-based teaching methods such as lectures and small-group teaching, therefore the lowest cost option (RBL) may be unacceptable to students. However, relatively little is known about student preferences and this assumption may be anecdotal. The literature on learning resources in general concentrates on the production of resources by staff, rather than consumption by students, although student feedback on teaching may form part of revised quality assurance procedures. Further research into and explanation of student preferences would therefore be useful. The model examines the amount of teaching that can be delivered for a fixed cost output, rather than considering input (for example, the staff time to deliver a fixed amount of teaching). The analysis focuses on capital rather than labour as the key factor in learning resources production. Increasing use of networked learning resources relies upon the substitution of capital for labour, but it is debatable whether it reduces total costs. Boucher (1998) argues that Chiddick et al. do not adequately define what type of costs, total or average, are considered, and neglect other fundamental economic concepts such as economies of scale, marginal costs and diminishing returns. This model makes no distinction between internally and externally developed RBL materials which both Chiddick et al. (1997) and Laurillard (1999) consider to be fundamental. Hobbs and Boucher suggest that the optimal solution of student preferences coincident with lowest cost requires improvements in the quality of RBL materials. Course Resource Appraisal Model Laurillard's (1999) Course Resource Appraisal Model is a simple framework which tabulates the allocation of student study hours between various learning activities. These relate to different media forms, which in turn map on to particular technologies as outlined in Table 4. Technologies PrintTV.Video, DVD Library, DVD, Web Seminar; Online group Laboratory, Simulation Essay, Product Model Table 4: Course Resource Appraisal Model, after Laurillard (2001) 10 Learning activities Attending Investigating Discussing Practising Articulating Media forms Narrative Interactive Communicative Adaptive Productive ALT-J Volume 10 Number 2 The model looks at how student workload is distributed across various learning activities and media and assesses the implications for the workload of three groups of OU staff, termed academic, production and presentation. In this model the radical change in the role, workload and relations between technical and academic staff when producing electronic learning resources are significant, as discussed below. The production of electronic learning resources is especially labour-intensive in terms of technical staff time used in development. This is the main factor in their expense. For instance, at the OU, changing 20 per cent of course material to electronic learning resources increases academic staff time by 40 per cent but doubles technical staff time compared with production of traditional learning resources (Laurillard, 2001), assuming that all electronic learning resources are created from scratch. This is a specific example for Laurillard (2001); in general the degree of change in workload is dependent on the nature of material developed. An alternative option to creating electronic learning resources from scratch is to use generic, customizable resources. If 60 per cent generic, customizable resources are used, academic staff time increases by only 10 per cent and technical staff time increases by 20 per cent. Therefore it is not a cost-effective use of staff time for an individual to produce their own materials, as they do for traditional learning resources (Laurillard, 1999). In traditional campus HEIs, where staff are unlikely to work with dedicated teams of production staff, the entire burden of resource development and production would probably fall on academic staff. This is confirmed by a national survey of the production of electronic learning resources (HEFCE, 1999), which found that staff were mostly developing their own materials for their own needs. In addition, there are complex cultural reasons why generic, customizable learning resources are rarely used by academics in traditional, campus-based HEIs. While the review so far has mainly considered individual models, we now discuss some common features of the models. All of the models, by definition, are highly abstract and diagrammatic. Several have a strong accounting or economic focus and are concerned with costs. Few of the models explicitly consider staff except as passive variables, and even fewer consider students, with the exception of Bacsich et al. (1999). Most of the models are based on OU course production team methods which, although similar to team production of online distance teaching materials, do not correspond well to learning resources production practices in traditional campus-based teaching. Laurillard's (1993) 'con- versational framework' provides the theoretical basis underlying several of the models (Chiddick et al.,\991; Oliver and Conole, 1999; Laurillard, 1999). Discussion: cultural factors influencing development and reuse of learning resources There are two ways in which staff may acquire learning resources without producing them themselves. These are (i) inter-institutional collaboration in the production of resources and (ii) the use or adaptation of generic, customizable resources (Laurillard, 1999; JISC, 2002). Neither of these options is perceived as attractive by academic staff (HEFCE, 1999), indicating a relatively widespread and entrenched autonomous and decentralized model of resource production. Louise Plewes and Kim Issroff Understanding the development of teaching and learning resources: a review Bates (2000) has coined the term the 'Lone Ranger and Tonto' approach to describe actual practice in the development of learning resources in North American HE. Here the academic ('Lone Ranger') works with or employs an IT-literate graduate student ('Tonto') to develop electronic learning resources (Bates, 1997). However, this practice does not often produce a usable end product as constant revisions to keep pace with technological change are necessary (Bates, 1997). In contrast to the six models reviewed, this approach describes actual practice and should, we argue, be developed further. If, as it seems from the examples given above, academic staff are mostly developing learning resources themselves, why is this the case? HEFCE (1999) identifies a series of well-known 'constraining and enabling factors' to the development of electronic learning resources. These include lack of IT skills, lack of time for IT training, conflicting priorities, management pressure, incentives and rewards, and lack of suitable examples (Tait and Mills, 1999). The opportunity cost of the use of academic staff time is a key issue. HEFCE (1999) suggests that it is not cost-effective to institutions for staff to develop electronic materials. However, inter-institutional collaborative production can reduce development costs to a single institution and spread the risk (Laurillard, 1999). Especially at first-year undergraduate level, where the core curriculum is common to many institutions, there is clear potential for collaboration (Laurillard, 1999). However, at higher levels where courses are research-led, difficulties arise which explain the lack of interest in this activity. The use of commercial resources is also possible. Purchasing generic commercial material is cheaper than in-house development or commissioning (Hunt and Clarke, 1997; Laurillard, 1999), but has tended to be a victim of the 'not-invented-here syndrome' where staff are reluctant to use materials which they feel do not bear their own personal stamp (Laurillard, 1999). However, Hammond et al. (1992; 160) suggest that the 'not-invented- here' label is flawed as it underestimates and undervalues the conventional process by which lecturers prepare and update their material. They refer to textbooks, monographs, the research literature, and their own research and experience for sources of factual information, ideas, representations, and organisation during the preparation of lecture notes, seminars and tutorials. While use of commercial materials saves development time, additional time is needed to evaluate and adapt resources (Laurillard, 1999). Also, there are few commercial resources developed for or targeted specifically at HE, and, conversely, there is little interest by commercial publishers in resources developed by HEIs (HEFCE, 1999). These issues relate to teaching staff reluctance to cede autonomy over teaching content and delivery (Ryan, Scott, Freeman and Patel, 2000). Electronic learning resources in particular raise these issues, given their potential to change radically the nature of academic work and university organization (see Noble, 1998). For example, the trend towards automation and commodification of HE (Noble, 1998) and publicly available online materials may result in increased scrutiny of these materials, and potentially the substitution of materials for teachers (Jones, 1999). 12 ALT-J Volume / 0 Number 2 Conclusions This paper has identified and discussed six models which may be used to understand the development and re-use of learning resources. We have focused specifically on aspects which these models do not address such as actual staff practices and the cultural context of development. There are few in-depth empirical studies of the everyday practices of academic staff in terms of learning resources production. Since these are rarely articulated, recorded or documented there has been little systematic investigation of the ways in which learning resources production is conceptualized. Unless we explore and document these conventional forms of work in the production of traditional learning resources, it is difficult to understand the value attached to them and how the increased use of electronic learning resources represents a threat to these traditional resource production practices. As a result of these concerns, there is a need to develop a phenomenographic account of staff everyday practices in the production of a variety of teaching and learning resources. Phenomenography is an emerging qualitative research method which is increasingly applied in educational research (such as Marton, 1981; Marton, Hounsell and Entwistle, 1984; Entwistle, 1997; Brew, 2001a; 2001b) to study empirically descriptive conceptions of the world around us. This includes the different ways in which people experience, perceive, understand and conceptualize the same phenomena. References Bacsich, P., Ash, C., Boniwell, K., Kaplan, L., Mardell, J. and Caven-Atack, A. (1999), The Costs of Networked Learning, Sheffield Hallam University, School of Computing and Management Sciences, http://www.shu.ac.uk/cnl/report1.html. Bacsich, P. and Ash, C. (2000), 'Costing the lifecycle of networked learning: documenting the costs from conception to evaluation', Active Learning, 8 (1), 92-102. Bacsich, P., Ash, C. and Heginbotham, S. (2001), The Costs of Networked Learning- Phase Two, Telematics in Education Research group on behalf of the School of Computing and Management Science, Sheffield Hallam University, http://www.shu.ac.uk/cnl/report2.html. Bates, A. W. (1994), Costing Distance Education Technologies: Developing a Methodology, Canada, Open Learning Agency. Bates, A. W. (1995), Technology, Open Learning and Distance Education, London: Routledge. Bates, A. W. (1997), 'The impact of technological change on open and distance learning', Distance Education, 18 (1), 93-109. Bates, A. W. (2000), Managing Technological Change: Strategies for College and University Leaders, San Francisco: Jossey-Bass. Becher, T. (1989), Academic Tribes and Territories: Intellectual Enquiry and the Cultures of Disciplines, Buckingham: SRHE/Open University Press. Boucher, A. (1998), 'Information technology-based teaching and learning in higher 13 Louise Plewes and Kim Issroff Understanding the development of teaching and learning resources: a review education: a view of the economic issues', Journal of Information Technology for Teacher Education, 7(1), 87-111. Brew, A. (2001a), 'Conceptions of research: a phenomenographic study', Studies in Higher Education, 26 (3), 271-85. Brew, A. (2001b), The Nature of Research: Inquiry in Academic Contexts, London: Routledge/Falmer. Chiddick, D., Laurillard, D., Quigley, G. and Wolf, D. (1997), 'New approaches to teaching: comparing cost structures of teaching methods', in NCIHE, Higher Education in the Learning Society, Report of the National Committee of Inquiry into Higher Education, 'The Dearing Report', London: HMSO, http://www.leeds.ac.uk/educollncihe/a2_001.html. Daniel, J. S. (1996), Mega-Universities and Knowledge Media: Technology Strategies for Higher Education, London: Kogan Page. DETYA and Ernst and Young (2000), 'A study to develop a costing methodology for the Australia higher education sector', http:llwww.detya.gov.au/highered/otherpub/costing/costing.pdf Entwistle, N. (1997), 'Introduction: phenomenography in higher education', Higher Education Research and Development, 16 (2), 127-34. Fames, N. (1993), 'Modes of production: Fordism and distance education', Open Learning, 8 (1), 10-20. Flashlight (2002), http://www.tltgroup.org/programs/flashlight.html Gibbs, G., Habeshaw, T. and Yorke, M. (2000), 'Institutional learning and teaching strategies in English higher education', Higher Education, 40, 351-72. Hammond, N., Gardner, N., Heath, S., Kibby, M., Mayes, T., McAleese, R., Mullings, C. and Trapp, A. (1992), 'Blocks to the effective use of information technology in higher education', Computers and Education, 18, 155-62. Haywood, J., Anderson, C., Coyle, H., Day, K., Haywood, D. and MacLeod, H. (2000), 'Learning technology in Scottish higher education - a survey of the views of senior managers, academic staff and "experts"', ALT-J, 8 (2), 18-28. HEFCE (1999), Communications and Information Technology Materials for Learning and Teaching in UK Higher and Further Education, Bristol: HEFCE. HEFCE (2001a), Analysis of Strategies for Learning and Teaching: Research Report by Professor Graham Gibbs, Bristol: HEFCE, http://www.hefce.ac.uk/Pubs/hefce/2001/ 01 _3 7a..htm. HEFCE (2001b), Strategies for Learning and Teaching in Higher Education: A Guide to Good Practice, Bristol: HEFCE, http://www.hefce.ac.uk/Pubs/hefce/2001/01_37.htm. HEFCE (2001c), Supply and Demand in Higher Education, Bristol: HEFCE, http://www.hefce.ac.uk/Pubslhefcel2001/01-62.htm. Hobbs, P. J. and Boucher, A. C. (1997), 'An analysis of teaching methods, costs and student preferences', Active Learning, 7, vii-x. 14 ALT-J Volume 10 Number 2 Hunt, M. and Clarke, A. (1997), A Guide to the Cost Effectiveness of Technology-Based Training, Coventry: National Council for Educational Technology and Sheffield: DfEE. Joint Information Systems Committee (JISC) (2002), Exchange for Learning (X4L) Programme, Circular 2/02 January 2002, http://www.jisc.ac.uk/pub02/c02_02.html. Jones, C. (1999), 'From the sage on the stage to what exactly? Description and the place of the moderator in co-operative and collaborative learning', ALT-J, 1 (2), 27-36. Jones, D. (2001), 'Technology costing methodology handbook', version 1.0, NCHEMS/ WCET, http://www.wiche.edultelecom/Projects/tcm/proj-projects.html. Laurillard, D. (1993), Rethinking University Teaching: A Framework for the Effective Use of Educational Technology, London: Routledge/Falmer. Laurillard, D. (1999), 'Investing in information technology pays big dividends', Planning for Higher Education, 27, 1-8. Laurillard, D. (2001), 'Exploring the principles of online learning', Presentation given in Cambridge, May 2001, http://www2.open.ac.uk/ltto/lttoteam/Diana/cambridge/index.htm. Lewis, B. N. (1971a), 'Course production at the Open University I: some basic problems', British Journal of Educational Technology, 1 (2), 4-13. Lewis, B. N. (1971b), 'Course production at the Open University II: activities and activity networks', British Journal of Educational Technology, 2 (2), 111-23. Marton, F. (1981), 'Phenomenography: describing conceptions of the world around us', Instructional Science, 10, 177-200. Marton, F., Hounsell, D. and Entwistle, N. J. (eds.) (1984), The Experience of Learning, Edinburgh: Scottish Academic Press. National Committee of Enquiry into Higher Education (NCIHE) (1997), Higher Education in the Learning Society, 'The Dearing Report', London: HMSO, http://www. leeds. ac. uk/educol/ncihel. National Board of Employment, Education and Training (NBEET) (1994), Costs and Quality in Resource-Based Learning On- and Off-Campus, Commissioned Report no. 33, Canberra: Australian Government Publishing Service, http://www.detya.gov.au.nbeet/. Neumann, R. (2001), 'Disciplinary differences and university teaching', Studies in Higher Education, 26 (2), 135-46. Noble, D. (1998), 'Digital diploma mills: the automation of higher education', First Monday: Peer-Reviewed Journal on the Internet, 3 (1), http://www.firstmonday.dk/issues/ issue3_1/noble/index.html. Oliver, M. and Conole, G. (1999), Assessing and Enhancing Quality Using Toolkits, EFFECTS Report no. 14, London: University of North London. Oliver, M., Bradley, C. and Boyle, T. (2001), 'The distributed development of quality courses for a virtual university', ALT-J, 9 (2), 16-27. Peters, O. (1998), Learning and Teaching in Distance Education: Analyses and Interpretations from an International Perspective, London: Kogan Page. 15 Louise Plewes and Kim Issroff Understanding the development of teaching and learning resources: a review Plewes, L. and Issroff, K. (2002), 'Academic staff attitudes towards the use and production of networked learning resources', in Proceedings of the Third International Conference on Networked Learning 2002, University of Sheffield, 489-96. Rumble, G. (1976), The Economics of the Open University of the United Kingdom, Open University Planning Office. Rumble, G. (1992), 'The competitive vulnerability of distance teaching universities', Open Learning, 7, 31-45. Rumble, G. (1997), The Costs and Economics of Open and Distance Learning, Kogan Page: London. Ryan, S., Scott, B., Freeman, H. and Patel, D. (2000), The Virtual University: The Internet and Resource-Based Learning, London: Kogan Page. Salmon, G. (2000), E-Moderating: The Key to Teaching and Learning Online, London: Kogan Page. Smeby, J.-C. (1996), 'Disciplinary differences in university teaching', Studies in Higher Education, 21 (1), 69-79. Tait, A. and Mills, R. (eds.) (1999), The Convergence of Distance and Conventional Education: Patterns of Flexibility for the Individual Learner, London: Routledge. Wagner, L. (1977), 'The economics of the Open University revisited', Higher Education, 6, 359-81. 16