ALT-J, Research in Learning Technology Vol. 17, No. 2, July 2009, 131–141 ISSN 0968-7769 print/ISSN 1741-1629 online © 2009 Association for Learning Technology DOI: 10.1080/09687760903033074 http://www.informaworld.com Guidelines for supporting re-use of existing digital learning materials and methods in higher education Judith Schoonenbooma*, Henk Sligtea and Eja Kliphuisb aUniversity of Amsterdam – SCO-Kohnstamm Institute, Amsterdam, The Netherlands; bINHOLLAND University – ICTO-OKR, Haarlem, The Netherlands Taylor and Francis LtdCALT_A_403480.sgm (Received 6 May 2008; final version received 16 November2008) 10.1080/09687760903033074ALT-J, Research in Learning Technology0968-7769 (print)/1741-1629 (online)Original Article2009Association for Learning Technology172000000July 2009Dr JudithSchoonenboomj.i.schoonenboom@uva.nl The literature on the re-use of learning materials has largely focused on the development of materials. This paper explores how re-use can be stimulated after learning materials have been developed and made available. We searched for and developed guidelines that support staff and/or management most frequently adopt in cases of (un)successful re-use of existing digital teaching and learning materials and methods by teachers in higher education. In a grounded theory approach, we collected existing guidelines from 11 literature studies, and developed new guidelines from 19 (mainly Dutch) case studies. Through constant comparison, we developed ‘low-level’ guidelines, which were classified into high-level guidelines. Five high-level guidelines turned out to be robust when subjected to repeated comparison with the case studies. Further, this paper elaborates existing guidelines by showing specific elaborations of them in our case studies. Keywords: educational innovation; higher education; teacher support; re-use; digital learning materials Introduction There is now an extensive literature on the re-use of learning objects, i.e. resources “usually digital and web-based, that can be used and re-used to support learning” (Wiley 2000). Up till now, the literature has focused on the development of materials, with significant attention being paid to the design of learning objects to optimise their re-use. This has led to a debate on the extent to which learning objects should contain such diverse elements as context (Quinton 2007), pedagogy (McCormick 2003) and learning objectives (Boyle 2003). Learning materials have been classified into various hierarchies (Cisco Systems 2003; Wagner 2002; Wiley 2000), and learning object design trade-offs have been formulated, e.g. between size and re-usability (South and Monson 2000). Attention has also been paid to encouraging re-use by making learning materials available through repositories, using metadata (Sonntag 2007). This has led to the formulation of principles for good metadata (NISO Press 2004); the develop- ment of metadata standards such as Learning Object Metadata (LOM) (Institute of Electrical and Electronics Engineers 2002); a discussion on the use of tagging versus metadata (Smith 2008); and the formulation of a trade-off between learning material size and costs of metadata (Wiley 2002). *Corresponding author. Email: j.schoonenboom@ond.vu.nl 132 J. Schoonenboom et al. However, much less attention has been paid to the question of how re-use can be stimulated after development. In this situation, one or more collections of existing learning materials are available, and the question is how can teachers be stimulated to re-use these existing materials? This question has become increasingly relevant in recent years, as an abundance of learning materials has become available through the Internet and through institutional and shared repositories. In this paper we will focus on the re-use of existing learning materials in Dutch higher education. Re-using existing learning materials in higher education is different from develop- ing re-usable learning materials, in several respects. Firstly, it is educational support staff and managers rather than developers who support re-use in this situation. Secondly, support staff have fewer possibilities than developers to create an optimal situation for re-use. They will be confronted with a range of existing materials, vary- ing in terms of fit to teachers’ needs, quality of the metadata and appropriateness of their size. Thirdly, while developers can stimulate re-use by developing high-quality learning materials, support staff focus must be on the teachers, coaching them towards successful re-use in their teaching. Although this post hoc stimulation of re-use has received less attention than the development of learning materials, several guidelines for it can be found in the litera- ture. An example is peer review (Campbell 2003; Lim 2007; Metros, Bennett, and Diaz 2001), which was shown to resolve teachers’ anxiety about the quality of the learning materials (Littlejohn 2003a). The aim of our study is to bring together the most important guidelines from the literature and develop additional guidelines from existing case studies. We develop a theory for supporting the re-use of existing learning materials, which is both ‘grounded’ in the data (Charmaz 2006) and ‘humble’ in the sense of Cobb et al. (2003, 10), the theory being “concerned with domain-specific [support] processes, … accountable to the activity of design”, and “provid[ing] detailed guidance in organis- ing [support]”. Although these guidelines may be matched onto stages of Rogers’ (2003) ‘grand theory’ of the innovation process, this is not the focus of our paper. Most of the research for our study was conducted in a six-month study for SURF, the Dutch foundation for ICT in Higher Education (Schoonenboom et al. 2007). Our main research question is: What guidelines do support staff and/or management most frequently follow in successful and unsuccessful cases of re-use of existing digital teaching and learning materials and methods by teachers in higher education? How can these guidelines be classified? In this research question, following a guideline can be done either explicitly or implicitly. Some guidelines were mentioned by participants as guidelines they had followed, others were derived from information on cases of re-use, and participants may not have been aware of them. We looked at successful and unsuccessful cases of re-use of existing learning materials, as we would expect that useful guidelines should contribute to the success of re-use (and ignoring them may lead to unsuccessful re- use). Our research aims to discover the guidelines most frequently followed. The more often a guideline is encountered in successful cases, the more likely it is that its success is not a coincidence, and the more useful it will be to educational support staff. Guidelines are classified not only to arrive at a more manageable presentation, but also to arrive at more general guidelines to be followed, and to inform practice in several ways. ALT-J, Research in Learning Technology 133 Our criteria for including learning materials and methods is whether they can be used by individual teachers in their own teaching. Examples include materials such as study tasks, courses, pictures or theses, or methods such as working in small virtual groups, concept mapping or the use of webcams. Outside the scope of this research lies, for example, the institution-wide deployment of competence-based learning, or the development of a master degree at a distance. Methodology Design Guidelines were derived through two methods. Firstly, we searched for guidelines in a sample of relevant literature. These guidelines were not treated as ‘proven important guidelines’, but as input for the formulation of guidelines. Secondly, we derived guidelines from a sample of case studies of successful and unsuccessful re-use of existing learning materials. We then applied a grounded theory approach (Charmaz 2006), in which we derived general guidelines from the literature and case studies samples together, using constant comparison. Finally, the resulting guidelines were compared with the recent literature. Sampling from practice We followed a multi-stage purposeful sampling strategy (Onwuegbuzie and Leech 2007). Our first sample contained case studies of re-use of existing digital learning materials and methods in Dutch higher education. Following the general principles of Robert Stake (1995, 2006), we selected those cases from which we expected to learn most. We used the following criteria for selecting case studies: (1) we knew the cases; (2) they were recent; (3) descriptions with some detail were available; (4) we knew something interesting was done on re-use of existing materials; (5) something had been learned and/or was assumed by the participants on success or fail factors with re- use; and (6) which together led to a variety of guidelines. On the basis of the first five criteria, we selected a first set of case studies. Since it turned out that the guidelines derived from these case studies were too limited in scope, we then used theoretical sampling (Glaser and Strauss 1967) to increase the variety of the guidelines, to ensure they covered the whole innovation process and to find more evidence and counterevidence for our guidelines-in-devel- opment. We added a second sample of case studies from an existing collection, which had been brought together in our earlier study Learning objects in practice (Schoonenboom et al. 2006). In this study, carried out between 2004 and 2006, we identified and contributed to solutions for important problems for those working with learning objects in Dutch higher education. In this study, a set of six Dutch case studies had been collected, which met the following criteria: (1) they were related to digital learning materials; (2) they had successfully applied for funding with the Dutch national SURF organisation or the Dutch Digital University; (3) they had been finished successfully; and (4) the results were used in practice. This second sample had in the original study already been supplemented by two other case studies of successful practice and two running experiments, in order to increase diversity and to increase the validity of the discovered guidelines. The literature that we used as data for developing guidelines was also taken from our study Learning objects in practice (Schoonenboom et al. 2006). At the start of this 134 J. Schoonenboom et al. project in 2004, four learning objects experts, each with four or more years of experi- ence (both at a national and an international level) conducted a literature review in the problems of working with digital learning materials. A description of the 19 case studies and 11 literature studies used in the analyses can be found in Schoonenboom et al. (2007). Procedure The procedure consisted of four steps. Figure 1 shows the results of each step, which in the text are indicated in bold. Figure 1. Summary of one practice, showing the elements of the derivation of guidelines: description of success, short descriptions of actions, low-level guidelines and high-level guidelines; numbers of high-level guidelines refer to their number in Figure 2. Step 1. Identification of ‘actions’ in case studies Detailed descriptions of the case studies were either collected or prepared by the researchers. Each description included a description of the success or lack of success of the practice. Within each example of practice, ‘actions’ were identified, which seemed to have yielded or contributed to its success or lack of success. ��� ����� �� ��� ������ � � �� ��������� �� ��� ���� �� � �������� � �������� � ��� �������� ��� ��� ��� �� � ���������� ����� �� ��� � ��� ������ � �������� �� �������� �� ��� ��������� ������ ���������� �������� �� ������ ��������� � � ���� �� ����� ������ ����� ����� �� !�������� ��� ��������� �� � � ��� ��� ����� �� � � ������� � ���� � ������� " ��� ����� ���� ����� #$%% ��������� ���� ���������& � � ��� ������ ��� ��������� ��� ���� �� � ����� '$%% ���������� � � ��� ���(��� ������ ���� � ���� ����� ��� � ���������� �� �� ����� �� �� � ��� �� �)� ��� ���������� �� ���� ���� ���� �� �� ������� ��� ������ � �*� �� ���� �������� �� ������ +����� ���� � � �,� -���� �� �� ��� � ���� � �������� ���� ���� �� �� � ������ ��� ����� �������� ���� � � ����� ��� ������� � ������������� ���� �� ��� ������ � ������� �� ������ �.� -���� �� �� ��� ���� �� �� ��������� ����� ���� �� � ������ � � ��� ����� ������� �� ������ � ������� �� ��� ��� /� ) 0���� �������� �������� � � ������ ������ ������� � ��� �� ����� ���� ����������� �������� �� �������� �� � ����� �����+����� ������� �1 )� /� , �� ��� �2�� � ���� �������� � � ����� �� ��� ������ ���� �� ������ �� ������ ��� ���� � � ��������� � � ������ �� ������ ���� �� ������ ����� � ����� ��� �� ��� ��� �������� ����� ��� �� ����� ������� �����+����� ������� �1 ,� /� . � ���� ��� ������� �� � ������ � �� ������ ����� �� ���� �������� � � ����� �� � � ������� �� ��� 3������ �� ��� �������� � � ������ � �������� �����+����� ������� �1 #� Figure 1. Summary of one practice, showing the elements of the derivation of guidelines: description of success, short descriptions of actions, low-level guidelines and high-level guidelines; numbers of high-level guidelines refer to their number in Figure 2. ALT-J, Research in Learning Technology 135 Step 2. Formulation of low-level guidelines For each action, a short description was created. The literature sample was scanned for existing guidelines, and short descriptions of these guidelines were prepared. From the short descriptions of both the existing guidelines in the literature and the practice actions, low-level guidelines were derived. The derivation of the low-level guidelines was an iterative process, using repeated comparison (Charmaz 2006; Glaser and Strauss 1967). First, a set of guidelines was formulated, and as many short descrip- tions as possible were grouped under these guidelines. Reflection on the remaining descriptions led to the formulation of new guidelines and so on, until each short description was grouped under one low-level guideline. Then, the literature and the detailed descriptions of the case studies were analysed again, looking for additional instances of actions that would fit under the low-level guidelines. Step 3. Formulation of high-level guidelines Finally, the low-level guidelines were categorised into high-level guidelines, such as ‘Provide easy access to the materials and methods’ in Figure 2. The same procedure of repeated comparison was used as in step 2. Step 4. Testing the validity of the guidelines The resulting high-level guidelines were tested for robustness. We performed a split- half test, in which we randomly divided the literature and practice cases into two. The split-half test showed that guidelines derived from 10 or more cases were robust in that they could be derived from either half of the cases. These five guidelines were kept, and the other non-robust guidelines were removed. Finally, we checked a recent three-volume work on learning objects for remarks on these guidelines (Harman and Koohang 2007; Koohang and Harman 2007a, 2007b). Validity Throughout our study, we had to deal with a variety of validity threats. Consequently, we used a variety of strategies to deal with these threats. ● Case studies were divided among the researchers, and each researcher analysed those practices with which he or she was most familiar (Stake 1995, 2006). Most of the case studies we knew very well and over a long period, thereby reducing the risk of misinterpretations (Becker 1970). ● We used multiple data sources; we used multiple cases, which were multiple in kind, including both descriptions of case studies and literature studies, and by multiple authors. This reduced the risk of biases, resulting from studying one specific case using a specific type of resource by one specific researcher. ● Descriptions of case studies had been validated by participants. This applied to all samples. ● To ensure that the guidelines would be robust enough to capture a variety of situations, we monitored the guidelines at any point in our research project, to see if they provided enough variation. When it turned out that the guidelines of our first sample did not explicitly refer to digital learning materials, we included a second sample of case studies and literature studies. 136 J. Schoonenboom et al. ● The descriptions and analyses of the six most demanding, initially selected case studies were either done by two researchers, or by one researcher and checked by another. ● Within each individual practice, we devoted one question in our analysis template to a systematic search for additional evidence and for rival explana- tions of the presumed relation between action and success (or failure). Evidence often included signs of growing and sustained use. Sometimes, the presumed relation was backed by evidence from the literature. ● With each guideline that we had derived from an individual practice, we checked its occurrence in all other case studies. Furthermore, we checked whether guidelines found in one case study were contradicted in other case studies, but we found no examples of this. ● Reformulations in the guidelines were thoroughly discussed between researchers. Outcomes Figure 2 shows the five high-level guidelines that we derived. After each guideline, the number of cases on which it is based is indicated in brackets. Figure 2. The five high-level guidelines for supporting the re-use of existing digital learning materials and methods; numbers indicate the number of the 30 case studies and literature studies on which each item is based. Guideline 1 is mostly relevant to managers who at the beginning of the process are considering re-use of existing learning materials. Guidelines 2–5 become rele- vant once the decision for re-use has been taken and is mostly relevant to support staff. The sections below each present one high-level guideline, with an explanation of how it was derived from its underlying low-level guidelines. A document is available in Dutch that shows for all guidelines the whole derivation process from detailed descriptions to actions to low-level guidelines to high-level guidelines (Schoonenboom et al. 2007). 1. Develop an appropriate vision on the incipient re-use of the materials and methods within the programme, department or institution From the beginning an appropriate vision of re-use should be developed by the teacher(s), department or institution involved. Our study showed that the following elements of this vision, which had been identified in the literature, occur frequently in the case studies: ��� ������� � ������ �� � � �� �� ��� �� � ��� ������ �� ��� � ��� �� � �� � ��� ����� ���� ��� ������ �� ��� ��� �� ������� ��� ���� �� � � ����� �� ��� � ��� �� ����!�� �"� ���� �� � ��� �� �� � � � ���� ��� �� �����# ��� ����� �� � �� �� �� �� ������� ��� ���� �� � ��� �� �� ���� $� � � � �� �� %�� �� ����� �� �������� �� &���� ��� $� � � �� � ��� �� �� ������� ����'�� �(� )� %�� ������ � � �� � ��� �� ��� � ��� �� �� ������� �� ��� �����*� ����+�� Figure 2. The five high-level guidelines for supporting the re-use of existing digital learning materials and methods; numbers indicate the number of the 30 case studies and literature stud- ies on which each item is based. ALT-J, Research in Learning Technology 137 (1) The materials should have a demonstrable added value for teaching (Rogers 2003), which is visible to the teachers and managers involved. Examples include materials that can be taken into account in students’ assessment and materials on which use feedback can be provided by either teacher or students. (2) Re-use of materials should be considered a gradual process of small changes. Teachers are more inclined to apply small innovations than to change the whole curriculum (Rogers 2003). (3) Re-use of materials should be viewed as a way of improving education, not as a way of reducing costs. In fact, costs may go up (Pennings et al. 2005). Our case studies show that institutions should consider costs for updating the mate- rials (Littlejohn, Jung, and Broumley 2003). Also, materials or methods that might be provided for free during the course of a project might require a license after the project has finished. New in our study is that it shows examples in practice of visions that may stimulate re-use. We found that this vision might be a big or a small idea. In one practice, a repository was established in support of the ‘big’ idea of problem-based learning. Existing learning materials were collected that together support the whole cycle of problem-based learning in the department involved. In another practice, the ‘small’ idea of visual learning was prominent, and learning materials and methods were collected that could contribute to this idea. Although earlier literature has identified ideas such as problem-based learning as a basis for developing new learning materials (many examples in Koohang and Harman 2007a; Wiley 2003), we found that this is also relevant in working with existing learning materials. 2. Provide easy access to the materials and methods If it is too difficult for teachers to get access to the digital learning materials and meth- ods, they will not use them. This point has received much attention in the literature on learning object development. As a first solution to this problem, the importance of good quality metadata has been emphasised, so that users find relevant, and neither too few nor too many learning materials (Koper 2003; Sonntag 2007). A second solu- tion is the use of ‘tailored wrappers’, information on the local context of use, which is added to existing learning materials (Quinton 2007). These two solutions show up in our case studies, but with two elaborations. First, two case studies show that it can be useful not to provide access to all learning mate- rials of an available repository, but instead to make a selection of materials that are deemed relevant to the teachers involved. Second, one practice made use of tailored wrappers which contained: information on which colleagues had already used these materials; information on where one could get help; own metadata with familiar local terms which was added on top of the existing metadata. 3. Provide materials and methods that are a good fit to teachers’ own needs and way of teaching If materials and methods are a good fit to teachers’ own needs and ways of teaching, then it will be easy for teachers to use them (Rogers 2003). On the other hand, learning materials cannot easily be used if, for example, the cultural gap with the teacher’s own 138 J. Schoonenboom et al. practice is too wide (Littlejohn 2003b), or each piece is too large, containing too much material that is not useful (Downes 2002). In line with the literature, our case studies show that if the conditions of use of the target institution are too different from those where the materials were developed, then re-use will not be possible. A good fit also implies that teachers should be able to use the materials selectively, only the elements they deem to be useful. Student needs must be considered too; for example, materials that fit the students’ future profes- sional practice can be particularly useful. Materials that are a poor fit include those that are too complex for the level at which they are intended to be used, or those that clash significantly with aspects of the pedagogy in use, or if they are built upon an ‘alien’ pedagogy. 4. Provide materials of good quality, and enable teachers and students to judge the quality of materials Learning materials should be of good quality. At the level of the repository, too much variation between learning materials is considered a problem (Downes 2002). Teach- ers and students should be enabled to judge their quality (Littlejohn 2003a); peer review is seen as a very useful method (Metros, Bennett, and Diaz 2001). In line with the literature, we found that the collection of materials in one repos- itory should be consistent, and with respect to the domain involved the materials should be as complete as possible, and of equal quality. For students, it is important that multimedia and interactive materials are provided (Ravenscroft, Boyle, and Jones 2004). Regarding judgement of quality by teachers and students, we found in the cases that we studied the use of peer review, in which peer teachers tested the usability of materials before use in the institution. In one practice, peer review was seen as especially important for materials on the web because of the unclear status of these materials. The second form of judgement in our case studies was rating and annotating by users to inform their peers on how to use the learning material involved (Sonntag 2007). In addition to what is known from the literature, we found in one example of prac- tice a clear limit to differences in the size of learning materials. Here, teachers did not appreciate that the results of one search could include materials as diverse as study tasks and pictures. In examining peer review, we found that teachers rely partly on whether they know the person who developed the materials and the status that this person has. 5. Enable necessary adaptations of the materials and methods to the context Teachers should be able to use the materials and methods in their teaching. An important element of this is that they should be able to adapt the materials and methods to their own teaching context. Several factors might hinder use. As a necessary condition, learning materials should ‘run’ in the institution’s virtual learning environment. Also, copyright problems concerning the re-use of the materials should be solved (Campbell 2003). Teachers must be provided with the necessary tools and support. Lack of user- friendly tools with which teachers can re-use learning objects is considered a problem (McNaught 2003). Also, time is an important constraining factor (Littlejohn, Jung, and Broumley 2003), and enough time should be provided for teachers to evaluate and test materials. ALT-J, Research in Learning Technology 139 In line with these findings in the literature, we found in our research that the user- friendliness of the materials and the occurrence of technical problems were also constraining factors. If adaptation takes too much time, teachers will not use the mate- rials. We also found the need for specific forms of support. Support should be provided to users at several levels of expertise, for both students, managers, beginning teachers and experienced teachers. Explaining the pedagogical possibilities of the materials helps, as this may not be clear from the materials themselves. Collective support should be complemented by individual support. In addition to what was found in the literature, our case studies showed that use of the materials is stimulated by removing two important practical hindrances to re-use by enabling teachers and students to work with the materials from home and by making the necessary software freely available. One additional form of support in the case studies was to provide support to beginners by letting them join more experienced colleagues. Conclusions This paper has contributed to the discussion on re-use of learning materials in several ways. It has brought together existing answers and developed new answers to ques- tions that had perhaps been solved for the development of learning materials from scratch, but not for situations in which teachers and their supporters are working with existing learning materials. Our case studies have shown that also in this situation, an overarching educational idea or vision is required. Furthermore, although the existing materials may not all be optimal, access can be optimised by making a relevant selec- tion, by adding own metadata on top of existing metadata and by bringing down the range of different levels of aggregation of learning materials that teachers can find in one search. Our study has grounded existing guidelines and provided an overview by showing which of these are important in practice, in that they occur often in our cases. It has specified existing guidelines, in that it has shown some specific elaborations of these guidelines in practice. Finally, our paper has demonstrated a method for deriving guidelines from case studies, testing their robustness and combining these with findings in the literature. References Becker, H.S. 1970. Sociological work: Method and substance. Chicago, IL: Aldine. Boyle, Tom. 2003. Design principles for authoring dynamic, reusable learning objects. Australian Journal of Educational Technology 19: 46–58. Campbell, L.M. 2003. Engaging with the learning object economy. In Reusing online resources: A sustainable approach to e-learning, ed. A. Littlejohn, 35–45. London: Kogan Page. Charmaz, K. 2006. Constructing grounded theory: A practical guide through qualitative anal- ysis. London; Thousand Oaks, CA: Sage Publications. Cisco Systems. 2003. Reusable learning object strategy: Designing and developing learning objects for multiple learning approaches. Version 4.5. http://www.e-novalia.com/materi- ales/RLOW__07_03.pdf Cobb, P., J. Confrey, A. diSessa, R. Lehrer, and L. Schauble. 2003. Design experiments in educational research. Educational Researcher 32: 9–13. Downes, S. 2002. The learning object economy. http://www.downes.ca/files/ Learning_Object_Economy.doc 140 J. Schoonenboom et al. Glaser, B.G., and A.L. Strauss. 1967. The discovery of grounded theory: Strategies for quali- tative research. New York: Aldine de Gruyter. Harman, K., and A. Koohang. 2007. Learning objects: Applications, implications, & future directions. Santa Rosa, CA: Informing Science Press. Institute of Electrical and Electronics Engineers. 2002. Draft standard for learning object metadata. New York: Institute of Electrical and Electronics Engineers, Inc. http:// ltsc.ieee.org/wg12/20020612-Final-LOM-Draft.html Koohang, A., and K. Harman. 2007a. Learning objects and instructional design. Santa Rosa, CA: Informing Science. Koohang, A., and K. Harman. 2007b. Learning objects: Theory, praxis, issues, and trends. Santa Rosa, CA: Informing Science. Koper, R. 2003. Combining re-usable learning resources to pedagogical purposeful units of learning. In Reusing online resources: A sustainable approach to e-learning, ed. A. Littlejohn, 46–59. London: Kogan Page. Lim, G. 2007. Instructional design and pedagogical considerations for the ins-and-outs of learning objects. In Learning objects and instructional design, ed. A. Koohang and K. Harman, 89–137. Santa Rosa, CA: Informing Science Press. Littlejohn, A. 2003a. An incremental approach to staff development in the reuse of learning resources. In Reusing online resources: A sustainable approach to e-learning, ed. A. Littlejohn, 221–33. London: Kogan Page. Littlejohn, A. 2003b. Issues in reusing online resources. In Reusing online resources: A sustainable approach to e-learning, ed. A. Littlejohn, 1–6. London: Kogan Page. Littlejohn, A., I. Jung, and L. Broumley. 2003. A comparison of issues in reuse of resources in schools and colleges. In Reusing online resources: A sustainable approach to e-learning, ed. A. Littlejohn, 212–20. London: Kogan Page. McCormick, R. 2003. Keeping the pedagogy out of learning objects. Paper presented at the 10th Biennial Conference of the European Association for Research on Learning and Instruction (EARLI), September 26–30, in Padova, Italy. http://www.eun.org/eun.org2/ eun/Include_to_content/celebrate/file/KeepingPedagogyOutOfLOs3v2.doc McNaught, C. 2003. Identifying the complexity of factors in the sharing and reuse of resources. In Reusing online resources: A sustainable approach to e-learning, ed. A. Littlejohn, 199–211. London: Kogan Page. Metros, S., K. Bennett, and V. Diaz. 2001. The promise and pitfalls of learning objects: Status of digital resource collections. Presentation given at EDUCAUSE 2001, October 28–31, in Indianapolis, IN. http://net.educause.edu/ir/library/powerpoint/NLI0201.pps NISO Press. 2004. Understanding metadata. Bethesda, MD: NISO Press. http:// www.niso.org/publications/press/UnderstandingMetadata.pdf Onwuegbuzie, A.J., and N.L. Leech. 2007. A call for qualitative power analyses. Quality and Quantity 41: 105–21. Pennings, L., M. van Staden, S. Limonard, and V. Frissen. 2005. Van bloei naar groei; condi- ties voor ontwikkeling en toepassing van business modellen voor webbased educatieve content. [From bloom to growth: Conditions for development and application of business models for web-based educational content]. Delft, The Netherlands: TNO. http:// www.ez.nl/dsresource?objectid=143709&type=PDF Quinton, S.R. 2007. Contextualisation of learning objects to derive meaning. In Learning objects: Theory, praxis, issues, and trends, ed. A. Koohang and K. Harman, 113–79. Santa Rosa, CA: Informing Science. Ravenscroft, A., T. Boyle, and R. Jones. 2004. Using learning objects to enhance blended learning. In JISC innovative e-learning practice case studies, ed. T. Boyle, J. Cook, and J. Clay. London: London Metropolitan University. http://www.jisc.ac.uk/ uploaded_documents/Learnobjs_ed.doc Rogers, E.M. 2003. Diffusion of innovations. 5th ed. New York: Free Press. Schoonenboom, J., H. Hermans, S. Poortman, F. Benneker, B. Emans, J. Meijer, F. de Vries, and P. Sloep. 2006. Learning objects in practice. 6 vols. Utrecht, The Netherlands: Stichting Digitale Universiteit. http://www.sco-kohnstamminstituut.uva.nl/zips/Learn- ing%20objects%20in%20practice.zip Schoonenboom, J., H. Sligte, E. Kliphuis, and L. van Elk. 2007. Hergebruik van materialen en methodieken: Een op praktijkvoorbeelden gebaseerd beslissingsschema voor het hoger ALT-J, Research in Learning Technology 141 onderwijs – digitaal beslissingsschema [Re-use of materials and methods: A practice- based decision scheme for higher education – digital decision scheme]. Utrecht, The Netherlands: Stichting SURF. Smith, G. 2008. Tagging: People-powered metadata for the social web. Berkeley, CA: New Riders. Sonntag, M. 2007. Syntax and semantics of learning object metadata: The IEEE/IMS LOM and beyond. In Learning objects: Theory, praxis, issues, and trends, ed. A. Koohang and K. Harman, 417–505. Santa Rosa, CA: Informing Science. South, J.B., and D.W. Monson. 2000. A university-wide system for creating, capturing, and delivering learning objects. In The instructional use of learning objects: Online version, ed. D. Wiley. http://reusability.org/read/chapters/south.doc Stake, R.E. 1995. The art of case study research. Thousand Oaks, CA: Sage. Stake, R.E. 2006. Multiple case study analysis. New York: The Guilford Press. Wagner, E.D. 2002. Steps to creating a content strategy for your organization. The eLearning Developers’ Journal, the eLearning Guild. http://www.elearningguild.com/pdf/2/ 102902MGT-H.pdf Wiley, D.A. 2000. Connecting learning objects to instructional design theory: A definition, a metaphor, and a taxonomy. In The instructional use of learning objects: Online version, ed. D.A. Wiley. http://reusability.org/read/chapters/wiley.doc Wiley, D.A. 2002. Learning objects need instructional design theory. In The ASTD e-learning handbook, ed. A. Rossett, 115–26. New York: McGraw-Hill. http://opencontent.org/docs/ astd.pdf Wiley, D.A. 2003. Learning objects: Difficulties and opportunities. Utah State University. http://opencontent.org/docs/lo_do.pdf