schoonenboom.pdf Australasian Journal of Educational Technology 2012, 28(2), 249-265 Four scenarios for determining the size and reusability of learning objects Judith Schoonenboom VU University Amsterdam The best method for determining the size of learning objects (LOs) so as to optimise their reusability has been a topic of debate for years now. Although there appears to be agreement on basic assumptions, developed guidelines and principles are often in conflict. This study shows that this confusion stems from the fact that in the literature, hardly any attention is paid to different types of reusability. This study not only shows that there are different types of reusability, but also that different types of reusability go together with different principles for developing LOs and for determining LO size. A grounded theory analysis was performed on separate statements on LO development in 71 selected publications from LO literature. This resulted in the development of four scenarios for LO development, each with their own definition of reusability and their own principle for determining the size of LOs. A field test with 11 experts showed that the scenarios are recognisable. In higher education, educational technologists often assist amateur LO developers. Educational technologists can use the scenarios to determine which definition of reusability is applicable to their situation and which principle for determining the size of LOs should be applied. Introduction This article is about reuse of learning materials and the relation between reuse and the size of the learning materials. I start with two examples from practice. The library of VU University offers online courses in information skills. A few years ago, these courses were redesigned to enable more flexible use. The courses were split up into modules that could be followed by students independently from each other. In addition, where possible, departments were allowed to personalise the course by adding content from their own disciplines. Naturally, this redesign raised questions regarding how to split up the courses and where to allow departments to personalise the course. A few years before, VU University participated in a large national project on knowledge engineering. The aim was to develop and to make available learning materials for Dutch university degree programs on knowledge engineering. In the Netherlands, knowledge engineering is a small discipline with only a few hundred experts who are used to sharing experiences and materials. The course of study in the degree programs is largely fixed, and Dutch experts agree to a large extent on what students should learn and what learning tasks fit these learning goals. In this case, the main question was determining the units in which learning materials should be offered and how they should be made available. To r e u s e means to use learning materials outside of their original context of development. In the literature, two types of reuse are distinguished. The first type is 250 Australasian Journal of Educational Technology, 2012, 28(2) reuse of content, of pieces of learning material. These pieces are called ‘learning objects’ (LOs). The second type of reuse occurs when expert teaching and learning practice is documented in a consistent notational form as a so-called ‘learning design’, which serves as a model or template for reuse by others (Bennett, Agostinho, Lockyer, Harper & Lukasiak, 2006; Lockyer, Bennett, Agostinho & Harper, 2009). This article is about reuse of learning objects. In both examples from practice, LOs are developed that are offered to potential users in a collection: a collection of modules on information skills, and a collection of learning materials on Knowledge Engineering, respectively. In both cases, the same problem emerged that will be central to this article: In developing LOs, how can one determine their size, or, to use the technical term, their granularity, so as to optimise reusability? These cases describe a situation that is typical for VU University and other universities. LO development is done by content experts, who often know little about LO development, and who are assisted by educational technologists who themselves are often not specialised LO developers. The most important question in practice, then, is how educational technologists can assist these amateur LO developers in determining the granularity of LOs. In the last decade, much discussion has occurred among experts about how to develop learning objects that have a high potential for reuse. From this discussion, a generally acknowledged principle has been derived for determining the granularity of LOs; to promote reuse, LOs should be relatively small (Downes, 2004; Littlejohn, 2003; Wiley, 2002). Unfortunately, recognition of these general principles has not led to accepted standard guidelines for determining the size of LOs, for two reasons. First, a large diversity exists among different authors as to what “as small as possible” means. In the literature, as small as possible can vary from single images to substantial parts of courses. Second, although a variety of guidelines for determining the size of LOs have been developed, there is no unequivocal agreement on these guidelines. Even worse, many guidelines contradict each other. For example, some authors base their advice for determining the size of LOs on physical characteristics. Thus, Robson (2001) recommends that LOs contain between five and fifteen minutes of learning material, whereas Muzio, Heins and Mundell (2002) contend that one computer screen should incorporate at least two LOs. Other authors base the size of LOs upon content, but this guideline does not lead to unanimous agreement either. One often proposed idea is that LOs should be based upon one learning objective. Metros (2005) states that to be considered an LO, a digital resource must include or link to a learning objective, a practice activity, and an assessment. Conversely, McCormick (2003) advocates keeping pedagogy out of LOs because adding pedagogy is the task of the teacher. Doorten et al. (2004) posit another view by arguing that LOs can be developed by decomposing existing materials into their smallest intrinsically meaningful parts. Littlejohn, Falconer and McGill (2008) provide many tips for developing LOs. However, their advice that LOs should be “of a critical size”, without defining what a critical size is, is not very illustrative. In sum, the contradictions found in the literature are not very helpful in developing LOs. One can base an LO on one learning objective, or one can leave the pedagogy out of the LO, but one cannot do both at the same time. This article has two goals. The first goal is to argue that there is a more fundamental reason why the formulated guidelines and principles provide so little help in Schoonenboom 251 developing LOs. The reason is the basic premise that there is one thing called ‘reusability,’ an assumption that is either implicitly or explicitly present in many studies (e.g., Littlejohn et al., 2008). A few studies, though, make a distinction between several types of reusability. Boyle (2003) distinguishes re-purposing, in which the LO is adapted to its new context of use, from re-use, in which the object is used in a new context without modification. Within the Quality for Reuse project (Q4R, 2010), a distinction is made between three types of reusability: pedagogical, technical, and socio-cultural reusability. Müller (2006) is one of the few authors who identifies the problem that different types of reusability can be in conflict with each other. Müller distinguishes between technical reusability, which tries to achieve platform and format independence of LOs, and instructional reusability, which aims to reuse one LO in different learning contexts. He points out that reusability is context dependent and that each context needs its own LO scenario. Viewing a scenario as a discrete sequence of steps inside a learning management system, Sicilia and Lytras (2005) propose that different scenarios put different requirements on technical standards and metadata, and that reusability is limited within a specific type of scenario. Unfortunately, they do not provide concrete examples of different scenario types. Although the existence of several types of reusability has been recognised in a few studies, there are no studies in which the existence of several types of reusability is linked to the existence of several ways in which an LO can be “as small as possible.” Filling this missing link is the second goal of this article. In this study, an analysis of the LO literature following the principles of grounded theory (Glaser & Strauss, 1967) will lead to the distinction of four types of reusability, each with a different definition of what “as small as possible” means. In this way, different definitions of reusability are linked to different principles for the development of LOs. Working out this idea leads to four different scenarios for the development of LOs. These four scenarios do not contain very concrete guidelines for developers for determining the size of LOs. They are, however, a tool for educational technologists to use to help LO developers determine which scenario is applicable to their situation, and thus which principle for determining the size of LOs applies. This analysis will show that the two cases at the beginning of this article, although superficially very much alike, belong to different scenarios, and that the size of LOs should be determined by different principles. As such, the research questions of this study are: Which global scenarios for LO development and use can be defined on the basis of differences in the definition of reusability? Which principles for deciding the size of LOs are valid for which scenarios? What other related differences exist between scenarios? Method Data collection The method of data collection in this study differs from a traditional review or meta- analysis. In a traditional review, studies are collected with the same or very comparable research questions, and a meta-analysis is performed by comparing the results of those studies. By contrast, the research goal of this study is to develop a 252 Australasian Journal of Educational Technology, 2012, 28(2) theory that does not yet exist in the field. Thus, in this case, the literature does not provide theories, but it provides statements on which a theory is built by the researcher. Therefore, the review in this study is more comparable with a realist synthesis (Pawson, 2006a, 2006b). A realist synthesis focuses on theory building, and the primary studies are viewed as case studies whose purpose is to test, to revise, and to refine the preliminary theory (Pawson, 2006b, p. 74). This view is shared by this study. The scenarios of this study are best characterised as a theory developed on the basis of statements in the literature according to the principles of grounded theory (Glaser & Strauss, 1967), which is subsequently tested, revised, and refined similarly to a realist synthesis. Contrary to a traditional review, the unit of analysis in a realist synthesis is not an entire study, but rather fragments of evidence within a study (Pawson, 2006a). Comparably, the unit of analysis in this study is the fragment within a study that contains a statement that might contribute to the development of different types of scenarios. Statements can be of various types and are not limited to statements on how to determine the size of LOs. For example, statements of different types on the goal or process of LO development can also serve as raw material for scenario development. The theory development process in this study has consequences for how studies are selected. In a traditional review, studies are selected on the basis of their research question and the quality of the research performed. This procedure is entirely justified; meta-analysis is based on the results of the individual studies, and therefore the quality of the building blocks of the meta-analysis depends directly on the quality of the research performed in the individual studies. This rationale does not apply, however, to a realist synthesis (Pawson, 2006a). In a realist synthesis, research quality can only be determined within the act of synthesis, and it is determined by the extent to which the particular fragment helps and is of sufficient quality to help in respect of clarifying the particular explanatory challenge that the synthesis has reached (Pawson, 2006a, p. 135). In other words, a study that has yielded low quality results with respect to its own research question may still contain fragments that are very useful with respect to the research question of the realist synthesis. The same applies to this study. A study on LOs that has yielded results of a low quality can still contain a statement on how to determine the size of LOs that is useful for developing other scenarios. With this concept in mind, in this study, primary studies were selected on the basis of their importance in the field of studies on LO development. For the scenarios to have ecological validity, they had to be based upon statements on how to determine the size of LOs that were most likely to have influenced LO development, and thus publications in this field. The most important studies on LO development were defined in the following way. Because an important part of LO literature is grey literature outside the academic journals, it was expected that many relevant studies would not be found using the standard academic literature databases. Therefore, the literature search was performed using the software package Publish or Perish (Harzing, 2010), which searches all citations in Google Scholar. I searched the engineering and social science literature using the keywords granularity, design and learning objects, and selected the 40 hits with the highest citation ranks. To this selection were added LO studies considered important by four Dutch internationally renowned LO experts (2006). Also included were useful references found in the selected studies. The literature search yielded 141 studies in total. The number of studies is obviously much Schoonenboom 253 larger than the number of 40 references that were initially selected. The main reason for this discrepancy is that some of the references consisted of edited books that contained several studies. Next, from the 141 selected studies, fragments were selected that contained statements on how to determine the size of LOs. Within the selected literature, relevant fragments were found in the full text of digitally available literature using the keywords granularity and size. Non-digital literature was searched with these keywords using the index. By reading the surrounding text of each keyword occurrence, text fragments that dealt with the size of LOs could be easily located. Finally, fragments were selected that were useful for the development of scenarios. Many fragments that were identified using full text search turned out not to be useful, as the word granularity or size was mentioned but not explained or discussed. In the end, the development of the scenarios was based on fragments in 71 studies. Because of this large number, these studies are not mentioned in the bibliography except for those studies that are referenced in this article. The 71 studies were of different kinds, as is shown in Table 1. Table 1: The 71 studies ordered by type Type of study Example N Theoretical discussion about the relation between reuse and granularity Silveira et al. (2007) 10 Theoretical discussion about LO development in relation to instructional design Cheal & Rajagopalan (2007) 14 Practical guidelines for developing LOs in relation to instructional design Hamel & Ryan-Jones (2002) 6 Report of lessons learned from running an LO project in practice Falloon, Janson & Janson (2010) 9 Technical description of an e-learning system and/or its components Cisco Systems (2003) 16 Discussion on quality criteria for LOs Littlejohn et al. (2008) 3 Reflection on the past or future of the LO movement Metros (2005) 4 Short introduction to LOs Robson (2001) 5 Total 71 Data analysis Scenario development The development of scenarios started with open coding (Glaser & Strauss, 1967); fragments were searched for statements on how to determine the size of LOs. This search resulted in a large number of statements, which were then clustered. Clustering means putting together various statements that, upon analysis, turned out to express the same thing. Subsequent scenario development was based on conflicting clustered statements (for example, “LOs are hierarchically organised from very small to very large” versus “one LO addresses one learning objective”). Each pair of conflicting statements was investigated to see whether they could serve as the basis of two separate scenarios. As a first step, for each potential scenario, an attempt was made to link it to a unique definition of reusability. Furthermore, each scenario was scrutinised for other possibly relevant characteristics of the LOs themselves, the collection of LOs, the development process, or the way in which the collection is put to use. This procedure was repeated several times to obtain scenarios that were exclusive and that were as different from each other as possible. Figure 1 displays the procedure. 254 Australasian Journal of Educational Technology, 2012, 28(2) Figure 1: Procedure for scenario development Test on face validity This procedure raises the larger issue of validity, as the scenarios were developed by the author alone, and complete scenarios were rarely found in the literature. As a first step, face validity was tested by three experts on e-learning. The experts considered the scenarios to be understandable, easily imaginable, clearly recognisable in practice, clearly distinct from each other, with differences between them corresponding to differences in reality, and potentially useful for learning object developers. Field test Subsequently, a field test was performed to test the reliability and usability of the scenarios. To investigate the reliability of scenario classification, eleven educational experts classified scenarios in texts on LOs. These experts were recruited from among my colleagues. None of these educational experts was a specialist in LOs and their experience with e-learning varied widely. The experts were deliberately chosen from a broader audience than the immediate target group of educational technologists in an effort to investigate not only whether the scenarios would be useful for the target group but also whether they would be understandable for a broader audience. Text fragments were selected that at least seemed to contain characteristics of one specific scenario. Some text fragments had been used for the development of the scenarios, (alternate) clustering of principles detection of two (alternate) conflicting principles for determining LO size use conflicting principles as base for separate scenarios declustering of principles can one definition of reuse be formulated? is the definition of reuse unique for this scenario? split scenario can scenarios be merged, with one definition of reusability and one principle for determining LO size? keep scenario; search for additional characteristics for each scenario yes no no yes yes no no Schoonenboom 255 whereas others were new. The experts were then presented with a description of the scenarios. For each of 22 fragments, they were asked to indicate which scenario they recognised. Their judgments were compared to each other and to the judgment of the author. The degree of agreement was calculated using Krippendorff’s (2004) alpha. Results – the four scenarios Four scenarios were distinguished, and their main aims, respectively, are sharing LOs, flexibility in developing instruction, reusability in different domains, and, as borrowed from Boyle (2003), LO cohesion. The scenarios are summarised in Table 2. Table 2: Characteristics of the four learning object scenarios Defining characteristics for all scenarios Partially defining and non-defining characteristics Label/ aim of LO develop- ment Definition of reusability LO is smallest object… Design process Size of LOs in collection Aggre- gation of LOs Sets of objects are used together Metadata Sharing The extent to which LOs can be used in treating the same topic in the same domain with the intended audience that will be used separately by the intended audience Collect and create shareable elements From small to large No No Very important for finding LOs Flexibility The flexibility with which the content of the repository can be recombined within a set that can be used in different sequences Disassem- ble and reassemble From small to large Yes Yes Less important for finding LOs; important for automatic coupling Different domains The number of different topics and domains in which an LO can be used [unclear] Collect and create shareable elements Small No No Very import- ant for finding LOs; creating understand- able descript- ions for differ- ent domains is very difficult Cohesion Occurs if the LO addresses one learning object- ive or one idea that addr- esses one learning objective Compose on the basis of one learning objective Relative- ly small; LOs differ little in size No No A good description of the learning objective suffices The Sharing scenario in the literature According to Downes (2004), the problem of determining LO size centres around what will be shared. He sees sharing as an important goal of developing LOs. Downes focuses on sharing between institutions and on sharing materials on particular topics between those who teach these topics. Thus, for Downes (2004), sharing is about 256 Australasian Journal of Educational Technology, 2012, 28(2) sharing content on the same topic at a long distance geographically. Quinton (2007) argues that the concept of LOs is not new in the sense that many lecturers and teachers are familiar with sharing and re-applying instructional resources to their teaching programs. What is new with LOs is that these resources are digitised, tagged, and saved to a repository so that they can be quickly located, accessed, and easily modified and updated. Therefore, for sharing, a repository in the sense of an accessible place for objects is essential. Hamel & Ryan-Jones (2002), for example, compare instructional designers who are designing content objects for their own training needs to those who design for sharing and who design content that is independent of a training context for sale or reuse by other designers. They argue that decisions on granularity will be different in both cases. Sharing LOs can lead to a beneficial ‘LO economy’. On one hand, shared objects need to be developed only once and not separately at each institution (Elliott & Sweeney, 2008). On the other hand, sharing gives contributors access to a much larger pool of resources, which they can use and reuse (Duncan, 2003). Most examples of the Sharing scenario referred to sharing between institutions (McCormick, 2003; Muirhead & Haughey, 2005). Sharing between institutions has also been dominant in the study author’s previous research (Schoonenboom, 2006a, 2006b; 2006). Characteristics of the Sharing scenario In Table 2, the first three columns present the defining characteristics of the Sharing scenario. • The Sharing scenario aims at optimising the sharing of materials between instructors who teach or develop courses on the same topics. • In the Sharing scenario, LOs should be so small that people will want to share them (Downes, 2004). Schoonenboom (2006b) gives a more precise definition of optimal LO size in the Sharing scenario, noting that pieces that will be used together by the audience should be put together. This so-called principle of common use was developed and applied by Jan Rasenberg in a collaborative project on teacher training in the Netherlands. In his view, the estimated use by the intended audience is the key factor that determines LO size. This principle of common use states that pieces that will always be used together by the intended audience should be kept together. Only if components will be used separately should they be developed into separate LOs. • Naturally, application of the principle of common use leads to LOs that differ in size depending on their type. Assignments, for example, are usually much smaller than theoretical sections, as assignments will be used separately more readily. In Rasenberg’s project, the study load per LO varied from 1 hour up to 30 hours. Thus the principle of common use can explain why, in the Sharing scenario, resources that are shared can vary between “as small as a single picture or a block of text, or as large as an entire teaching programme” (Quinton, 2007, p. 145). • In accordance with the principle of common use, LOs in a repository are designed to be used separately. However, this categorisation does not preclude them from being used together, and one may, for example, use Exercise 1 together with Schoonenboom 257 Exercise 9. However, this incidental use of combinations is not recorded in the repository as a shared relationship between LOs. • Essentially, in the Sharing scenario, the focus is on sharing between institutions. Because the intended users are different people than the developers, good metadata is essential for finding the LOs. As sharing is intended between an audience that teaches the same topic, metadata can be based upon the jargon that the intended audience utilises. • The process of developing LOs consists of selecting shareable elements from existing materials and developing new shareable elements. Quinton (2007) views these as resources that can be combined and recombined with other resources and are, themselves, generic pieces of knowledge (or “knowledge bits”) that the educator can pass onto others. The Flexibility scenario in the literature Douglas (2001) describes how a larger LO, which includes a discussion of a concept, an application, and a problem to test the learner’s understanding, may be divided into three separate objects. This approach leads to flexibility for the user, who may choose one or two of the LOs, or use them in a different order. The idea is that, through disassembling a larger object into its constituent parts, parties can reassemble new larger objects from these parts. Muzio et al. (2002) remark that this process of disassembling and reassembling mimics the known actions of instructors. If they are given access to instructional materials, users tend to break down these LOs into their constituent parts and reassemble them in ways to support their individual instructional goals. Boyle (2003), who uses the term “re-purposing” for this flexibility, relates repurposing to storing LOs at several layers or levels of aggregation. At the lowest level, simple objects are found. For example, in the previous sample, three simple objects exist: a discussion, an application, and a problem. At higher levels compound objects are found that are composed of at least two simple objects (for example, a discussion plus an application). According to Boyle (2003), compound objects provide a richer pedagogical experience than simple objects. The idea of compound hierarchically- organised LOs is found throughout LO literature, including, prominently, the Learnativity model (Wagner, 2002). Many authors argue that simple objects should be as small as possible to maximise the possibilities of putting them back together in a different order (Muzio et al., 2002; Quinton, 2007; Silveira et al., 2007). Conlan, Dagger and Wade (2002) even propose development of textual LOs as small as paragraphs so that the instructional designers can add, remove, or resequence content at this level to produce tailored courses. Several authors point out that differences in the order of presenting learning materials exist between various pedagogical approaches. For example, Hamel and Ryan-Jones (2002) suggest presenting problems first in a problem-based approach, and presenting conceptual materials first in other approaches (see also Quinn, 2000; Quinton, 2007). Others relate differences according to the preferences of various learners. Hamel and Ryan-Jones (2002) note that providing alternative sequences in combination with a standard route enables learners to choose their preferred learning path (see data on the relation to learners' needs in Muirhead & Haughey, 2005; Quinn, 2000; Quinton, 2007). 258 Australasian Journal of Educational Technology, 2012, 28(2) An example of the Flexibility scenario in practice is given by Farrell and Carr (2007). They describe how they decomposed an instruction on information literacy on a university library website into smaller instructional units and rebuilt the instruction at several levels of aggregation as follows: the course, the instructional unit (e.g. “finding books”), and the learning objective (”search the catalogue”). In this way, students were given maximum freedom to start at the topic and level of aggregation they liked and study a sequence of LOs of their own choosing. Characteristics of the Flexibility scenario • The Flexibility scenario aims at optimising the flexibility with which the content of the repository can be recombined. • In this case, the size of LOs is determined by the smallest piece that can be placed into a different order or removed from the set. This determination then controls the size of the simple LOs; compound objects are comprised of various combinations of simple objects. • In the Flexibility scenario, LOs are developed in a process of disassembling and reassembling. Existing or newly developed larger LOs are split into their smallest constituent parts, the simple LOs. These simple objects are subsequently recombined into new, larger, compound LOs. This process of disassembling and reassembling in the Flexibility scenario has consequences for the repository. • The repository will contain hierarchically-layered objects that vary in size: specifically, compound objects together with their constituting simple objects. • The repository will contain sets of LOs that are meant to be used together, namely sets of simple objects that are the result of disassembling one larger object. As both simple and compound objects are stored in the repository, the Flexibility scenario is very labour intensive because all objects have to be described and stored separately (namely as a simple object). It will be clear that using the criterion of maximum flexibility for determining LO size can lead to very small LOs, and even, according to Conlan et al. (2002), down to the size of a paragraph. • In the Flexibility scenario, sharing LOs with outsiders is not a main goal. Thus, metadata in the form of key words comprehensible for human users are less important. In this scenario, metadata can be used by computer systems to select and to sequence LOs automatically based on a given pedagogical approach or a learner’s preference for a specific learning path. The Different domains scenario in the literature Although there is agreement that LOs should be reusable in different contexts (Duncan, 2003; Polsani, 2004; Silveira et al., 2007; Wiley, 2001), many references to this principle in the relevant literature do not specify what these different contexts are. Sometimes, the phrase different contexts refers to use with different content areas or use in different domains. In this setting, the claim is often made that the smaller the LO, the greater its potential for reuse. McGreal (2004), for example, states that a video clip of the World Cup could form part of lessons in kinesiology, sports science, politics, history, or media studies. Cheal and Rajagopalan (2007) claim that a simple art image Schoonenboom 259 could be used in a course on art history, religious studies, history, journalism, or in an art studio (in order to discuss its style and function, religious context, its historical context, as evidence for an event, or to incorporate or to modify it within another’s artwork, respectively). Quinn (2000) explains how an LO that discusses how vehicles behave differently with and without anti-lock brakes might be used in several different educational domains, such as the physics of friction, automotive design, or insurance liability. In these examples, reuse means reuse in domains different from the domain in which the LO had been developed. Wiley (2001) calls this type of reuse “inter-contextual use.” Wiley defines the potential of an LO for inter-contextual use as “the number of different learning contexts in which the LO might be used, that is, the object’s potential for reuse in different content areas or domains”. The counterpart of inter-contextual reuse, according to Wiley (2001), is intra-contextual use, which is the number of times the LO might be reused within the same content area or domain. Many authors state that smaller LOs have the greatest potential for inter-contextual reuse. Additionally, LO type is important. Some authors consider images to be the LOs with the greatest potential for inter-contextual reuse. It is no coincidence that two of the three examples of inter-contextual reuse mentioned above concern stills or moving images. Conole, Evans and Sims (2003) state that “because of their high degree of granularity, images, more than any other LOs, have the greatest potential for creative, inter-contextual reuse across multiple subject areas”. The Different domains scenario has been around from the very beginning of the LO literature. Its appeal likely derives from a “primal” scenario of reuse, and certainly it is the one that most appeals to one’s imagination. However, as opposed to the other three scenarios, I did not find applications of the Different domain scenario in practice. Characteristics of the Different domains scenario • The Different domains scenario seeks to maximise different topics and domains in which the content of the repository can be used. As in the other scenarios, LOs should be as small as possible. However, the characteristics of this scenario have not been elaborated in great detail. • It remains unclear how the decision on LO size is made. • In the Different domains scenario, excellent metadata are of the utmost importance. A potentially useful LO should be traceable by people outside the domain in which the LO had been developed. At the same time, however, attaching key words will be very difficult. It will be necessary to avoid the use of jargon as much as possible because people outside the domain very often will not master the domain’s jargon. • In the Different domains scenario, smaller LOs are not aggregated into larger objects, and the repository does not contain sets of LOs that should be used together. • Learning objects are created by collecting or developing objects that are expected to be used within a number of different domains. 260 Australasian Journal of Educational Technology, 2012, 28(2) The Cohesion scenario in the literature The scenarios presented until now “focus on how LOs are created, used, and stored, rather than on what LOs look like” (Smith, 2004, p. 1). On the basis of this last criterion, Smith (2004) uses a definition of LOs that is often found in the literature, namely that LOs should be based on one learning objective (see e.g., Cisco Systems, 2003). Boyle (2003) calls this principle cohesion. In this context, “based upon” means that LOs should address one learning objective (Hamel & Ryan-Jones, 2002). By working their way through the LO, learners should achieve that singular learning objective (Boyle, 2003). There are a few variants of this idea of cohesion. Polsani (2004) states that an LO should contain only one or a few related ideas. In the latter case, one of the ideas should be the main idea, and the others should be derived from it or be dependent upon it. South and Monson (2001) state that LOs should centre on a single, core concept. As an example, they discuss an interactive simulation that allows a learner to manipulate a pressure gauge, the shape of a container of liquid in which it is submerged, and the depth of that liquid. Some authors relate the principle of cohesion to the potential for reuse. South and Monson (2001) argue that LOs that centre on a single core concept have the greatest potential for instructional reuse because this categorisation is the smallest level at which the object still has significant instructional utility. Characteristics of the Cohesion scenario • In the Cohesion scenario, each LO addresses one idea or one learning objective. • There is no hierarchical layering of aggregated objects. • Learning objects are meant to be used independently of each other, and the repository does not contain sets of objects that should be used together. In the repository, learners and teachers search for LOs on the basis of a learning objective they wish to achieve. • Concerning the metadata, a clear description of the LO is especially important. Other information on LO type and size is less important because each LO is supposed to lead to the acquisition of one specific learning objective. • Learning objects are developed on the basis of their learning objective by asking what instruction and exercise are necessary to achieve a particular objective. Results – the expert tests The overall results of the field test were on first sight rather mediocre: the reliability score for recognising the scenarios was 0.65 (Krippendorff’s alpha). On closer inspection, however, recognition problems turned out to be concentrated in specific experts and fragments. One of the 11 experts differed in his judgments from the other experts, and three out of 22 fragments led to much more varied choices of scenarios than the others. Removing this deviant expert and these fragments removed 21% of the data and led to an acceptable Krippendorff’s alpha of 0.79. Schoonenboom 261 Conclusions In this paper, four global scenarios for the development of LOs and determining LO size have been presented (research question 1). Three of the four scenarios have their own principle for deciding on LO size (research question 2). LOs are the smallest objects that (a) will be used separately by the intended audience (Sharing); (b) can be used within a set in different sequences (Flexibility); (c) address one learning objective (Cohesion). Between the four scenarios, several more differences exist (research question 3), which are listed in Table 2. Each scenario is characterised by its own unique aim of LO development and definition of reusability. Furthermore, Table 2 shows that aggregated objects at several hierarchical levels are only found in the Flexibility scenario. The question whether one aims at reuse between different domains or reuse within one domain determines not only the extent to which LOs can differ in size but also how difficult it will be to attach the right metadata. The process of LO development is different with each scenario. If the most important goal is cohesion, LOs are developed to address a specific learning objective. When striving for maximum flexibility, larger LOs are disassembled into smaller objects, and these are then used to build compound objects with different selections and sequences of objects. If the aim is sharing or reuse in different domains, shareable elements are collected or designed. The results of the expert tests are very encouraging for two reasons. First, as the experts were both educational technologists and educationalists without a technical background, the scenarios were generally recognisable by a broader audience. Second, there was no reason to expect that recognising the scenarios would be easy. They had not been identified as such in the existing literature and developing them required a very thorough and difficult analysis. The conclusion is that the scenarios are well recognisable for a broader audience, although not for everyone and not in all cases. Discussion This study is useful for learning object practice in several ways. Firstly, the scenarios can be used to classify existing cases of LO development and use. With respect to the examples at the beginning of this article, the case of the university library is clearly a Flexibility scenario, and it closely resembles the example that was found in literature. In principle, students follow all or a substantial subset of modules, and the aim is to give them as much flexibility as possible in selecting their own subset of modules to follow their own route. The development process consists of disassembling the whole library course into separate small modules and to build from these modules different selections and routes, which might eventually be stored as separate learning objects in the collection. Additionally, where applicable, variants of modules were created that contained examples from the disciplines of the various departments. Assuming that the students start at the library’s course page, students can browse for modules, so metadata for finding separate modules are of less importance. The size of the modules can instead be determined by investigating what flexibility is needed in following the modules: Which parts should students wish to select separately, or follow in a different order? The knowledge engineering case is a clear example of the Sharing scenario. Contrary to the library case, students are not expected to follow all or a large part of the materials in the collection, and students’ preferences for a specific order do not really play a role. 262 Australasian Journal of Educational Technology, 2012, 28(2) Rather, the teacher is the person who selects a very small subset of materials to their own needs. There is no large course that is split up into smaller modules. Learning materials are delivered separately, and reusable learning materials are collected. The size of LOs is determined on the basis of the principle of common use. In this case, teachers wanted to use materials separately at the level of the study task: tasks with a study load of approximately 15 hours consisting of a task, learning goals, a test of prior knowledge, resources, and tools. Statements from the literature can be classified using the scenarios as well, and contradictions found in the literature can be explained through the four scenarios. In their LO pilot, Muirhead and Haughey (2005) found far fewer examples of object aggregation than expected. This discrepancy can be explained because their pilot fits the Sharing scenario while LO aggregation fits the Flexibility scenario. Furthermore, Muirhead and Haughey (2005) observe that the LOs in their pilot, contrary to expectation, were not all very small. They varied in size, ranging from individual images to complete courses. This range also fits in well with the Sharing scenario, in which the size of LOs is determined by what objects the intended audience will want to use separately. A repository with only small LOs would fit in with the Different domains scenario or as the lowest layer of simple objects within the Flexibility scenario. The case study of Farrell and Carr (2007), discussed earlier in this paper, uses aggregated objects at several levels. Their main goal was to make the access to their redeveloped library course as flexible as possible for various students. Sharing with others or use outside their own domain were not considered. Still more important, the scenarios can be used in making decisions during LO development. Educational technologists can use the four scenarios and their differences as displayed in Table 2 to assist LO developers. Table 2 can be used to determine which definition of reusability is applicable in a specific situation and on the basis of this determine which scenario (and thus which principle for determining the proper size of an LO) should be applied. In practice, the four goals will sometimes go together. One can imagine that an LO developer strives for the largest possible flexibility in presenting a specific unit of learning, and, at the same time, aims at sharing with others the simple objects of this unit of learning together with the compound objects built from them. Possibly, this co- occurrence of goals is one of the reasons why these four scenarios have not been explicitly identified before. Yet, it is conceivable that, in practice, one goal will often be more important than the others, and in that case the four scenarios and Table 2 can be helpful to LO developers. Finally, Table 2 shows that, in practice, it might be difficult to aim at achieving various goals within one LO project because the various goals put different requirements on the LOs and their development process. Acknowledgments Thanks to the Dutch Digital University for funding the project Learning objects in practice. Thanks to Frank Kresin, Silvester Draaijer, members of the HogCog research group, and members of the research group Higher Education for their comments on an earlier draft of this paper. Thanks to Joyce Aalberts, Joyce Brouwer, Gerdien Jansen, Hanna Kuijs, Thea van Lankveld, Hester Radstake, Christoffel Reumer, Sjoerd Sinke, Pieter Swager, and Mark te Wierik for their contribution to the expert test. Schoonenboom 263 References Bennett, S., Agostinho, S., Lockyer, L., Harper, B. M. & Lukasiak, J. (2006). Supporting university teachers create pedagogically sound learning environments using learning designs and learning objects. IADIS International Journal on WWW/Internet, 4(1), 15-25. http://www.iadis.net/dl/final_uploads/2006410102.pdf Boyle, T. (2003). Design principles for authoring dynamic, reusable learning objects. Australian Journal of Educational Technology, 19(1), 46-58. http://www.ascilite.org.au/ajet/ajet19/boyle.html Cheal, C. & Rajagopalan, B. (2007). A taxonomy showing relationships between digital learning objects and instructional design. In A. Koohang & K. Harman (Eds.), Learning objects and instructional design (pp. 59-88). Santa Rosa, CA: Informing Science Press. Cisco Systems (2003). Reusable learning object strategy: Designing and developing learning objects for multiple learning approaches. Version 4.5. http://www.e- novalia.com/materiales/RLOW__07_03.pdf Conlan, O., Dagger, D. & Wade, V. (2002). Towards a standards-based approach to e-learning personalization using reusable learning objects. In G. Richards (Ed.), Proceedings of World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education 2002, Montreal, Quebec, Canada (pp. 210-217). Chesapeake, VA: AACE. https://www.cs.tcd.ie/Owen.Conlan/publications/eLearn2002_v1.24_Conlan.pdf Conole, G., Evans, J. & Sims, E. (2003). Use and reuse of digital images in teaching and learning. In A. Littlejohn (Ed.), Reusing online resources: A sustainable approach to e-learning (pp. 156-170). London: Kogan Page. Doorten, M., Giesbers, B., Janssen, J., Daniels, J. & Koper, R. (2004). Transforming existing content into reusable learning objects. In R. McGreal (Ed.), Online education using learning objects (pp. 116-127). London: Falmer Press. Douglas, I. (2001). Instructional design based on reusable learning objects: Applying lessons of object-oriented software engineering to learning systems design. Paper presented at the Annual Frontiers in Education Conference, Reno NV, October 10-13. http://dx.doi.org/10.1109/FIE.2001.963968 Downes, S. (2004). Learning objects: Resources for learning worldwide. In R. McGreal (Ed.), Online education using learning objects (pp. 21-31). London: Falmer Press. Duncan, C. (2003). Granularization. In A. Littlejohn (Ed.), Reusing online resources: A sustainable approach to e-learning (pp. 12-19). London: Kogan Page. Elliott, K. & Sweeney, K. (2008). Quantifying the reuse of learning objects. Australasian Journal of Educational Technology, 24(2), 137-142. http://www.ascilite.org.au/ajet/ajet24/elliott.html Falloon, G., Janson, A. & Janson, R. (2010). Learning objects and the development of students' key competencies: A New Zealand school experience. Australasian Journal of Educational Technology, 26(5), 626-642. http://www.ascilite.org.au/ajet/ajet26/falloon.html Farrell, K. & Carr, A. E. (2007). A blended model of instructional design for learning objects. In A. Koohang & K. Harman (Eds.), Learning objects and instructional design (pp. 359-406). Santa Rosa, CA: Informing Science Press. Glaser, B. G. & Strauss, A. L. (1967). The discovery of grounded theory: Strategies for qualitative research. New York: Aldine de Gruyter. Hamel, C. J. & Ryan-Jones, D. (2002). Designing instruction with learning objects. International Journal of Educational Technology, 3(1). http://www.ascilite.org.au/ajet/ijet/v3n1/hamel/ Harzing, A.-W. (2010). Publish or Perish, version 3.0. http://www.harzing.com/pop.htm 264 Australasian Journal of Educational Technology, 2012, 28(2) Krippendorff, K. (2004). Content analysis: An introduction to its methodology (2nd ed.). Thousand Oaks, CA: Sage Publications. Littlejohn, A. (2003). Issues in reusing online resources. In A. Littlejohn (Ed.), Reusing online resources: A sustainable approach to e-learning (pp. 1-6). London: Kogan Page. Littlejohn, A., Falconer, I. & McGill, L. (2008). Characterising effective elearning resources. Computers & Education, 50(3), 757-771. http://dx.doi.org/10.1016/j.compedu.2006.08.004 Lockyer, L., Bennett, S., Agostinho, S. & Harper, B. (Eds.) (2009). Handbook of research on learning design and learning objects: Issues, applications, and technologies. http://dx.doi.org/10.4018/978- 1-59904-861-1 McCormick, R. (2003). Keeping the pedagogy out of learning objects. Paper presented at EARLI 2003. http://www.eun.org/eun.org2/eun/Include_to_content/celebrate/file/ KeepingPedagogyOutOfLOs3v2.doc McGreal, R. (Ed.). (2004). Online education using learning objects. London: Falmer Press. Metros, S. E. (2005). Learning objects: A rose by any other name…. EDUCAUSE Review, 40(4), 12- 13. http://www.educause.edu/ir/library/pdf/erm05410.pdf Muirhead, B. & Haughey, M. (2005). An assessment of the learning objects, models and frameworks developed by the Le@rning Federation Schools Online Curriculum Content Initiative Australia: Phase two. Melbourne, Australia: The Le@rning Federation. http://www.thelearningfederation. edu.au/verve/_resources/muirhead_haughey_0105_file.pdf Müller, K. (2006). Structuring of content repositories for adaptive e-learning content: Reusabilty through metadata and ontologies. Hamburg, Germany: Hamburg University of Technology, Information and Mediatechnologies. http://www.sts.tu-harburg.de/pw-and-m- theses/2006/muell06.pdf Muzio, J. A., Heins, T. & Mundell, R. (2002). Experiences with reusable e-learning objects: From theory to practice. The Internet and Higher Education, 5(1), 21-34. http://dx.doi.org/10.1016/S1096-7516(01)00078-1 Pawson, R. (2006a). Digging for nuggets: How ‘bad’ research can yield ‘good’ evidence. International Journal of Social Research Methodology, 9(2), 127-142. http://dx.doi.org/10.1080/13645570600595314 Pawson, R. (2006b). Evidence-based policy: A realist perspective. London: SAGE. Polsani, P. R. (2004). Signs and objects: Modeling learning objects on Peirce’s theory of signs. In R. McGreal (Ed.), Online education using learning objects (pp. 104-115). London: Falmer Press. Q4R (2010). Q4R workflow models: Quality for reuse - from analysis to daily use of a high quality LOR. http://www.q4r.org/Q4RBestPractices/Q4RWorkflowModels/tabid/103/ language/en-US/Default.aspx Quinn, C. (2000). Learning objects and instruction components. Educational Technology & Society, 3(2). http://www.ifets.info/journals/3_2/discuss_summary_0200.pdf Quinton, S. (2007). Contextualisation of learning objects to derive meaning. In A. Koohang & K. Harman (Eds.), Learning objects: Theory, praxis, issues, and trends (pp. 113-179). Santa Rosa, CA: Informing Science. Robson, R. (2001). Learning object tutorial. http://www.eduworks.com/resource-library- 2/learning-object-tutorial/ Schoonenboom, J. (2006a). De omvang van leerobjecten (Vol. 4). Utrecht, The Netherlands: Stichting Digitale Universiteit. http://hdl.handle.net/1820/809 Schoonenboom 265 Schoonenboom, J. (2006b). A model for determining the size of learning objects. In Kinshuk, R. Koper, P. Kommers, P. Kirschner, D. G. Sampson & W. Didderen (Eds.), Proceedings of the 6th IEEE International Conference on Advanced Learning Technologies, Kerkrade, The Netherlands (pp. 46-50). Los Alamitos, CA: IEEE. http://dx.doi.org/10.1109/ICALT.2006.21 Schoonenboom, J., Hermans, H., Poortman, S., Sloep, P. & Benneker, F. (2006). Leerobjecten in de praktijk: Handreikingen voor het hoger onderwijs. Utrecht, The Netherlands: Stichting Digitale Universiteit. Sicilia, M. & Lytras, M. (2005). Scenario-oriented reusable learning object characterisations. International Journal of Knowledge and Learning, 1(4), 332-341. http://dx.doi.org/10.1504/IJKL.2005.008355 Silveira, I. F., De Araujo Jr, C. F., Amaral, L. H., De Oliveira, I. C. A., Schimiguel, J., Ledon, M. F. P. & Ferreira, M. A. G. V. (2007). Granularity and reusability of learning objects. In A. Koohang & K. Harman (Eds.), Learning objects and instructional design (pp. 139-170). Santa Rosa, CA: Informing Science Press. Smith, R. S. (2004). Guidelines for authors of learning objects. Austin, TX: The New Media Consortium. http://archive.nmc.org/guidelines/NMC%20LO%20Guidelines.pdf South, J. B. & Monson, D. W. (2001). A university-wide system for creating, capturing, and delivering learning objects. In D. A. Wiley (Ed.), The instructional use of learning objects: Online version. Bloomington, IN: Association for Educational Communications and Technology. http://reusability.org/read/chapters/south.doc Wagner, E. D. (2002). Steps to creating a content strategy for your organization. The eLearning Developers' Journal. http://www.elearningguild.com/pdf/2/102902MGT-H.pdf Wiley, D. A. (2001). Connecting learning objects to instructional design theory: A definition, a metaphor, and a taxonomy. In D. A. Wiley (Ed.), The instructional use of learning objects: Online version. Bloomington, IN: Association for Educational Communications and Technology. http://reusability.org/read/chapters/wiley.doc Wiley, D. A. (2002). Learning objects need instructional design theory. In A. Rossett (Ed.), The ASTD e-learning handbook (pp. 115-126). New York: McGraw-Hill. http://opencontent.org/docs/astd.pdf Author: Judith Schoonenboom, Department of Research and Theory in Education VU University Amsterdam, De Boelelaan 1105, 1081 HV Amsterdam, The Netherlands Email: judith.schoonenboom@vu.nl Please cite as: Schoonenboom, J. (2012). Four scenarios for determining the size and reusability of learning objects. Australasian Journal of Educational Technology, 28(2), 249- 265. http://www.ascilite.org.au/ajet/ajet28/schoonenboom.html