Australasian Journal of Educational Technology, 2017, 33(5). 136 Designing and integrating reusable learning objects for meaningful learning: Cases from a graduate programme Joyce Hwee Ling Koh National Institute of Education, Nanyang Technological University, Singapore E-learning quality depends on sound pedagogical integration between the content resources and lesson activities within an e-learning system. This study proposes that a meaningful learning with technology framework can be used to guide the design and integration of content resources with e-learning activities in ways that promote learning experiences, characterised by five dimensions: active, constructive, intentional, authentic, and collaborative. The pedagogical uses of these meaningful learning dimensions to support the design and integration of reusable learning objects as content resources will be explicated and exemplified through three cases related to the instruction of theories, principles, and professional skills respectively in a graduate programme. Design notes and surveys of students’ perception of learning experiences are used as data sources to understand how the five meaningful learning dimensions are being implemented by instructors and perceived by students. The strategies for supporting meaningful learning with reusable learning objects in higher education contexts are discussed. Introduction Meaningful learning refers to learning that involves the active participation of students in experiences that are cognitively engaging (Ausubel, 1963). To engender meaningful e-learning, it is important that digital content resources are thoughtfully integrated with learning activities as a learning system (Bannan-Ritland, Dabbagh, & Murphy, 2000; Boyle, 2010; Ritzhaupt, 2010). Reusable learning objects (RLOs) are digital content resources with learning activities and assessment to fulfil specific learning objectives (Wiley, 2001). Designed to be independent and interoperable, learning units that can be reused across lessons, courses, or even programmes, RLOs can potentially lower the costs of accessing content resources for e-learning. While the independent nature of RLOs supports reusability, their effectiveness can be curtailed if integrated without clear pedagogical intents (Sridharan, Deng, & Corbitt, 2010). Even though there are pedagogical categories of RLOs (e.g. Churchill, 2007), there are few established e-learning strategies at the lesson-level (Hew & Cheung, 2014) to guide the meaningful synthesis and integration of RLOs. The dimensions of meaningful learning with technology proposed by Howland, Jonassen, and Marra (2013) can be used as a pedagogical framework to guide the design and use of RLOs for e-learning. These authors purport that meaningful learning with technology involves the engagement of students in experiences characterised by active exploration, authentic problem-solving, constructive thinking, collaboration, and intentionality. These dimensions can be used as guidelines to integrate RLOs in ways that support meaningful e-learning. This paper therefore discusses three cases of how the meaningful learning with technology dimensions of Howland et al. (2013) could be applied to the integration of RLOs to support e-learning in a graduate programme. The pros and cons of the integration strategies as well as the implications of using RLOs to support e-learning in higher education contexts will be discussed. Literature review What are RLOs? RLOs are conceptualised as accessible, reusable, interoperable, and adaptable learning resources to facilitate developmental cost savings (Bannan-Ritland et al., 2000). Therefore, RLOs need to be designed as independent learning units that are free from context as well as links with external resources (Green, Jones, Pearson, & Gkatzidou, 2013). While RLOs have sometimes been loosely defined as digital content resources, Wagner (2002) argues that resources such as photographs, audio, and video files are better considered as content assets for formulating basic content related to concepts, facts and procedures. Australasian Journal of Educational Technology, 2017, 33(5). 137 According to Wagner (2002) and McGreal (2004), RLOs are collections of basic content targeted at the attainment of specific learning objectives. Several studies examine how RLOs can be better classified to facilitate accessibility and reusability. Wiley (2001) and Churchill (2007) categorise learning objects by their pedagogical functions which include presentation, practice, and simulations. Hendrix et al. (2012) focus on serious games as learning objects and propose that serious games can be meta-coded by attributes such as learning objectives and game type. On the other hand, Bannan-Ritlan et al. (2000) argue that constructivist learning means that student-produced artefacts should be included within RLOs. These studies focus on the pedagogical outcomes of RLOs. However, Wagner (2002) asserts that learning experiences need not be constrained to RLOs as these can be enhanced with different tools and activities within the learning environment. In fact, learning effectiveness is found to be highly influenced by how RLOs are being integrated with learning activities (Boyle, 2010). It appears that the pedagogical integration of RLOs can be fairly complex, going beyond the mere consideration of RLO functionality. Ways of integrating RLOs Several studies provide insights about how RLOs could be integrated. RLOs have firstly been used as standalone resources to facilitate independent learning (Sridharan et al., 2010; Watson, 2010). In such contexts, RLOs are typically designed with feedback mechanisms to facilitate students’ practice and self-evaluation. Watson (2010), for example, assigned RLOs to specific students for remedial purposes. RLOs have secondly been used to support blended modes of e-learning where there are both classroom-based and online lesson activities (Vincenti, Braman, & Hilberg, 2013; Watson, 2010). In computer science courses for example, lecturers use RLOs as tools to demonstrate programming techniques during lectures and students use them as simulations for practice after lectures (Vincenti et al., 2013). A third way of using RLOs is as an on-demand resource as exemplified by Gee, Strickland, and Salazar (2014) for the training of occupational therapists. In this study, students access RLOs for content review before field-based practice where needed. When discussing the critical success factors for e-learning, Sridharan et al. (2010) emphasise that learning quality is brought about by how content resources such as RLOs can support “student-centered interactive e-learning” (p. 264). Extant e-learning pedagogical models, however, tend to address programme-level concerns such as institutional support, infrastructure, content readiness, instructor readiness, and student readiness (Hew & Cheung, 2014). E-learning studies which discuss blended learning models suggest several ways for sequencing online and face-to-face learning experiences. For example, Twigg’s (2003) supplemental blend and Staker’s (2011) self-blend propose that online resources can be used to supplement existing face-to-face classes. On the other hand, Twigg’s (2003) buffet blend and Staker’s (2011) flex blend suggest that students can mix-and-match online and face-to-face options according to their needs. Even though some ideas for integrating RLOs can be gleaned from blended learning models, Hew and Cheung (2014) comment that these may not provide sufficient guidance as lesson-level strategies. The integration of RLOs to support meaningful learning still remains a gap in extant research. Meaningful Integration of RLOs Ausubel (1963) explains that meaningful learning involves students in the cognitive interpretation of their experiences rather than the regurgitation of information. Similarly, Howland et al. (2013) envision meaningful technology-integrated learning as having five dimensions. These are: active, authentic, constructive, intentional, and collaborative. The active dimension refers to the use of technology to support data manipulation and discovery whereas the authentic dimension suggests that real-world problems should be the subject of such explorations. Technology-based learning experiences should target higher-order thinking as denoted by the constructive dimension. These should also be intentional where students are given opportunities to be in control of their learning goals. Finally, the collaborative dimension suggests that technology-based learning experiences should support the social construction of knowledge through engagement with either expert or peers. Howland et al. (2013) argue that when guided by these five dimensions of learning, technology tools become mindtools that aid cognition and learning rather than serve as tools for presentation of content information. Australasian Journal of Educational Technology, 2017, 33(5). 138 The need for meaningful learning during e-learning and blended learning has been mentioned (Tsai, Shen, & Chiang, 2013). Yunianta, Yusof, Othman, and Octaviani (2012) argue that the five dimensions of Howland et al. (2013) can be used to analyse how different kinds of e-learning activities contribute to meaningful e-learning. There is therefore scope to consider if the dimensions of meaningful learning can be applicable for guiding the integration of RLOs. Theoretical framework and research questions Meaningful learning rubric Koh (2013) proposes that depending on instructors’ lesson goals, each of the meaningful learning dimensions conceptualised by Howland et al. (2013) can be given different levels of prominence in a lesson. This is articulated as a meaningful learning rubric that rates each meaningful learning dimension with different levels. As explained in Koh (2013), the rating scale of the meaningful learning rubric was derived from content analysis of 270 lesson activities from 55 technology-integrated lesson plans designed by Singapore pre-service teachers. The rubric was also used to support 37 in-service teachers’ technology-integrated lesson planning and they have found that the dimensions of meaningful learning served as effective pedagogical scaffolds for helping them to set targets regarding the kinds of student-centered technology-integrated learning they wanted to achieve (Koh, Chai, & Lim, 2016). Building upon the approach of Koh (2013), it is proposed that the meaningful learning dimensions of Howland et al. (2013) can be defined for different levels of RLO integration. However, as suggested by Bannan-Ritland et al. (2000), it is necessary to consider RLOs as components within instructional systems. Therefore, the meaningful learning dimensions need to be applied by considering students’ entire blended learning experiences from both RLOs as well as the associated learning activities beyond the RLOs. By adapting from the meaningful learning rubric of Koh (2013), this study suggests that the five dimensions of Howland et al. (2013) can be defined for the integration of RLOs for different levels as follows: • Active – In e-learning contexts, student participation is a key indicator of learning quality (McLoughlin & Lee, 2008). Correspondingly, learning experiences are highly active when RLOs and the associated learning activities provide students with opportunities to be actively doing as opposed to merely receiving information from media and technology. Examples of doing with technology could be manipulation and exploration of information, creation of artefacts, engagement in practice exercises or engagement in discussion and reflection. • Constructive – While the active dimension deals with the participation of students, the constructive dimension analyses the level of their cognitive engagement, that is, being active does not imply that students are constructive (Chi, 2009). Specifically, the learning experiences provided by RLOs and the associated learning activities need to engage students in divergent thinking where content information is being interpreted and synthesised to derive new forms of understanding. A learning experience may be highly active but when it largely engages students in the regurgitation of content, it is still considered to be low in the constructive dimension. In this respect, it is proposed that students’ cognitive engagement with the lesson content could be determined through the levels stated by Bloom’s taxonomy (Krathwohl, 2002). Lower levels of the constructive dimension would involve learning experiences at the knowledge and comprehension levels of Bloom’s taxonomy whereas experiences involving levels such as evaluation and creation of the taxonomy could be considered as being high on the constructive dimension. • Intentional – The intentional dimension is akin to analysing how learning experiences addresses the motivations and goals of students, which are important indicators of quality in e-learning (McLoughlin & Lee, 2008). This dimension addresses the extent to which students are engaged in the understanding of learning gaps, setting of learning goals, and the resolution of learning gaps (Koh, 2013). Therefore, high levels of intentionality within a lesson supported by RLOs could be interpreted as the presence of multiple opportunities for feedback and remediation. • Authentic - Howland et al. (2013) explain that authenticity in learning implies that learning is situated in the real-world tasks, cases, or problems that one would apply the learning to. Koh (2013) emphasises that authenticity is enhanced with students finding personal applications for the knowledge learnt. Therefore, the authentic dimension can be considered as low when RLOs Australasian Journal of Educational Technology, 2017, 33(5). 139 and associated lesson activities involve the mere presentation of real-world phenomenon whereas learning experiences providing students with opportunities to analyse and find personal meanings of the phenomenon are considered to be higher in the authentic dimension. • Collaborative – Collaboration has been emphasised as an e-learning strategy for evoking student participation (Ossiannilsson & Landgren, 2012; Sridharan et al., 2010). The collaborative dimension of meaningful learning emphasises the need for tasks that engages students to negotiate different perspectives and meanings with each other (Howland et al., 2013). Therefore, the more RLOs and associated learning activities provide students with opportunities to engage in divergent conversations, the stronger the collaborative dimension is being featured within the lesson design. Adapting the rubric of Koh (2013), three levels for each meaningful learning dimension – low, medium, and high, are proposed as follows (Table 1): Table 1 Meaningful learning rubric adapted from Koh (2013) Low Medium High Active Students primarily involved in receiving content through information transmission Students have some opportunities to engage in doing e.g. manipulating and exploring information, making, practicing, reflecting, discussing etc. Students primarily involved in doing e.g. manipulating and exploring information, making, practicing, reflecting, discussing etc. Constructive Students engage in knowledge and comprehension of lesson content Students use lesson content to support application or analysis Students use lesson content to support evaluation or creation Authentic Minimal use of real-world examples related to lesson content Students analyse real-world examples related to lesson content Students articulate personal application of lesson content Intentional Students have minimal opportunities for feedback and remediation Students engage in one cycle of feedback and remediation Students engage in multiple rounds of feedback and remediation Collaborative Predominantly individual work, OR Group-based engagement but with convergent discussions Some level of engagement in divergent group discussion Predominantly engaged in divergent group discussions When using this rubric, the levels attained by the RLO as well as through the combination of the RLO and its associated activities in an entire blended learning experience could be analysed for each meaningful learning dimension (Cases 1, 2, and 3 in the Findings section). In this way, the meaningful learning dimensions of Howland et al. (2013) could be used as pedagogical anchors to guide the integration of RLOs. By using this framework, the utility of RLOs could be extended beyond the different kinds of inherent pedagogical affordances as described by authors such as Wiley (2001) and Churchill (2007). Where blended models of e-learning (e.g. Staker, 2011; Twigg, 2003) addresses design at the programme-level, the meaningful learning framework extends pedagogical design to the lesson-level. This could address the general dearth of lesson-level strategies as identified by Hew and Cheung (2014). Australasian Journal of Educational Technology, 2017, 33(5). 140 Research questions Given the preceding analysis, the two research questions examined in this study are: 1. How can the meaningful learning dimensions be used to design and integrate RLOs? 2. What are students’ perceptions of their learning experiences from RLO-integrated lessons that are designed with the meaningful learning dimensions? Methodology Context A higher education institution in Singapore launched a new Masters programme in August 2015 targeted at developing training professionals’ competencies for designing, conducting, and managing professional education. The programme included courses such as instructional design, training needs analysis, workplace learning, quality assurance of learning, as well as coaching and mentoring. This was a part-time programme attended by training professionals in the evenings. In order to provide students with opportunities for flexible, self-paced modes of learning, online and blended learning strategies were adopted throughout the different courses of the programme. The institution sought to enhance its online resources by developing RLOs as content resources. The RLOs were designed as independent, self-contained learning units that could be used and reused flexibly by the instructors across courses. Wiley’s (2001) definition was used for the RLOs where each RLO contained presentations of key content, activities that promoted exploration of content for understanding, and self-assessment for monitoring of learning. The core instructors teaching the programme jointly identified specific topics for the development of the RLOs in the areas of learning theories, instructional design processes, instructional strategies, and data collection methods. These instructors served as subject matter experts during the development of the RLOs. Design meetings were held with the respective subject matter experts, programme coordinator, and the production manager at the beginning of the development cycle for each RLO to conceptualise its design and integration strategies. After development was completed, the RLOs were stored in a central depository where all programme instructors could freely assign and reuse the RLOs in their courses. Data source and analysis This study draws upon data related to the development and implementation of three RLOs with students. To answer research question 1 that is related to how the meaningful learning dimensions guided the design of each RLO, notes made during design meetings were examined. A total of ten design meetings were held to develop these RLOs. The design notes documented the decisions and rationales related to the design of each RLO and the strategies for their subsequent integration. Using a case study approach, each RLO was examined as a case. The meaningful learning with technology framework was used as the theoretical model to support explanation building (Yin, 2003). The five dimensions of meaningful learning were used to categorise the emergent themes from the design notes with respect to the design rationales and integration strategies adopted. Consideration of student learning difficulties also emerged as a theme when the design team articulated the development rationales for each RLO. Research question 2 was answered through the administration of perception surveys. As approved by the institutional ethics board, perception surveys were conducted with students who used each RLO to gather their feedback regarding their RLO learning experience and was used to provide initial insights about the RLO design. A total of 64 graduate students were surveyed and 54 responses were gathered, constituting a response rate of 84.38%. The surveys were embedded within each RLO and presented for the voluntary response of students whenever they completed all the learning activities of the RLO. There were two parts in each survey. The first part consisted of Likert scale items. As the graduate programme trained professionals, it was important for the RLOs to help these professionals make theory-practice linkages. Therefore, the Likert scale items assessed students’ perceptions regarding their ability to understand and apply the RLO content to their practice as well as whether they enjoyed the RLO learning experience. These Likert scale items were analysed with descriptive statistics. The second part of the survey comprised open-ended questions that asked for students’ comments about what they liked and what they Australasian Journal of Educational Technology, 2017, 33(5). 141 felt could be improved in terms of the RLO design. The open-ended responses gathered were first broken down into sentences as units of analyses. Each sentence was coded by the five dimensions of meaningful learning as an overarching theme. In the process of coding, categories related to other aspects such as content structures, navigation, and the use of media emerged and were incorporated into the coding protocol. The frequencies of units coded for the different categories were then counted up and compared. Findings Case 1: Meaningful RLO integration for theoretical learning The RLO on the principles of behaviourism was developed to support a course that aims to help training professionals appreciate the practical uses of different learning theories. It was used by two instructors teaching different sections of the same course. RLO design rationale Examination of the design notes for this RLO revealed that the team chose to develop this RLO because they observed that students faced difficulties with remembering the many concepts associated with behaviourism. Students also found it difficult to make theory-practice linkages. In terms of the meaningful learning dimensions, the design notes revealed the team wanting students to work with the subject content in an active manner. In terms of the constructive dimension, the team primarily targeted the knowledge and comprehension of behaviourist concepts as well as some level of authentic application to real-world examples. The team also emphasised the need for the intentional dimension where students could obtain immediate feedback to remediate their understanding of behaviourist concepts. Using these dimensions as a guide, the team conceptualised the RLO as a drill-and-practice game. The game mission was to amass sufficient number of logs to rebuild a broken bridge through answering questions correctly. The game was set at two levels to address the comprehension and application levels of Bloom’s taxonomy in terms of the constructive dimension (Figure 1). The first level addressed the comprehension of concepts such as positive reinforcement, punishment, and negative reinforcement whereas the second level tested students’ ability to apply behaviourist concepts to common instructional situations. The intentional dimension was supported with feedback for each question and students could replay the game if they were dissatisfied with their scores. Figure 1. The two game levels for the RLO on behaviourism RLO integration strategy While the game mission of the RLO was not designed to be authentic to students’ work contexts, the instructors sought to deepen this dimension by supporting students’ RLO experiences with individual reflections and instructor debriefs. Instructors required students to post individual reflections on the Blackboard discussion forum after completing the RLO. In these individual reflections, students were asked to identify the aspects of behaviourism they experienced through the game, explain how they have or could have applied two behaviourism concepts in teaching and learning situations, and comment on the limitations of behaviourism by using examples from their professional practice. In terms of the authentic Australasian Journal of Educational Technology, 2017, 33(5). 142 dimension, students’ experiences with the game was instantiated as authentic encounters with various principles of behaviourism modelled through the game rules. For example, positive reinforcement was modelled through students earning logs to repair a broken bridge whenever they had correct answers whereas punishment was encountered when they lost logs with incorrect answers. To enhance the constructive dimension, these experiences were extended to a personal level where students had to apply and evaluate the efficacy of behaviourism with respect to their own training experiences. There were opportunities for students to experience the intentional dimension through instructor debriefs of students’ misconceptions at the following class session after the instructors have reviewed students’ individual reflections. Table 2 summarises how this RLO-integrated learning experience with respect to the five dimensions of meaningful learning of Howland et al. (2013) by drawing upon the rubrics of Koh (2013). Table 2 Rating of lesson for theoretical learning by meaningful learning dimensions Low Medium High Active Students primarily involved in receiving content through information transmission Students have some opportunities to engage in doing e.g. manipulating and exploring information, making, practicing, reflecting, discussing etc. Students primarily involved in doing e.g. manipulating and exploring information, making, practicing, reflecting, discussing etc. Constructive Students engage in knowledge and comprehension of lesson content Students use lesson content to support Application or Analysis Students use lesson content to support Evaluation or Creation Authentic Minimal use of real-world examples related to lesson content Students analyse real-world examples related to lesson content Students articulate personal application of lesson content Intentional Students have minimal opportunities for feedback and remediation Students engage in one cycle of feedback and remediation Students engage in multiple rounds of feedback and remediation Collaborative Predominantly individual work, OR Group-based engagement but with convergent discussions Some level of engagement in divergent group discussion Predominantly engaged in divergent group discussions Level attained by RLO Level attained by both RLO and other learning activities Level attained by other learning activities It can be seen that the integrative use of the RLO, individual reflections and instructor debriefs resulted in a lesson design that could be considered as being high on the active, constructive, authentic, and intentional dimensions. This lesson design provided opportunities for students to remain highly active and intentional throughout the learning process rather than receiving information passively. There was a conscious structuring of lesson activities to deepen the aspects of the constructive and authentic dimensions that may be difficult to incorporate within the RLO. Despite that, the design of this learning experience did not focus on the collaborative dimension as students worked on the RLO and reflections individually. Given the curriculum time, the instructors felt that students would have had sufficient engagement with behaviourism through the learning activities planned. As instructors had to devote time Australasian Journal of Educational Technology, 2017, 33(5). 143 for other learning theories, they did not plan to engage students further in group discussions during the face-to-face session. Case 2: Meaningful RLO integration for the learning of principles The RLO on Writing Learning Objectives was developed as a content resource to help students review the principles of writing good learning objectives. This was because students needed to write learning objectives for project work requiring them to design training materials. This occurred in several courses within the programme. This RLO was therefore used by two faculty members who were teaching a basic instructional design course and an e-learning design course respectively. RLO design rationale The team observed that the principles of writing good learning objectives were fairly easy for the students to understand. However, the challenge for students was in the translation of these principles to the contexts of their projects. Therefore, the team sought to develop this RLO so that curriculum time could be optimised for project application, feedback, and remediation. The team also felt that students could be provided with more examples of learning objectives written for different kinds of training content. Given these considerations, the active, constructive and authentic dimensions were selected to anchor the RLO development. For the active dimension, the team emphasised the need for the RLO to engage students in identifying and repairing faulty learning objectives. This was supported through the designation of content application as the goal of the constructive dimension. To anchor the RLO with an authentic context, an introductory animated scenario was created where students were shown the problems faced by Cindy, an instructional designer, when she did not write clear and measurable learning objectives. The team also emphasised the need for students to be adept at writing learning objectives related to training content that are conceptual, procedural, and attitudinal in nature. Therefore, students were asked to repair learning objectives related to training programmes in Food Science, Food Safety in the Kitchen, as well as Food Service Management respectively. To further model how learning objectives are created in training contexts, students were asked to choose an appropriate learning objective after viewing a video of Cindy explicating the training requirements that she has received from her client as part of their assessment (Figure 2). Figure 2. RLO on writing learning objectives RLO integration strategy Both the instructors assigned the RLO as a self-paced learning resource to be completed prior to their face-to-face sessions. During the class sessions, both instructors conducted a short review of the principles for writing learning objectives following which the students worked in groups to design learning objectives for their project. Each group was asked to present their learning objectives and the instructors as well as their peers provided feedback for improvement. Table 3 shows the rating of this RLO integration strategy with the dimensions of meaningful learning. Table 3 Australasian Journal of Educational Technology, 2017, 33(5). 144 Rating of lesson for learning of principles by meaningful learning dimensions Low Medium High Active Students primarily involved in receiving content through information transmission Students have some opportunities to engage in doing e.g. manipulating and exploring information, making, practicing, reflecting, discussing etc. Students primarily involved in doing e.g. manipulating and exploring information, making, practicing, reflecting, discussing etc. Constructive Students engage in knowledge and comprehension of lesson content Students use lesson content to support application or analysis Students use lesson content to support evaluation or creation Authentic Minimal use of real-world examples related to lesson content Students analyse real-world examples related to lesson content Students articulate personal application of lesson content Intentional Students have minimal opportunities for feedback and remediation Students engage in one cycle of feedback and remediation Students engage in multiple rounds of feedback and remediation Collaborative Predominantly individual work, OR Group-based engagement but with convergent discussions Some level of engagement in divergent group discussion Predominantly engaged in divergent group discussions Level attained by RLO Level attained by both RLO and other learning activities Level attained by other learning activities The opportunities for students to engage with the active and intentional dimensions at high levels were maintained as students transitioned between the RLO and classroom. Multiple opportunities for feedback and remediation were catered for through the RLO as well as through peer and instructor feedback during the in-class discussions. The RLO integration strategy also ensured that the constructive and authentic dimensions were developed from medium to high levels from RLO to classroom. The hands-on tasks assigned by instructors aimed to raise the level of cognitive challenge for students as they had to transition from repairing learning objectives in pre-specified scenarios within the RLO to creating learning objectives for the specific training contexts of their projects. The requirements and constraints of an actual training design project presented students with the need to consider how learning objectives could be scoped within these contextual challenges. Not only did this deepen students’ engagement with the authentic dimension, it also opened the space for some level of divergent group-based discussions. In fact, instructors observed the students to be negotiating how their learning objectives could be better articulated to support their project goals during class sessions. Therefore, this RLO integration strategy served to deepen the collaborative dimension from low to medium as students transitioned between the RLO and the classroom. Case 3: Meaningful RLO integration for the learning of skills The RLO on Conducting Interviews was designed as part of an initiative to convert a course for data collection and analysis to blended learning. This course was initially conducted through a 5-day workshop during summer semesters. To facilitate the attendance of working professionals who were completing the graduate programme on a part-time study mode, the team aimed to transition to a blended learning mode requiring 3 days of in-class attendance. To facilitate this transition, the team planned for the development of RLOs to cover content related to designing and conducting surveys, interviews, and observations. This Australasian Journal of Educational Technology, 2017, 33(5). 145 RLO was designed to address the dos and don'ts of conducting interviews and was implemented by one instructor. RLO design rationale The team recognised that interview skills need to be perfected through hands-on practice. While this should be the focus during the classroom sessions, the multimedia affordances of RLOs could be used to model best practices. This was one way of stimulating cognitive rehearsal in the students before actual enactment in the classroom. Given these objectives, the constructive and authentic dimensions were given foremost consideration when designing this RLO. The RLO was first used to model the dos and don'ts of conducting interviews through animated video clips that demonstrate typical problems faced during interviews and strategies for handling them (Figure 3). This catered to the learning of content at the comprehension level of Bloom’s taxonomy in terms of the constructive dimension. Figure 3. RLO on conducting interviews After reviewing the dos and don'ts, the students were presented with the assessment. This comprised an animated video clip of an interview conducted by a training executive. Students analysed this video clip to identify the correct and incorrect ways of conducting interviews. This activity enabled students to engage with the lesson content at the analysis level of Bloom’s taxonomy. Therefore, the RLO was designed to engage students in both the low and medium levels of the constructive dimension of meaningful learning (Table 4). In terms of the authentic dimension, the aim was to model the use of interviews with work contexts that were familiar to students. As training needs analysis is one of the tasks typically performed by training professionals, the instructors chose to use Sarah, a training executive, as the lead character for its introductory scenario. The dos and don'ts of conducting interviews were modelled through animations of interview snippets that Sarah had conducted with the waiters and managers of a restaurant chain to understand its training needs. In the assessment section, students were asked to analyse an interview that Sarah had conducted with the customers of the restaurant to identify problems and good practices. These elements were targeted at embodying a medium level of authenticity where students were presented with opportunities to analyse concrete problems that could surface during the performance of a real-world task, that is, training needs analysis. RLO integration The instructor asked the students to complete the RLO before the class met during the workshop. At the workshop, students were broken into groups and assigned an interview topic. After designing the interview questions, the groups were paired to interview and critique each other’s interview techniques with respect to the dos and don'ts that they have learnt through the RLO. Each group was asked to complete a post-activity reflection of their performance as well as to create their own set of dos and don’ts for conducting interviews. Australasian Journal of Educational Technology, 2017, 33(5). 146 Table 4 Rating of lesson for learning of skills by meaningful learning dimensions Low Medium High Active Students primarily involved in receiving content through information transmission Students have some opportunities to engage in doing e.g. manipulating and exploring information, making, practicing, reflecting, discussing etc. Students primarily involved in doing e.g. manipulating and exploring information, making, practicing, reflecting, discussing etc. Constructive Students engage in knowledge and comprehension of lesson content Students use lesson content to support application or analysis Students use lesson content to support evaluation or creation Authentic Minimal use of real-world examples related to lesson content Students analyse real-world examples related to lesson content Students articulate personal application of lesson content Intentional Students have minimal opportunities for feedback and remediation Students engage in one cycle of feedback and remediation Students engage in multiple rounds of feedback and remediation Collaborative Predominantly individual work OR Group-based engagement but with convergent discussions Some level of engagement in divergent group discussion Predominantly engaged in divergent group discussions Level attained by RLO Level attained by both RLO and other learning activities Level attained by other learning activities By focusing the RLO on demonstration and practice and supporting it with hands-on classroom activities, students were engaged in highly active learning as they had opportunities to examine cases, test their understanding and to apply their knowledge through actual hands-on practice. The intentional level in this integration strategy was deemed to be high as students not only had opportunities to encounter feedback and remediation through the RLO but also from the critique of instructors and peers. This integration strategy could deepen the constructive and authentic dimensions of meaningful learning. As students critiqued their peers and developed their own heuristics for conducting interviews, they were given opportunities to engage in the evaluation and creation levels of Bloom’s taxonomy which characterised the high level of the constructive dimension. In terms of the authentic dimension, students not only had opportunities to plan and conduct interviews, they also reflected and created personal takeaways to guide their future practice. This lesson design moved students’ engagement with the authentic dimension from medium to high as they transitioned from the RLO to classroom. In contrast to the collaborative task designed for the lesson on writing learning objectives, the collaborative task designed for students in terms of conducting interviews was more open-ended and complex because students needed to design interview protocols, strategise and critique interviews as well as to create their personal heuristics for conducting interviews. Therefore, these tasks engaged students predominantly in divergent conversations where different viewpoints and ideas need to be negotiated. The in-class activities raised students’ exposure to the collaborative dimension from low to high as they moved from the RLO to the classroom. Australasian Journal of Educational Technology, 2017, 33(5). 147 Student evaluation of RLO experience Surveys of students were conducted after the completion of the RLOs that they were assigned to. Positive ratings were obtained for the design of RLOs in terms of supporting content understanding, application of content through practice exercises, perceived ability to explain subject content, perceived transfer to workplace, and student enjoyment of learning (Table 5). Ratings were based on a 5-point Likert scale where 1 (strongly disagree), 2 (disagree), 3 (neither agree nor disagree), 4 (agree), and 5 (strongly agree). Students largely had strong agreement about the usefulness of examples, practice exercises and confidence in explaining the subject content. Students also agreed that they could apply the lesson content at the workplace and that learning through the RLO was enjoyable. Table 5 Student rating of RLO learning experiences (n = 54) Questions M SD The examples in the learning object helped me to understand the subject content 4.50 0.61 The practice exercise in the learning object helped me to apply the subject content 4.44 0.60 After going through the learning object, I am able to explain the subject content 4.33 0.61 After going through the learning object, I am able to apply what I learnt to the workplace 4.24 0.67 I enjoy learning through the learning object 4.22 0.72 Table 6 shows the qualitative comments from the open-ended survey questions about what students liked about the RLOs. The findings corroborated the intended design objectives of associated meaningful learning dimensions. About 28% of the coded comments were related to the intentional dimension which mentioned students perceiving opportunities for self-paced learning and remediation through the RLO. About 26% of the coded comments appreciated the authenticity of examples and opportunities to build understanding as per the authentic and constructive dimensions respectively. About 43% of the positive comments described other factors such as the content being clearly structured and the learning experience being engaging and fun. Table 6 Students’ positive comments about RLOs (n = 54) Themes Frequency % Meaningful Learning Dimensions Active – It is interactive 2 2.70% Authentic – Provides practical examples 10 13.51% Constructive – Builds understanding and application of content 9 12.16% Intentional – Feedback, reinforcement, and ability to engage in multiple tries to fix learning gaps 21 28.38% Others Content is clearly structured 9 12.16% Stimulates interest in content – Engaging and fun 15 20.27% User friendly – easy to navigate 5 6.76% Links to other learning activities – Supports the textbook reading, reflection questions helps reinforce learning from RLO 3 4.05% Total units coded 74 100.00% Table 7 shows the students’ comments from the open-ended survey questions that were related to areas for improvement. Interestingly, about 24% of the coded comments declared that there was nothing to improve. The major categories of improvements sought were related to technical areas such as speed for loading of videos, student preferences for use of music and graphics, and clarity of instructions. Students also made requests for additional practice exercises and examples which were related to the intentional and authentic dimensions respectively. Australasian Journal of Educational Technology, 2017, 33(5). 148 Table 7 Areas of improvement for RLOs (n = 54) Themes Frequency % Meaningful Learning Dimensions Intentional – More practice exercises 6 11.11% Authentic – More examples 3 5.56% Others Nothing to improve 13 24.07% Technical improvement e.g. loading time of videos 11 20.37% Enhance navigation 2 3.70% Enhance clarity of instructions 6 11.11% Enhance media (e.g. changes to music and graphics, ability to print content) 10 18.52% Enhance interactivity (e.g. sequence of practice questions) 3 5.56% Total units coded 54 100.00% Discussion The preceding analysis shows that the five dimensions of meaningful learning developed by Howland et al. (2013) could be used to guide the design of RLOs as well as their integration for blended modes of e-learning. These dimensions provide instructors with pedagogical guidelines to better synthesise online and face-to-face activities towards the aim of deepening meaningful learning. Through this study, we derive several guidelines for the design and integration of RLOs in higher education contexts. Design RLOs to facilitate active and intentional learning The RLOs used in this study comprise features of presentation and practice objects that are described by Churchill (2007). It appears that when RLOs use interactivity to engage students predominantly in exploration and practice, it enables the active dimension to be engendered at a high level. A high level on the intentional dimension needs to be supported by the presence of comprehensive feedback and opportunities for multiple retries. Whether presented as an online game or through the analysis of work-based scenarios, consideration of the active and intentional dimensions imply that RLOs need to engage students in doing, identifying, and remediating learning gaps rather than the mere reading of content information. The survey results indicate this to be particularly important for graduate students. In fact, interactivity and learner choice are important pedagogical considerations when designing RLOs to support student-centered learning (Cochrane, 2007). Use multimedia to model authentic contexts of use The design and presentation of content in ways that motivate the interest of target learners are pertinent indicators of RLO quality (Gordillo, Barra, & Quemada, 2014; Sanz-Rodriguez, Dodero, & Sánchez-Alonso, 2010). The use of multimedia in the RLOs provides instructors with options to go beyond text and graphics for content presentation. This study shows that when media elements are designed to model contextual use of subject content, the RLO contributes an authentic dimension to students’ engagement with subject content. Regardless of whether the RLOs were addressing theories, skills, or principles, the need for integration of authentic contexts was found to be important for the students of this study who were working professionals. However, as the scenarios and cases featured in RLOs were predetermined by the instructors, it is noted that the design of RLOs did not enable learning experiences to reach a high level on the authentic dimension. Nevertheless, by working through RLOs, students could still be exposed to some form of cognitive rehearsal prior to actual application. In this study, the analysis of video-based animations was found to be particularly useful for supporting this. Extend the constructive and authentic dimensions from activity to activity This study shows that synergies among blended learning activities could be designed with an aim to deepen meaningful learning dimensions for students as they transition from one learning activity to another. It appears that high levels of the constructive and authentic dimensions are challenging to Australasian Journal of Educational Technology, 2017, 33(5). 149 achieve through students’ encounter with RLOs alone. The evaluation and creation levels of Bloom’s taxonomy as well as the translation of learning to personal practices appear to be more easily reached through activities such as online reflections, in-class activities and group discussions. As emphasised by Sridharan et al. (2010), effective e-learning in higher education results from the effective synergy among online and face-to-face activities as a learning system. Therefore, it is important to strategise how the constructive and authentic aspects of the blended learning experience could be extended beyond RLOs. Create opportunities for collaboration beyond RLOs It can be seen that while RLOs are well-poised to handle individual practice and learning, they are not as well-endowed in terms of social affordances. As such, RLOs have some limitations for supporting the collaborative dimension of meaningful learning. Many studies have found that blended experiences to be superior to fully online experiences as it allows opportunities for social participation and collaborative learning (Means, Toyama, Murphy, & Baki, 2013). It is therefore important to plan for collaborative activities that extend the students’ RLO experience. In this study, collaborative activities predominantly occurred during classroom sessions. However, collaborative activities could also be mediated through the use of online discussions. The guidelines suggested above are especially relevant in the context of graduate education. In this study where the students are training professionals, the blended learning experiences within the programme need to help them to become reflective practitioners who are adept at manoeuvring theory-practice linkages in team-based work contexts (Smeby & Heggen, 2014). This study shows that RLOs designed and integrated with the meaningful learning dimensions could help students to consider workplace transfer as they engage with subject content. The examples described in this study illustrate how meaningful learning dimensions could be used to plan e-learning experiences at a lesson level. This extends current e-learning studies which typically adopt an institutional perspective (Hew & Cheung, 2014). This study shows that the meaningful learning dimensions could be useful for planning specific blended learning experiences related to different kinds of subject matter. Limitations and future research The current study has several limitations that can also serve as areas of future research. Firstly, the RLOs used in this study were designed as practice objects. It is not clear if the ratings on the meaningful learning dimensions would differ if other kinds of RLOs such as simulation objects were used. Therefore, an area of future research would be to analyse blended learning experiences supported by different kinds of RLOs. This could be one way of validating if the guidelines suggested for the integration of RLOs remain applicable. Secondly, the context of the study was limited to RLOs designed for a graduate programme designed for training professionals. The applicability of the meaningful learning dimensions for blended learning in undergraduate and K-12 contexts were not examined in this study. Furthermore, this study sought to discuss three in-depth cases of how RLO integration could be supported by the meaningful learning dimensions. As the study is exploratory in nature, the applicability of the dimensions as well as the classifications of the different levels cannot be generalised. Further validation through a larger sampling of blended learning cases at different educational levels could be carried out as future research. The final limitation of the study is that student feedback was only gathered through a survey that focused on their perception of the design of the RLOs. As the graduate programme was targeted at helping professional trainers develop theory-practice linkages, the Likert scale questions focused on the extent to which students felt they could understand and apply the content of the RLOs. It is recognised that these questions were largely targeted at the constructive and authentic dimensions of meaningful learning. While students’ appreciation of the active and intentional dimensions emerged through the open-ended questions of the survey, the addition of Likert scale questions for these dimensions could enhance the survey design. Therefore, it is proposed that the Likert scale survey items be revised to include all meaningful learning dimensions in future studies. As this is an exploratory study, more rigorous evaluation studies comprising student surveys and interviews could also be implemented in future to compare the efficacy of blended learning courses developed with and without the use of the meaningful learning dimensions. Australasian Journal of Educational Technology, 2017, 33(5). 150 Conclusion This study has proposed that blended learning experiences could be designed through the use of meaningful learning dimensions. These dimensions could guide instructors to design and integrate content resources such as RLOs to create experiences that are active, authentic, constructive, intentional, and collaborative. This approach could be further explored and developed in future studies. Acknowledgement The development of RLOs in this study is funded by the ICT Governance Committee, National Institute of Education (NIE), Nanyang Technological University, Singapore. The views expressed in this paper are the author’s and do not necessarily represent the views of NIE. References Ausubel, D. G. (1963). Cognitive structure and the facilitation of meaningful verbal learning. Journal of teacher education, 14(2), 217-222. https://doi.org/10.1177/002248716301400220 Bannan-Ritland, B., Dabbagh, N., & Murphy, K. (2000). Learning object systems as constructivist learning environments: Related assumptions, theories, and applications. Retrieved from http://reusability.org/read/chapters/bannan-ritland.doc Boyle, T. (2010). Layered learning design: Towards an integration of learning design and learning object perspectives. Computers & Education, 54(3), 661-668. https://doi.org/10.1016/j.compedu.2009.09.026 Chi, M. T. H. (2009). Active-constructive-interactive: A conceptual framework for differentiating learning activities. Topics in Cognitive Science, 1(1), 73-105. https://doi.org/10.1111/j.1756-8765.2008.01005.x Cochrane, T. (2007). Developing interactive multimedia learning objects using QuickTime. Computers in Human Behavior, 23(6), 2596-2640. https://doi.org/10.1016/j.chb.2006.08.007 Churchill, D. (2007). Towards a useful classification of learning objects. Educational Technology Research and Development, 55(5), 479-497. https://doi.org/10.1007/s11423-006-9000-y Gee, B. M., Strickland, J., & Salazar, L. (2014). The role of reusable learning objects in occupational therapy entry-level education. The Open Journal of Occupational Therapy, 2(4), 6. https://doi.org/10.15453/2168-6408.1108 Gordillo, A., Barra, E., & Quemada, J. (2014). Towards a Learning Object pedagogical quality metric based on the LORI evaluation model. Proceedings of the 2014 IEEE Frontiers in Education Conference (FIE), Madrid-Spain, 1-8. https://doi.org/10.1109/FIE.2014.7044499 Green, S., Jones, R., Pearson, E., & Gkatzidou, S. (2013). Accessibility and adaptability of learning objects: responding to metadata, learning patterns and profiles of needs and preferences. ALT-J, 14(1), 117-129. https://doi.org/10.1080/09687760500479811 Hendrix, M., Protopsaltis, A., Rolland, C., Dunwell, I., de Freitas, S., Arnab, S. Petridis, P., and LLanas, J. (2012, February). Defining a metadata schema for serious games as learning objects. Paper presented at the eLmL 2012, The Fourth International Conference on Mobile, Hybrid, and On-line Learning, Valencia, Spain. Hew, K. F., & Cheung, W. S. (2014). Using blended learning: Evidence-based practices. Singapore: Springer. https://doi.org/10.1007/978-981-287-089-6 Howland, J. L., Jonassen, D., & Marra, R. M. (2013). Meaningful learning with technology (4th ed.). Hudson, WA: Pearson Higher Education. Koh, J. H. L. (2013). A rubric to analyze teachers’ conceptions of meaningful learning in ICT lesson planning. Australasian Journal of Educational Technology, 29(6), 887-900. https://doi.org/10.14742/ajet.228 Koh, J. H. L., Chai, C. S., & Lim, W. Y. (2016). Teacher professional development for TPACK-21CL: Effects on teacher ICT integration and student outcomes. Journal of Educational Computing Research. https://doi.org/10.1177/0735633116656848 Krathwohl, D. R. (2002). A revision of Bloom's taxonomy: An overview. Theory into Practice, 41(4), 212-218. https://doi.org/10.1207/s15430421tip4104_2 McLoughlin, C., & Lee, M. J. W. (2008). The three P’s of pedagogy for the networked society: Personalization, Participation, and Productivity. International Journal of Teaching and Learning in https://doi.org/10.1177/002248716301400220 http://reusability.org/read/chapters/bannan-ritland.doc https://doi.org/10.1016/j.compedu.2009.09.026 https://doi.org/10.1111/j.1756-8765.2008.01005.x https://doi.org/10.1016/j.chb.2006.08.007 https://doi.org/10.1007/s11423-006-9000-y https://doi.org/10.15453/2168-6408.1108 https://doi.org/10.1109/FIE.2014.7044499 https://doi.org/10.1080/09687760500479811 https://doi.org/10.1007/978-981-287-089-6 https://doi.org/10.14742/ajet.228 https://doi.org/10.1177/0735633116656848 https://doi.org/10.1207/s15430421tip4104_2 Australasian Journal of Educational Technology, 2017, 33(5). 151 Higher Education, 1, 10-27. McGreal, R. (2004). Learning objects: A practical definition. International Journal of Instructional Technology and Distance Learning (IJITDL), 9(1). Retrieved from http://auspace.athabascau.ca/handle/2149/227 Means, B., Toyama, Y., Murphy, R., & Baki, M. (2013). The effectiveness of online and blended learning: A meta-analysis of the empirical literature. Teachers College Record, 115(3), 1-47. Ossiannilsson, E., & Landgren, L. (2012). Quality in e‐learning–a conceptual framework based on experiences from three international benchmarking projects. Journal of Computer Assisted learning, 28(1), 42-51. https://doi.org/10.1111/j.1365-2729.2011.00439.x Ritzhaupt, A. D. (2010). Learning Object Systems and Strategy: A description and discussion. Interdisciplinary Journal of E-Learning and Learning Objects, 6(1), 217-238. Sanz-Rodriguez, J., Dodero, J. M. M., & Sánchez-Alonso, S. (2010). Ranking learning objects through integration of different quality indicators. IEEE transactions on learning technologies, 3(4), 358-363. https://doi.org/10.1109/TLT.2010.23 Smeby, J.-C., & Heggen, K. (2014). Coherence and the development of professional knowledge and skills. Journal of Education and Work, 27(1), 71-91. https://doi.org/10.1080/13639080.2012.718749 Sridharan, B., Deng, H., & Corbitt, B. (2010). Critical success factors in e-learning ecosystems: a qualitative study. Journal of Systems and Information Technology, 12(4), 263-288. https://doi.org/10.1108/13287261011095798 Staker, H. (2011). The rise of K-12 blended learning: Profiles of emerging models. Boston, MA: Innosight Institute. Tsai, C. W., Shen, P. D., & Chiang, Y. C. (2013). Research trends in meaningful learning research on e‐learning and online education environments: A review of studies published in SSCI‐indexed journals from 2003 to 2012. British Journal of Educational Technology, 44(6), E179-E184. https://doi.org/10.1111/bjet.12035 Twigg, C. A. (2003). Models for online learning. Educause review, 38(5), 28-38. Vincenti, G., Braman, J., & Hilberg, J. S. (2013). Teaching introductory programming through reusable learning objects: a pilot study. Journal of Computing Sciences in Colleges, 28(3), 38-45. Wagner, E. (2002). Steps to creating a content strategy for your organization. In B. Brandon (Ed.), Best of the eLearning guild's learning solutions: Top articles from the eMagazine's first five years (pp. 103-120). San Francisco, CA: Pfeiffer. Watson, J. (2010). A case study: Developing learning objects with an explicit learning design. Electronic Journal of e-Learning, 8(1), 41-50. Wiley, D. A. (2001). Instructional use of learning objects. Bloomington, IN: Association for Educational Communications and Technology. Yin, R. K. (2003). Case study research: Design and methods (1st ed.). Thousand Oaks, CA: Sage Publications. Yunianta, A., Yusof, N., Othman, M. S., & Octaviani, D. (2012). Analysis and categorization of e-learning activities based on meaningful learning characteristics. International Journal of Social, Behavioral, Educational Economic, Business and Industrial Engineering, 6(9), 2430-2435. Corresponding author: Joyce Hwee Ling Koh, joyce.koh@nie.edu.sg Australasian Journal of Educational Technology © 2017. Please cite as: Koh, J.H.L. (2017). Designing and integrating reusable learning objects for meaningful learning: Cases from a graduate programme. Australasian Journal of Educational Technology, 33(5), 136-151. https://doi.org/10.14742/ajet.3072 http://auspace.athabascau.ca/handle/2149/227 https://doi.org/10.1111/j.1365-2729.2011.00439.x https://doi.org/10.1109/TLT.2010.23 https://doi.org/10.1080/13639080.2012.718749 https://doi.org/10.1108/13287261011095798 https://doi.org/10.1111/bjet.12035 mailto:joyce.koh@nie.edu.sg https://doi.org/10.14742/ajet.3072 Joyce Hwee Ling Koh National Institute of Education, Nanyang Technological University, Singapore Introduction Literature review What are RLOs? Ways of integrating RLOs Meaningful Integration of RLOs Theoretical framework and research questions Meaningful learning rubric Table 1 Research questions Methodology Context Data source and analysis Findings Case 1: Meaningful RLO integration for theoretical learning RLO design rationale RLO integration strategy Table 2 Case 2: Meaningful RLO integration for the learning of principles RLO design rationale RLO integration strategy Table 3 Case 3: Meaningful RLO integration for the learning of skills RLO design rationale RLO integration Table 4 Student evaluation of RLO experience Table 5 Table 6 Table 7 Discussion Design RLOs to facilitate active and intentional learning Use multimedia to model authentic contexts of use Extend the constructive and authentic dimensions from activity to activity Create opportunities for collaboration beyond RLOs Limitations and future research Conclusion Acknowledgement References Corresponding author: Joyce Hwee Ling Koh, joyce.koh@nie.edu.sg