falloon.pdf Australasian Journal of Educational Technology 2010, 26(5), 626-642 Learning objects and the development of students’ key competencies: A New Zealand school experience Garry Falloon The University of Waikato Annick Janson Ecosynergy Group Ltd, New Zealand Robin Janson Ecosynergy Group Ltd, New Zealand This paper outlines a study investigating the impact of the use of learning objects on the development of two key competencies from the revised New Zealand Curriculum Framework (Ministry of Education, 2007). It specifically focuses on the key competencies of ‘thinking’ and ‘relating to others’, and explores how teachers in an urban intermediate school (year 7 and 8) planned and integrated learning objects into a wider ‘community’ social studies topic, and the impact this integration had on student competency development. Outcomes from the study indicate that learning objects can, for some students, provide a motivating and engaging learning experience through which thinking capabilities and relationship skills can be enhanced, but that such development is dependent upon several factors including the design, content, and level of interactivity of the objects, how closely they align with learning goals, the careful selection of student groupings, and how access to the learning objects is organised and managed. The study also revealed some issues with the use of new digital data collection tools, particularly related to the accuracy of coding of visual information, and how to best translate this into text format for publication without ‘diluting’ its richness and meaning. ICT in schools Trends towards the integration of information and communication technology (ICT) and new digital learning tools into schools in New Zealand and internationally, are not new. However, while the array of technological devices available for use in the classroom has expanded rapidly in recent years, according to some authors (for example, Capper, 2001; Parr & Ward, 2004; Robertson, 2003; Wood, 2003; Wright, 2009) there is still a level of confusion existing as to the precise nature of ‘outcomes’ for students attributable to the use of such technology, and few empirical studies categorically stating significant learning advantages from their inclusion in school curriculum. Furthermore, it is significant to note that this apparent confusion is not a recent phenomenon, with authors such as Capper (2001), Postman (1993), Brown (1997) and Page (1999) earlier commenting that schools were falling victim to a ‘technocentric’ approach to technology introduction. That is, so called technology ‘integration’ was more akin to ‘technology injection’, and based upon little more than building a competitive advantage in market driven, student numbers-based school funding systems. According to Capper, Falloon 627 Most school approaches to ICT are dominated by the desire to purchase and deploy the technology itself. This appears to be driven by a mix of competitive pressures between schools and a desire to ensure that current students do not ‘miss out’ (on what they might be missing out is often unstated). The pressure to adopt is therefore driven by the technology itself, rather than by any assessment of educational needs (Capper, 2001, p. 6). In the New Zealand context, early papers by Brown (1997) also pointed to what was viewed as the ‘competing rationale’ for inclusion of ICT in school curriculum, citing official New Zealand Ministry of Education documents of the time as giving mixed and confusing messages relating to the reasons for ICT use in schools. These rationales, identified as vocational, economic, commercial, marketing, cost-effectiveness, social, transformational and pedagogical (Brown, 1997), he claimed, were conflicting, fundamentally flawed, lacked sound educational purpose, and were merely illustrative of the level of confusion about ICT integration existing at the central policy making level. However, studies by authors such as Culp, Honey and Spielvogel (2003), Heinecke, Blasi, Milman and Washington (1999), McCombs (2000), McNabb, Hawkes, and Rouk (1999), and Xiufeng, Macmillan and Timmons (1998), argue that such problems relate not so much to whether or not ICT should be included in classrooms, but more to how the impact of that inclusion should be measured or assessed. Indeed, they comment that the tendency to link technology inclusion with such measures as standardised or summative test scores is inherently problematic, and fails to both acknowledge other significant variables which may impact upon such outcomes, and the wider role ICT can play in the development of cognitive, affective, and social functioning. According to Heinecke et al. (1999), If, on the other hand, one views the goals of education as the production of students who can engage in higher-order, problem-based inquiry, new potential for entirely different uses of technology emerge… we can evaluate these outcomes, but it is more complicated than the standardised test route. Standardised tests are an efficient means for measuring certain types of learning outcomes, but we must again ask ourselves, are these the outcomes we value for the new millennium? To a certain extent we are living out the decisions reflected in previous evaluation methods which constrain the purpose and effectiveness of technology in education (Heinecke et al., 1999, p. 3). Later international and New Zealand research on the use of ICT in schools tends to support this perspective. In a 2007 BECTA study in the UK, significant social and educational benefits for using ICT in the classroom were identified, including improvement in the quality of interactions between students and teachers, greater levels of learner independence, improved student engagement and motivation, and greater student participation in the learning process (Somekh, Underwood, Convery, Dillon, Jarvis & Lewin, 2007). Similar outcomes were noted in a 2008 Australian study by Hartnell-Young and Vetere, who explored the use of mobile technologies with indigenous aboriginal populations in the Northern Territory. They concluded that the use of such technologies supported students to integrate elements of their cultural knowledge into the classroom. However, the effectiveness of this was very much related to the teacher’s willingness to adopt pedagogical approaches which not only validated this knowledge, but also allowed the students greater input into plotting the direction of inquiries, leading to higher levels of engagement and motivation. This finding is supported in a recent study of gifted and talented students using online technologies by Nicholas and Ng (2009), who concluded that one of the most powerful motivating factors associated with the use of ICT relates to the notion of ‘community 628 Australasian Journal of Educational Technology, 2010, 26(5) formation’, and its ability to facilitate social interaction and share information with authentic audiences. According to Wright (2009), these outcomes closely align with “the Key Competencies as outlined in The New Zealand Curriculum (NZC), especially regarding ‘Relating to others’, ‘Participating and contributing’ and ‘Managing self’” (Wright, 2009, p. 20), and offer teachers greater clarity and direction as to where ICT could add value to their classroom programs. Key competency development The development of students’ key competencies (Rychen & Salganik, 2003) in primary and secondary schools has formed a cornerstone of many recent education initiatives around the world. The Organisation for Economic Cooperation and Development (OECD) led DeSeCo (Definition and Selection of Competencies) project, sought to identify the type of knowledge, skills, attitudes and values students would need to deal with the rapidly changing demands of an increasingly globalised and interconnected world of the 21st century. According to the OECD, … a competency is more than just knowledge and skills. It involves the ability to meet complex demands, by drawing on and mobilising psychosocial resources (including skills and attitudes) in a particular context. For example, the ability to communicate effectively is a competency that may draw on an individual’s knowledge of language, practical IT skills, and attitudes towards those with whom he or she is communicating… (OECD, 2005, p. 4) The DeSeCo initiative proposes an overarching framework comprising three broad competency categories: using tools interactively, interacting in heterogeneous environments, and acting autonomously (Rychen & Salganik, 2003). Many countries have adopted the principles of this framework and have subsequently contextualised it to suit their local circumstances and priorities. In New Zealand, for example, the Ministry of Education produced its latest curriculum revision integrating many major components of the DeSeCo competencies within its Key Competency framework, which, amongst other targets, sets goals for students to be competent at using languages, symbols and texts, participate and contribute meaningfully to society, relate positively to others, and develop self management and independent lifelong learning skills (Ministry of Education, 2007, p. 12). The New Zealand Curriculum views Key Competencies as important and relevant to students’ future lives and careers, and for addressing wider concerns such as sustainability, citizenship, enterprise, and globalisation. Consistent with more contemporary views of the role ICT can play in learning as identified in the earlier brief review of literature, the aim of the following study was to explore the potential of one type of digital learning resource, namely learning objects (LOs), to support the development of a selection of Key Competencies from the revised New Zealand Curriculum framework (Ministry of Education, 2007). Key competency development and learning objects According to Wiley (2000), LOs are simply defined as, “any digital resource that can be reused to support learning” (p. 7). While this definition is relatively broad, he goes on to comment that generally LOs are characterised by their “purposeful use (by either an instructional designer, an instructor, or a student), to support learning” (Wiley, 2000, p. 8) and that they possess attributes such as reusability and interactivity, are able to be Falloon 629 accessed and used by a number of learners, and are often “deliverable over the Internet” (ibid, p. 3). Wiley (2000) also comments that for LOs to be effective in supporting learning, they need to be associated with an instructional design theory that acknowledges their unique capacities such as reusability, their ability to combine a range of diverse learning elements in different ways and to access additional sources of information external to the object itself, and their applicability to a range of different learning contexts. Recent development work by the Le@rning Federation in Australia and New Zealand has explored the potential of LOs to enhance the learning experience for students in primary and lower secondary schools. Early results from these studies revealed generally positive outcomes from the integration of LOs for the uptake of factual and development of conceptual knowledge, as well as students’ abilities to transfer this knowledge between contexts and across disciplines (Freebody, Muspratt & McRae, 2007). However, these studies also revealed that the extent to which this occurred was variable according to specific design features of the LOs, such as their capacity to provide meaningful feedback, the variety and nature of media embedded in the object (often linked to levels of student engagement), and student literacy skill levels. Other factors influencing effectiveness included the pedagogical practices of the teachers, or, as Freebody et al. (2006, p. 11) put it, the extent to which there was evidence of “new learning environments being put to ‘old’ pedagogical work”. Other exploratory studies in New Zealand (for example Falloon, 2006a, 2006b) have investigated the efficacy of LOs for supporting factual and conceptual knowledge development when used in collaborative learning contexts. These studies indicated that when LOs were used in small group or pair situations they encouraged and supported high levels of discussion, debate, and social negotiation of meaning, appearing to contribute to learning outputs of accuracy and depth. Additionally, these studies showed tentative indications that students engaged early and in depth with the content being explored, and were prepared to ‘experiment’ with different ways of negotiating their way through information and presenting outcomes. It was this observed characteristic that acted as a catalyst for this study - namely, exploring the potential of LOs to support the ‘Thinking’ Key Competency - and note any relationship which might exist between this and LOs’ capacities to stimulate social interaction and meaning-making, when used in collaborative group settings. Background and purpose This study is the most recent in a range of studies supported by Microsoft (NZ) as part of their Partners in Learning (PiL) program. The PiL program commenced in 2005, with one of its main aims being to explore the potential for using LOs to support specific curriculum objectives and outcomes. To date, research has centred on trials using ‘off the shelf’ LOs from organisations such as the BBC and the Le@rning Federation, and has adopted a ‘best fit’ approach to matching content and functionality with the teachers’ learning goals and objectives. Whilst acknowledging that this approach may decrease the ‘precision’ of the resource in meeting specific and unique learning objectives, it was seen as a practical and sustainable solution which would yield outcomes more likely to be consistent with normal classroom practices. Additionally, the advent of the revised national curriculum (Ministry of Education, 2007) with its inclusion of a Key Competency framework, has enabled the researchers 630 Australasian Journal of Educational Technology, 2010, 26(5) to adopt a wider perspective on where LOs may be able to ‘add value’ to learning, and make a contribution to the curriculum’s more global objectives relating to the needs of 21st Century learners and citizens. Within the context of this study, these specifically relate to exploring any relationship which might exist between student interaction with the objects and the development of Thinking and collaborative (Relating to others) capabilities. ‘Thinking’ is defined in the curriculum framework as, … using creative, critical and metacognitive processes to make sense of information, experiences and ideas. These processes can be applied to purposes such as developing understanding, making decisions, shaping actions, or constructing knowledge. (Ministry of Education, 2007, p.12) while ‘Relating to others’ refers to, “students’ abilities to listen actively, recognise different points of view, negotiate, and share ideas” (Ministry of Education, 2007, p. 12). The significance of this study This study is significant for three reasons. Firstly, as Roblyer (2005) comments, fundamental research on the impact of technology on learning is essential to encourage its uptake in the classroom. He claims that it is only when the value of technology for student learning is able to be described and demonstrated, that teachers will begin to include it ‘automatically’ in their programs. As is the case with most innovations, teachers incorporating LOs into their learning programs will inevitably encounter some implementation issues, and will need to invest extra time and energy to overcome the technological challenges that come with such changes (Freebody, et al., 2007). Describing how teachers have successfully overcome such challenges, and the benefits they have gained from the implementation of LOs, may support the uptake of this learning technology through the sharing of practical and theoretical knowledge. Additionally, making links with the Key Competencies – specifically examining how LOs may support the development of Thinking skills in collaborative situations - may encourage the use of LOs through an enhanced sense of ‘understanding of purpose’ and clarity of positioning within the curriculum framework. Secondly, one implication of using digital learning resources such as LOs is the need to develop new research methods and approaches in order to evaluate their effectiveness (Boyd and Watson, 2006). To meet this challenge, new digital data collection tools were used in this study that enable ‘natural’ data to be gathered relating to student activity when engaged with the learning objects. A software application called SnagIt was installed on students’ computers that allowed for the real time collection of all screen activity (mouse movements, menu selection, etc.) and associated verbal interactions, as the students negotiated their way through the LOs. These data were then stored on DVDs for later analysis. In this way, it was possible to gain data that more accurately represented what went on ‘behind the screens’, rather than being influenced by direct researcher presence, mediated through interpretation into field notes, or generated from interviews alone. The use of this software made it possible to map out the students’ ‘learning journeys’, and note the extent and nature of interaction between them and the impact this had on decision making. Thirdly, this study provides a vehicle through which any interaction of the Key Competencies can be explored when contributing to a learning performance. Consistent with the notion that Key Competencies should be viewed as ‘complex Falloon 631 performances’ since they encompass values, skills, knowledge, and attitudes (Delandshere & Petrosky, 1998), and that within any learning experience multiple Key Competencies may be operationalised (Ministry of Education, 2007), this study provides an ideal opportunity to ‘unpack’ any relationship which may exist between the competencies, and note any contribution one makes to the development of others. Research focus and questions The primary goal of this study was to investigate any relationship which might exist between student use of learning objects (LOs) and the development of the Key Competencies of Thinking and Relating to others (Ministry of Education, 2007). Data collection was guided by the following research questions: 1. What evidence exists which suggests that LOs are able to support the development of thinking and collaborative skills when used in small group situations? 2. What is the nature of student interaction when using LOs, that may contribute to the development of thinking and collaborative skills? 3. What implications can be drawn from this study which may assist teachers to optimise any benefits from LO use in their classrooms? Research method A case study methodology was used for this study. Case studies have been described by Erickson as, … the intensive investigation of a single object of social inquiry such as a classroom… and that it holds major advantages in that it allows the immersion of oneself in the dynamics of a single social entity and enables the uncovering of events or processes that one might miss with more superficial methods. (Erickson, 1986, p. 238). Whilst acknowledging the limitations of case studies in terms of their lack of ‘generalisability’, it needs to be noted that the primary purpose of this study was not to arrive at universal or empirical conclusions about the value of LO use, but rather to undertake preliminary exploration which could be used to inform future studies of a similar nature. As Burns (1997) comments, case studies have a number of purposes or functions within educational research. Due to their intense and subjective nature, he states that they are particularly suited to acting as preliminaries to major investigations by providing a “source of hypothesis for future research” (Burns, 1997, p. 365), or in assisting in developing deeper understandings “of the class of events from which the case has been drawn” (p. 366). It is for this reason that such a methodology has been selected as suitable for this study. Data collection and analysis Data were collected between May and August 2009, using a combination of electronic and conventional qualitative research tools. The screen capture application SnagIt was used to produce video and audio recordings of the students as they worked with the LOs, while both teachers and students were interviewed about their experiences during and after the unit, using semi-formal interview schedules. These were supported by the collation of comprehensive researcher field notes which were 632 Australasian Journal of Educational Technology, 2010, 26(5) recorded by the research team during the course of the study. These field notes focused primarily upon observations relating to the nature of student interaction whilst using the objects, the manner in which decisions about navigation pathways and/or content being developed for the objects were arrived at, and identifying features and elements of the LO which the students found particularly easy, challenging, or engaging. In total, 146 audio and/or audiovisual clips were collected, comprising 68 SnagIt screen captures, 43 interviews (35 students and 8 teachers and other staff) and 35 video clips of learning episodes taken during in-class observations. The SnagIt screen capture video was coded against an analytical framework developed from Bloom’s Taxonomy (Cognitive domain) revised by Anderson and Krathwohl (2001) (see Table 1). Whilst acknowledging the limitations of this framework and its essentially descriptive nature, it was useful in this instance as it provided an analytical ‘lens’ through which the actions and discourse of the students could be interpreted, and instances of higher levels of thinking indexed against components of the learning object and interactions with it, and each other. In total, 23 hours of video and associated audio were captured and used in this analysis – the purpose being to identify and ‘map out’ the timing, application and level of thinking exercised by the students as they interacted with the LOs. To assist with improving the accuracy of this analysis, samples of the data were pre-moderated by the research team before all footage was triple blind coded, with the results and decisions then being compared and adjusted to ensure consistent interpretation. These results were then developed into learning journey ‘maps’ which provided graphical representations of student activity when using the LOs, and allowed the researchers to note and record the level of thinking observed, when in the learning episode it occurred, and what events led up or contributed to the performance. A sample of one of these maps is included in the Findings section of this paper. Table 1: The data coding framework (adapted from Anderson and Krathwohl, 2001) Level Description 6. Creating Synthesising or building a structure or pattern from diverse elements. Putting parts together to form a whole, with emphasis on creating new meaning or structure. 5. Evaluating Checking and critiquing using standards. Making judgments about the value of ideas or materials. Higher order thinking 4. Analysing Separating material or concepts into component parts so that its organisational structure may be understood. Distinguishing between facts and inferences. 3. Applying Using a concept in a new situation or unprompted use of an abstraction. Applying what was learned (or other knowledge) to novel situations. 2. Under- standing Comprehending the meaning, translating, interpolating, and interpreting instructions and problems. Stating a problem in one's own words. Lower order thinking 1. Remem- bering Recognising and recalling data or knowledge. Falloon 633 Context and organisation This study took place during two school terms in a New Zealand urban Intermediate school (years 7 and 8 - ages typically 11 to 12.5 years). The ethnic composition of the school was representative of its geographical location and included Maori, Asian, African (Somali refugees) and other ethnic minorities, comprising 51% of the school’s roll. The three participating classes totalled 75 students, and included students from diverse socio-economic backgrounds working at a range of achievement levels. The school principal and ICT coordinator facilitated access to the school to work with the participating teachers. Prior to the trial, the research team met with the teachers and carried out individual planning sessions and group professional development, focussing on LO integration with planning, and linking with learning outcomes. The participating teachers chose the Cartown LO (from the New Zealand Digistore) to use within their ‘Our Community’ unit. This learning object was a simulation in which students needed to develop and communicate a perspective related to the possible imposition of a traffic congestion toll in a city, designed to discourage cars from coming into the central city during peak times. The LO followed an Environmental Education/Sustainability theme, and integrates substantial elements of the Social Studies, Language and Literacy curriculum areas. In the first or ‘teaching’ part of the LO, students explore the social, economic and environmental impact of a congestion toll. Having collated information from various sources, they then complete the second part of the LO which is an online survey in which they identify the media that influenced them most, before expressing their initial position as to the introduction of the toll. In the third or ‘action’ part of the LO, they collate additional information from various media to support their perspective, before submitting a final summary statement. At the conclusion of the simulation, their initial perspective and the final statement they chose to communicate are compared by the LO, and they are prompted to re-evaluate their decisions if their communicated message is inconsistent with their initial stance. The students were paired to work with the LO, taking turns in the school’s ICT suite. Findings and discussion The findings below are presented as short summaries of action and discourse associated with specific events during the students’ learning episodes, that have been coded under each of the classifications of the analytical framework (Table 1). While it is not possible to totally convey in a written description the full meaning from the ‘live’ data recorded on video using the SnagIt screen capture application, descriptors have been generated against each classification to provide illustrative examples of student discourse and activity. Table 2 summarises these. Whilst acknowledging the limitations of this initial analysis in terms of its generalisabilty, in reviewing Table 2’s summary, evidence does exist that for this group of students, using this particular LO, the experience provided them with an opportunity to utilise and refine a range of thinking skills of both lower and higher order. While not all students in the trial displayed equal functioning at all levels, the demands of, and prompts embedded within the LO, required them to interact critically with the information on screen, discussing the usefulness (or otherwise) of both the content, and the mode used to deliver it. They were then required to reach negotiated and agreed-to decisions about which information was useful in constructing and 634 Australasian Journal of Educational Technology, 2010, 26(5) Table 2: Examples of lower and higher order thinking revealed in LO learning journey transcripts, and coded against the analytical framework (see Table 1) Framework categorisation Example - what students did Screenshot of clip Creating: Designing a media clip to convey their message to a target audie- nce, by analysing and inte- grating data of several types from different sourc- es. Revising initial draft to improve the communicati- on of perspectives. Reorga- nising elements into a new output. Construct and plan a newspaper layout – students record, summarise and blend different media and content to create a layout appropriate to their target audience and key messages. Pairs effectively discuss, interact and decision-make to reach a negotiated and agreed to outcome. (J&M: 12.00-13.09/14.30-15.30) Evaluating: Effective collaboration in jointly evaluating outcomes. Judging the value of the media resources by applying standards and criteria. Discuss, evaluate and select from a range of media resources (TV, radio, computer, newspaper etc) to get information appropriate to their key messages and target audience. Appraising the ‘fitness for purpose’ of information, and the media used to convey it. (Y&J: 3.25-4.45)H ig he r or de r th in ki ng Analysing: Comparing and deconstructing information related to the imposition of a road toll. Developing understanding of, and identifying relationships between, different sources and types of information. Discuss the different ways information is conveyed within the LO via news, chat rooms and blogs - and their relevance to, and impact upon, decision-making. Identify the mode/s of communication which they consider to be most effective, and why. (Y&J:5.31- 7.00) Applying: Using a concept or method learnt in one setting and applying it to another. For example, applying Inquiry Learning style questioning across more than one curriculum area. Applying knowledge about carbon dioxide emissions learnt in a class literacy unit to supporting their perspectives in this task (teacher rated) (M(2)&A: 2:39-2:48) Understanding: Making meaning and drawing implications from newspaper headlines. Explaining the main ideas and concepts contained in provided information. “Drivers of trucks will have to pay large amounts of money to do their jobs” (‘Daily News’ - LO). Discussion of the implications this may have ‘downstream’, e.g. the need to charge more for goods in shops, effects on inflation, employment etc. (Y&J: 7.16-8.50) Lo w er o rd er th in ki ng Remembering: Recalling specific information about different modes of transp- ort. Discussing the impact these may have on the env- ironment (e.g. differing levels of pollution, differ- ing energy consumption). Discussing the different modes of transport in their city and analysing their strengths and weaknesses in relation to air pollution and energy consumption (J&M: 0.23-2) Falloon 635 supporting their perspectives, and in generating appropriate content and an effective layout to convey this to their selected key audience. The way this LO was designed in terms of the nature of its output (ie: the presentation and justification of a perspective based on the analysis of varied information from a number of sources) demanded student critique of information, and required them to justify and explain their decisions regarding the selection and use of that information. If they failed to do that, it was not possible to successfully complete the exercise. While the rigour of the critique and the criteria applied in reviewing the information (and subsequently how it was used in developing the outcome) was certainly not uniform across the group, in all observed cases evidence existed that students were willing to ‘give it a go’, engaging with each other in discussing, thinking about, and developing strategies to solve the problems required by the LO. The quality of this interaction - specifically the knowledge each member of the pair was able to contribute to decision making and how decisions were negotiated - and when this occurred in the students’ learning journeys, were instrumental in the formation of accurate and high quality outcomes. Collaboration (Relating to others) and the promotion of higher order thinking Hipkins (2006) in her introduction to the Key Competencies, promotes the perspective that they should not each be treated as isolated entities, but rather attempts should be made in teacher planning and teaching to draw them together in an integrated manner within units of learning. Data from this study indicated the efficacy of this approach, with significant learning benefits being evident from students interacting and working in pairs on the LOs - the associated discourse appearing to contribute significantly to ‘raising the bar’ for both the frequency and complexity of thinking processes used. Data supporting this claim was gathered in a number of ways, including teacher interviews and comments, and the analysis of LO (SnagIt) transcripts. Data gathered during this study illustrated that for some students working in collaborative pairs, the level of thinking skills displayed, and the frequency and placement of these within the learning episode, may have had a positive impact upon the development of outcomes, many of which were of superior quality and accuracy. Figure 1 shows, for one example, how the occurrences of the six levels of thinking distributed around the recording of Relating to others interactions. Learning episodes were coded on a relative timeline (percentage of time elapsed during the episode, rather than ‘real’ time) to allow for comparisons between students to be made. The distribution in the example below is consistent with results from earlier trials (Falloon, Janson & Janson, 2008), and indicate how Relating to others appears to influence the exercise of higher order thinking, with the tendency being for such events to occur relatively adjacent to Relating to others interactions. An appraisal of the outcomes produced by these students (as interpreted through the LO’s evaluation function) also indicated generally higher levels of accuracy in the matching of evidence with perspectives being promoted. This finding was also supported by interview responses from some students, one who stated in response to the question, when do you prefer working together as opposed to alone on LOs? - “when it is hard for me and I don’t get the questions – it is good to be able to talk to someone else about it to get it worked out” (student H, interview, 0:34-1:24.). In stating this, and as commented on by one teacher during an interview, observing that interaction is occurring does not necessarily mean that it is quality interaction which will contribute to an enhanced performance. That is, some interactions may in fact only be ‘appearances’ of quality Relating to others – with one peer dominating the 636 Australasian Journal of Educational Technology, 2010, 26(5) Figure 1: A sample learning journey (students H & L) showing Relating to others interactions (arrows) and ‘Thinking’ levels (bars from 1 to 6 as per Table 1) on a relative timeline interaction, or with too much ‘off task’ discussion occurring, which does little to contribute to the outcome. This observation prompted the teacher to comment that it is vital not only to note the occurrence of Relating to others, but also, where possible, to monitor its quality and assess its contribution to the overall performance (DW, 2008, interview transcript, p. 16). He further commented that teachers should be mindful of group composition and selection, effectively ‘managing’ as much as possible the extent to which non-productive interaction can occur: … students’ contributions to the peer study dynamic may be unequal, even though at a glance the students seem to be equitably involved in the use of the LO… students participated willingly in pairs set up randomly rather than in self-chosen groups. This possibly allowed them to focus more on the work than on socialising. In some cases, I would imagine friction between students who don't work together might be an issue, but in this case the participants evaluated cooperated well, reinforcing the notion that ‘Relating to others’ was successfully occurring during the trial, in most cases. (DW, 2008, interview transcript, p. 3). Interview data from participating teachers also indicated that some students were integrating knowledge developed in other subjects and units, into their work with the LOs. This specifically related to the application of information analysis and processing skills that had been developed during the course of the year through inquiry learning- based science and literacy topics. In commenting on the SnagIt video capture, DW stated that this was a very satisfying outcome for the teachers, who often thought this might have been occurring, but had previously had no concrete evidence of it. At a professional level, DW commented that the study provided a valuable opportunity for detailed classroom observation, and an “in depth analysis of class work" (DW, 2008, interview transcript, p. 4). In elaborating upon this observation, he stated that the opportunity to have ‘time out’ to scrutinise the working methods and strategies of his students, both through direct observation and viewing the SnagIt captures, had alerted him to the fact that some students may be "missing the point, completing work Falloon 637 without understanding and strategising to get work finished, rather than to do it with accuracy or comprehension" (DW, 2008, interview transcript, p. 4.). This observation prompted him to reflect upon the extent to which he evaluated not only what students produced during a unit, but also how they went about it. He commented that with a heavier emphasis in the school on assessment of a summative nature, it is easy to lose sight of the ‘processes of learning’, in favour of outcomes that may more easily be measured (DW, 2008, interview transcript, p. 6). According to DW, a further relevant outcome was the extent to which students in the initial trial group (his class) gained sufficient confidence to act a peer tutors for students in other classes who were involved in subsequent trials. Of the 34 students who participated in the first two days of the trials, 15 subsequently asked to become ‘mentor peers’ and assist other students. This interaction assumed two forms – leading and facilitating interactions in which mentors explained to a group of their peers what the exercise was about and what to expect while completing it, and one on one sessions working with their ‘mentees’ using the LO. While the fact that this occurred by itself may not be highly significant, what the teachers noted through viewing the SnagIt captures and reading the associated transcripts, were the strategies the students were applying to this task. An analysis of the transcripts indicated that students were not ‘telling’ or ‘showing’ others what to do (or doing it for them), but rather they had adopted questioning strategies aligned with inquiry learning approaches that had been previously modelled by their class teachers, and were using them in their interactions as mentors. In one such example, the mentor used phrases such as, “what other information do you think you need to make up your opinion?”, “what do you think should be done about this problem?” and “what information can you use to back up what you are saying?” (M&A, 2008, SnagIt transcript, 1:00-2:39). Some teachers commented on the value of these interactions during informal group feedback sessions (DW, 2008, recorded discussion, line 8) and during more structured interviews. After observing one student mentor’s skilful use of strategic questioning, HK commented, "she's done this before - setting her (peer) up for success" (HK, 2008, transcript, line 9). As for the previous example, the advent of the SnagIt data provided valuable confirmation of intuitively held beliefs. Other findings There was little doubt that the majority of students found using the LOs highly engaging and motivating, and in many instances they were keen to use their own time in order to improve the quality of their outcome. In a follow up interview, one teacher described the typical attitude students held towards using the LO as … they (the students) willingly participated and were highly motivated to use the learning objects. In class, they would ask when they would get the chance to use them and would complain if only a few were chosen… (DW, 2008, Teacher transcript, 2, (4) line 2). In one case, the SnagIt data revealed one pair redoing an exercise five times in order to reach a more satisfactory completion point, and in all recorded instances, students undertook this extra work willingly, without appearing to lose interest or motivation. In a second example, high levels of engagement motivated one group of students to work through the LO twice, deciding on their own accord that they wanted to expand their experience by trialling two different communication media, rather than just one. They then compared the outcomes from each trial before settling on a final submission. 638 Australasian Journal of Educational Technology, 2010, 26(5) Another teacher concluded that, “the LOs were clearly highly engaging (although differentially) ... after a point some students gave up, whilst others were focused on completing ‘correct’ or quality work” (HK, 2008, interview transcript 2, (4) line 16). While some of these characteristics at a general level could well be attributed to the individual students, such comments point to the need to carry out further research into the design of LOs to better understand how they might act as useful ‘intelligent scaffolds’ for students who struggle to persevere when solving more complex problems. With the increasing capacity of digital technologies to generate meaningful feedback, LOs could be powerful tools to formatively assist in the development of specific problem solving capabilities. However, presently it remains that ‘teacher watch’ is essential for ensuring the optimum benefit from LOs, so that the differences between completion ‘modes’ and the ways students work with them, are clearly understood. Working with LOs during this study also alerted teachers to the potential of working with such resources more often and in different areas of the curriculum, and was seen as a valuable professional development opportunity. As DW commented: Having been gifted the opportunity to be assisted in experiencing and analysing LO use in a real context, it has encouraged me to continue using them as an excellent resource where previously I had dismissed them as being another thing I did not have time to investigate or trial ... I have since then used the resources to compare my usual teaching of fair testing in science to the LO version - which was far superior and much easier. (DW, 2008, Teacher transcript , p. 1). The school capitalised upon this enthusiasm as teachers presented their work at staff meetings and later at a teacher conference. They strongly advocated including LOs in school-wide teaching practice, citing cognitive as well as motivational and collaborative benefits: … I will continue to teach using LOs, as the medium engages the students, helps easily distracted or disinterested students, encourages socio-dialogical collaboration, and allows quality resources to be used without hours of preparation time for the teacher. (DW, 2008, Teacher transcript, p.14.) Some teachers who participated in this initial trial subsequently took on mentoring roles with other staff members, to assist their peers to learn from their experiences and to help prevent known technological and logistical obstacles associated with implementation. Conclusion Whilst acknowledging the limited scope of this study, the data provides tentative indications of the efficacy of programs integrating learning objects, and signals that their use could provide valuable opportunities to support development of the ‘Thinking’ and ‘Relating to others’ Key Competencies. The study also indicates some value could be gained from students using LOs in pairs or smaller groups, as the associated interaction and discourse (RTO) appeared to encourage and promote the use of higher order thinking, often leading to more accurate and higher quality outcomes. The study also provides some early insights into the manner in which the Key Competencies can complement each other when using digital resources, and how Falloon 639 they are able to be developed ‘in tandem’ when aligned to a project-based learning model. Additionally, data indicates that opportunities for developing thinking capabilities through the integrated use of LOs can contribute to student learning ‘across the board’, and when associated with ‘Relating to others’, can provide a useful vehicle for the development of collaborative problem solving capacity. Learning objects can provide opportunities for students to assume roles and responsibilities embedded in realistic simulations, and through doing so may help them develop competencies in a variety of areas – both academic and social. While this study focused primarily on the ‘Thinking’ and ‘Relating to others’ competencies, it also revealed opportunities to explore the potential of LOs to enhance competencies in other areas, such as ‘Managing self’ and ‘Participating and contributing’ (citizenship). As Hipkins (2006) points out, students need to be exposed to learning experiences which engage them in authentic social decision making and problem solving, so that they “understand the importance of contributing to the quality and sustainability of social, physical and economic environments” (p. 51). While recognising the need for further research of this nature, it may well be that LOs provide an accessible means for teachers to support the achievement of such objectives. Additionally, this study indicated clearly the motivational value for students using learning objects, and it also highlighted areas for teachers to be mindful of to ensure that maximum benefit is being gained from their use. These included monitoring the nature of collaboration between peers to determine the level and quality of engagement of each group member, being vigilant in noting the depth and sophistication of students’ understandings developed from their use of objects to ensure they are not engaging only at a ‘surface level’ or with the object’s ‘entertainment’ features, ensuring that the concepts explored in the objects closely align with learning goals, and considering carefully the composition of working groups to mitigate possible issues associated with peer domination. In addition, the study also highlighted the benefit of using new digital data collection tools such as SnagIt, and demonstrated the value of such tools for gathering authentic data that can provide an accurate picture of student activity and interaction as they go about learning tasks using computers. However, their use also raised some issues with regard to the objectivity of data coding and the recording of outcomes in text form. While in this study every effort was made to ensure consistency in the interpretation of the SnagIt clips (triple-coded), the mere act of interpreting the graphical and audio information contained in them was subjective. While this issue is consistent with interpretive studies of this type, it does need to be considered as a caveat in drawing any firm conclusions from it. Additionally, in writing up this study, it quickly became apparent that a lot of the richness and meaning of data contained in the SnagIt clips became ‘diluted’ in their translation to printed text and still images. That is, valuable cues such as voice intonation and expression, visual indications of ‘trial and error’ type problem solving strategies, and informal but meaningful verbal exchanges between the students - and their impact upon decision making and outcome development, could not be clearly displayed. There is a developing need for academic publishers to acknowledge the worthiness of such data, and explore possible ways this could be incorporated into publications and accessed by readers. With the expansion of Web 2.0 technology and improved broadband access, it is hoped the availability and accessibility of such data may become commonplace in the future. 640 Australasian Journal of Educational Technology, 2010, 26(5) Acknowledgment The authors gratefully acknowledges the support of the staff and students of Peachgrove Intermediate School in the development of this paper. References Anderson, L. W. & Krathwohl, D. R. (Eds.) (2001). A taxonomy for learning, teaching, and assessing: A revision of Bloom's Taxonomy of Educational Objectives. Longman: New York. Boyd, S. & Watson, V. (2006). Unpacking the key competencies: What does it mean for primary schools? Paper presented at the NZARE Conference: Rotorua. [verified 21 Jun 2010] http://www.nzcer.org.nz/pdfs/15238.pdf Brown, M. A. (1997). Educational technology: I’ve been thinking. Computers in New Zealand Schools, 9(1), 45-48. Burns, R.B. (1997). Introduction to research methods (3rd ed.). Melbourne: Addison Wesley Longman. Capper, P. (2001). Report on the implementation of ICT in New Zealand schools (unpublished). Wellington, New Zealand: WEB Research. Culp, K. M., Honey, M. & Spielvogel, R. (2003). Achieving local relevance and broader influence. In G. Haertal & B. Means (Eds), Evaluating educational technology: Effective research designs for improving learning (pp. 75-95). New York: Columbia University, Teachers College Press. Delandshere, G. & Petrosky, A. (1998). Assessment of complex performances: Limitation of key measurement assumptions. Educational Researcher, 27(2), 14-24. Erickson, F. (1986). Qualitative methods in research on teaching. In M. C. Wittrock (Ed.), Handbook of research on teaching (3rd ed., pp. 119-161). New York: Macmillan. Falloon, G. W. (2006a). The Microsoft Learning Object Project (MiLO) - End of Project Iteration 2 Report (MiLO 2) (unpublished). A research report prepared for Microsoft Ltd. Auckland, New Zealand. Falloon, G. W. (2006b). Project Evaluation Report: The Microsoft Digital Learning Object Pilot (Project MiLO) (unpublished). A research report prepared for Microsoft Ltd. Auckland, New Zealand. Falloon, G., Janson, A. & Janson, R. (2008). Digital learning objects: Towards an understanding of their value in supporting key competencies within the New Zealand curriculum framework (2007). Paper presented at ACEC2008 Conference, Canberra, Australia. [verified 21 Jun 2010, 5.2 MB] http://conferences.acce.edu.au/uploads/documents/store/conferences/conf_P_ 889_Learning%20Object%20paper%20Falloon%20and%20Janson%20final.doc Freebody, P., Muspratt, S. & McRae, D. (2007). Evaluating The Le@rning Federation’s online curriculum content initiative. Melbourne, Australia: The Le@rning Federation. [verified 21 Jun 2010] http://www.thelearningfederation.edu.au/verve/_resources/freebody_final_report_2007.pdf Hartnell-Young, E. & Vetere, F. (2008). A means of personalising learning: Incorporating old and new literacies in the curriculum with mobile phones. Curriculum Journal, 19(4), 283-292. Heinecke, W., Blasi, L., Milman, N. & Washington, L. (1999). New directions in the evaluation of the effectiveness of educational technologies. Paper presented at the Secretary’s Conference on Educational Technology 1999. [viewed 17 Jul 2009, verified 21 Jun 2010]. http://www2.ed.gov/rschstat/eval/tech/techconf99/whitepapers/paper8.html Falloon 641 Hipkins, R. (2006). The nature of key competencies: A background paper. Wellington: New Zealand Council for Educational Research. [verified 22 Jun 2010] http://www.nzcer.org.nz/default.php?products_id=1805 McCombs, B. (2000). Assessing the role of educational technology in the teaching and learning process: A learner-centered perspective. Paper presented at The Secretary’s Conference on Educational Technology: University of Denver Research Institute. [viewed 1 May 2009, verified 21 Jun 2010]. http://www2.ed.gov/rschstat/eval/tech/techconf00/mccombs_paper.html McNabb, M., Hawkes, M. & Rouk, U. (1999). Critical issues in evaluating the effectiveness of technology. Paper presented at the National Conference on Educational Technology: Washington, D.C. [viewed 16 May 2009, verified 21 Jun 2010] http://www2.ed.gov/rschstat/eval/tech/techconf99/confsum.pdf Ministry of Education (2007). The New Zealand Curriculum. Learning Media Ltd. Wellington: New Zealand Government. http://nzcurriculum.tki.org.nz/Curriculum-documents/The- New-Zealand-Curriculum Nicholas, H. & Ng, W. (2009). Engage secondary school students in extended and open learning supported by online technologies. Journal of Research on Technology in Education, 41(3), 305- 328. OECD (2005). The definition and selection of key competencies. Executive summary. [verified 21 Jun 2010] http://www.oecd.org/dataoecd/47/61/35070367.pdf Parr, J. & Ward, L. (2004). Evaluation of the Digital Opportunities project FarNet: Learning communities in the Far North (unpublished). Wellington, New Zealand: Ministry of Education. Postman, N. (1993). Technopoly: The surrender of culture to technology. New York: Vintage Books. Robertson, J. W. (2003). Stepping out of the box: Rethinking the failure of ICT to transform schools. Journal of Educational Change, 4(4), 323-344. Roblyer, M. D. (2005). Educational technology research that makes a difference: Series introduction. Contemporary Issues in Technology and Teacher Education, 5(2), 192-201. [verified 21 Jun 2010] http://www.editlib.org/f/21850 Rychen, D. & Salganik, L. (2003). A holistic model of competence. In D. Rychen & L. Salganik (Eds.), Key competencies for a successful life and a well-functioning society. Cambridge, MA: Hogrefe and Huber. Somekh, B., Underwood, J., Convery, A., Dillon, G., Jarvis, J., Lewin, C. et al. (2007). Evaluation of the ICT Test Bed project (p. 17). Final report June 2007. Centre for ICT, Pedagogy and Learning Education and Social Research Institute, Manchester Metropolitan University, Division of Psychology, Nottingham Trent University: BECTA. [viewed 18 June 2009, verified 21 Jun 2010] http://www.evaluation.icttestbed.org.uk/files/test_bed_evaluation_2006_learning.pdf Watson, N. & Kilcher, A. (2004). PiL Program Evaluation: Guidance for academic program managers and evaluators. Redmond: Microsoft Corporation. Wiley, D. A. (2000). Connecting learning objects to instructional design theory: A definition, a metaphor, and a taxonomy. In D. A. Wiley (Ed.), The instructional use of learning objects: Online version. [viewed 16 Feb 2009, verified 22 Feb 2010] http://reusability.org/read/chapters/wiley.doc Wood, D. (2003). Think again: Hindsight, insight and foresight on ICT in schools (Report prepared for the European School Network consortium). Nottingham: University of Nottingham. [verified 22 Jun 2010] http://www-old.eun.org/insight-pdf/ernist/David_Wood_Think_Again.pdf 642 Australasian Journal of Educational Technology, 2010, 26(5) Wright, N. (2009). e-Learning and New Zealand Schools: A literature review (unpublished). A report prepared for the eLearning Unit. Wellington, New Zealand: Ministry of Education. Xiufeng, L., Macmillan, R. & Timmons, V. (1998). Assessing the impact of computer integration on students. Journal of Research on Computing in Education, 31, 189-204. Authors: Dr Garry Falloon Department of Professional Studies in Education The Faculty of Education, The University of Waikato Hamilton, New Zealand Email: falloong@waikato.ac.nz Dr Annick Janson, Director Leadership Division Ecosynergy Group Limited (egl) 30 Aurora Terrace, Hamilton, New Zealand Email: annick@egl.ac.nz Web: http://egl.ac.nz/ Dr Robin Janson, Director Ecology Division Ecosynergy Group Limited (egl) 30 Aurora Terrace, Hamilton, New Zealand Email: robin@egl.ac.nz Web: http://egl.ac.nz/ Please cite as: Falloon, G., Janson, A. & Janson, R. (2010). Learning objects and the development of students' key competencies: A New Zealand school experience. Australasian Journal of Educational Technology, 26(5), 626-642. http://www.ascilite.org.au/ajet/ajet26/falloon.pdf Errata 28 May 2011: AJET regrets that authors Dr Annick Janson and Dr Robin Janson were omitted from the version of this article mounted on 16 July 2010.