barwell.pdf Australasian Journal of Educational Technology 2011, 27(Special issue, 5), 765-780 Marking machinima: A case study in assessing student use of a Web 2.0 technology Graham Barwell University of Wollongong Chris Moore Deakin University Ruth Walker University of Wollongong The model of learning best suited to the future may be one which sees learning as the process of managing the different kinds of participation an individual might have in complex social systems. Learning capability and engagement is thus dependent on the relationship between an individual identity and social systems. We report on the incorporation of machinima, a Web 2.0 technology, as part of an interdisciplinary and collaborative project where the focus is not on the mastery of the tools or the acquisition of predetermined knowledge, but on the development of learning engagement. We provide the case study of a pilot project involving students across two Arts disciplines collaborating via the game, World of Warcraft, to produce an animated adaptation of one of Geoffrey Chaucer’s Canterbury Tales. Their contributions were differently assessed according to the pre-existing requirements of their home disciplines. We argue that the assessment in such projects, in conjunction with innovations and experimentation with Web 2.0 technologies, should shift from an emphasis on product to process. We believe that this has a sound pedagogical and theoretical foundation, and also fits better with the increasingly digitalised, unfixed and interdisciplinary world that students will face on graduation. Introduction Cultural impacts of digital technologies in the education sector are evident in the changing nature of learning and teaching models, in workload pressures in higher education, and in increasingly unreliable employment markets and shifting work environments for graduating students. Learning tasks that incorporate rapidly evolving technologies encourage students to anticipate and adapt to the shifting Web 2.0 environments and the collaborative practices associated with them. While we resist the more extreme claims of technological determinism, which posit that the current and future generation of students ‘think’ differently from previous generations due to their acceptance of digital mobile technology cultures and practices (Prensky, 2001), we recognise that digital technologies do affect the way students today act, research, communicate, study and learn. At the very least, the role of social media and mobile technologies has been naturalised in students’ everyday lives, in ways that present both challenges and opportunities in teaching that were not imaginable by previous generations of educators. 766 Australasian Journal of Educational Technology, 27(Special issue, 5), 2011 Learning is not always easy for educators to measure and the introduction of the mobile, social and participatory elements of Web 2.0 further intensifies the complexity of its assessment. Biggs’ (2007) principles of constructive alignment require the educator to determine in advance the levels of understanding students should achieve and how they should be demonstrated in assessable activities, which meet pre- determined learning objectives. Assessment based on this kind of approach does not always lend itself to innovative activities which depend on Web 2.0 technologies, unless those activities are modelled on very narrowly defined outcomes, such as mastery of a particular tool. This kind of expectation of mastery is, as Prensky (2001; 2010) notes, one reason why educators are fearful of new technologies, unless they themselves are confident users of the tool in question. The Biggsian assessment model thus does not provide room for the incorporation into established assessment strategies of student experimentation with Web 2.0 technologies. In proposing that a future saturated in constantly changing digital technologies is particularly uncertain, Wenger (2006) argued that the model of learning best suited to such a future is one which sees learning as being demonstrated in managing the different kinds of participation an individual might have in complex social systems. Learning capability thus lies in the relationship between individual identities and social systems, and the development of this relationship relies on ‘the recognition and cultivation of constellations of communities across relevant dimensions’ (Wenger, 2006; p.4). If Wenger’s approach is to be adopted (and we argue that it should be) in relation to the use of Web 2.0 technologies in higher education, then we must expand our approach from assessment based solely on the Biggsian model of constructive alignment to accommodate the ways in which students demonstrate a capacity to take up new technologies and practices, to blend traditional and digital literacies, and to find ways of working with a variety of stakeholders. One of the aims for the teaching and learning experiment reported in this paper was to determine the feasibility of assessment based on Wenger’s approach. Accordingly, the pilot project worked from the perspective, highlighted by Prensky (2010), that it is not the educator’s responsibility to be expert in the technologies, or to instill excellence in production, in order to support student learning. The project involved an interdisciplinary Web 2.0 project based on World of Warcraft as we trialled a collaboration in a machinima production between two small cohorts of students in two different humanities disciplines. Machinima refers to the production of animation by capturing and editing the video produced within a digital game environment. This project sought to utilise Web 2.0 flexibility in meeting the demand for collaborative activities in conjunction with pre-existing assessment criteria. Central to the aim of the project was the desire to increase student motivation and promote creative experimentation, and support students adapting to available technologies as they contribute in ways that demonstrate their original disciplinary understanding as well as critically reflect on their own and others’ performance. This shift in an assessment emphasis from product to process, we argue, fits better with the increasingly digitalised, unfixed and interdisciplinary world the students will inevitably enter. We propose that it is possible to draw a series of generalisable observations about assessment of Web 2.0 work, despite the limitations of their being based solely on a small-scale pilot project. The pilot project had a very clear set of outcomes, both positive and negative, which are discussed in regards to the development of learning capability and engagement and the strategies of assessment employed for the two disciplinary cohorts. Barwell, Moore and Walker 767 Literature review Learning and teaching with machinima and online games Virtual worlds are a well-established, innovative environment for learning and teaching (Jarmon et al., 2009; Whitton & Hollins, 2008; Eschenbrenner et al., 2008). They may be effective for interactive learning because they model problems encountered in everyday practice that are typically emergent and not well-defined (Jonassen, 2003; Brown, 2004). The current lack of guidelines and availability of best practice recommendations for the educational use of virtual worlds, however, hinders wider implementation. As Schiller (2009) notes, the literature hasn’t yet demonstrated the extent to which student work in virtual worlds has impacted on their learning outcomes. While Second Life has been widely adopted by those interested in exploring virtual environments to facilitate education, gaming worlds such as World of Warcraft (WoW) have received less attention. After experimenting with the use of WoW to create a course that critically examined identity, gender, culture and citizenship, Pirius and Creel (2010) report high levels of student engagement and acknowledge the play-based learning approach results in knowledge that is dynamic but highly challenging to manage effectively. Machinima has been used for educational purposes, notably within focused media studies programs, but not in interdisciplinary contexts (Morozov, 2008). Machinima both relies on and facilitates engagement between students and, as an interventionist media, highlights the point made by Middleton and Mather (2008; p.207) that ‘digital media works best when simply developed for deployment within a blended curriculum to inform learning activity, and where the media are specifically designed to set challenges, seed ideas, or illustrate problems’. Machinima calls on a range of technical, discursive and evaluative literacies; with its origins in fan activity there is no ‘correct’ way to produce, view or enjoy the results. As such, producing machinima draws on an appreciation of the interaction between various media (navigation of virtual spaces, recording and editing video, uploading and tagging online content) and group activities (negotiation, synchronous/asynchronous contributions, and performance). The interactive aspect of the technical skills, as well as the negotiation of a social game-playing environment, can be tested by an appropriately designed assessment task. Collaborative Web 2.0 learning activities The intent of this machinima project was to find an activity that would not just facilitate collaboration, but call on students’ active learning and problem-solving skills as part of an interdisciplinary team, both online and offline. The effectiveness of peer- based cooperation and collaboration in promoting student learning, understanding and achievement has been widely discussed (e.g. Slavin, 1995; Dillenbourg et al., 1996; Scanlon, 2000) with a broad consensus of its positive effects, if not with agreement on why and under what conditions it is most effective. In summary, however, it seems that successful learning outcomes from collaborative tasks rely on student motivation, social cohesion, and engagement on working towards a shared outcome, rather than a focus on a final product. Determining these outcomes in a Web 2.0 context is less well-developed, especially in relation to machinima and virtual worlds. In a recent discussion of the assessment of 768 Australasian Journal of Educational Technology, 27(Special issue, 5), 2011 collaborative learning through wikis, Judd et al. (2010) note there is limited literature about cooperative and collaborative interactions in such environments. Of course, compared with wikis, student roles in a machinima project involve a broader range of networking activities: negotiation of tasks between and inside groups, testing, script production and exchange, storyboard development, meetings, performance, recording, editing, uploading and discussion. Other literature about the educational use of collaborative technologies similarly focuses on wikis and blogs (Elliot, 2008), or even online comments to uploaded YouTube videos (Ng & Hussain, 2009) rather than machinima and virtual worlds, which remain relatively unexplored (e.g. they are an aside in the overview of Web 2.0 teaching and learning resources from Gray et al., 2010). Assessing Web 2.0 and active learning In developing this project, we were conscious that the mere incorporation of digital technologies into a teaching and learning context does not make the resulting educational practice ‘Web 2.0’ (Dohn, 2009), and strove instead to emphasise the practices and processes inherent to social media, such as participation and collaboration. Collaborative online activities can improve learning experiences most effectively if they can equip students for participation in their life-long learning futures (Boud & Falchikov, 2006), and are well integrated into the curriculum and its assessment . This follows on from Biggs’ (2007) principles of constructive alignment which recognise the relationship between engagement with learning and assessment. Here, assessment design depends on a pre-determination of the expected levels of student understanding and their demonstration in assessable activities to meet learning objectives. This equation, however, directly limits the degree to which experimentation with Web 2.0 technologies to support pedagogical innovation can occur, as our interdisciplinary machinima project required an alternative approach to the relationship between the Web 2.0 technology, the students’ collaboration, and the assessment tasks of the two disciplines. In developing the communities of practice model of education, Wenger (2006) highlights a social identity theory of learning. This model focuses on increasing learning capacity through mutual negotiation rather than depending on a hierarchical arrangement, moving ‘from learning as the mere acquisition of stuff to learning as a changing experience of participation’ (p.2). Wenger offers the blog as metaphor for this type of learning, where the judgment of its success is not based on size, output or hits but on the ‘work it enables through widespread appropriation of its perspective’ (p.9). Schiller’s (2009) proposal for an assessment framework, based on an evaluation of a Second Life project, comprised a variety of items, such as reflection essays and chat transcripts, to measure learning motivation and attitudes. Whether applied to blogs, wikis or virtual worlds, the teaching focus must shift from the teacher to the student, in ways that empower the students’ learning performance (Lee & McLoughlin, 2007). Weimer (2002) identified five main principles that shift instructional practice in this way: balance of power, function of content, the role of the teacher, the responsibility for learning, and the purpose and principles of evaluation. In the conventional, teacher-centred approach, assessment is used to monitor learning, with an emphasis on tests to measure declarative knowledge or applications to small problems, rather than the depth of understanding or skills the students have acquired. A learner- Barwell, Moore and Walker 769 centred methodology deploys a variety of assessment items, assessed in multiple stages, to enhance students’ potential to promote learning and give them opportunities to develop self and peer assessment skills. Just as Schiller (2009) had used Wenger’s model to design a collaborative student project in Second Life, so did we in the ‘hands off’ design of the learning activity in the machinima project, which required students to engage at a higher cognitive level of learning to problem solve the complex task. Here, responsibility for negotiating the level of contribution between the two cohorts rested on the students themselves, after being provided with communication opportunities, resources and technical support. With this approach, we negotiated the delicate balance between teacher control and student autonomy, which has been stressed by Drexler (2010) as key for the ongoing challenge in encouraging students to self-regulate their learning with new technologies. Interdisciplinary assessment tasks There is some debate about the term ‘interdisciplinary’ (Weingart & Stehr, 2000). Mansilla et al. (2009; p.337) identify ‘interdisciplinary understanding’ as the capacity to integrate knowledge and modes of thinking in two or more disciplines to produce a cognitive advancement - such as solving a problem or creating a product - that would have been impossible through single disciplinary means. With quality interdisciplinary work grounded in disciplinary understanding and expertise, they argue that students need to be encouraged to move beyond the view of disciplines as fixed bodies of information to understand the constructed and dynamic nature of knowledge production. There is a growing interest in interdisciplinarity in the higher education sector, as graduates are increasingly expected to be able to work effectively in organisational teams that are often interdisciplinary and with diverse functional areas of responsibility (Kruck & Teer, 2009). Web 2.0 technologies have helped ignite this interest in interdisciplinary work, as they encourage unprecedented information sharing and collaborative project development. However, most of the collaborative project work mentioned in the literature is relatively homogenous, drawn from a single class or disciplinary background, with few that used interdisciplinary teams (Napier & Johnson, 2007). Further effort is needed in exploring opportunities to work in ways that more accurately reflect what students will experience when they enter the workforce, and how their use of Web 2.0 can facilitate interdisciplinary project development. There is also a considerable body of literature dealing with differences among academic disciplines regarding collaborative work, with fields like humanities having low consensus on the central paradigms, and thus being less receptive to collaborative projects (Creamer, 2004; p.204), despite the demands for interdisciplinary teams in the workforce. Less work has been done on assessing interdisciplinary student projects, let alone interdisciplinary online collaborative projects. Case study and methodology The project, which ran over one session in 2010, had several aims. One was to test the practicality and pedagogic value of machinima making in an interdisciplinary context. 770 Australasian Journal of Educational Technology, 27(Special issue, 5), 2011 Part of this involved trialling the assessment of collaborative and cooperative interactions among project participants, thus contributing to an area of Web 2.0 pedagogy which is not yet well covered in the literature. We were further interested in assessing an interdisciplinary project in a field like humanities, which traditionally focuses on individual achievement. Since the project design was based on Wenger’s approach to learning, our strategy in the assessment was to implement a learner- centred approach as far as practicable. While we anticipated that a novel project would encounter technical and logistical difficulties, we did not know if these would be overcome until some weeks after session began. Because the Faculty required that the nature of assessment tasks be finalised before this, we were unable to negotiate the tasks with the student participants, so designed them together, pooling our respective experience and keeping within the confines of pre-existing tasks which already had Faculty approval. We tried to lessen the impact of a teacher-centred approach by focusing more on student reflections on their involvement, rather than on the finished product, which was not directly assessed. Our project involved two small groups of student volunteers: five Digital Communication students from a subject titled ‘New Media Communication’, and four English Studies students from a Chaucer subject. The small number of student participants from each disciplinary program was an unavoidable aspect of the pilot project, which required students to choose to participate in the new assessment task to test its viability. We acknowledge the limitations of this self-select methodology, which, given the students’ enthusiasm, may have influenced results. As timetabling made it impossible to share the same class times, a regular weekly meeting time was scheduled in a computer laboratory, with the support of a technical officer familiar with online gaming and machinima production. Although the university’s information technology services were keen to see the project succeed, they were unable to run the game on the university lab computers. Laptops with the software downloaded on to them were made available, but only at specific times since they were in high demand and students couldn’t take them home. This proved an unsatisfactory strategy. We therefore provided students with copies of the game to install on their own computers and the necessary subscriptions to W o W for the duration of the project. The participants were also provided with a pack of resources explaining the project. A Facebook page was set up, a common meeting time was negotiated, and group email contact initiated. These communication options were provided for students to negotiate the stages of their machinima creation (selection of story, script, performance, and technical production). The English students were to provide a script while the Digital Communication students produced the machinima, but these roles were not rigidly enforced. Figure 1 shows a passage from one script, while Figure 2 shows a scene from the machinima. Participants were given free reign in determining the negotiating process, the production schedule, and whether or not they would proceed to a final output in the form of a publicly posted YouTube video (see Figure 3). This approach drew on a learner-centred methodology, which deploys a variety of assessment items, assessed in multiple stages, to enhance students’ potential to promote learning and give them opportunities to develop self and peer assessment skills. Although we did not formally require peer assessment as part of this first pilot study, peer engagement and self review was a key component of the process of learning during the machinima project. Barwell, Moore and Walker 771 Figure 1: Example of script development by one English student followed by the original Chaucer Towards the end of session, the non-teaching member of the research team conducted structured group interviews. The first group interview included four English Studies students and the second had five participating Digital Communication students. The assessment tasks themselves, blog posts and reflective essays, provided further evidence of student response to the project. Additionally, interviews were conducted with the Digital Communication tutor and the project’s technical support advisor. Figure 2: Tavern scene from the machinima Pardoner’s Tale The role of assessment The assessment was designed to provide a focus on the learning process, rather than a fixed content of knowledge, in order to provide an environment that adapts to development needs and social influence on learning (Schiller, 2009). It was anticipated that this approach would encourage students to engage in active learning. The machinima was not to be considered as an assessable task in and of itself, but to fit into the already predefined criteria for the assessment tasks of the individual subjects. Following the learner-centered approach participants were provided extensive background documentation on machinima making with relevant examples of machinima. 772 Australasian Journal of Educational Technology, 27(Special issue, 5), 2011 Figure 3: Uploaded final video to YouTube The students were specifically encouraged not to be too ambitious and informed that they would not be assessed on the basis of the technical merits or ‘quality’ of the machinima they produced. It was made clear to the participating students that the process of collaborative production and their reflections were more important than the finished product. The aim of the pilot project was to produce an outcome, but acknowledged the distinct possibility this would not occur in the allotted time. The students were encouraged to consider uploading their machinima (however unfinished or incomplete) to YouTube for general viewing, but this was not a requirement of participation or assessment. Ultimately the students themselves elected to set up their own YouTube account to upload their work, and continued to finish their machinima after the assessments in their home subjects had been concluded. Failing to produce a finished machinima would have demonstrated the pilot project was too ambitious. Students were given support, encouragement and assistance from teachers and technical staff, but it was ultimately their decision to upload the final work to YouTube. The Digital Communication students’ machinima project work could count as part of their major assessment that also included weekly blog tasks and reflection pieces, and in-class exercises; for the English students, they could write their essay on the issues they encountered in turning a Chaucer tale into a machinima. The web project assessment criteria for Digital Communication students included the degree of their Barwell, Moore and Walker 773 engagement, their reflection on theoretical materials (in terms of readings and lecture content), approaches and discussions encountered in the subject, aspects of design, and their demonstration of an ability to manage their project as documented in their weekly blogs. The essay assessment criteria for the English students included the quality, sophistication and perceptiveness of analysis, theoretical awareness, and familiarity with relevant secondary material. These requirements were announced in the subject outlines made available at the start of session. All students were also informed of the focus groups that were scheduled at the end of session, which would be an opportunity for reflection on their experience of the group project. Findings and discussion The value of machinima The English students reported they volunteered to participate because ‘We thought it would be fun’, while one Digital Communication student said, in consensus with her group, that ‘It was definitely the most interesting thing [she’d] done’ in her university studies. The Digital Communication teaching staff were equally invested and enthusiastic about the machinima project. When asked about the value of machinima for working with different groups of students, one teacher summed up: I think it definitely is useful in interdisciplinary settings as it forces everyone to come to a common ground and then work from there. The beauty of machinima is it doesn’t favour any one particular academic discipline... it was great to see how each individual interpreted and manipulated the setting they were given. In a similar comment, an English literature major responded to a question on the need for activities based around assessment tasks that propel the acquisition of digital literacies but within a properly rigorous pedagogical context: From the point of view of being a technophobe, and being in a learning institution, absolutely I need to be forced into learning how to use new technologies and be introduced to them gently, in a supportive environment where someone can tell me what I am doing wrong. Students from both disciplines agreed that the machinima project was an invaluable experience, to be introduced to new online technologies and possibilities in a way that was not directly assessed, but part of a range of discipline-specific assessment tasks. The English students were somewhat disappointed that they didn’t have even more opportunity to get involved in the process of making the machinima video, as the collaboration between the two students groups fractured into cooperative rather than collaborative tasks (Dillenbourg et al. 1996; Scanlon, 2000), with the English students working on adapting the Chaucer stories and the Digital Communication students working on the technical elements of the machinima production. This indicates there was an opportunity for even more cross-disciplinary work focusing on collaboration than was achieved, despite the restriction on access to the software from the university computers, class scheduling and the reliance on the Digital Communication students’ own hardware. The benefits of interdisciplinary and collaborative work The interdisciplinary project allowed students to interact not only with a group in the same class and field of study, but with another student cohort. Initially, students 774 Australasian Journal of Educational Technology, 27(Special issue, 5), 2011 reported being resistant to group projects, saying ‘They suck as a rule, generally. You know, when a group of people have to get together and present a body of work as a group, it is never ever any sort of even spread of work’. However, the students were alert to the rationale about why they were given these as assessment tasks: ‘it is practice for the real world’. This particular task, with its cross-disciplinary components and challenging technical parameters, made students even more conscious of the need to communicate and cooperate. As one student pointed out, the asynchronous aspect of machinima production as well as the need to have a strong and carefully rationalised storyline ‘kind of forces you to cooperate with each other. So if you don’t cooperate it is obviously going to fall apart’. The teacher who worked closely with the project by offering technical support during the meeting times commented that the interdisciplinary team work was an important factor in the project: It was great to see collaboration between different students who possessed different skill sets such as sound/video editing, familiarity with video games, understanding of Chaucer’s works and writing styles. The English students had to work with the Digital Communication students in the selection of an appropriate Chaucer tale, as their original selection was revealed to be inappropriate for the given online gaming environment: We had to reduce an entire tale into two pages or something of script, we had to consciously think ‘well, what’s important, what meaning do we want to bring to it’ so we definitely had to think about that. And just in that initial process that [another English student] and I had of going through it with the digital media students - we realised that The Miller’s Tale wasn’t going to work... Yeah, because we were saying ‘and then she hangs her arse out the window, and then this happens’. Although it was a far more interesting story, it didn’t work in World of Warcraft. The English students were initially uneasy about their own capacity to contribute to the technical aspects of the project, with their comparatively limited digital literacies. However, they reported gaining insight and appreciation of their own expertise with Chaucer: I got a little buzz because one of the digcom students took my copy of Chaucer and I said ‘you will have to look at the footnotes [for translations of Middle English]’. And he looked at it blankly for a few seconds and handed it straight back to me. And I thought, yeah, actually we are quite on top of this language now and we know the stories and we can go off that. The interdisciplinary aspect of the project development meant that students had to be conscious of their own disciplinary knowledge base and skills, and work strategically together in order to effectively communicate ideas across to the other student cohort, so that ‘everyone is on the same playing field’, and a successful outcome would be achieved. This meets the requirements for a successful interdisciplinary approach detailed above in the literature review. In our discussion with both focus groups, the students were most animated in their responses to questions about the collaborative work across the two student cohorts, as they commented on their good or sometimes frustrating experiences and the logistical hurdles encountered when using the Web 2.0 technologies. Barwell, Moore and Walker 775 Logistical issues As this was a pilot project, there were some logistical issues to overcome, not only to set up the technology, but also to establish communication lines between student cohorts and to encourage synchronous collaboration. Although students appreciated that WoW was chosen for its collaborative possibilities, and some were already familiar with it, they were frustrated by the institutional roadblocks in accessing suitable technology and that they couldn’t get direct net access during designated meeting times without staff help. There were also coordination challenges in a multiplayer game, so that some would have preferred a more individually controllable platform like The Sims. However, they also realised that this would tempt them to fall back on independent work rather than negotiating the more difficult collaborative engagement: ‘you would have group work on a project like that but you could control it all by yourself’. While Skiba and Barton (quoted in Williams & Chinn, 2009; p.172) have stressed the value of harnessing the digital preferences of ‘net generation learners’, they also emphasise the importance of positive responses to experiential and interactive learning. Both of these factors were engaged in the development of the machinima project that led to a meaningful interdisciplinary experience which called on their prior experience with Web 2.0 for some students, introduced new opportunities and literacies to others, and called on the collaborative and problem solving skills of all. For both cohorts of students the biggest problems were coordination, communication and participation in the project development. As one student commented, ‘The only problem with this project, and I will say this frankly, was communicating with everyone, and everyone actually pulling their weight’. Because only small groups of students from both classes, with very different timetables, were involved, it was decided that they would work together outside normal class hours with a range of online options. However, in practice, the students avoided using email or Facebook, or only used it erratically. One student agreed with the principle that working online would have been less frustrating overall if students had committed to it more fully, particularly as the project itself was about filming a video in synchronous time in an online environment, and wished that they had all met ‘in game’. Despite the issues with working out the best ways to communicate, students eventually negotiated ways to work together as a group. As one English student reported: It was just like, what do you call it, modes of communication. Like maybe that could have been improved, but when we did meet up we actually were working or collaborating very well, and were learning off each other. Digital literacies and learning outcomes The two interviewed teachers had slightly different perspectives on the students’ capacities. While one reflected on his exposure as a student to very limited educational technology, the comments of the other teacher touch more directly on the assessment strategy we had adopted. While she valued the challenge offered by innovative and collaborative projects that called on students’ engagement with Web 2.0, she pointed out that many of the Digital Communication students had chosen to study this discipline not because they were 776 Australasian Journal of Educational Technology, 27(Special issue, 5), 2011 already experts, but because they felt that they didn’t know enough and wanted to learn. She commented: I think that students’ digital literacies are greatly over exaggerated by commentators. Most of the students I see, even in Digital Communication, say ‘I have no idea how to make a blog, I have no idea how to do this’. So they take the class because they don’t know... Students in my experience are a lot less digitally literate that you might assume. This teacher pointed out that the machinima project drew on a very wide range of technical and communication skills, as even navigating WoW calls on a range of skills that are not necessarily apparent at first glance, so that “interacting in virtual worlds comes very naturally to some people and very painfully for others”. The project required students to appreciate not just how individual Web 2.0 technologies worked, but to find the best ways to use them to support the wider goals of their home disciplines. Both teachers thought that participating students had developed their digital literacies, particularly as the frustrations of negotiating the collaborative project drew attention to the value of communication in both offline and online environments: At the very least they developed an appreciation for how difficult it is to effectively communicate using digital media. There is a lot of work that needs to go into such projects and now that they have experienced it, I believe that they have developed new problem-solving and technical skills as a result. While the teachers had varying views about their students’ digital literacies, and the students themselves were self-conscious of their own presumed level of technical expertise, the machinima project worked from the assumption that they were all capable of learning and of developing some level of proficiency in Web 2.0 tools in order to communicate effectively about project development. However, if all the participating students had been expected to have a full and commensurate prior understanding of the use of Web 2.0 technologies in order to meet pre-determined learning objectives, the machinima project would have side-stepped a key value of the experiment, which included ‘the informal processes of conversational brainstorming, meaning making and pursuing ideas’ (Wenger, 2006; p.31). Negotiating levels of online and offline interaction was part of this process. Effectiveness of the assessment strategy The assessment tasks were presented, structured and marked differently for the two different student cohorts. As one teacher pointed out, the students were given open- ended opportunities to negotiate what steps they needed to complete to get to the final machinima product and to negotiate their assessment requirements. The interdisciplinary collaboration between the two student groups is the key point where a clearer assessment strategy needs to be developed for future iterations of the project. The discussion in the group interviews revealed some early frustrations as each cohort of students waited for the other to take the initiative in leading the project. There was a perception from the Digital Communication students that the English students were less committed because they would be assessed at the end of the session, which explained their initial slowness in developing a script for them to work from: It just felt like for the English students the project meant less to them that it did to us, so they were more lax about it and were like, oh yeah, whenever we get around to it. Barwell, Moore and Walker 777 On the other hand, the English students were surprised that the Digital Communication students weren’t working collaboratively with them at the start of the project to develop the script: The media students were very dependent on us to set the pace in a way. Like, I was surprised that they didn’t really have much input. At times, particularly in the early stages of the project development, it seemed that both groups devalued the other’s contribution: the Digital Communication students were reliant on the English students for the script but considered themselves the main actors and producers of the machinima task, while the English students were hesitant to get involved in the making of the machinima video, event though they were interested in the performance and technical aspects of creating the video: ‘That was kind of, that was their expertise. That was the only thing they could do, so I guess we couldn’t really do that, take over that aspect of the project’. Another English student highlighted the value of the collaborative project, focusing on the negotiation between the interdisciplinary teams and its impact on the learning process. She thought: That we were responsible for our own learning in a way, like we had to because there weren’t any teachers guiding the process and because we had to interact with people who had no idea what Chaucer was about. We actually had to know the subject very well in order to explain to them what we were doing. So it kind of forced us to read the text deeply - more than we might have just in class. It was notable that the although some Digital Communication students had already been marked for their participation in the project at the time of the interviews, they continued on to finish the machinima video in consultation with other students, and were keen to participate in the group discussion. The final product, the machinima video, was understood to be the least significant learning outcome of the project, as was reflected in the minimal attention paid to it in the marking criteria and assessment instructions. Therefore the willingness of students to persist in this project was closely tied to their willingness to adapt their own expectations about the teaching and learning aspect of this task. As Chapman (2003) argues, there are two interpretations of the term, ‘student engagement’: the first definition focuses on a student’s willingness to participate (which she aligns to procedural compliance to instructions or assessment imperatives) while the second considers students’ use of meta-cognitive and self- regulatory strategies to monitor and guide their own learning processes. The self- selecting students from both English and Digital Communication cohorts enacted this second level of engagement in their work with the machinima project, particularly those Communication students who persevered with the task beyond the end of the marking stage to produce the YouTube video, or the Chaucer students who stayed for the project duration even if they decided to write an essay on another topic. Conclusion This pilot project worked from the perspective, highlighted by Prensky (2010), that it is not the educator’s responsibility to be expert in the technologies or to instill excellence in production in order to support student learning. This project responded to Web 2.0 demands on flexibility in approach and in assessment criteria that are mindful of motivation and creative experimentation, as students adapt to available technologies 778 Australasian Journal of Educational Technology, 27(Special issue, 5), 2011 and their collaborators, contribute in ways that demonstrate their original disciplinary understanding, and also critically reflect on their own and others’ performance. This shift in an assessment emphasis from product to process, we argue, fits better with the increasingly digitalised, unfixed and interdisciplinary world the students will inevitably enter. While we recognise that our observations about assessment of Web 2.0 work are limited in being based on a highly specific pedagogical experiment with machinima, we did see clear evidence of the development of learning capability and engagement. Though we were constrained by institutional factors in determining the assessment, we were convinced that the reflective nature of the tasks enabled the students to demonstrate the nature of their learning and the educators to assess how fully the participants had engaged in a complex interdisciplinary project. Students incorporated the technology and collaborative practice in their learning, which involved blending traditional and digital literacies as well as individually reflecting on the demands of collaborative learning activities. Using Web 2.0 technologies in innovative ways in higher education can lead to positive outcomes despite the inevitable logistical difficulties and, arguably, without formalising them explicitly as ‘assessment’ tasks. Our experience in running the trial project left us convinced that Wenger’s model of learning is more productive in a Web. 2.0 environment than one focusing on end product and mastery of tools. For interdisciplinary projects, we think it vital that the assessment be designed to test the development of learning engagement using versions of existing assessment tasks. That is, by incorporating Web 2.0 technologies as part of existing tasks where the assessment is not focused on technological mastery, but is instead connected to already established learning outcomes, the emphasis would be more squarely on the social practices inherent to Web 2.0 (Dohn, 2009) rather than on the technologies themselves. In a second iteration of the project we plan to include an early task for both cohorts, where they would reflect on their own and their peers’ contributions. This would follow a first formal meeting of the cohorts to negotiate how the project would develop and to avoid inappropriate disciplinary division but still ground student participation in line with established criteria of the different subjects. Ideally the two cohorts would be able to do project work both in and outside class time. In disciplines which focus on individual effort, the challenges of interdisciplinary collaboration and assessment will make demands on students and educators, but these can be addressed successfully with careful planning and where the participants are both enthusiastic and thoughtful. References Biggs, J. (2007). Teaching for quality learning at university: What the student does. (3rd ed). Maidenhead: McGraw-Hill/Society for Research into Higher Education and Open University Press. Boud, D. J. & Falchikov, N. (2006). Aligning assessment with long-term learning. Assessment and Evaluation in Higher Education, 31(4), 399-413. http://dx.doi.org/10.1080/02602930600679050 Brown, K. (2004). Technology: Building interaction. TechTrends, 48(5), 36-38. http://dx.doi.org/10.1007/BF02763528 Chapman, E. (2003). Alternative approaches to assessing student engagement rates. Practical Assessment, Research and Evaluation, 8(13). http://pareonline.net/getvn.asp?v=8&n=13 Barwell, Moore and Walker 779 Creamer, E. (2004). Collaborators’ attitudes about differences of opinion. Journal of Higher Education, 75(5), 556-571. http://dx.doi.org/10.1353/jhe.2004.0028 Dillenbourg, P., Baker, M., Blaye, A. & O’Malley, C. (1996). The evolution of research on collaborative writing. In E. Spada & P. Reiman (Eds), Learning in humans and machines: Towards an interdisciplinary learning science. Oxford, Elsevier, 189-211. Dohn, N. (2009). Web 2.0: Inherent tensions and evident challenges for education. International Journal of Computer-Supported Collaborative Learning, 4(3), 343-363. http://dx.doi.org/10.1007/s11412-009-9066-8 Drexler, W. (2010). The networked student model for construction of personal learning environments: Balancing teacher control and student autonomy. Australasian Journal of Educational Technology, 26(3), 369-385. http://ascilite.org.au/ajet/ajet26/drexler.html Elliot, B. (2008). Online collaborative assessment: Assessing student online collaboration. [viewed 15 May 2011, verified 13 Aug 2011]. http://www.scribd.com/doc/9375123/Online- Collaborative-Assessment Eschenbrenner, B., Nah, F. F. & Siau, K. (2008). 3D virtual worlds in education: Applications, benefits, issues, and opportunities. Journal of Database Management, 19(4), 91-110. http://dx.doi.org/10.4018/jdm.2008100106 Gray, K., Waycott, J., Thompson, C., Clerehan, R., Sheard, J., Hamilton, M. & Richardson, J. (2010). Using social web (Web 2.0) activities for student assessment: Resources for university learning and teaching. https://web2assessmentresources.wikispaces.com/ [viewed 23 Feb 2011, verified 14 Aug 2011]. Jarmon, L., Traphagan, T., Mayrath. M. & Trivedi, A. (2009). Virtual world teaching, experiential learning, and assessment: An interdisciplinary communication course in Second Life. Computers & Education, 53(1), 169-182. http://dx.doi.org/10.1016/j.compedu.2009.01.010 Jonassen, D. (2003). Using cognitive tools to represent problems. Journal of Research on Technology in Education, 35(3), 362-381. Judd, T., Kennedy, G. & Cropper, S. (2010). Using wikis for collaborative learning: Assessing collaboration through contribution. Australasian Journal of Educational Technology, 26(3), 341- 354. http://www.ascilite.org.au/ajet/ajet26/judd.html Kruck, S. E. & Teer, F. P. (2009). Interdisciplinary student team projects: A case study. Journal of Information Systems Education, 20(3), 325-329. [abstract only] http://jise.org/Issues/20/20N3P325-abs.pdf Lee, M. J. W. & McLoughlin, C. (2007). Teaching and learning in the Web 2.0 era: Empowering students through learner-generated content. International Journal of Instructional Technology and Distance Learning, 4(10), 21-34. http://www.itdl.org/Journal/Oct_07/article02.htm Mansilla, V. B., Duraisingh, E. D., Wolfe, C. R. & Haynes, C. (2009). Targeted assessment rubric: An empirically grounded rubric for interdisciplinary writing. Journal of Higher Education, 80(3), 334-352. http://dx.doi.org/10.1353/jhe.0.0044 Middleton, J. & Mather, R. (2008). Machinima interventions: Innovative approaches to immersive virtual world curriculum integration. ALT-J, 16(3), 207-220. http://repository.alt.ac.uk/id/eprint/812 Morozov, A. (2008). Machinima learning: Prospects for teaching and learning digital literacy skills through virtual filmmaking. Proceedings of the World Conference on Educational Multimedia, Hypermedia and Telecommunications. Chesapeake, VA: AACE. 5898-5907. http://www.editlib.org/p/29201 Napier, N. & Johnson, R. D. (2007). Technical projects: Understanding teamwork satisfaction in an introductory IS course. Journal of Information Systems Education, 18(1), 39-48. [abstract only] http://jise.org/Issues/18/V18N1P039-Abs.pdf 780 Australasian Journal of Educational Technology, 27(Special issue, 5), 2011 Ng, H. Z. & Hussain, R. M. R. (2009). Empowering learners as the owners of feedback while YouTube-ing. Interactive Technology and Smart Education, 6(4), 274-285. http://dx.doi.org/10.1108/17415650911009254 Pirius, L. K. & Creel, G. (2010). Reflections on play, pedagogy, and World of Warcraft. EDUCAUSE Quarterly, 33(3). http://www.educause.edu/EDUCAUSE+Quarterly/ EDUCAUSEQuarterlyMagazineVolum/ReflectionsonPlayPedagogyandWo/213663 Prensky, M. (2001). Digital natives, digital immigrants. On The Horizon, 9(5). [viewed 10 Dec 2010]. http://www.marcprensky.com/writing/Prensky%20- %20Digital%20Natives,%20Digital%20Immigrants%20-%20Part1.pdf Prensky, M. (2010). Teaching digital natives: Partnering for real learning. Thousand Oaks, California: Corwin Press. Scanlon, E. (2000). How gender influences learners working collaboratively with science simulations. Learning and Instruction, 10(6), 463-481. http://dx.doi.org/10.1016/S0959- 4752(00)00009-8 Schiller, S. Z. (2009). Practicing learner-centred teaching: Pedagogical design and assessment of a Second Life project. Journal of Information Systems Education, 20(3), 369-381. [abstract only] http://jise.org/Issues/20/20N3P369-abs.pdf Slavin, R. E. (1996). Research on cooperative learning and achievement: What we know, what we need to know. Contemporary Educational Psychology, 21(1), 43-69. Weimer, M. (2002). Learner-centred teaching: Five key changes to practice. San Francisco: Jossey-Bass. Weingart, P. & Stehr, N. (2000). Practicing interdisciplinarity. Toronto, Ontario: University of Toronto Press. Wenger, E. (2006). Learning for a small planet: A research agenda. http://ewenger.com/research/LSPfoundingdoc.doc [viewed 12 Jan. 2011]. Whitton, N. & Hollins, P. ( 2008). Collaborative virtual gaming worlds in higher education. ALT-J, 16(3), 221-229. http://dx.doi.org/10.1080/09687760802526756 Williams, J. & Chinn, S. J. (2009). Using Web 2.0 to support the active learning experience. Journal of Information Systems Education, 20(2), 165-174. [abstract only] http://jise.org/Issues/20/V20N2P165-abs.pdf Authors: Graham Barwell, Faculty of Arts University of Wollongong, Wollongong, NSW 2522, Australia Email: gbarwell@uow.edu.au Chris Moore, School of Communication and Creative Arts Deakin University, 221 Burwood Highway, Burwood, Vic. 3125, Australia Email: c.moore@deakin.edu.au Ruth Walker, Learning Development University of Wollongong, Wollongong, NSW 2522, Australia Email: rwalker@uow.edu.au Please cite as: Barwell, G., Moore, C. & Walker, R. (2011). Marking machinima: A case study in assessing student use of a Web 2.0 technology. In J. Waycott & J. Sheard (Eds), Assessing students' Web 2.0 activities in higher education. Australasian Journal of Educational Technology, 27(Special issue, 5), 765-780. http://www.ascilite.org.au/ajet/ajet27/barwell.html