R-NEST: Design-Based Research forTechnology-Enhanced Reflective Practice in Post-Secondary, Professional Education Australasian Journal of Educational Technology, 2015, 31(5). 572 R-NEST: Design-based research for technology-enhanced reflective practice in initial teacher education Bonnie Thompson Long and Tony Hall School of Education, National University of Ireland, Galway This paper reports research into developing digital storytelling (DST) to enhance reflection within a specific professional learning context – that of a programme of teacher education - while concomitantly producing a transferrable design framework for adaption into other, similar post-secondary educational contexts. There has been limited substantive, evaluative design-based research investigating empirically the potential of digital storytelling for reflection in professional, post-secondary education. Consequently, there has also been a lack of robust and reusable models to guide and inform design-based research in this context. This paper illustrates the development of a repeated study, undertaken on a longitudinal basis, over 3 years, and on a large scale, involving 323 pre-service teachers. The design-based research developed at the three key stages along the triadic spectrum of maturity: from (1) analysis and exploration, through (2) design and construction, to (3) evaluation and reflection (Kopcha, Schmidt, & McKenney, 2015).The innovation reported here is now a mature intervention, constituting a core part of the professional educational formation of pre-service teachers within a two-year, graduate teacher education programme. Further, the R-NEST design framework, which emerged from this longitudinal design-based research, enumerates key criteria and principles for designing, implementing and evaluating DST to enhance reflective practice in post-secondary professional education. Introduction Reflective practice constitutes a key developmental dimension of teacher professional education internationally (Organisation for Economic Co-operation and Development, 2005). However, it can prove a very challenging and problematic process for pre-service teachers (Calderhead, 1989; Hatton & Smith, 1995; Korthagen, 2001; MacLeod & Cowieson, 2001; Moon, 1999). There exists significant potential in combining storytelling with technology, using creative, narrative media to enhance formative processes of reflection and learning from practice (Barrett, 2005a; Moon, 1999). The specific educational problem or challenge that this research addresses is to explore and examine critically if and how these innovative media, specifically digital storytelling (DST), can be deployed and developed to ameliorate the difficulties that teachers can experience in trying to reflect meaningfully on their learning from their teaching practice, and in the overall development of their professional identity as educators. Further, we hoped to be able to engage in rigorous, critically informed design-based research (DBR), whereby we might work towards the development of a robust and reusable solution. We scaled our innovation to involve a large population of learners, and at least three significant, iterative cycles of design, deployment, and evaluation. There exists a dearth of systematic, longitudinal, and large-scale design research in this context, and consequently a lack of criteria, frameworks, and models to guide and inform educational designers and technologists. The primary aim of this repeated study has been to help to address these two lacunae. Conceptualising a prototype design model “Digital Storytelling is a modern expression of the ancient art of storytelling. Digital stories derive their power by weaving images, music, narrative and voice together, thereby giving deep dimension and vivid color to characters, situations, experiences, and insights” (Rule, Digital Storytelling Association, as cited in Barrett, 2005a). It is possible to construe and define DST in myriad ways. In this research, however, one particular genre of DST was employed consistently over the lifetime of the multi-cycle, repeated 3- year design study: a short, 3–5–minute video, produced by someone who is not a media professional, and normally constructed as a thought piece on a personal experience (Matthews-DeNatale, 2008). This was inspired by the model for DST originally developed by the Center for Digital Storytelling (Lambert, 2009). The creation of the digital story includes incorporating multimedia components such as still images, music, video, and a narration, which is usually the author’s own voice (Dogan & Robin, 2008). While the digital story is created using digital technologies, the story itself is the most important element Australasian Journal of Educational Technology, 2015, 31(5). 573 in the DST process (Lambert, 2009; Matthews-DeNatale, 2008). Our particular focus in this research was to use technologies that are highly usable, and not tools that are proprietary, expensive, or difficult to learn, and thus less likely to be used by teachers. Gilbert (2002) classified these kinds of information and communications technologies (ICTs) as low threshold applications. The growing mobility and ubiquity of computer applications and devices that are “eminently easy to use” (Hall, 2012, p. 111) can enable learners to create, produce and share rich multimedia artefacts and projects. This research has been inspired by leading researchers in reflection and DST, principally Barrett (2005a, 2005b, 2005c) and Moon (1999, 2004), and their ideas about creative and novel representations of reflection, including the potential of combining storytelling with digital technology. Furthermore, the study reported here was motivated strongly by the researchers’ own professional-biographical background as teacher educators, education technologists, and design researchers. As teacher educators, we have had several years’ first-hand experience of the difficulties and challenges faced by our students in engaging meaningfully in reflection, particularly through exclusively written formats. Reeves, Herrington, and Oliver (2005) suggested that design research is the most appropriate method for studying instructional technology in higher education. Hofer and Owings Swan (2006) pointed to the potential of DBR as a systematic and impactful approach, specifically germane to the development and use of DST. They stated: It will be important for researchers and practitioners to [be] systematic in developing an efficient, effective instructional model to implement digital storytelling projects in a range of settings for a range of purposes…A Design-Based Research approach, in which iterative, formative evaluation helps to both improve an instructional approach and develop theory of teaching and learning, may be particularly effective in assisting this effort. (p. 4) A further core dynamic of the initial design process was the multi-ontological framework or theorising that accompanied the early stages of DBR, in scoping out the initial, prototype R-NEST (reflection, narrative, engagement, sociality, and technology) design model. Figure 1 illustrates the provenance of the nascent R-NEST framework, which emerged in the synthesis of four main activities on the part of the researchers. Figure 1. Emergence of the nascent R-NEST framework Given the practicalities of a journal paper, it is not possible to enumerate all aspects of the very extensive review and ideation work that occurred during the early stages of the research. For a complete discussion of the design study, the reader is directed to the monograph (Thompson Long, 2014) in which all steps in the three-year cyclical process are outlined in detail. Our conceptualisation of reflection in the early stages of the research exemplified the extensiveness of the initial analysis and exploration. This aspect of the process was informed by a comprehensive review of key researchers and theorists, synthesising the seminal early thinking about reflection, principally Dewey (1910, 1916, 1933); the critical work of Lortie (1975), Schön (1983, 1987), Moon (1999), and Korthagen and Wubbels (2001); and research on the use of storytelling and multimodal, digital methods to support Australasian Journal of Educational Technology, 2015, 31(5). 574 teachers’ reflective writing, for example, Moon (2004), Lambert (2009), and Kajder and Parkes (2012). In addition to reflection, our multi-ontological framework encompassed the other four key components of the R-NEST design model: narrative, engagement, sociality, and technology. In this conceptualisation and review, we engaged with key, relevant literature, including, among others, Bruner (2002) for narrative; Csikszentmihalyi (1991) for engagement; McDrury and Alterio (2002) for collaborative storytelling in higher education; and Norman (1998) and Gilbert (2002) for usability and technology. A prototype R-NEST framework emerged through the confluence of design analysis and exploration activity. The salient, interrelated themes of the initial R-NEST model included: • narrative design and the potentially important role of storytelling as a medium for identity development in teacher education; • the central importance of collaborative learning among pre-service teachers, especially in relation to personal stories of change (Lambert, 2009) and reflection thereon; • easy-to-use technology and easy-to-access and use, rich media content; and • creative engagement in the process. R-NEST then guided and was shaped by the iterative design and evaluation of the three interventional, sequential cycles of DBR. Implementing the R-NEST model Informed by the nascent R-NEST framework, the overarching approach for the design and construction of the iterative and cyclical DBR process was to begin with a small-scale, exploratory pilot intervention; then scale up to a mainstream intervention should the pilot prove promising/successful; and finally, implement a capstone intervention to help verify the process overall and the emergent R-NEST design model. Figure 2. Repeated, R-NEST multi-cycle DBR A central aim of the R-NEST intervention – as it was implemented, evaluated and redesigned over the three design cycles – was to try to achieve the highly integrative dynamic of design and research activities and outputs in McKenney and Reeves’ (2012) generic model for educational design research. The generic model (McKenney & Reeves, 2012) “explicitly depicts an integrated cycle of research and design activities and outputs, which interacts both directly and indirectly with practice” (McKenney & Visscher- Voerman, 2013, p. 14). Each R-NEST iteration – pilot, mainstream and capstone – was characterised by three interconnected phases of (1) analysis and exploration; (2) design and construction; and (3) evaluation and reflection. Situating R-NEST in the context of McKenney and Reeves’ (2012) integrative model, it was hoped the reciprocity of design and research activities would contribute both to the efficacy of the local intervention with pre-service teachers and the conceptualisation of a transferable, theoretical understanding of DST for reflective practice. If the pilot deployment showed potential, R-NEST would be further developed and scaled up, and the ensuing design iterations would potentially expand the impact, implementation and spread of the DST intervention. The McKenney and Reeves’ (2012) model “depicts two main outputs from an intertwined empirical and regulative cycle, a practical one and a theoretical one” (McKenney & Visscher-Voerman, 2013, p. 14). The proximal, practical output is the “designed intervention, which may be a process, a product, or (most Australasian Journal of Educational Technology, 2015, 31(5). 575 often) a combination of the two”, and the distal, theoretical output is “an empirically founded set of design heuristics which can inform similar endeavors” (McKenney & Visscher-Voerman, 2013, p. 14). The envisaged proximal output of R-NEST would be twofold, both process and product. First, at the level of the local teacher education programme, the targeted process output of a maturing R-NEST intervention would be a situated design for DST, effectively integrated as part of the teacher education curriculum at our university. Second, the product goal would be that R-NEST would enable our pre-service teachers to create high quality educational design artefacts in the form of digital stories that critically reflected their learning from practice. The aimed-for distal output of R-NEST would be a transferrable ontology of orienting criteria and principles that could be adopted and adapted to “inform similar endeavors” (McKenney & Visscher-Voerman, 2013, p. 14), and potentially enhance reflective practice in other cognate professional education contexts and disciplines. Design cycle 1 – pilot Context The DST innovation was introduced in 2010 at our university, within the state-approved, one-year teacher education programme in Ireland, the Professional Diploma in Education (PDE). A different cohort of students engaged with the intervention in each of the 3 years of the repeated DBR. The piloting of the innovation involved 18 self-selected pre-service teachers who participated in the first year of the repeated study on a voluntary basis. The country’s graduate teacher education programme has now been extended and reconceptualised as a two-year master’s degree, the Professional Master of Education (PME), principally to facilitate a more extensive preparation for teaching, including more substantial teaching practice placement in schools. The DST design is now a mature part of graduate teacher education at our university, although it is important to note that the research reported here relates only to the introduction and development of R-NEST within the one-year PDE programme. Using R-NEST, we have planned future research on the effects of DST within the reconceptualised two-year programme. The setting of the DST intervention was a graduate course that leads to a consecutive teacher education qualification. This means that graduates of different disciplines can apply for entry to the programme once they have a third-level or university degree with sufficient, relevant study credits in a recognised area of Ireland’s secondary school curriculum (e.g., English, mathematics, history, biology, economics). Entry to the programme is competitive and based on a centrally administered points system, calculated on the basis of one’s performance in one’s degree, and also if one has additional graduate qualifications or documented relevant experience cognate to education and teaching. The programme is a popular choice for college graduates; the numbers of students taking the PDE each year during the course of this research exceeded 200. The three foundational parts of the programme are (a) educational sciences and the theoretical disciplines within education, for example, sociology and psychology of education; (b) professional studies, which includes educational technology, curriculum design, and reflective practice; and (c) subject methodology, wherein the pre-service teachers develop their skills in pedagogical content knowledge (Schulman, 1986) and how to teach their respective subject(s) in the classroom. The graduate programme caters to a broad range of students who want to become teachers, and the student population each year is very diverse, including both immediate college graduates and those returning to study after an extended period or a career change. Students come from a wide array of disciplines and backgrounds, and the programme prepares teachers across a broad set of school subjects, including the biological and physical sciences, languages, ICTs, the arts, and humanities. Students select at least one school subject to specialise in, which is determined by their degree studies in college. The majority of students are eligible to specialise in the teaching of two subjects on the Irish school curriculum. Popular teaching subject combinations for graduates are English and geography, and maths and science. The purpose of the initial phase of the R-NEST research with the 18 students involved in the pilot intervention was to examine the potential of DST to enhance reflection in teacher education, and whether the innovation merited further development for the pre-service teaching group as a whole. Jakes and Australasian Journal of Educational Technology, 2015, 31(5). 576 Brennan (2005) suggested to “start small” when implementing DST for the first time, noting that the process of DST is both support intensive and system intensive, and “requires a great deal of student support as well as a stable technology infrastructure” (p. 5). The DST pilot project with the first cohort of students took place in February and March, 2010. Five 1- hour lessons on the different components of the DST process were devised, based on salient DST research literature (Banaszewski, 2005; Lambert, 2009; Matthews-DeNatale, 2008; Ohler, 2008; Porter, 2004b) that had been reviewed and synthesised as part of the initial conceptualisation of the R-NEST framework. In designing and developing their digital stories, the pre-service teachers were asked to: • reflect on your educational journey to this point and your decision to become a teacher; • re-evaluate learning goals and learning philosophies at the start of your teacher education, evaluate achievement of these learning goals; • trace any transformations in your learning and teaching beliefs, values, attitudes, and assumptions, how these changes have come about; • relate what/why/how different elements of the teacher education programme contributed to your learning and teaching; • highlight significant landmark achievements/improvements you have made to your learning and teaching (please draw from your journal, lesson plans [lesson designs for teaching practice] and evaluations [reflections on teaching, both individual and collaborative – with their mentor/tutor]. Main instruments An assessment rubric was created to evaluate the completed digital stories. The rubric was based on other DST rubrics reviewed in the literature, specifically schemes created by Barrett (2005a) and the personal expression analytical student scoring guide (Porter, 2004a), the Western Massachusetts Writing Project (Hodgson, 2010), and the guidelines from Integrating digital storytelling into your classroom (2006). Results were examined to explore the effects of DST on students’ reflection on practice. The R-NEST framework was utilised to frame the analysis of findings in which both confirming/positive and disconfirming/negative evidence were sought. The evaluation of the DST innovation evolved concurrently with the design over the 3 years, and included artefacts such as students’ completed digital stories, their working portfolios, online discussion boards, a post-DST questionnaire, and qualitative feedback. The data derived from these evaluations were the subject of critical analysis, informed by R- NEST. Main findings Significant corroborating/positive data emerged from the pilot project. Analysis of students’ working portfolios (a collection of their planning materials for their digital stories) and their responses on the post- DST questionnaire revealed that students spent much more time engaging in the process of reflection than they would have done simply writing an essay on the topic. Engagement can be evidenced by the amount of time and effort spent on a task (Jones, 2003; Sandholtz, Ringstaff, & Dwyer, 1994). Students were asked on the post-DST questionnaire to estimate the amount of time they spent on the different tasks associated with the process. Figure 3 shows the results of student answers to this question. Australasian Journal of Educational Technology, 2015, 31(5). 577 Figure 3. Hours spent by pilot project students on the different tasks involved in creating a digital story The amount of time and effort invested in the creation of the digital stories, an average of 30 hours, shows a high degree of engagement in the DST process. In addition, several comments made by the students in the questionnaire show high levels of engagement in the process of creating their digital story. Csikszentmihalyi (1991) described high levels of engagement in an activity as being in a state of flow. A state of flow occurs when the difficulty of the task is matched well with our ability to tackle it. If the task is too hard, we may feel anxious. If it is too easy, we can become bored. When in a state of flow, concentration is intense, self-consciousness disappears and time becomes distorted (Csikszentmihalyi, 1991). Student comments evidence engagement in the DST process, even where they encountered challenge or frustration in developing their digital story: Loved it. It took me a long time but I really enjoyed it and I didn't dread it the way I do assignments. (Student 10) You become engrossed in every little detail and become a perfectionist. It ended up taking a lot longer than planned but it was worth it. (Student 11) I have never cursed so much at my computer as I have in the last few days but now that it is over I am so glad that I did it because I have learned so much from it. I really hope next year’s students have the chance to do it. (Student 2) Knowing that the process could be difficult, and possibly off-putting for students, they were asked what they thought of the DST process overall. These answers were then coded as a positive, negative, or neutral response. The results of this question show that students generally enjoyed the process, with 11 out of 12 answering positively. Other responses to the questionnaire reveal that all of the students found DST a motivating and worthwhile experience, and recorded high technology self-efficacy at the end of developing their project. An analytic assessment rubric was created to assess the digital stories for depth of reflection, based on Moon’s (2004) generic framework for reflective writing. The scale ranged from the lowest level of descriptive to the highest level of reflective (2) (see Moon, 2004, pp. 214–216). Although the majority of students who completed a digital story during the pilot project felt that the use of DST significantly 0 2 4 6 8 10 12 Using the video editing software Sourcing images Sourcing sound or music Recording the voice-over Creating a storyboard Writing the script Creating a story map Brainstorming Number of questionnaire respondents D ST T as k Hours spent on DST creation 0 hours 1-2 hours 3-4 hours 5-6 hours 7-8 hours 9-10 hours 10+ hours (N = 12) Australasian Journal of Educational Technology, 2015, 31(5). 578 enhanced reflection on their practice, the completed DST projects did not evidence this, as shown in Figure 4. Figure 4. Pilot project depth-of-reflection rubric assessment results It was observed in the analysis that the students who scored the highest for levels of reflection were those who diverged most from the DST assignment brief. It was evident that the task that had been set for the students did not sufficiently match the desired outcome of deep reflection. The formal assignment brief asked for a summing-up of the year, looking back at original goals and teaching philosophies, and assessing whether or not these goals were met. While this allowed for some reflection, the task required a broad, retrospective assessment of the year. If we wanted students to produce something that was deeply reflective, we needed to set a task that would allow them to delve more deeply into an experience; into the thoughts, feelings, and motivations, that led to and emerged from that experience. We needed to provide students with a task that more closely aligned to key reflective activities suggested by Moon (2004), including linking reflection to an action or application, emphasising second-order reflection, and collaboration and reflecting with others. Changes based on findings Importantly, for the DBR process as a whole, the pilot demonstrated and established that there was potential with DST as a technology-enhanced learning process for supporting reflective practice in initial teacher education. However, results from the pilot also point to important changes that would need to be made if we were to enhance and extend the impact of DST on pre-service teachers’ engagement in, and understanding of, reflective practice. Our evaluation using R-NEST and the depth-of-reflection rubric reveal that the reflections generally were still too descriptive and superficial, rather than more deeply analytical and reflective. Although the pilot had established the potential of the DST innovation, significant educational design work remained. It is not possible within the constraints of this paper to describe fully all the changes that were made for design cycle 2. However, the most salient redesigns to R-NEST are summarised below. Reflection Emerging from the analysis of the pilot intervention, it became evident that students needed a task that would allow them to delve more deeply into an experience and explore their own motivations, feelings, and emotions. Reviewing the other reflective aspects of the PDE programme in the context of R-NEST, we realised that the Critical Incident analysis section of the professional practice portfolio might lend itself very well to supporting more deeply reflective digital stories by the pre-service teachers. This critical incident analysis required students to select a formative occurrence or episode from their reflective journal (which they kept during the year) and tell a story about the incident that took place. 0 2 4 6 8 Descriptive Descriptive with some reflection Reflective (1) Reflective (2) 0 8 5 3 N um be r o f s tu de nt D ST s Levels of reflection Depth of reflection results (N = 16) Australasian Journal of Educational Technology, 2015, 31(5). 579 They were to discuss why this was a defining moment for them and to reflect critically on the incident, discussing emotions, feelings, and reactions related to it. Students were also asked to draw on pertinent academic literature, relevant to understanding the subject of the incident. A new R-NEST assignment brief and rubric were thus developed, predicated on critical incident theory (Tripp, 1993). Process and product While assessing the digital stories for levels of reflection, the researchers also found that the depth of reflection evident in the final digital stories was not of paramount, preeminent importance. This direction in our thinking resulted from the fact that most of the students who took the survey felt that the DST process had enhanced their reflection on practice, even though many of them did not create digital stories that evidenced deep reflection. This finding prompted our return to the relevant research literature where it is noted (see Gravestock & Jenkins, 2009; Sandars, Murray, & Pellow, 2008) that reflection can take place at all stages of the creation of a digital story. These authors placed the emphasis on the process and not necessarily on the product. We, therefore, felt that more information was needed from the students regarding what they thought they gained from the process of creating their digital stories. The possibility of adding questions dealing with this to the survey was considered. But as these anonymous data could not be linked back to the students and the DSTs they created, another form of feedback was sought. Gravestock and Jenkins (2009) suggested the use of additional evidence outlining the steps taken to create the digital story to show evidence of deeper learning, “as it may be possible for a student to engage in quite high levels of learning and reflection, but for this not to be manifest within the final digital story” (p. 269). It was decided, therefore, to ask students in the second iteration of the design to include, as part of their DST working portfolio, an 800–1000-word reflective feedback piece that explained both the process of making their digital story and how they felt about the product. Narrative – story structure In the creation of their DSTs, many students did not follow the story structure suggested to them in the form of a story map, nor did many follow the format of a personal narrative. It was thought that there were two main reasons for this: (a) the essay criteria ultimately used to assess the digital stories led students to revert to an essay-type format and (b) not enough time was spent in the initial DST lessons on story structure. The criteria for personal narrative, and story structure in general, were perhaps addressed too superficially in the introductory DST lesson. It was felt that more time needed to be devoted to teaching this aspect in the second iteration of the DST study if students were to create actual digital stories. Sociality and peer feedback: the story circle During the pilot project, many students were unprepared for and unenthusiastic about the collaborative story circle stage of the DST process. This resulted from the more informal, unstructured design of the story circle in the pilot study. The researchers felt the students did not get as much out of the story circle process as they could have. A return to the literature on story sharing for learning purposes was undertaken and potential solutions were found, particularly in McDrury and Alterio’s (2002) Learning through Storytelling in Higher Education. They emphasised the importance of providing students with opportunities to share their practice stories as this “encourages a reflective process, especially when storytelling is accompanied by dialogue and occurs in formalised settings” (p. 111). Further, McDrury and Alterio suggested a formal, structured story-sharing session, where tellers share a pre-determined story and listeners engage tellers in reflective dialogue. This is in contrast to informal story sharing where significant learning may be more limited, as occurred with the story circle in the pilot. Design cycle 2 – mainstream Context In the second design cycle, the research focus became how the pilot DST innovation could be scaled up and mainstreamed for the benefit of the student cohort as a whole. Design cycle 1 involved students only Australasian Journal of Educational Technology, 2015, 31(5). 580 who self-selected into the pilot and who were enthusiastic about the process and the teacher education programme in general. A related, key concern of the second design cycle was to explore whether DST had potential wider impact among pre-service teachers, including those who may be innovation averse (Baylor & Ritchie, 2002), and with that, less enthusiastic about educational technology and reflective practice. Of the 208 students who submitted a DST during the second design cycle, 143 gave permission for their materials to be used in the data analysis. Ten students returned a permission slip stating that they did not want their DST materials included in the research. Fifty-five students did not return a permission slip, so their material was not included for data analysis. The most significant design change for the second iteration of the DST project was a total redesign of the task students were asked to complete in the creation of their DSTs. As stated above, it was decided to use the existing critical incident analysis section of the professional practice portfolio as the basis for their DSTs. An updated and revised DST assessment rubric was created to reflect these changes in the assignment. Tripp (1993) advocated the use of critical incidents in teaching as an effective way to develop an understanding of, and control over, professional judgement and practice. Critical incidents in teaching can come from out of the ordinary, highly significant events, but they can also come from ordinary routine events in a teacher’s practice. Through critical reflection on the incident, the wider implications or wider context of the incident, such as that of the school or the community, can be brought to the surface. Tripp considered everything that happens in a classroom as a potential critical incident, “we just need to analyse it critically to make it one” (p. 28). Interpretation of the incident can lead to a transformation of experience, which Tripp felt happens “when one renders teaching practices into discourse” (p. 28). Moon (2004) suggested the use of critical incidents as a form of second-order reflection that can focus reflection, and defined second-order reflection as any reflective activity that requires a learner to look through previously written reflective work and to “write a deeper reflective overview” (p. 148). This reprocessed material can be more valuable as students are allowed the chance to reflect on their primary reflections, which can lead to deeper levels of reflection and improved learning (Moon, 2004). Main instruments As in the pilot project, items used for data analysis included the students’ completed digital stories, their working portfolios, the online discussion board, and a post DST questionnaire. In addition, for design 2, students were asked to complete an 800 to 1000–word reflective feedback piece on what they thought of the DST process and the product they created. These essays were coded and analysed using the qualitative data analysis software NVivo. Coding of student essays The student feedback essays were downloaded from the university’s learning management system. Essays from students who did not give permission for their materials to be used in the research were removed from the batch of essays to be analysed. The essays were anonymised and batch imported into NVivo. The coding of the students’ reflective feedback essays was both concept-driven and data-driven (Gibbs, 2007). Several codes arose from the literature, based on the emerging R-NEST framework. These concept-driven codes can be seen in Figure 5. Australasian Journal of Educational Technology, 2015, 31(5). 581 Figure 5. Taxonomy of concept-driven codes created for qualitative data analysis, design cycle 2 The students’ feedback essays were also coded in vivo. This refers to using the words occurring in the text to label the codes, as ideas emerge from the participants’ comments (Richards, 2010). The nodes (individual codes) created in this way were saved to an open coding folder in NVivo. When the first open coding run through was finished with the second design cycle cohort, there were 234 codes. As some of these were coded using the words occurring in the text to label the codes, there were many codes that were redundant. These were reviewed and similar codes were condensed. Several codes were renamed for clarity. Once codes have been created, the researcher can begin to arrange them into a coding hierarchy (Gibbs, 2007). The coding framework originally designed by the first author based on the R-NEST framework was used to organise the nodes into parent themes. All codes were categorised under these themes. Once the codes were organised in this way, they were analysed further. All the passages coded at a certain node could be called up in one document. The first author was then able to read through the passages several times to synthesise what the students were saying about their DST experiences. Main findings The updated and revised DST assessment rubric proved a much better instrument to assess the digital stories in the second iteration of the design. This applied especially to evaluating the DSTs for levels of reflection, as indicators for depth of reflection, suggested by Moon (2004), were now incorporated in the critical incident criteria. While using the depth of reflection rubric to assess the pilot student projects (results shown in Figure 4), the researchers felt that it did not represent the multimodality of the digital stories, as it did not include the reflection that took place with the incorporation of multimedia aspects of the DST process or the reflection students experienced as they framed and re-framed their stories. Therefore, it was decided to use the reflective aspects of the updated DST rubric to devise a depth of reflection score for the students for the second design cycle. These included the criteria for the critical incident, planning materials, the reflective feedback essay, and the use of multimedia in a reflective manner. The scores from these criteria in the assessment rubric were totalled up, and a reflective score was given. Based on the grading scheme used by the university, a rating of high (70% to 100%), medium Australasian Journal of Educational Technology, 2015, 31(5). 582 (50% to 69%) or low (10% to 49%) levels of reflection were assigned. Figure 6 shows the results of the level of reflection scores attained by the students, based on their rubric scores. Figure 6. Student levels of reflection achieved in design cycle 2 Students were asked in the questionnaire if they thought creating a DST was a reflective process. The results of the pre-service teachers’ responses can be seen in Figure 7. Figure 7. Student responses to the statement “Creating a DST is a reflective process” More than four-fifths of students who took the questionnaire (86%) felt that creating a DST was a reflective process. Students were very positive about the use of DST for reflection in their reflective feedback essays. They described DST as a different and engaging way of reflecting. Similar to the pilot project, students described the reflective writing that they were asked to do on the course as a “chore”. The following quotations exemplify students’ comments on the refreshing alternative that reflecting through the DST afforded: 53%43% 4% Design cycle 2 level of reflection scores High Medium Low (N = 133) 53% 33% 10% 4% Creating a DST is a reflective process strongly agree agree neutral disagree (N = 55) Australasian Journal of Educational Technology, 2015, 31(5). 583 I feel that of all personal reflections during the PDE this one has been the most productive and worthwhile as the process was different, creative, clear, and enjoyable. Throughout the year it seems that all of us have become sick of the word “reflection”. In being asked to reflect constantly it becomes a chore and the benefits are harder to see. This task proved to be an enjoyable alternative to writing page after page of personal reflections. (2010–2011 Student 103) DST is definitely an excellent way to get PDE students to reflect. I found that writing in my Reflective Journal was akin to a chore and that what I was writing was more like a diary entry than a critical reflection. I did not deeply engage with that particular process all of the time – some nights I was more motivated to reflect than others. Whereas with the DST, I know I have reflected comprehensively on my critical incident. (2010–2011 Student 12) In terms of engagement, time spent by students on the DST averaged 30 hours in the pilot study. This increased by 1 hour in the second design cycle, with the students involved in the second cycle of the DST intervention spending an average of 31 hours in the development of their respective digital stories. Over one-third of the students noted in their feedback essays that the DST enabled them to reflect more deeply than they had done in other reflective assignments on the course. The DST process afforded students the opportunity for secondary reflection, taking an incident from their reflective journal and delving deeper into it: I had already reflected on the bullying ‘critical incident’ in my reflective journal and 3rd weekly reflection, but making the DS made me look at it from a whole other angle. I had to look much deeper at the feelings behind my reactions and made the connections to my own past experiences with bullying. (2010–2011 Student 10) Students discussed the manner in which the DST process caused them to be more critical of their own actions. Through the creation of their DST, they realised how they could have done things differently, and that their teaching changed as a result. The students commented on how this was a very uncomfortable realisation for them, questioning their own actions and seeing the incident from other points of view: The critical incident story along with the visual representation of the story demanded that I completely immerse myself in the moment all over again. I felt I had to give everything - honesty, emotion, humiliation and criticism. Only by opening up in such a manner was I able to see the opportunity for personal and professional development. Rewriting the incident, taking the time to think of my actions and question them, reconsidering the position of the other person in the story, all helped me to see the situation from multiple viewpoints. While I was still writing the story and producing the video, I feel that I truly stepped into my student’s shoes. His situation and my behaviour played out before my eyes each time I read the script or watched the movie play. Feeling a genuine connection with the incident is what I believe genuinely helped me to engage with this experience. (2010– 2011 Student 94) An element that added significantly to the depth of students’ reflection was the multimedia aspect of the process, and this became very evident in the feedback received from students in the second, mainstream iteration of R-NEST: Making my digital story has allowed me to look at the process of critical reflection in a different way – firstly through writing the script as a story and secondly by sourcing images and music that convey exactly what I am feeling and thinking/saying. It allowed for a much deeper type of reflection to that carried out previously through my Weekly Reflections and Post Lesson Evaluations. (2010–2011 Student 109) I feel that the process of constructing the digital story provided a much more conducive method for reflection than any other form of assessment on the PDE. I had to fully engage with my emotions and my experience of the critical incident when I was writing the personal narrative, choosing words, images and music, deciding on the special effects and Australasian Journal of Educational Technology, 2015, 31(5). 584 the pace, even the inflection of the voice. These aspects of the digital story were all influenced by my reflection on the critical incident as I wanted the digital story to be in every way sensory to how I experienced the incident. (2010–2011 Student 127) The choice of multimedia such as images, music and sound, as well as the recording of their own voice, added significantly to students’ reflection. In describing the different way they reflected while incorporating multimedia into their DST, students used terms such as “focused,” “intensified,” “greater clarification,” and “greater insight.” Student 131 explained in the following excerpt how the multimedia augmented her reflection: With doing the Digital Story I went deeper into the reflection than I think I ever have done, not just in the PDE, but in general. For each picture that I was looking for I went deeper into my thoughts and, more importantly, my emotions. Instead of simply writing down the words ‘that made me feel lost’, as I would have done in previous reflections, I went deeper and deeper into how I really felt, and what exactly made me feel this way. This was not difficult; however, as I searched and searched through pictures I could measure my emotions by them. For example, I would see a picture portraying anger and think that I felt angrier than that depicted, or perhaps felt less angry than it portrays. Therefore, I was not only reflecting on the emotion of anger, but I was also able to contemplate the extent to which I felt this. As with the reflective scores from the DST rubric, story format scores from the assessment rubric (see the Appendix) were also calculated, and a narrative score was given. These scores were based on analysis of how well students used a clear, cogent, and compelling narrative format for their DSTs, including such key elements as a beginning, middle, and end; a climactic or dramatic moment (the critical incident); a resolution or denouement; and a strong narrator’s voice evidencing meta-reflection. Nearly two-thirds of the students (63%) met the narrative criteria at the highest level, while a further one-third (33%) scored in the medium level. Only 4% of students scored in the lowest band. The extra instruction on story format at the beginning of the process, the use of the critical incident as the basis of the story, and the extended use of the graphic organisers all contributed to the students utilising the story format for their digital stories. Although the preponderance of student feedback was positive and praiseworthy of the DST intervention, critical and negative evidence also emerged in the analysis of design cycle 2. Of the 143 students who gave permission for their materials to be used for research, two students in particular were negative towards the DST process, stating: On completing the digital story I must admit that most of my original fears were founded. I spent more time on it than any other part of the Professional Practice portfolio, and this does not reflect the weighting of the marks. I felt it was quite a stressful task to complete as I was very busy with other assignments. (2010–2011 Student 91) Creating the digital story was something that I did not enjoy. I found it very difficult and more time consuming than many other assignments on the course. (2010–2011 Student 102) Notwithstanding, both of these students found some good in the process, and did feel that they benefited from the experience: I struggled with the technical side of the project mostly however I felt that it had many benefits in the reflection side of the task. I enjoyed getting to the root of my critical incident and felt the reflection on it was very beneficial … In the end while the process was quite trying, the finished product was a success. (2010–2011 Student 91) But that is what I found most useful about creating the digital story, it allowed me to realise how much I have progressed and developed as an effective teacher over the past year. (2010–2011 Student 102) Australasian Journal of Educational Technology, 2015, 31(5). 585 Changes based on findings Reflection R-NEST analysis of the questionnaire results and feedback essays demonstrate that the use of the critical incident as the topic for the DST was successful. The critical incident worked well in conjunction with the narrative elements of the DST process and allowed students to reflect deeply on their chosen incident. Based on the success of the critical incident analysis within the second design cycle, it was therefore decided that this aspect of the DST design should remain the same for the third iteration of the intervention. One aspect that the researchers found lacking in the DSTs during assessment in the second cycle was the required quotations from the academic literature on the students’ chosen DST topic. In an effort to encourage students to consult the academic literature as part of the reflective process, the 2010–2011 DST assignment brief called for the inclusion of “at least three quotations from academic literature about teaching and learning that hold significant meaning for you in relation to this incident”. However, instead of quoting relevant academic literature, many students used generic education quotes from Dewey, Einstein, or Aristotle, to name just a few, which they sourced from online quote websites. This did not meet the purpose of including the requirement of academic quotations in the DSTs. The researchers felt that acceptable types of academic literature would need to be emphasised with students when discussing the assignment brief in the third iteration of the design. Narrative For the most part, the additional emphasis on story format, combined with the graphic organisers provided, enabled students to produce more cogent and reflective stories for the narrative of their DSTs during the second iteration of the design. However, one aspect needed greater emphasis at the beginning of the process: the storyboard. Some students, eager to proceed with image selection, skipped this step. However, once they realised it was a required part of the process, they went back after the DST was finished and created one. Others simply misunderstood the purpose of using the storyboard as a planning tool. For the third iteration of the design, we needed to make sure the planning purpose of the storyboard was highlighted and reinforced. It was decided to include examples of storyboards in the initial DST lessons, from students who had given permission for the use of their materials, to show how other students had effectively used the storyboard in planning their DST projects. Engagement Students who demonstrated possible disengagement with the DST process were unhappy with the amount of marks allocated to the project. It was therefore decided to increase the allocation of marks to 40 for the DST assignment in the third iteration of R-NEST. Sociality Results from the questionnaire and the student feedback essays regarding the supportive social aspects of the DST process were mostly positive. The formal structured process implemented for the story circle session was very successful, and it was decided to continue with this format of peer feedback for the third iteration of the DST design. Additional data Further feedback on the intervention was provided by the external examiner for the teacher education programme who observed in an official report to the university: “The range of assessment techniques is broad and effective, being particularly creative around the use of information technology (IT) for the story-boards [DST] where there was truly some excellent practice in operation. This arose as part of the/Professional Practice elements, inviting the students to be reflective practitioners, and was a first-rate means of underscoring the integration of philosophy of education in tackling a critical classroom incident or issue so that personal reflection and action came to the fore, and illuminating the important integrative thread through the processes of professional reflection. The vehicle for this, through IT competency and skill, also underscored the team’s (teaching staff’s) emphasis on the marriage of process and product.” Australasian Journal of Educational Technology, 2015, 31(5). 586 Design cycle 3 – capstone Based on the success of the DST innovation in both the pilot and mainstream design cycles, the decision was made by the course committee to incorporate it as a stand-alone and significant credit-bearing assignment within the teacher education programme. In the third cycle, the focus became fine-tuning the R-NEST intervention and seeking to verify and consolidate its impact on students’ capacity for reflection. As in the pilot project and the second iteration of the design, items used for data analysis included the students’ completed digital stories, their working portfolios, the online discussion board, and a post-DST questionnaire. Students were also asked to complete their 800- to 1000-word reflective feedback piece on what they thought of the DST process and the product they created. These essays were analysed using NVivo. Of the 197 students who submitted a DST during the third design iteration, 162 gave permission for their materials to be used in the data analysis. Summary of results Similar to the results in the second cycle of the R-NEST design, students found that the DST enabled them to reflect more deeply than they had done in other reflective assignments on the teacher education programme. The reasons students gave for this deeper level of reflection were: • the additional time taken to reflect while creating the DST; • the self-questioning required during the process; • having a chance to assess their own actions more thoroughly; • looking at the incident from different time frames and from different perspectives; • assessing personal beliefs and philosophies; • connecting theory to practice; and having the chance to step back and see the broader context. Students reported through their feedback essays how all of this was amplified and enhanced by the use of multimedia to create their DSTs. The following excerpts from three students exemplify the highly positive experience of the pre-service teachers during their engagement in the DST process as part of the third implementation cycle: The Digital Story has brought about deeper reflection as I thought carefully about my critical incident … Some of my weekly reflections were typed up and neatly stored away, however reflections need to be engaged with over and over again. The Digital Story allowed this thorough engagement resulting in complex thinking and reflection on my critical incident. (2011–2012 Student 55) On completion of the assignment I can say without hesitation it has been the most enjoyable and beneficial project I have ever completed. I believe that it not only improved my ICT skills but also gave me the chance to be reflective at a deeper level … Completing this assignment really gave me a sense of accomplishment and made me excited to learn. It is the first time that I have really stood up after doing an assignment and said I am proud of what I achieved. (2011–2012 Student 138) I believe the creative process involved in making this digital story has truly required me to become a “reflective practitioner” as a student teacher. (2011–2012 Student 75) Although student feedback was overwhelmingly positive, the evaluation of design cycle 3 also provided negative data, comparable to that received from students in the evaluation of design cycle 2. The credit weighting of the DST project had been increased for the third design cycle, but this persisted as an issue. One student was critical of the lack of marks awarded for the assignment and responded that she was thoroughly disengaged throughout the process. This negative feedback in the third iteration of the repeated study illustrates that there remained aspects of the intervention to be worked on in future deployments of R-NEST, including the workload demanded in the development of the digital stories and the associated academic credit. Furthermore, where students reported negatively about the DST process, Australasian Journal of Educational Technology, 2015, 31(5). 587 this appeared to be related to their own innovation risk aversion and limited engagement with the teacher education programme as a whole. As the R-NEST intervention continues to mature, the aim will be to try to engage all the pre-service teachers who undertake the DST project. The ultimate aim will be to achieve 100% participation in the process, especially among those teachers who may not be positively disposed towards teacher education and educational technology. This will be done initially by addressing the workload issue and the positioning and weighting of R-NEST within the wider teacher education programme. Of course, it may just not be possible to achieve 100% satisfaction with all aspects of the process, but this is what the researchers are aiming for as R-NEST further develops as a technology-enhanced approach to reflective practice in professional education. Conclusions Analysis of the third, capstone iteration of the DST intervention demonstrated the impact of the final design, building on the preceding two DST innovations. We now outline in Figure 8, the emergent R- NEST framework, which both informed, and was informed by, the three iterative, sequential cycles of design-based research. The design criteria and sensitivities, and informants and resources, illustrated in Figure 8, constitute key contributions of the R-NEST model. The design criteria and sensitivities provide practical guidelines for how we can use strategies for deeper reflection, such as critical incident theory (Tripp, 1993), together with specifically designed collaborative and narrative activities and methods. In particular, as outlined in Figure 8, we recommend using story circles and story maps, emphasising personal narrative format, focusing on a single incident from practice, and utilising second-order reflection. Reflective practice predominates as a paradigm for critical professional learning in teacher education. Creative and innovative reflective methods are important to prompt and sustain alternative and novel ways for teachers to think about their profession (Gore, 2015). ICTs have been shown to expand the possibilities for representing, sharing and discussing reflections on teaching (Cochran-Smyth et al., 2015). R-NEST enumerates and exemplifies how low-threshold applications (Gilbert, 2002) can be practically designed and synthesised with specific collaboration and narration strategies to enhance teachers’ generative agency as reflective practitioners. It highlights how scaffolding and support are needed throughout the process, particularly in relation to ICTs. We also recommend making explicit the value of DST in terms of the reflective and ICT skills that learners will gain from engaging in DST. The R-NEST model also provides useful recommendations about the timing of the DST process, as well as the importance of choice in learners’ selection of creative technology. It is important to note the interdependence of the R-NEST criteria; in our experience, the more R-NEST criteria and principles that are implemented, the greater the potential impact of the educational technology design. Rather than a definitive, exhaustive or finished model for technology design, deployment and evaluation, R-NEST is a working framework, to be situated and localised within cognate, respective educational design contexts. Methodologically, in adapting R-NEST for other educational settings, we would advocate the use of design-based research, or a close variant – educational design research approach. In the first instance, as recommended by Jakes and Brennan (2005), it might be most appropriate to begin with an exploratory, pilot implementation of DST. This will enable the educational designer or technologist to establish the proximal potential of the R-NEST design model while identifying exigencies that need to be addressed in subsequent iterations of the design in situ. Whether an initial small pilot, or a large, full-scale deployment, R-NEST will help to frame and guide the design, development and evaluation of DST for reflective practice in professional education. Australasian Journal of Educational Technology, 2015, 31(5). 588 Figure 8. The R-NEST model Situating R-NEST in the context of McKenney and Reeves’ (2012) model for educational design research, there are four major outputs from the design-based research outlined in this paper. Figure 9 illustrates the emergence of the R-NEST model and related outputs over the three design cycles. Students’ Perspective Wider ITE Context ITE Programme Perspective Collaboration of Teacher Educators External Evaluation REFLECTION: • Focuses on a single incident from practice • Uses second-order reflection as the basis for the digital story • Allows students time to reflect at the beginning of the DST process • Both DST process and product are important NARRATIVE: • Encourages personal narrative format needed in reflective writing • Provides basic story structure information before beginning the DST process • Utilises narrative graphic organisers such as story maps and story boards • A time parameter or limit can enhance the final digital story ENGAGEMENT: • Engages students in the reflective process through novel digital media • Makes explicit the value of the reflective and ICT skills to be gained for teaching • Implements DST project promptly after first teaching practice experience • Compacts production period as much as possible within the parameters of the ITE programme • Breaks the DST process into smaller, manageable steps SOCIALITY: • Includes opportunities for collaborative reflection in the DST process • Follows a formal, structured story sharing session for the story circle • Provides tutor feedback early in the construction of the digital story • Provides an opportunity for students to share their completed digital stories • Scaffolds technology lessons to minimise students’ anxiety or fear of using new hardware/software TECHNOLOGY: • Uses low-threshold technologies for the creation of DSTs • Provides access to good quality hardware for the voiceover recording process • Gives students a choice of recording technologies • Provides technology support to avoid disengagement due to technological difficulties Supporting design informants and resources R -N E S T d e si g n c ri te ri a a n d s e n si ti v it ie s Australasian Journal of Educational Technology, 2015, 31(5). 589 Figure 9. Illustration of R-NEST design-based research, based on the generic, integrative model for educational design research (McKenney & Reeves, 2012) In terms of design research, the first key takeaway or output of the R-NEST DBR is the enumeration of a cyclical, iterative process of design which demonstrates how DST can be introduced, developed and improved to enhance reflective practice in professional education. Second, R-NEST has engaged our students in the production of creative, multimodal artefacts that evidence critical reflection on teaching. The third and fourth significant contributions are the R-NEST curriculum for DST (timetables, resources, rubrics, etc.) and R-NEST design model, which other educational technologists and design researchers can adapt and deploy to develop DST within their own context. We would envision the R-NEST curriculum for DST in teacher education as a medial design output—a connective resource—embodying conceptually the R-NEST process while supporting practically the implementation of the R-NEST design model. The DST intervention—informed by R-NEST—is now in its sixth year of implementation. As a maturing intervention in our university, it continues to support and promote students' reflective practice. A key contemporary issue for ICT in education is understanding how collaborative learning can be mediated and enhanced by technology (Cress, Stahl, Ludvigsen, & Law, 2015). As Stahl (2015) observed, this entails: “looking at how groups of students interact with various technological artifacts and observing their meaning-making processes, their enacting of the technologies and their problem solving as mediated by the technologies” (p. 15). The DBR process illustrated in this paper represents a deep and systematic analysis of student teachers’ enactment of DST technology. The R-NEST design framework expands our broader, ontological understanding of design-based research for technology-enhanced reflective practice in post-secondary, professional education. As a prototype design model, R-NEST can help to guide other design researchers and educational technologists who want to support their learners to produce high- quality reflective digital artefacts. References Banaszewski, T. (2005). Digital storytelling: Supporting digital literacy in grades 4 – 12. (Unpublished Master’s thesis, Georgia Institute of Technology, Atlanta, USA). Retrieved from http://techszewski.blogs.com/techszewski/files/TBanaszewski_DS_thesis.pdf Barrett, H. (2005a). Digital storytelling research design. Retrieved from https://sites.google.com/site/digitalstorysite/ Australasian Journal of Educational Technology, 2015, 31(5). 590 Barrett, H. (2005b). Storytelling in higher education: A theory of reflection on practice to support deep learning. Paper presented at the Society for Information Technology and Teacher Education International Conference 2005, Phoenix, AZ, USA. Retrieved from http://www.editlib.org/p/19329/ Barrett, H. (2005c). White paper: Researching electronic portfolios and learner engagement. Retrieved from http://electronicportfolios.org/reflect/whitepaper.pdf Baylor, A. L., & Ritchie, D. (2002). What factors facilitate teacher skill, teacher moral, and perceived student learning in technology-using classrooms? Computers & Education 39(4), 395–414. doi:10.1016/S0360-1315(02)00075-1 Bruner, J. (2002). Making stories: Law, literature, life. Cambridge, MA: Harvard University Press. Calderhead, J. (1989). Reflective teaching and teacher education. Teaching and Teacher Education, 5(1), 43–51. doi:10.1016/0742-051X(89)90018-8 Cochran-Smith, M., Villegas, A.M., Abrams, L., Chavez-Moreno, L., Mills, T., & Stern, R. (2015). Critiquing teacher preparation research: An overview of the field, part II. Journal of Teacher Education, 66(2), 109–121. doi:10.1177/0022487114558268 Cress, U., Stahl, G., Ludvigsen, S., & Law, N. (2015). The core features of CSCL: Social situation, collaborative knowledge processes and their design. International Journal of Computer-Supported Collaborative Learning, 10(2), doi:109–116. 10.1007/s11412-015-9214-2 Csikszentmihalyi, M. (1991). Flow: The psychology of optimal experience. New York, NY: Harper Collins. Dewey, J. (1910). How we think. Boston, MA: D.C. Heath & Co. Dewey, J. (1916). Democracy and education. New York, NY: Macmillan. Dewey, J. (1933). How we think: A restatement of the relation of reflective thinking to the educative process. Boston, MA: Houghton Mifflin. Dogan, B., & Robin, B. (2008). Implementation of digital storytelling in the classroom by teachers trained in a digital storytelling workshop. Paper presented at the Society for Information Technology and Teacher Education International Conference 2008, Las Vegas, Nevada. Retrieved from http://www.editlib.org/p/122283/ Gibbs, G. (2007). Analyzing qualitative data. London: Sage. Gilbert, S. W. (2002, February 12). The beauty of low-threshold applications. Campus Technology. Retrieved from http://campustechnology.com/articles/2002/02/the-beauty-of-low-threshold- applications.aspx Gore, J. M. (2015). Effective and reflective teaching practice. In N. Weatherby-Fell (Ed.), Learning to teach in the secondary school (pp. 68–85). Port Melbourne: Cambridge University Press. Gravestock, P., & Jenkins, M. (2009). Digital storytelling and its pedagogical impact. In T. Mayes, D. Morrison, H. Mellar, P. Bullen, & M. Oliver (Eds.), Transforming higher education through technology-enhanced learning (pp. 249–264). York: The Higher Education Academy. Hall, T. (2012). Emplotment, embodiment, engagement: Narrative technology in support of physical education, sport and physical activity. Quest, 64(1), 105–115. doi:10.1080/00336297.2012.669324 Hatton, N., & Smith, D. (1995). Reflection in teacher education: Towards definition and implementation. Teaching and Teacher Education, 11(1), 33–49. doi:10.1016/0742-051X(94)00012-U Hodgson, K. (2010). Rubric assessment. Retrieved December 9, 2009, from http://www.umass.edu/wmwp/DigitalStorytelling/RubricAssess ment.htm Hofer, M., & Owings Swan, K. (2006). Digital storytelling: Moving from promise to practice. In C. Crawford, R. Carlsen, K. McFerrin, J. Price, R. Weber, & D. Willis (Eds.), Proceedings of Society for Information Technology & Teacher Education International Conference 2006 (pp. 679–684). Chesapeake, VA: Association for the Advancement of Computing in Education. Retrieved from http://www.editlib.org/p/22122/ Integrating Digital Storytelling in your Classroom. (n.d.). Digital storytelling rubric. Retrieved December 9, 2009, from http://its.ksbe.edu/dst/PDFs/Rubrics/digstorysample.pdf Jakes, D., & Brennan, J. (2005). Digital storytelling, visual literacy and 21st century skills. Paper presented at the Tech Forum New York, New York. Retrieved from http://te831us.wiki.educ.msu.edu/file/view/How%20to%20Digital%20Storytelling.pdf Jones, J. (2003). “Measuring time”: Let’s dwell on it – Three perspectives on exhibits that foster high dwell time. St Paul, MN: ASTC. Kajder, S., & Parkes, K. (2012). Examining preservice teachers’ reflective practice within and across multimodal writing environments. Journal of Technology and Teacher Education, 20(3), 229–249. Retrieved from http://www.editlib.org/p/37489/ Australasian Journal of Educational Technology, 2015, 31(5). 591 Kopcha, T. J., Schmidt, M., & McKenney, S. (Eds.). (2015) Educational Design Research [Special issue]. Australasian Journal of Educational Technology, 31(5). Retrieved from http://ajet.org.au/index.php/AJET Korthagen, F. (2001). Helping individual student teachers become reflective: The supervisory process. In F. Korthagen (Ed.), Linking practice and theory: The pedagogy of realistic teacher education (pp. 108–130). Mahwah, NJ: Earlbaum. Korthagen, F., & Wubbels, T. (2001). Evaluative research on the realistic approach and the promotion of reflection. In F. Korthagen (Ed.), Linking practice and theory: The pedagogy of realistic teacher education (pp. 88–107). Mahwah, NJ: Earlbaum. Lambert, J. (2009). Digital storytelling: Capturing lives, creating community (3rd ed.). Berkeley, CA: Digital Diner Press. Lortie, D. C. (1975). Schoolteacher: A sociological study. Chicago, IL: University of Chicago Press. MacLeod, D. M., & Cowieson, A. R. (2001). Discovering credit where credit is due: Using autobiographical writing as a tool for voicing growth. Teachers & Teaching, 7(3), 239–256. doi:10.1080/13540600120078193 Matthews-DeNatale, G. (2008, January 28). Digital storytelling: Tips and resources. Educause. Retrieved from http://www.educause.edu/Resources/DigitalStoryMakingUnderstandin/162538 McDrury, J., & Alterio, M. (2002). Learning through storytelling in higher education. London: Dunmore Press. McKenney, S., & Reeves, T. (2012). Conducting educational design research. London: Routledge. McKenney, S., & Visscher-Voerman, I. (2013) Formal education of curriculum and instructional designers. Educational Designer, 2(6), 1–20. Retrieved from http://www.educationaldesigner.org/ed/volume2/issue6/article20/ Moon, J. (1999). Reflection in learning and professional development. London: Kogan Page. Moon, J. (2004). A handbook of reflective and experiential learning: Theory and practice. London: Routledge Falmer. Norman, D. (1998). The invisible computer: Why good products can fail, the personal computer is so complex, and information appliances are the solution. Cambridge, MA: MIT Press. Ohler, J. (2008). Digital storytelling in the classroom: New media pathways to literacy, learning and creativity. Thousand Oaks, CA: Corwin Press. Organisation for Economic Co-operation and Development (OECD). (2005). Teachers matter: Attracting, developing and retaining effective teachers. Paris: Author. Retrieved from http://www.oecd- ilibrary.org/education/teachers-matter-attracting-developing-and-retaining-effective- teachers_9789264018044-en Porter, B. (2004a). personal expression analytical student scoring guide. Retrieved November 7, 2009, from http://www.digitales.us/evaluating/scoring_guide.php Porter, B. (2004b). DigiTales: The art of telling digital stories. Sedalia, CO: BJP Consulting. Reeves, T. C., Herrington, J., & Oliver, R. (2005). Design research: A socially responsible approach to instructional technology research in higher education. Journal of Computing in Higher Education, 16(2), 96–115. doi:10.1007/BF02961476 Richards, L. (2010). Handling qualitative data: A practical guide. London: Sage. Sandars, J., Murray, C., & Pellow, A. (2008). Twelve tips for using digital storytelling to promote reflective learning by medical students. Medical Teacher, 30(8), 774–777. doi:10.1080/01421590801987370 Sandholtz, J. H., Ringstaff, C., & Dwyer, D. C. (1994). Student engagement: Views from technology-rich classrooms. Retrieved from http://www.apple.com/nl/images/pdf/acotlibrary/rpt21.pdf Schön, D. A. (1983).The reflective practitioner New York, NY: Basic Books. Schön, D. A. (1987). Educating the reflective practitioner. San Francisco, CA: Jossey-Bass Inc. Shulman, L. S. (1986). Those who understand: Knowledge growth in teaching. Educational Researcher, 15(2), 4–31. doi:10.3102/0013189X015002004 Stahl, G. (2015). The group as paradigmatic unit of analysis: The contested relationship of CSCL to the learning sciences. The learning sciences: Mapping the terrain. Cambridge: Cambridge University Press. Retrieved from http://GerryStahl. net/pub/ls. pdf. Thompson Long, B. (2014). Designing digital storytelling: Creative technology for reflection in initial teacher education (Doctoral dissertation). School of Education, National University of Ireland, Galway, Ireland. Retrieved from http://aran.library.nuigalway.ie/xmlui/handle/10379/4463 Tripp, D. (1993). Critical incidents in teaching: Developing professional judgement. London: Routledge. Australasian Journal of Educational Technology, 2015, 31(5). 592 Corresponding author: Bonnie Thompson Long, bonnie.long@nuigalway.ie Australasian Journal of Educational Technology © 2015. Please cite as: Thompson Long, B., & Hall, T. (2015). R-NEST: Design-Based Research for Technology- Enhanced Reflective Practice in Initial Teacher Education. Australasian Journal of Educational Technology, 31(5), 572-596. Australasian Journal of Educational Technology, 2015, 31(5). Rubric based on School of Ed. critical incident criteria and assessment rubric provided by the Center for Digital Storytelling, http://www.storycenter.org, used with permission. Adheres to NUI grade bands. 593 Appendix: Digital Storytelling (DST) Evaluation Rubric Categories Excellent to Supreme (7-10) Good to Very Good (5-6) Satisfactory (4-4.5) Unacceptable (0-3) Content (Critical Incident) Rationale for choice of critical incident and context Clear rationale for choice of particular critical incident, identifies what initial beliefs were about incident, interprets possible significance of incident in context of school and wider society. Rationale for choice of particular critical incident apparent, identifies what initial beliefs were about incident, interprets possible significance of incident in context of school and wider society. Rationale for choice of particular critical incident somewhat apparent, attempts to identify what initial beliefs were about incident and significance of incident in context of school and wider society. No rationale evident for choice of particular critical incident, does not identify what initial beliefs were about incident nor possible significance of incident in context of school and wider society. Outline of incident Clearly describes key features of incident, chronology of events in the incident are clearly understandable. Describes key features of incident, chronology of events in the incident are stated. Somewhat describes key features of incident, chronology of events in the incident are unclear. Does not describe key features of incident, chronology of events in the incident are not explained. Demonstrates learning that involves the whole person Critically reflects and shows evidence of learning that involves the whole person; clearly shows how the incident impacted on their emotions, thoughts, beliefs and actions. Evidence of reflection and learning that shows how the incident impacted on their emotions, thoughts, beliefs and actions. Little evidence of reflection that shows how the incident impacted on their emotions, thoughts, beliefs and actions. No evidence of reflection, does not show how the incident impacted on their emotions, thoughts, beliefs and actions. Draws on other perspectives and time frames Critically reflects and draws on other perspectives about incident, including literature & colleagues. Considers incident in different ways & within different time frames. Reflects and draws on other perspectives about incident. Considers incident in different ways and within different time frames. Little evidence of reflection on other perspectives about incident, or consideration of incident in different ways or within different time frames. No evidence of reflection on other perspectives about incident, or consideration of incident in different ways or within different time frames. Demonstrates change in thoughts or actions Clearly conveys how critical incident has changed their thoughts and/or actions. Conveys how critical incident has changed their thoughts and/or actions. Somewhat conveys how critical incident has changed their thoughts and/or actions. Does not convey how critical incident has changed their thoughts and/or actions. Evidence of integration of theory and practice Incorporates at least three quotations from academic literature about teaching and learning that hold significant meaning for them in relation to critical incident. Incorporates two quotations from academic literature about teaching and learning that hold significant meaning for them in relation to critical incident. Incorporates one quotation from academic literature about teaching and learning that holds significant meaning for them in relation to critical incident. Does not incorporate quotations from academic literature about teaching and learning in relation to critical incident. http://www.storycenter.org/ Australasian Journal of Educational Technology, 2015, 31(5). Rubric based on School of Ed. critical incident criteria and assessment rubric provided by the Center for Digital Storytelling, http://www.storycenter.org, used with permission. Adheres to NUI grade bands. 594 Categories Excellent to Supreme 5 Good to Very Good 3 Satisfactory 1 Unacceptable 0 Planning Working Portfolio Working Portfolio includes complete and detailed planning materials: Brainstorming sheet, Story drafts, Story map, Storyboard, Final script, Lists of resources used, Reflective write-up Working Portfolio includes most required planning materials: Brainstorming sheet, Story drafts, Story map, Storyboard, Final script, Lists of resources used, Reflective write-up Working Portfolio includes some required planning materials: Brainstorming sheet, Story drafts, Story map, Storyboard, Final script, Lists of resources used, Reflective write-up Working Portfolio does not include any of the required planning materials. Storyboard Complete and detailed evidence of planning throughout entire storyboard, including sketches, sequencing, pacing, script, images, music and sound. Evidence of planning through most of the storyboard, including sketches, sequencing, pacing, script, images, music and sound. Evidence of planning through some of the storyboard, including sketches, sequencing, pacing, script, images, music and sound. Little to no evidence of planning, including minimally completed sketches, sequencing, pacing, script, images, music and sound. Reflective write-up Reflective write-up is within the 800- 1000 word count. Write-up clearly conveys the author’s feelings on the making of the digital story, explaining both the process of making the film and how they feel about the product. Reflective write-up is 5% above or below the expected word count. Write-up conveys the author’s feelings on the making of the digital story. Reflection write-up is 10% above or below the expected word count. Write-up somewhat conveys the author’s feelings on the making of the digital story. Reflection write-up is 20% above or below the expected word count, or, is not included in the working portfolio. Mechanics Citation of Sources and Permission All sources are cited completely and accurately in the credits. All copyrighted material, if used, is identified individually. (Google images is NOT cited as the source.) Most sources are cited completely and accurately in the credits. Some sources are cited completely and/or accurately in the credits. No sources are cited in the credits. Or, Google images has been cited as the source. Length Length of digital story is between the required 3-5 minutes. Length of digital story is 30 seconds shorter or longer than the required 3-5 minutes. Length of digital story is one minute shorter or longer than the required 3-5 minutes. Length of digital story is more than one minute shorter or longer than the required 3-5 minutes. http://www.storycenter.org/ Australasian Journal of Educational Technology, 2015, 31(5). Rubric based on School of Ed. critical incident criteria and assessment rubric provided by the Center for Digital Storytelling, http://www.storycenter.org, used with permission. Adheres to NUI grade bands. 595 Categories Excellent to Supreme 5 Good to Very Good 3 Satisfactory 1 Unacceptable 0 Grammar and spelling Grammar and spelling are correct (for the dialect chosen) and contribute greatly to clarity, style and story development. Grammar and spelling are mostly correct (for the dialect chosen) and contribute to clarity, style and story development. Grammar and spelling are somewhat correct, but errors detract from the story. Repeated errors in grammar and spelling detract greatly from the story. Story Structure Dramatic Question Use of strong dramatic question; opening statements demonstrate thoughtfulness and creativity and engage audience in an interesting and subtle fashion. Use of dramatic question; opening statements demonstrate thoughtfulness and creativity and engage audience. Use of dramatic question; opening statement somewhat engages audience. No dramatic question apparent; opening statement does not engage the audience or has no relationship to the rest of the story. Personal narrative Story is clearly told in the first person, conveys why events are important and how they affected the author, expresses the author’s thoughts and feelings throughout, and includes many relevant sensory details. Story is told in the first person, conveys why some events are important and how they affected the author, expresses the author’s thoughts and feelings, and includes some relevant sensory details. Story is mostly told in the first person, reason behind importance of events and how they affected the author is lacking, author’s thoughts and feelings are not well expressed, includes few relevant sensory details. Story is not told in the first person, importance of events and how they affected the author is missing, author’s thoughts, feelings and/or relevant sensory details are not included. Economy of story The story is told with exactly the right amount of detail throughout. It does not seem too short nor does it seem too long. Only language necessary to further plot and complete story arc is used. Story is mostly told with the right amount of detail throughout. However, it does need slightly more detail in some sections, or drags somewhat in others. Very little unnecessary language is used. The story seems to need more editing. It is noticeably too long or too short in more than one section. Some unnecessary language is used. The story needs extensive editing. It is too long or too short in many sections. A great deal of unnecessary language is used. Resolution of dramatic question Dramatic question is clearly resolved. Story is concluded through the use of details that allow the audience to interpret the message of the story. The audience feels satisfied and is given the opportunity to think about the content. The story concludes with enough information to provide a response to the dramatic question. The audience feels satisfied and the conclusion does not sound preachy. The story concludes with the resolution to the dramatic question. The audience feels satisfied, although the conclusion may be moralizing or preachy. The conclusion does not address the dramatic question, is not a logical conclusion given the content of the story, or the story trails off without a response to the dramatic question. http://www.storycenter.org/ Australasian Journal of Educational Technology, 2015, 31(5). Rubric based on School of Ed. critical incident criteria and assessment rubric provided by the Center for Digital Storytelling, http://www.storycenter.org, used with permission. Adheres to NUI grade bands. 596 Categories Excellent to Supreme 5 Good to Very Good 3 Satisfactory 1 Unacceptable 0 Use of Technology Images complement and help convey the ideas in the script Implicit imagery used to convey information that is not contained in the script but that adds to storyline and sense of satisfaction with the story. Tone of the visuals is aligned with tone of the story or is juxtaposed to the story with specific intent. Some use of implicit imagery to convey information not contained in the script. Images enhance the audience’s experience of the story. Tone of most visuals is aligned with the tone of the story. Limited use of implicit imagery to convey information not contained in the script. Most images reflect the voiceover exactly and do not provide additional information. Tone of some visuals is aligned with the tone of the story. Many images undermine intent of story. Almost all images reflect the voiceover exactly and do not add any new information to the script. Tone of visuals is not aligned with the tone of the story. Soundtrack (optional) contributes to the message of the story Soundtrack choice enhances sense of satisfaction with the story and makes it more interesting. Soundtrack does not interfere with ability to hear voiceover and adds greatly to the emotional tone of the story. Soundtrack enhances sense of satisfaction with story. Soundtrack does not interfere with ability to hear voiceover and adds to emotional tone of story. Soundtrack somewhat enhances story. Level of soundtrack interferes with ability to hear voiceover. Soundtrack interferes with ability to hear voiceover and/or undermines purpose of story or makes it impossible to understand story. Voiceover supports purpose and tone of story Voiceover is clearly audible, voice inflections and pacing draws audience in and creates intimacy with authentic emotion. Voiceover is clearly audible. Voice inflections and pacing in most of the script makes it easy to listen to and engage with the story. Voiceover is mostly audible. Some interest created with inflection and pacing. Voiceover is difficult or impossible to hear or is missing. Interest is lost due to lack of inflection and pacing. Student utilizes video editing software effectively Exceptional use of movie editing software. Titles, transitions and effects used effectively and greatly enhance the experience of watching the digital story. Effective use of movie editing software. Titles, transitions and effects used effectively and enhance the experience of watching the digital story. Titles, transitions and effects under or over used, and can distract from story. Titles, transitions and effects under or over used, or not used at all. No evidence of knowing how to apply movie editing effects. Possible points: With Soundtrack: 130 Without Soundtrack: 125 / 130 points / 125 points Final %: Points out of 40: http://www.storycenter.org/ Conceptualising a prototype design model Implementing the R-NEST model Design cycle 1 – pilot Context Main instruments Main findings Changes based on findings Reflection Process and product Narrative – story structure Sociality and peer feedback: the story circle Design cycle 2 – mainstream Context Main instruments Coding of student essays Main findings Changes based on findings Reflection Narrative Engagement Sociality Additional data Design cycle 3 – capstone Summary of results Conclusions References Appendix: Digital Storytelling (DST) Evaluation Rubric