colasante.pdf Australasian Journal of Educational Technology 2011, 27(1), 66-88 Using video annotation to reflect on and evaluate physical education pre-service teaching practice Meg Colasante RMIT University This case study examined the integration of a media annotation tool (MAT) into the learning and assessment activities of an undergraduate teaching (physical education) course. The media form or artefact for annotation was video recordings demonstrating individual learners’ teaching practice. The learners categorised (marked sections) and annotated their videos and received peer and teacher feedback within the tool. Their use of MAT was analysed to determine if this learning environment was effective in the context of the case, to critically reflect upon and evaluate pre-service teaching practice. The research site was RMIT University, Melbourne, and data was collected from the pilot users of MAT, a third year class and their teacher/key academic, using pre- and post-surveys and interactive process interviews (combined sessions of direct observation and semi-structured interviews). The data indicated MAT was effective for the main learning purpose of the case, but also identified some areas for further consideration. Introduction RMIT University developed a media annotation tool named MAT (co-designed by the author as the educational designer). MAT allows video-based artefacts to be uploaded and annotated online, and in later stages will incorporate other media forms. Essentially, it enables learner selection and categorisation of areas of video, with each selected area marked with a coloured ‘Marker’ along the video timeline. Each Marker links its video segment to its own annotation area, which comprises text- entry/dialogue panels structured to build into a cycle of learning. The various panels are titled: ‘Notes’, ‘Comments’, ‘Conclusion’, ‘Teacher Feedback’, and ‘Final Reflections’, and can be progressively opened and closed depending on the learning activity. The first learner cohort to formally pilot M A T was a third year class in an undergraduate physical education teaching program, studying the subject ‘Methods of teaching physical and sport education’. This subject lists among its learning outcomes the ability to critically reflect and evaluate physical education teaching practice. Achieving reflective practice is recognised as significant across teacher training including physical education (Chorney, 2006). The artefacts uploaded into MAT were each learner’s video recording of individual teaching episodes recorded during their professional experience placements. Access to each others’ video recordings within MAT was limited to peer groups of five to six learners, plus the class teacher. The learners analysed their teaching videos, and received peer and teacher feedback within M A T . This was expected to promote critical reflection and evaluation, in a collaborative learner approach. Colasante 67 Figure 1 provides an example (test site) of how a timeline of a video was marked-up at various points of analysis with a range of coloured and categorised ‘Markers’. It presents as reviewing the first ‘red’ marker, where red denotes an ‘Introductory Activity’ marker type or analysis category. It has been named ‘test marker’ and is anchored to the first 30 seconds of video. The only annotation attached to this Marker in the ‘Notes’ panel, is the word ‘testing’. The remaining annotation area can be opened by clicking on the various panels. Figure 1: A MAT test site (additional yellow labels support introductory text) Key research questions Several questions were asked of this research, including: 1. How do undergraduate teaching (physical education) students use the media annotation tool (MAT) to help achieve the intended learning outcome, to critically reflect and evaluate their teaching practice? 2. How effective is MAT from the learners’ perspective for their learning activities? 3. How effective is MAT from the course coordinator’s (key teacher/academic) perspective for the learning activities? 68 Australasian Journal of Educational Technology, 2011, 27(1) Aim of research The aim of this research was to explore and explain the use and effectiveness of MAT (stage 1: video). It endeavoured to reveal how a MAT environment might be used by the learners and teachers to support student learning in tertiary level study, in this case, to promote critical reflection. The research approach employed a single case study methodology, using mixed data collection methods (survey, observation and semi-structured interviews), as a pilot study in preparedness for a potential multiple- case study. There is still a sense of frontier about online annotation tools in tertiary education. Zahn, Pea, Hesse and Rosen expect that researching learning affordances “of advanced video tools [e.g., for segmenting, editing and annotating, compared to general video playing tools on computer] will remain an exciting and challenging field in the learning sciences” (2010:434-35). This research adds to the limited body of knowledge related to the learning effectiveness of annotation tools. Specifically, it focuses on learning stimulated by a video-based artefact, supported in an electronic environment with a structured learning cycle of reflection, annotation, discussion and feedback, in an undergraduate teaching course to critique teaching. Literature review Media and technology Video can present as a passive learning medium (e.g., Jonassen, Howland, Moore & Marra, 2003; Laurillard, 2002). Active learner engagement with video could be better promoted with electronic learning environments that enable discussion. Some authors note that anchoring discussion to specific parts of the video (or other artefact) encourages focussed discussion, providing clear linkage between learner contributions and particular components of the artefact (Kienle, 2006; Suthers, Vatrapu, Medina & Dwyer, 2008; van der Pol, Admiraal & Simons, 2010). Further to this, Laurillard (2002) states that “[f]or video, the synergy between image and argument may only work when the image is given time to be ‘sensed’, or the event experienced, and there is separate time for the argument to be ‘critically appreciated’, or the concept described”, allowing for reflection, which “is then available to the activity of modifying their description, should they be invited to do this by additional instructions or notes” (p.104). To facilitate constructivist learning online, educational technologies need to enable active learning with some learner-control, and learners should work with meaningful materials and construct their own meaning from them, writes Ally (2004), drawing from multiple learning theorists. Ally adds that this should not remain a single person experience; learners need to interact with other learners and there should be guidance from the teacher, providing points of testing and confirmation of newly developed ideas. Learners must have socio-constructive interaction in the form of dialogue plus reflection, and new educational technologies should enable this (Garrison & Anderson, 2003; Laurillard, 2002; Lin, Hmelo, Kinzer & Secules, 1999). Fahy (2004) acknowledges “the contribution online media often make to constructivist teaching … [by] expanding the range and variety of experiences usually available in classroom-based learning” enabling learning with real world situations (p.149). Such experiences can be provided for today’s learner via an array of contemporary artefacts. Representations of real Colasante 69 world situations in digital media, such as video, can form a basis for focused or converging discussion. Supporting reflective teaching practice For 20 years or more, “reflection and reflective practices have been regarded as standards towards which teachers and teacher educators must strive” (Gore & Zeichner, 1991, Laboskey, 1994, Rodgers, 2002, in Orland-Barak, 2005:26). The Victorian Institute of Teaching (2005-2011) has set standards for both graduating and registered teachers, both of which include a standard for “[t]eachers [to] reflect on, evaluate and improve their professional knowledge and practice”. Teacher educators are encouraged to “find ways to imbue preservice teachers with the intellectual and professional experiences necessary to enable them to reflect on critical levels” (Yost, Sentner, & Forlenza-Bailey, 2000:40), and “to use critical reflection as a teaching, metacognitive tool to … [develop] professional knowledge” (Rodriguez, Sjostrom, & Alvarez, 1998:5). Discernible in teacher education literature are three key factors to promote critical reflection: • peer discussion (King, 2008; Rodriguez et al., 1998; Hatton & Smith, 1995, in Yost et al., 2000:43) • teacher guidance and feedback (Dawley, 2007; Hatton & Smith, 1994; Tillema & Smith, 2000, in Orland-Barak, 2005:28) • linking theory to practice (Hatton & Smith, 1994; King, 2008; Orland-Barak, 2005; Rodriguez et al., 1998; Whipp, 2003; Yost et al., 2000). Strategies to promote a convergence of theory and practice have included structured, supervised, autobiographical writing of teaching experience (King, 2008; Rodriguez et al., 1998; Yost et al., 2000). However, these traditional methods — along with action research, case studies, microteaching, supervised practice, or interviews — are variable in ability to stimulate reflection (Hatton & Smith, 1994). Such activities still have merit in that they help to provide a ‘voice’, or a discussion with self, to “attempt to describe, explain, question, explore and challenge” (Valli 1992, in Hatton & Smith, 1994:9-10), albeit “effectiveness of such techniques depends on very careful structuring” (Hatton & Smith, 1994:12). Media and technology can play a role in offering structure for reflection. Feedback in video format is recommended in teacher education — indeed has been providing teachers alternate access to classroom interplay in (USA) teacher education from the 1960s (Sherin & Han, 2004). It is recommended for its benefits such as to enable visualisation, facilitate reflection and improve performance, as Table 1 supports. Lin et al (1999) discuss the design of technology to support reflection, advocating in- built features that provide scaffolds to support reflective thinking, and “that technology, properly designed and used, enables us to realise reflective learning environments that were not previously possible” (p.44). Whipp (2003) found from a range of studies that “online discussions need to be carefully structured to support high levels of reflection” and her own research concluded that “[i]t may be helpful to offer students a particular framework for critical reflection” (p.331). 70 Australasian Journal of Educational Technology, 2011, 27(1) Table 1: Benefits of providing feedback on teaching practice in video format to allow “preservice teachers to see and hear their teaching behaviours” (King, 2008:22) Enable visualisation to provide a view that the person teaching does not generally have because “self-observation is limited” (Fanselow, 1990:189) to enable repeat viewing to “conduct a rigorous self-assessment, and reflect profoundly upon that assessment” and provide “an effective stimulus to successfully drive preservice teachers to deeper levels of reflection” (King, 2008:28) Facilitate reflection to base reflection on “accurate video recorded data from teaching practice activities” (Kong, Shroff & Hung, 2009:547) to facilitate “honest and open self-critique … [to potentially] yield very important and revealing evidence that teachers can use to improve their craft” (Chorney, 2006:25) Improve performance “to self-evaluate their teaching from filmed lessons, and to modify their teaching behaviours to be more mastery involving” (Morgan & Kingston, 2007:127) Annotation in a new digital, video and collaborative context Annotation tools designed over the previous 10-15 years enable various artefacts to be annotated, such as web pages, digital text, images, audio, video, or a combination of media forms (e.g., Glover et al, 2007; Hwang et al, 2007; Nokelainen et al, 2005; Jung et al, 2006; McMahon & Dunbar, 2003; Northwestern University, n.d.; in Colasante & Fenn, 2009). During this time, some authors have cautioned to not over-structure engagement and collaboration processes within such tools, to avoid stifling learner control or creativity (e.g., Baker & Lund, 1996; Dwyer & Suthers, 2006). Kong, Shroff and Hung (2009) developed an online video system for pre-service teaching reflection that involved careful structuring from classroom technology to online video management. An online guiding framework comprising 15 evaluation criteria, grouped under four umbrella categories or ‘dimensions’, was used to prompt reflection. This structure was to prevent students getting ‘lost’ in the reflective process, but with no empirical data as yet to confirm this. Electronic annotation of video has been in limited use in education (Butler, Zapart & Li, 2006), despite recognition that video is rendered “more effective as learning resources when segmented and integrated with annotations from other media types”, thereby enabling ‘active consuming’ and knowledge construction (Jayawardana, Hewagamage & Hirakawa, 2001). However, there are recent examples of video annotation emerging across the teacher education sector, particularly for peer review of teaching. Rich and Hannafin (2009) report on several that are currently available for use, including Video Traces, which emerged in interactive museum activities (see Stevens & Hall, 1997) before use in teaching analysis. Other examples include a ‘video case library prototype’ for in-service teacher review (So, Lossman, Lim & Jacobson, 2009), and FD Commons Annotator that facilitates video and electronic hand-written feedback for academics (Tsukahara et al., 2009). Colasante 71 Methodology The project employed case study research methodology, seeking insight from learning experiences within a given context. The case was a class of undergraduate students using MAT in their learning, plus their teacher/key academic. They formed the pilot case or first adopter cohort of MAT, providing informative baseline data to inform wider uptake and further evaluation. Studies of single cases are beneficial in a range of scenarios, including “a pilot case that is the first of a multiple-case study” (Yin, 2003:42). Information from pilot studies can also test the design approach and provide a source to look for differences in the data of further research (Richards, 2005). A limitation in this research was the number of possible participants of M A T users. However, case study research is not based on population sampling; rather the case(s) and their units of analysis are carefully chosen. Selection of the MAT participant group was purposive, where “cases are hand-picked for a specific reason such as use of a new product” (Lewin, 2005:219), and where “the processes being studied are most likely to occur” (Denzin & Lincoln, 2000:370, cited in Silverman, 2005:129). This class provided a significant opportunity to capture data from inaugural or pilot users of a new learning tool, enabling a baseline study to be conducted within a specific learning context. A further limitation was the choice to use volunteers from within the participant cohort as key informants for the more intensive data collection processes of observations and interviews. This ‘non-probability sampling’ (Lewin, 2005) from within the purposive case relied on those willing and able to commit to a half-hour, mutually arranged appointment and therefore held no guarantee that they were representative of the wider case. Participants The participants were undergraduate physical education (PE) students enrolled in a third year practical placement subject (31 students and their class teacher). The learners were studying the subject: ‘Methods of teaching physical and sport education’, which required critical reflection and evaluation of physical education teaching practice. The cohort was expected to be dominated by young adults with good education and literacy levels, at least sufficient to pass entry requirements into undergraduate teaching and proceed successfully to third year. Twenty six consented to participate, with 23 ultimately participating; seven of whom also participated in individual ‘interactive process interviews’ (IPIs/observation-interview sessions) as did their teacher. While few more than seven students volunteered for the IPIs, the IPIs ceased after seven as data saturation was reached. Overview of data collection methods The case study employed mixed-methods of data collection yielding both qualitative and quantitative data. The participants were involved in: 1. Completion of a two-part survey (students only, n = 23): a. Pre-survey: collected pre-MAT detail b. Post-survey: collected information about experiences using MAT. 2. Individual ‘interactive process interviews’ (IPIs), involved observation then interview (students n = 7, teacher n = 1): 72 Australasian Journal of Educational Technology, 2011, 27(1) a. Direct observation: each participant used MAT and ‘thought aloud’ simultaneously b. Semi-structured/interactive interview: open ended questions encouraged each participant to discuss their experiences of using MAT. Pre- and post-survey The survey of two parts comprised questions designed substantially in quantitative format, administered as paper-based questionnaires. The question types included multiple-choice/answer followed by question sets tabled against five-point Likert scales of mainly ‘Strongly agree’ through to ‘Strongly disagree’, plus free form answer areas. The pre-survey (part 1), administered at semester start, was designed to harness learner profiles, participant opinion on previous methods of reflection and evaluation of their teaching practice, and their anticipation of a new tool to facilitate this. The two Likert-styled question sets provided a space for additional comments. This was to allow room to harness free form “real views of the respondents” (Fowler, 2009:101). The post-survey (part 2), administered at semester end, included questions related to the learners’ use of M A T, and their perspectives of how effective MAT was in their learning. Questioning included whether MAT helped them to learn, collaborate, and reflect on and evaluate their teaching practice. Like the pre-survey, each of the five Likert-styled question sets provided a free form space for additional comments, but the post-survey also culminated with several open ended questions asking, for example, what about MAT was most/least helpful to their learning and advice for others. Interactive process interviews To gain as full an understanding as possible, an innovative data collection method was required. This was in recognition that in case study research “strategies can be modified, creating hybrid strategies in which multiple sources of evidence are most likely to be relevant” (Yin, 2003:98). The ‘interactive process interview’ (IPI) was designed as a combination of consecutive direct observation and semi-structured (interactive) interview. In the MAT research, these occurred individually with student participants (n=7) and their teacher (n=1), in the second half of the semester. In each IPI, the observation component featured first. The participant was instructed to use MAT as they normally would and ‘think-aloud’ as they performed their activities. The interview phase directly followed, providing an opportunity for the participant to discuss M A T from their own perspective, prompted by open-ended questions if needed for topic coverage. The IPIs were audio-recorded with participant permission. The activities under observation included methods of M A T use and strategies to achieve their learning aims. While the participants were asked to think-aloud to capture audio data at the point of use, there is some noted difficulty in effectively and simultaneously completing an activity and reporting on it accurately (Stratman & Hamp-Lyons (1994), in Matsuta, n.d.). Think-aloud protocol in various adaptations has been used in other educational technology research despite such recognised limitations (e.g., Kozma & Russell, 1997). The aim of the interview phase was to harness each participant’s perception via reflections on: Colasante 73 • their learning (or teaching) with MAT • their intentions on actions made in the preceding observation phase • barriers to/value for learning, particularly in relation to reflection and evaluation of teaching • scenario questions (if required to stimulate full address of research issues (Wood, 2000)). The interviews of students as key informants were conducted until a point of recognition that further interviewing ceased to provide new or contradictory information, and were just reconfirming detail (Wood, 2000). Ethics Ethics approval was granted to conduct this research prior to data collection. Pseudonyms have been used for participants. Research findings The research findings — from survey and IPI results (coded and themed) — are presented here as: • learner profile, including attitudes on learning and anticipation of a new learning tool • an overview of how the learners used MAT • influential features of MAT • MAT’s effectiveness in the learners’ perspective, then the teacher’s perspective. Learner profile The learners in the case presented as largely homogenous. The 23 survey respondents comprised a female to male ratio of 13:10, English was the first language of all surveyed, and all except one were born in Australia. They were predominantly of a typical undergraduate age range, that is, 18 to 21 years, with six mature aged respondents, albeit four of these in the immediate post-typical (22-25) bracket. As could probably be expected from young adults of the current decade, their ICT confidence level was moderate or better, and almost all reported daily access to computers and the Internet. While almost all respondents reported satisfaction with their previous methods of reflecting on and evaluating their teaching practice (Figure 2), responses to other questions suggest they were generally ready to investigate the use of a tool like MAT in their learning. Respondents were generally positive towards wanting to view and examine teaching performance via video (both own teaching and their peers’), and towards discussing theory as related to their own and their peer’s teaching practice. As illustrated in Figure 3, the students wanted to give and receive feedback on teaching, although marginally more sensitive to receiving feedback from their peers. 74 Australasian Journal of Educational Technology, 2011, 27(1) 0 2 4 6 8 10 12 14 16 18 20 22 strongly agree agree neutral disagree strongly disagree Previous methods of analysing teaching practice Count I have been quite satisfied with my ability to reflect on practical experience Methods have been very effective for achieving critical reflection and evaluation Figure 2: Perspective on previous methods of analysing teaching practice 0 2 4 6 8 10 12 14 16 18 20 strongly agree agree neutral disagree strongly disagree I would like feedback on my teaching practice, and to give feedback to others on their teaching Count feedback from teacher feedback from peers feedback to peers Figure 3: Study preferences: receiving and giving feedback The students’ anticipation of a new tool to facilitate reflection and evaluation of teaching practice was also largely positive (see Figure 4). Colasante 75 0 2 4 6 8 10 12 14 16 strongly agree agree neutral disagree strongly disagree I would look forward to using / anticipate effectiveness of: Count an online tool to help me critically reflect and evaluate my teaching practice 'MAT' to help me reflect and evaluate of my teaching practice Figure 4: Anticipated effectiveness of MAT Learners’ use of MAT towards achieving the intended learning outcome Note: The features of MAT are referred to regularly in this sub-section. Figure 1 (see introduction) provides a screen capture of MAT, which helps illustrate these features. By analysing both the observation record sheets of the students’ use of MAT and transcripts of the IPIs (then coding and re-coding the data), typical methods of use emerged. Overall, there was generally a similar approach taken by the learners to critically reflect and evaluate their teaching practice; divergence was evident in only few parts of the process. While influencing factors included both the framework of MAT and guidance by the teacher, how others (peers and teacher) used this learning environment also affected the learners’ approach (see Colasante, 2010). Although some used MAT slightly differently, the commonality of approach is collated and presented in overview in Figure 5, then further explained. The activity overview is presented in Figure 5 as an artefact-centred model, given that student work in MAT revolved around their videos — both own and peers — in their MAT small groups. Each of the steps in their process is explained further, as follows. A: Access video in MAT Once each learner’s teaching practice video (pre-test) was uploaded into MAT, they accessed it via the MAT website, in their own ‘Group’ area. B: Annotate own video Learners watched their own video of teaching practice, then marked it with the pre- titled (and colour coded) ‘Marker’ types provided. The titles, or categories of analysis, were pre-determined by the teacher as key ‘beginner teaching factors’ of ‘Introductory 76 Australasian Journal of Educational Technology, 2011, 27(1) activity’, ‘Demonstrations’, ‘Checking for understanding’, ‘Transition’, ‘General feedback’, ‘Specific feedback’, ‘ALT-PE’ (academic learning time) and ‘Teacher position’. A Marker was created at each learner-selected point of the video and dragged along the timeline for the length to be captured. They then typed in a sub- title/tag name of their choice for each (e.g., the name of a specific PE activity, or any other sub-title, such as ‘Test Marker’ in Fig.1). Some created as many as 20+ Markers. The first annotation panel anchored to each Marker was ‘Notes’. Here the learners typically entered detail based on what was happening in that video segment. For example, [I wrote in Notes] how that benefits my teaching in the hope that someone else would comment back to me (Renee). Figure 5: Learning cycle of activities within MAT C: Comment on peer’s video The learners waited one to three weeks for the peers in their MAT group to analyse their videos. They then watched a marked segment of a peer’s video (or watched the video in full) then read anchored Notes (or visa versa), and decided whether to provide feedback to their peer in the second annotation panel of ‘Comments’. They repeated this for other marked segments of video, then for other peers in their group. D: Write conclusions Eventually, the learners returned to their own video of teaching practice and looked for feedback from group members in the ‘Comments’ panels. They noted the number of comments received as nil to three per Marker. An example of three responses received for one issue under analysis is: [My peers] gave me some feedback on how they thought I could improve the game. So, three people commented on this, they said it’s a “great idea; the activity lets the student practice the skills they’ve learnt” and suggested another rule I could put into the game. Another person said “they were on task and enjoying the activity” which Colasante 77 was really good to hear. And, the last person said “the students were active” and “they looked like they were having fun” and they suggested another way I could do the game by dividing them up into teams or giving them more space (Donna). Once the Notes and Comments panels were closed in MAT, the learners could enter conclusions to each of their marked-up video segments. Not all wrote a conclusion for each Marker, but when doing so, they used the ‘Conclusion’ panel typically to do one of three things: summarise; respond to feedback; or link forward to how they would apply this learning in future teaching. Some students explicitly enjoyed reading conclusions by others, for example: I’m just reading one of … [a peer’s] conclusions. He’s talking about using worksheets, which is a good idea, helps reinforce what the students have learnt, and you can check it afterwards and you can work out what they do and do not know and if there’s a pattern. So, this one’s really good! He’s done quite a lot of Conclusions so that gives everybody something to read, gives him something to read too, when he’s trying to improve his skills …; he can learn that I used to make this mistake and now I don’t (Brett). E: Review; Read teacher’s feedback The learners returned to their videos in MAT to seek out teacher feedback. Due to technical issues the ‘Teacher Feedback’ annotation panel was opened late, and all students observed had not yet received entries in this area. The learners checked through their Markers to review their teaching reflections/evaluation and look for any missed entries from their peers. They demonstrated a tendency to review whole cycles, i.e., read all the annotations written in order of Notes, Comments, Conclusions, plus check for Teacher Feedback, against each segment of aligned video, sometimes re- watching the video segment. Most tended to review their own work before their peers’. As the ‘Final Reflection’ panel was not yet available to the learners, they were shown this feature via a test site. They responded that their use of this would depend on other factors, e.g.: the importance of the teaching issue under focus, and/or the specific feedback they received from their teacher. For example, if they’re [the teacher is] just saying ‘agree with all the above’, I don’t think there’s a reason for you to say ‘ok’, that you agree as well. But if they’re giving more constructive feedback other than what the people in my group did then, yeah, I think it would be a good thing to respond to (Desi). F: Record post-test video At the time of observation, the learners were near the end of their first cycle of analysis of their pre-test teaching episode, and yet to begin analysing their second video (post- test). Some had recorded their post-test and were yet to upload it into MAT, others were due to record within days. They generally expected to follow the same process of teaching analysis as for their pre-test. Additional use A resourceful use of MAT by three observed learners was the visual use of Markers across the timeline, representing a summary or graph of the given lesson. For example, you can actually see how long you spend on a certain part of your lesson … and that indicates to me what I’ve written on my lesson plans that I’ve planned weeks before, I’m sticking to my time management (April). 78 Australasian Journal of Educational Technology, 2011, 27(1) [regarding] ‘General feedback’, I gave some at the start of the lesson and I can see I gave a fair bit at the end, and then in the middle it was very sparse and … I need to concentrate on giving feedback of any kind, all the way through the lesson (Brett). Influential features When asked an open-ended survey question on their top most valued features in MAT, the learners nominated feedback from others plus their own videos (‘media – watching self’, see Figure 6). Given a second choice (also Figure 6), the learners began to reach beyond their own analysis to that of their peers; however, own Markers dominated. 0 2 4 6 8 10 12 Feedback - from others Media - watching self Markers - mine Notes - mine Media - watching others Notes - others Which of the media annotation tool (MAT) features did you value most in your learning? Count 1st choice 2nd choice Figure 6: Most valued MAT features, ranked first and second Conversely, another open-ended survey question asked what was most helpful to their learning with MAT (Figure 7). Note the most frequent grouping of responses, viewing teaching, comprised a mix of own, our (or other collectives), and unspecified who’s teaching. Multi-themed responses from individuals were each counted, resulting in more than 23 responses. Similarly, during interviews four out of seven learners nominated video of their own teaching as the most valued thing about M A T . Some caution also emerged. For example, that all positive feedback from using MAT may not necessarily provide a true picture of your teaching, in that: maybe it was just that I had a good lesson when it was filmed, but …watching my video and looking at the Comments, there’s nothing for me to really work on, from this lesson …[and] Whereas if I was being videoed more often or in a different school or different classes, then I’d be able to pick up a lot more for me to work on (Desi). Another offered the conundrum of watching herself teach as both the best and worst thing about MAT, expressing some of her vulnerabilities she felt in watching herself, then adding: Colasante 79 [deep intake of breath] Not that I like looking at myself, but I liked being able to assess myself, I liked being able to pick up on where I could improve, or what I was doing OK, because I find that hard to see sometimes …; to disassociate when I’m teaching and see myself teach, so this was a good way of doing that. And it was private, I didn’t have to show — oh well it wasn’t — we had to work in groups, but it could be private, it could just be between you and the lecturer and not out there for everyone to see (Nicole). 0 2 4 6 8 10 12 14 16 18 View yourself (+/- others) teach Feedback from others Mark/ specify aspects of teaching Analyse teaching Whole annotation cycle Timeline of activites What was it about MAT that was most helpful to your learning? Count Figure 7: Most helpful factor, by open question While peer feedback was appreciated (Figure 6), the quality of peer feedback was also raised as a barrier issue by all but one interviewee. For example: a lot of them [peer Comments] weren’t very constructive, they were just agreeing with what I’d said rather than giving me help (Brittany). Additionally, when asked to offer advice for future students using MAT, almost all students interviewed raised improving the quality of peer feedback. For example: just try to be as specific as you can in your feedback to other group members and really be open-minded about other people’s feedback that they’re giving you … ‘cause it will make you a better teacher in the future if you do (April). While this was a strong theme across the interviewed sample, it must be noted that only a minority in the wider post-survey cohort entered quality of comments against the open questions on what about MAT was least helpful to their learning (four out of 23), with other responses mainly technical in nature (also raised across interviewee sample) or left unanswered. Learner perspective of MAT’s effectiveness Data from both the post-survey (n = 23) and the student interactive process interviews ( n = 7) helped address the question of how effective M A T was in the learners’ 80 Australasian Journal of Educational Technology, 2011, 27(1) perspective for their learning activities. This sub-section presents these findings in themes of base perceptions, followed by effectiveness for the intended learning outcome (reflection and evaluation of teaching practice). Post-survey responses revealed that most learners’ base perceptions were that MAT was an excellent tool for viewing and annotating media; however, there were a minority who were unsatisfied using MAT (Figure 8). 0 2 4 6 8 10 12 strongly agree agree neutral disagree strongly disagree Perceptions of MAT Count MAT was an excellent tool for viewing and annotating media I liked using MAT in my learning and assessment activities I would recommend MAT for other students to use Figure 8: Overall perceptions 0 2 4 6 8 10 12 14 16 strongly agree agree neutral disagree strongly disagree unanswered Using MAT in this course allowed me to: Count be challenged in an interesting way view role modelling of others as they worked through issues build or construct meaning from my learning experiences Figure 9: Learning enabled by MAT Colasante 81 A majority agreed MAT allowed them to be challenged in an interesting way, and none disagreed that M A T allowed them to view role modelling of others as they worked through issues, and allowed them to construct meaning from the learning experiences (Figure 9). Most learners surveyed considered the amount of interaction with their peers in MAT was appropriate, while approximately one-third agreed that there was an appropriate amount of interaction with their teacher (Figure 10) possibly reflecting the late availability of the teacher feedback panel due to technical start-up issues (see ‘E’ in learners use of MAT section). 0 2 4 6 8 10 12 14 16 18 20 strongly agree agree neutral disagree strongly disagree The amount of interaction with my peers/ teacher in MAT was appropriate Count Peers (fellow students) Teacher (academic) Figure 10: Amount of interaction 0 2 4 6 8 10 12 14 16 strongly agree agree neutral disagree strongly disagree The use of MAT in this course has allowed/helped me to: Count reflect on my teaching practice reflect on and evaluate my teaching practice be involved in activities relevant to my eventual professional practice understand key theoretical concepts related to my teaching Figure 11: Effectiveness for the key learning outcome 82 Australasian Journal of Educational Technology, 2011, 27(1) The major intended learning outcome for activities involving MAT was to critically reflect on and evaluate teaching practice, to which, Figure 11 illustrates an almost unanimous positive response from the learners surveyed. Further confirmation for effectiveness for the intended learning outcome was received during interviews, where all seven students interviewed agreed MAT helped them to reflect on and evaluate their teaching practice, supporting this with comments such as: it’s perfect for it [to critically reflect and evaluate teaching] … particularly because the lecturers can see it as well, so that means that you need to be more honest (Nicole). I was able to learn what my strengths were and what my weaknesses were … I found it really helpful (Brett). better really than anything we’ve really done; we’ve had to do a similar assignment, but we didn’t have MAT, and it was hard to look at your teaching through a video without having any Markers … whereas now I’ve got direction for what we are looking for (Renee). I enjoyed it because when you finish teaching and the supervisor says “oh you did this and that” you sort of think “did I really?” … when you can actually watch it and laugh at how bad some of the stuff is that you are doing it really sticks in your mind ‘cause you have a visual of watching yourself walk through the middle of an activity and you know not to do it again (Brittany). it was really good, because, in other methods you forget … But in this you can see, you can actually go back … [to] certain parts in the lesson and you might notice things you didn’t notice when you were teaching … critically analysing what you’ve done and when other people analyse it too, that’s even better because they might pick up other things as well, so instead of just you analysing it, you’ve got five other people helping you out as well (Donna). An academic’s perspective of MAT’s effectiveness The class teacher, ‘Carl’, provided one teacher’s perspective on the effectiveness of MAT. In response to whether MAT helped his students to critically reflect on and evaluate their teaching, Carl responded largely positively: I think they would find it helps their reflection [and] the visual stuff’s very powerful. … a lot of them have come up to me and said ‘didn’t realise I did that; I was amazed’ [and] most of them thought they were teaching a lot better than what they actually saw themselves as (Carl) Further qualifying with comments such as: I’m not sure they see the total benefit at the moment, because of the way it’s gone [start-up technological issues], … I think the future groups certainly will … [benefit] more (Carl). Carl recommended MAT for other academics to use, and nominated the Markers as the most helpful feature of MAT, which aided learning focus and teacher monitoring. Colasante 83 Discussion This discussion focuses on the effectiveness of MAT in the context of the case studied, a third year class in an undergraduate physical education (PE) teaching program. It also discusses some learning and teaching issues that emerged from the study, related to uses of media and collaboration within the tool. Effectiveness of MAT in the context of the case The case under examination presented as a largely homogenous cohort, e.g., most students were of typical post-secondary university age, but with an almost even gender mix. They tended to appreciate previous methods to reflect on and evaluate their teaching performance, while receptive to new methods to achieve this. The data collected from the learner and teacher participants indicated positive findings on MAT’s effectiveness for learning within the confines of its given context, to critically reflect and evaluate teaching practice. A mainly common method of MAT use emerged from the observation and interview data. This involved the learners accessing their own video of recorded teaching practice — within their peer group area in MAT — to commence reflection and evaluation. They would view their own video, selecting and marking sections that they categorised with pre-determined teacher categories (and further titled with their own tag names). They wrote about their teaching represented in the video segment in the ‘Notes’ annotation panel. Later they viewed their group members’ teaching videos, read their linked annotations, and decided whether to provide peer feedback via the ‘Comments’ annotation panels. Eventually returning to their own video, they read feedback received from peers. They further annotated some or all of their video segments by writing replies, summaries, or theory-to-practice type annotations in the ‘Conclusion’ panel. From here they sought feedback from their teacher via the ‘Teacher feedback’ annotation panel. They reviewed entire analysis cycles in their own and their peers’ videos. They did not enter ‘Final reflections’, as this area was not yet available. The learners anticipated following a similar process for their second recorded teaching episode. Responses regarding interaction in MAT were largely positive, although there was stronger satisfaction with the amount of peer feedback compared to quality, and the teacher feedback panel requires further trialling. There was support for M A T in learning via enabling construction of meaning from learning experiences and viewing role modelling of others. The learners overwhelmingly agreed that MAT was effective in facilitating reflection on and evaluation of their teaching practice; therefore, MAT seems to have been an appropriate intervention to support this intended learning outcome. In the context of the case, the teacher was largely satisfied with the effectiveness of MAT. The two things learners most valued about MAT were feedback from others (despite issues raised regarding quality) and viewing their own teaching. However, emergent issues relating to both use of media and collaboration in M A T involve learning and teaching issues worthy of further consideration. 84 Australasian Journal of Educational Technology, 2011, 27(1) Learning and teaching consideration 1 – Media issues While there was considerable appreciation for the learning ability to view teaching via MAT, the vulnerability of seeing one’s self performing via video also emerged. Thomas (2003) cautions that although useful for reflection, viewing self practice can be confronting. To minimise vulnerability while watching a video of own performance, Thomas advises establishing a non-judgmental viewing group, where “supportive peers can create a safe environment” (2003:2). Wherever promoting a safe learning environment with peers is insufficient, it might be appropriate in some situations to close the network tighter, as suggested by one interviewee, where the interaction could be made ‘private’ between teacher and learner. In the Kong et al., (2009) video reflection experience, student teachers only accessed by login videos of their own teaching practice. However, much can be gained from viewing others in video, even simply learning from the strengths of others (Sinh, 2010). The importance of recording across multiple contexts was made explicit in one interview. Uneasy about her analysis, the interviewee recommended multiple recordings as no weaknesses were exposed in her first (pre-test) teaching episode. Note that recorded sessions may not reveal all contextual issues regardless, e.g., “environmental factors that are not captured, or generally elements that are difficult to reassess” or re-analyse later without the full context of ‘being there’ (Butler et al., 2006:21). Multiple sampling can provide a range of contexts, even if entire contextual factors are not captured. In this pilot case, it was the original intent of the teacher for the learners to each record and analyse three teaching episodes, reduced to two due to technical delays. Learning and teaching consideration 2 – Collaboration issues Peer feedback in MAT was both valued and criticised, with some preference for more constructive comments. However, this is not unlike findings in the So, Hung and Yip (2008) study, where not all student peer comments where necessarily constructive in nature; approximately one-third equated to remarking “good”, another third “good” plus offering suggestions, and the remainder offering suggestions. While ‘competitive argumentation’, where “participants offer competing views … and work together to make sense of a given situation” might be ideal for in-service teacher professional development (Schoenfeld, Smith & Arcavi, 1993, in Sherin & Han, 2004:165), it may be argued as a high goal for student teachers. However, Jonassen et al. (2003) would argue that “given the opportunity, students of all ages readily experiment with technologies, articulate their own beliefs, and construct, co-construct, and criticize each other's ideas” (p.14). Perhaps further guidance on what constitutes good quality, constructive feedback could be offered in learner preparation on how to use MAT. The aim would be to further support the MAT structure to encourage ‘critical friends’ to approach the reflection activities more collaboratively than individually (Hatton & Smith, 1994). The need for personal versus shared annotations in MAT should be determined per learning activity, by considering benefits for others to read and collaborate, compared to inhibitors. Inhibitions to collaborative annotation could be from a “fear of saying wrong things thereby showing their misunderstanding of the issue at hand” (Lapique & Regev, 1998:3), or “feelings of vulnerability which follow from exposing one’s beliefs to others, with a tendency for self blame for any perceived weaknesses uncovered through reflection” (Wildman & Niles, 1987, in Hatton & Smith, 1994:13). However, Colasante 85 Marshall and Brush (2002) found that people apply more effort with annotations when intended for sharing, compared to more spontaneous personal annotations. Promoting a ‘safe’ environment for peers to contribute in MAT is important, as — similarly to video recorded practice — vulnerability needs to be reduced as a barrier to communication by promoting non-judgmental peer discussion groups (Thomas, 2003). Conclusion and next steps Although annotation tools are not new technology, they are still rather frontier in education. Certainly MAT is new, and differentiated from other annotation tools with a structured annotation cycle, requiring examination to ascertain its place in learning. This research explored the use and effectiveness of MAT (stage I: video) in a specific tertiary learning context. It involved a single-case study to evaluate the effectiveness of integrating MAT into an undergraduate teaching (physical education/PE) class, to facilitate critical reflection on and evaluation of videoed teaching practice. The research justified its single-case approach as forming pilot research to a future multiple-case study. The participants were third-year undergraduate PE students, plus their teacher, as inaugural users of MAT in their learning. The case under examination used MAT for learner analysis of video recorded teaching practice. The tool provided a structured learning cycle that explicitly enabled annotations by learner, peers and teacher to promote interaction. It promoted active learning with meaningful materials to construct meaning from them. While learner perception of M A T was largely positive in this study, there was minority dissatisfaction in some areas. The study illustrated that integrating MAT was an effective intervention for the case, that is, for the main purpose of reflection and evaluation of teaching practice. There was particular appreciation for the ability to view self teaching and receive feedback from others. The learners appreciated the ability to analyse their videos of teaching practice; to categorise the video and anchor annotations to segments of the video in cycles of notation and feedback. There were issues raised of vulnerability in viewing and analysing performance, and issues regarding quality of collaborative input from peers. The data illustrates the framework for collaborative annotations in M A T provided the opportunity for socio-constructivist learning in peer to peer networks, but more so where learners made concerted effort to contribute constructively. Further research Having been a largely successful integration into the case of this research, questions are now raised as to other tertiary contexts in which to examine MAT. Such contexts might vary across disciplines, learner types and tertiary sectors (vocational and higher education). The findings of this pilot study serve to inform a wider study in comparative findings, learning and teaching considerations, and research design. Acknowledgment This research was conducted under the supervision of Dr Damian Blake, School of Education, Deakin University. This paper benefited from graphic support by Geoff Marchiori, and insightful and valued feedback from the anonymous reviewers. 86 Australasian Journal of Educational Technology, 2011, 27(1) References Ally, M. (2004). Foundations of educational theory for online learning. In T. Anderson & F. Elloumi (Eds.), Theory and practice of online learning (pp. 3-31). Canada: Athabasca University. http://cde.athabascau.ca/online_book/ Baker, M. & Lund, K. (1996). Flexibly structuring the interaction in a CSCL environment. Paper presented at the EuroAIED Conference, Lisbon. [verified 20 Feb 2011] http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.108.5419&rep=rep1&type=pdf Butler, M., Zapart, T. & Li, R. (2006). Video annotation – improving assessment of transient educational events. Paper presented at the 2006 Informing Science and IT Education Joint Conference, Salford, UK, 25-28 June. [verified 20 Feb 2011] http://informingscience.org/proceedings/InSITE2006/ProcButl168.pdf Chorney, D. (2006). Teacher development and the role of reflection. Physical & Health Education Journal, 72(3), 22-25. Colasante, M. (2010). Future-focused learning via online anchored discussion, connecting learners with digital artefacts, other learners, and teachers. In Curriculum, technology & transformation for an unknown future. Proceedings ascilite Sydney 2010. http://www.ascilite.org.au/conferences/sydney10/Ascilite%20conference%20proceedings %202010/Colasante-full.pdf Colasante, M. & Fenn, J. (2009). ‘MAT’: A new media annotation tool with an interactive learning cycle for application in tertiary education. Paper presented at the World Conference on Educational Multimedia, Hypermedia and Telecommunications (ED-MEDIA) 2009, Honolulu, 22-26 June. Dawley, L. (2007). The tools for successful online teaching. [viewed 8 April 2008; available from several sources] http://ebookee.org/The-Tools-for-Successful-Online-Teaching_324424.html Dwyer, N. & Suthers, D. (2006). Consistent practices in artifact-mediated collaboration. Computer-Supported Collaborative Learning, 1, 481-511. [verified 20 Feb 2011; open access] http://www.springerlink.com/content/20885327pk7x1141/fulltext.pdf Fahy, P. (2004). Media characteristics and online learning technology. In T. Anderson & F. Elloumi (Eds.), Theory and practice of online learning (pp. 137-171). Canada: Athabasca University. http://cde.athabascau.ca/online_book/ Fanselow, J. (1990). "Let's see": Contrasting conversations about teaching. In J. Richards & D. Nunan (Eds.), Second language teacher education (pp. 182-). New York: Cambridge University Press. Fowler, F. (2009). Survey research methods (4th ed.). Thousand Oaks, California: SAGE Publications, Inc. Garrison, D. R. & Anderson, T. (2003). E-learning in the 21st century: A framework for research and practice. Milton Park, Oxon: Routledge. Hatton, N. & Smith, D. (1994). Facilitating reflection: Issues and research. Paper presented at the 24th Conference of the Australian Teacher Education Association, Brisbane, 3-6 July. http://www.eric.ed.gov:80/ERICWebPortal/contentdelivery/servlet/ERICServlet?accno=E D375110 Jayawardana, C., Hewagamage, K. & Hirakawa, M. (2001). Personalization tools for active learning in digital libraries. The Journal of Academic Media Librarianship, 8(1). [verified 20 Feb 2011] http://wings.buffalo.edu/publications/mcjrnl/v8n1/active.pdf Jonassen, D., Howland, J., Moore, J. & Marra, R. (2003). Learning to solve problems with technology; A constructivist perspective (2nd ed.). Upper Saddle River, New Jersey: Pearson Education, Inc. Colasante 87 Kienle, A. (2006). Integration of knowledge management and collaborative learning by technical supported communication processes. Education and Information Technologies, 11(2), 161-185. King, S. (2008). Inspiring critical reflection in preservice teachers. Physical Educator, 65(1), 21-29. Kong, S. C., Shroff, R. H. & Hung, H. K. (2009). A web enabled video system for self reflection by student teachers using a guiding framework. Australasian Journal of Educational Technology, 25(4), 544-558. http://www.ascilite.org.au/ajet/ajet25/kong.html Kozma, R. & Russell, J. (1997). Multimedia and understanding: Expert and novice responses to different representations of chemical phenomena. Journal of Research in Science Teaching, 34(9), 949-968. Lapique, F. & Regev, G. (1998). An experiment using document annotations in education. Paper presented at the WebNet 98 World Conference of the WWW, Internet and Intranet Proceedings. http://www.eric.ed.gov:80/PDFS/ED427713.pdf Laurillard, D. (2002). Rethinking university teaching: A framework for the effective use of learning technologies (2nd ed.). London.: RoutledgeFalmer. Lewin, C. (2005). Elementary quantitative methods. In B. Somekh & C. Lewin (Ed.), Research methods in the social sciences. London: SAGE Publications. Lin, X., Hmelo, C., Kinzer, C. & Secules, T. (1999). Designing technology to support reflection. Educational Technology Research & Development, 47(3), 43-62. Marshall, C. & Brush, A. (2002). From personal to shared annotations. Paper presented at the Conference on Human Factors in Computing Systems, Minneapolis, Minnesota, USA Matsuta, K. (n.d.). Think-aloud protocols: A means of observing cognitive processes of language learners, pp. 67-74. [viewed 15 May 2009, verified 21 Feb 2011] http://www.lib.yamagata- u.ac.jp/you-campus/koeki/kiyou-koeki/3/3-pA67-74.pdf Morgan, K. & Kingston, K. (2007). Development of a self-observation mastery intervention programme for teacher education. Physical Education & Sport Pedagogy, 13(1), 109-129. Orland-Barak, L. (2005). Portfolios as evidence of reflective practice: What remains 'untold'. Educational Research, 47(1), 25-44. Rich, P. J. & Hannafin, M. (2009). Video annotation tools: Technologies to scaffold, structure, and transform teacher reflection. Journal of Teacher Education, 60(1), 52-67. Richards, L. (2005). Handling qualitative data: A practical guide. London: SAGE Publications Ltd. Rodriguez, Y., Sjostrom, B. & Alvarez, I. (1998). Critical reflective teaching: A constructivist approach to professional development in student teaching. Paper presented at the Annual Meeting of the American Association of Colleges for Teacher Education, New Orleans, Louisiana, 25-28 February. http://www.eric.ed.gov:80/ERICWebPortal/contentdelivery/ servlet/ERICServlet?accno=ED418054 Sherin, M. G. & Han, S. Y. (2004). Teacher learning in the context of a video club. Teaching and Teacher Education, 20(2), 163-183. Silverman, D. (2005). Doing qualitative research (2nd ed.). London: SAGE Publications. Shih, R.-C. (2010). Blended learning using video-based blogs: Public speaking for English as a second language students. Australasian Journal of Educational Technology, 26(6), 883-897. http://www.ascilite.org.au/ajet/ajet26/shih.html So, H.-J., Lossman, H., Lim, W.-Y. & Jacobson, M. (2009). Designing online video based platform for teacher learning in Singapore. Australasian Journal of Educational Technology, 25(3), 440-457. http://www.ascilite.org.au/ajet/ajet25/so-2.html 88 Australasian Journal of Educational Technology, 2011, 27(1) So, W. W., Hung, V. H., & Yip, W. Y. (2008). The digital video database: A virtual learning community for teacher education. Australasian Journal of Educational Technology, 24(1), 73-90. http://www.ascilite.org.au/ajet/ajet24/so.html Stevens, R., & Hall, R. (1997). Seeing tornado: How video traces mediate visitor understandings of (natural?) phenomena in a science museum. Science Education, 81(6), 735-748. Suthers, D., Vatrapu, R., Medina, R. & Dwyer, S. (2008). Beyond threaded discussion: Representational guidance in asynchronous collaborative learning environments. Computers & Education, 50(4), 1103-1127. Thomas, H. (2003). Transcripts for reflective teaching. Paper presented at the 16th English Australia Education Conference, Melbourne, 2-4 October. http://www.englishaustralia.com.au/ea_conference03/proceedings/pdf/035F_Thomas.pdf Tsukahara, W., Hori, S., Kato, Y., Egi, H., Terada, T. & Nakagawa, M. (2009). Comparison of peer reviewing between paper-and-pen based review sheet and an annotation support system "FD Commons Annotator". Paper presented at the World Conference on Educational Multimedia, Hypermedia and Telecommunications (ED-MEDIA) 2009, Honolulu, 22-26 June. van der Pol, J., Admiraal, W. & Simons, P. (2010). Peer evaluation in online anchored discussion for an increased local relevance of replies. Computers in Human Behavior, 26(3), 288-295. Victorian Institute of Teaching (2005-2011). Professional standards. [viewed Oct 2009, Jan 2011] http://www.vit.vic.edu.au/standardsandlearning/Pages/professional-standards.aspx Whipp, J. (2003). Scaffolding critical reflection in online discussions: Helping prospective teachers think deeply about field experiences in urban schools. Journal of Teacher Education, 54(4), 321-333. Wood, C. (2000). Methodology for field work: Interactive interviews. The IAI/UM Summer Institute on Interdisciplinary Science in the Americas. [viewed 16 Apr 2008, verified 21 Feb 2011] http://www.rsmas.miami.edu/IAI/Inst2000/lectures/wood_jul20/reading/qual_appr_1.pdf Yin, R. (2003). Case study research: Design and methods (3rd ed.). California: SAGE Publications. Yost, D., Sentner, S. & Forlenza-Bailey, A. (2000). An examination of the construct of critical reflection: Implications for teacher education programming in the 21st century. Journal of Teacher Education, 51(1), 39-49. Zahn, C., Pea, R., Hesse, F. W. & Rosen, J. (2010). Comparing simple and advanced video tools as supports for complex collaborative design processes. The Journal of the Learning Sciences, 19(3), 403-440. Meg Colasante Academic Development Group College of Science, Engineering and Health, RMIT University GPO Box 2476, Melbourne 3001, Australia Email: meg.colasante@rmit.edu.au Web: http://www.rmit.edu.au/ Please cite as: Colasante, M. (2011). Using video annotation to reflect on and evaluate physical education pre-service teaching practice. Australasian Journal of Educational Technology, 27(1), 66-88. http://www.ascilite.org.au/ajet/ajet27/colasante.html