doherty.pdf Australasian Journal of Educational Technology 2011, 27(3), 381-396 Evaluating the impact of educational technology professional development upon adoption of Web 2.0 tools in teaching Iain Doherty The University of Auckland Our paper reports upon and critically analyses findings from a two-year research project into the provision of continuing professional development for making purposeful use of Web 2.0 tools in teaching. Based on our research, we make recommendations for delivering effective continuing professional development workshops whilst also acknowledging the limitations of this form of continuing professional development. The advantages of alternative forms of professional development are discussed and implications for practice are outlined. Introduction The Learning Technology Unit at the Faculty of Medical and Health Sciences, University of Auckland has been delivering Educational Technology Professional Development (ETPD) workshops to teach staff how to use Web 2.0 tools or social media (Anderson, 2007) purposefully in their teaching since delivering a successful trial workshop in late 2007 at the university’s annual teaching and learning showcase. There are many other professional development modules including: in situ training; consulting; peer assessment; peer mentoring; and student assessment of teaching (Prebble, et al., 2004). However, workshops are a cost effective means of teaching participants particular skills and in creating awareness of new ideas and there is some evidence that they can be effective in changing participants’ teaching practices (Prebble, et al., 2004; Timperley, Wilson, Barrar & Fung, 2007). At the time of writing we have delivered 16 workshops within the Faculty of Medical and Health Sciences and at the University’s Centre for Academic Development. Here we report research results for 12 of those workshops delivered from June 2008 through to December 2009. In line with the design research approach detailed in the next section, we have reported our research results in summary form over the past two years (Doherty, 2010a, 2010b; Doherty, Blake & Cooper, 2009; Doherty & Cooper, 2009). This article reports the data in full, presents additional data, analyses the data in its entirety and includes new explanations for the findings along with a discussion of the broader implications of our research. Research Research approach There are many approaches to educational research and various methodological approaches that are driven by different aims (Reeves, 2000). Our research aim was to 382 Australasian Journal of Educational Technology, 2011, 27(3) develop effective ETPD workshops whilst also constructing principles to guide future ETPD development efforts. For this reason we made use of the design research approach (Collins, Joseph & Bielaczyc, 2004; Oh & Reeves, 2008; Reeves, 2006; Reeves, Herrington & Oliver, 2005) – rather than an action research approach (Reeves, 2000) for our ETPD research. Action research consists of “theorizing in practice” (Laurillard, 2008) and focuses on local problems and issues and does not try to generate broader theory and was not, therefore, appropriate for our purposes. In contradistinction, the goal in design research is to understand how a particular learning design affects dependent variables in a real world setting (Collins, et al., 2004). In our case the dependent variables included participants’ workshop experiences, participants gains in knowledge/ skills, and participants implementing what they learned. Research method We had four research aims when we designed our research project. First, we wanted to evaluate participants’ experiences of the workshop. Secondly, to evaluate whether participants’ achieved the desired learning outcomes. The purpose of the first two evaluations was to determine whether the necessary conditions had been met for participants to put into practice what they had learned (Guskey, 2000). Our third aim was to determine whether participants had in fact put into practice what they had learned. Our fourth aim was to identify factors that impacted positively and negatively on participants’ intentions to put into practice what they had learned. We used an online pre-course evaluation to baseline participants’ prior knowledge of and use of Web 2.0 tools. We used an online post-workshop evaluation to gather data on participants’ experiences and to determine whether participants had achieved the desired learning outcomes. The post-course questionnaire initially consisted of a set of standard evaluation questions derived from the University’s question bank. However, in line with the principles for design research, we revised the post-course evaluation instrument over time. We also reflected in action and on action (Schön, 1987), making a mental note of what worked and what did not work as we delivered the workshops. We captured these “impressions” by writing up summaries of each workshop immediately after the workshops were over. Finally, we followed up with participants approximately three months after the workshops to determine whether they had put what they had learned into practice, through conducting semi-structured interviews that were recorded by the interviewer and transcribed by a third party. Web 2.0 workshops Background There is a substantial amount of research literature that provides principles for the effective delivery of continuing professional development (CPD). Prebble (Prebble, et al., 2004) and Timperley (Timperley, et al., 2007) both provide useful syntheses of the research evidence, whilst Wayne et al. (Wayne, Yoon, Zhu, Cronen & Garet, 2008) and Desimone (2009) both provide useful summaries of features that make professional development effective. Desimone’s conceptual framework for teacher professional development is particularly useful in identifying the necessary conditions for successful CPD. These conditions include: professional development content that focuses on how and what students learn; active rather than passive learning; coherence with teachers’ beliefs and attitudes; appropriate duration; and collective participation. Doherty 383 Chalmers and Keown (2006) along with Wayne et al. (2008) concur on a number of the necessary conditions identified by Desimone, whilst emphasising job embedded professional development as helpful for improving classroom practice. We had identified a number of the conditions for successful professional development specified by Desimone (2009) and Wayne et al. (2008) and we aimed to develop workshops that would be effective in opening participants to new ideas, whilst also providing them with new skills and usable knowledge. Our workshop format was grounded in active learning techniques (Desimone, 2009; Fink, 2006; Timperley, et al., 2007) in which participants focussed on a core pedagogical problem or project of their own choosing (Chalmers & Keown, 2006; Desimone, 2009; Tan, Hu, Wong & Wettasinghe, 2003; Timperley, et al., 2007) that would engage participants in authentic and meaningful learning (Timperley, et al., 2007) in a collaborative or social participatory environment (Wayne, et al., 2008). Finally, whilst Desimone (2009) conceives coherence as a fit between professional development content and teachers’ beliefs and attitudes about teaching, we conceived fit in terms of providing conditions within which participants would be challenged to think about their teaching beliefs as a prelude to constructing and evaluating their new knowledge (Chalmers & Keown, 2006). The overall aim of our voluntary Web 2.0 workshops was to teach educators how to make purposeful use of Web 2.0 tools in their teaching. There were two facilitators for each workshop and the maximum number of participants for each workshop was specified as 15, in order to ensure a reasonable facilitator to participant ratio. Participants for the Centre for Academic Development workshops came from arts and science faculties from across the university. Participants at the Faculty of Medical and Health Sciences workshops were all teaching staff from the Faculty. Each of the workshops was scheduled for three hours and delivered in a computer training room set up with user computers and a data projector. The learning materials were delivered using a commercial wiki (http://virtuallythere.wikispaces.com). The rationale for using Wikispaces was that we wanted an exemplar educational wiki that we could present to participants as they learned about wikis for themselves. Design One The first workshop design was used for Workshop 1 (Doherty, et al., 2009) and centred on a model of active learning (Fink, 2006), in which activities are constituted by experience (doing and observing) and by dialogue (conversation and self-reflection). Our overall strategy was to create an authentic learning environment (Herrington & Oliver, 2000; Reeves, Herrington & Oliver, 2002) in which participants would learn knowledge and skills in a way that would reflect the use of the knowledge and skills in real life. Whilst there will be debate about the authenticity of our environment – particularly given a workshop setting of limited duration – we take the view that teaching participants how to use Web 2.0 tools through encouraging them to think about the pedagogical uses of these tools in their own teaching context constitutes an authentic learning activity. If there is a question here it is one of degree. We also wanted to ensure that we achieved the three types of educational presence outlined by Rourke et al. (Rourke, Anderson, Garrison & Archer, 1999) and later by Hutchins (2003): cognitive presence which occurs though frequent and effective interaction with content; teaching presence which occurs through frequent and effective interaction with the facilitator; and social presence which occurs through frequent and effective interaction between participants. 384 Australasian Journal of Educational Technology, 2011, 27(3) Initially we engaged participants in self-dialogue and dialogue with others by asking them to reflect, in pairs, on why they were attending the workshop and what they hoped to gain from attending the workshop. Participants were then asked to report back to the wider group. Our purpose was to encourage participants to think about their reasons for their attendance so that they might connect their learning with their own particular needs. These activities also helped to establish a social presence through having participants engage with one another and with the facilitators. In order to establish an authentic context we then talked about how the Web 2.0 tools might be used purposively in teaching to develop students’ personal qualities, skills and attributes, as detailed in the University of Auckland Graduate Profile. The authentic context was also designed to ensure frequent and effective interaction with workshop content. We had a “reflective surprise moment” when we discovered that participants were not familiar with the Graduate Profile. As a result we changed the focus question and asked participants to progress through the workshop thinking about how the Web 2.0 tools might be used to meet a particular teaching and learning problem that they faced in their own teaching. Thus, although we had revised our context, we offered an alternative authentic context that would also serve to establish and maintain cognitive and teaching presence. The facilitators demonstrated the main features of each of the tools – a blog (Blogger), a wiki (Wikispaces) and a social bookmarking service (Delicious) – in turn for approximately 20 minutes, inviting questions and comments on technical and pedagogical issues. The facilitators moved around the classroom after each demonstration to help participants as they worked with each of the tools and to discuss technical and pedagogical issues. These activities were designed to promote teacher and cognitive presence. Although we had established an authentic setting within which the participants might learn how to use the technologies appropriately in their teaching, the reality was that the question of the teaching and learning context was eclipsed by the fact that the participants focused almost wholly on learning the technical aspects of the three technologies. There was very little reflective thinking as evidenced by the fact that questions put to facilitators almost wholly concerned the practical challenges of working with the new tools. Time management was an issue, which meant that the facilitators did not engage participants in dialogue on application of the tools in teaching at the end of each practical session. Therefore, social presence and cognitive presence were not sufficiently established. Overall, it was the judgment of the facilitators that there was insufficient effective interaction between: participants and participants; participants and facilitators; and participants and content. That is, social, cognitive and teacher presence were not sufficiently established. The desired outcome was partially achieved; participants learned how to use each of the tools but did not reflect sufficiently on the application of the tools to their teaching. However, the evaluations of this workshop – reported below – were very good. Design Two In terms of the dependent variables discussed above – participants’ experiences, participants achieving learning outcomes, and participants putting what they learned into practice – we re-designed the workshop to try to improve in each of these three areas. Based on our experience of not achieving the desired levels of social, cognitive and teacher presence, we re-designed the workshop using the work of Dunlap et al. (2007) on designing learning environments for deep and meaningful student-content Doherty 385 interaction. However we extended their work to cover student-teacher interactions and student-student interactions. Our concern lay with generating interactions that would encourage Bloom’s higher level cognitive processes of application, analysis, synthesis and evaluation (Atherton, 2005; Bloom, 1956; Clark, 2008). Our specific design conjecture (Sandoval, 2004) was that incorporating a variety of carefully planned and well managed triggering, exploration, reflective, integration, resolution, and metacognitive interactions would lead to greater levels of engagement, and to higher- order thinking on the part of the participants about use of the tools in teaching (Doherty & Cooper, 2009). Examples of the interaction types together with the form of implementation and expected intermediate outcomes (Sandoval, 2004) are detailed in Table 1. Table 1: Interaction types Interaction type [Strategy] Implementation [Embodied conjectures] Intermediate outcomes Triggering interactions enable participants to see a problem or lead to a sense of puzzlement. Have participants work in pairs to reflect on why they were attending the workshop and to consider what they hoped to gain from the workshop. Have participants post their responses to the wiki discussion board. Provide an explanation of what constitutes “deep learning” and set participants the specific task of reflecting on how the three Web 2.0 tools might be used in their own teaching context. Participants engage with one another. Meaningful answers posted to wiki discussion board. Questions about their own teaching context throughout workshop. Engaged paired discussions. Exploration interactions encourage participants to follow their own paths through content and to pursue their own areas of interest. Have participants create their own blogs, wikis, and social bookmarks, whilst keeping in mind the question of using the tools in their own teaching practice. Participants’ questions are a balance of technical and pedagogical. Reflective inquiry inter- actions enable particip- ants to ask questions, challenge assumptions and critically examine the implications of their actions. Encourage participants to ask the instructors questions – both concerning the technologies and their pedagogical value – throughout the workshop. Provide the opportunity for reflective inquiry through group discussion. Participants are focused on both the technologies and their pedagogical use. Integration interactions enable participants to connect ideas and create solutions. At the end of each practical session divide participants into pairs to talk about educational uses of the tools. Participants engage with one another and with facilitators. Resolution interactions enable participants to apply new ideas and assess solutions. Ask each participant to post to the discussion board on their thoughts on uses of the tools for their own teaching context and discuss as a group. Discussion board postings evidence uses of tools for participant learning. Metacognitive interactions enable participants to reflect on their own cognitive processes or to think about their own thinking. At the end of the workshop return the participants to their reflections from the initial exercise and ask them if and how their perceptions of the technologies for teaching and learning had changed during the workshop. Discussion that reflects metacognitive thought processes. 386 Australasian Journal of Educational Technology, 2011, 27(3) The second workshop design was used in Workshops 2, 3, 4 and 5 and followed the model of active learning detailed above for Workshop 1. However, the design conjecture detailed above was reified in terms of: reducing the demonstration time for the various tools to cover only key functionality, in order to make more time for reflection and discussion; providing an example of a teaching challenge, by including content on the workshop wiki concerning how the tools might be used to promote deep learning on the part of learners in participants’ own teaching settings; providing specific examples on the workshop wiki of ways in which blogs, wikis, and social bookmarking have been used in teaching, in order to help participants’ to reflect on their own practice; introducing paired discussions whilst incorporating a wiki discussion board to “capture” the participants’ thoughts in order to present them in a tangible manner for group discussion. Our overall aim – or intervention outcome (Sandoval, 2004) – was for participants to feel prepared to use the three tools in their teaching at the end of the workshop. We surmised that designing the workshop environment around various interactions would facilitate meaningful participant-content interaction, meaningful facilitator- participant interaction and meaningful student-student participation leading to greater levels of engagement and higher order thinking on the part of the participants. To assess that conjecture, we will report the facilitators’ observations during the workshop along with evidence from the discussion board postings. When working with the technologies – exploration and reflective inquiry interactions – participants focused on mastering the particular tools with inquiry concerning teaching practice being limited. However, during each of the discussion sessions – integration interaction – participants engaged both with one another and with the facilitators. We judged that the discussion board postings – the resolution interactions – showed evidence of higher order thinking with a number of participants reflecting on the use of the tools in their own context whilst also considering the various ways in which the tools might be used for teaching (Doherty & Cooper, 2009). We saw this workshop as a positive step forward in terms of participants engaging with one another, with facilitators and with how the Web 2.0 tools might be used in teaching. Based on the perceived success of the new learning design, we made the judgment that we might refine the learning design to further engage participants with teaching and learning questions. Design Three Our third workshop design – used in Workshops 6 through 10 – made use of the active learning techniques and the various types of interactions outlined in the previous section (Doherty, 2010a, 2010b). However, we made two major changes for this workshop design. First, we introduced a learning design template and let participants know that we expected them to create a learning design during the second half of the workshop, to purposefully integrate one or more of the Web 2.0 tools into their teaching. Our design conjecture was that working with the learning template would result in participants gaining a deeper understanding of how to use the Web 2.0 tools purposefully in their teaching. In terms of the dependent variables in this study we were still concerned with participants’ experiences and with the learning outcomes. In particular we wanted to see evidence of the three forms of presence along with learners feeling prepared to use the Web 2.0 tools in their teaching. The template itself required educators to specify: the learning outcomes for a course; the learning activities for the course; the assessment methods for the course; and the ways in which Doherty 387 Web 2.0 tools would be purposefully integrated into their course. We also introduced a social networking environment to expose participants to another technology and to give them a place to post their workshop reflections. Facilitators engaged participants in a discussion of constructive alignment (Biggs, 2008) with the purpose of bringing participants to understand how Web 2.0 tools might be meaningfully integrated into a learning design (Jones, 2007). The system of constructive alignment is designed to ensure that there are clear and logical relations between learning outcomes, teaching methods and assessment methods so that all aspects of the system are in accord in supporting student learning. Constructive alignment is, therefore, a suitable vehicle for promoting higher order thinking about the use of technologies in teaching. The first half of the workshop followed the active learning model and included interactions to encourage participants to think about use of Web 2.0 tools for teaching and learning. Participants were then asked to create a learning design that employed a Web 2.0 tool or tools of choice in a purposeful way. Workshop facilitators supported participants in the use of the tools as the participants worked on their learning designs. At the end of the workshop, participants were asked to reflect on their learning design and to post their thoughts to the social networking environment. Finally, participants were asked to give a short presentation to the group about their learning design. Participants posted reflections along with their presentations and these showed clear evidence of understanding how the Web 2.0 tools might be used in teaching. Results for participants’ experiences and learning outcomes are presented below. Research results We are reporting on the post-workshop evaluation results for the three ETPD designs that we have used over a two-year period. We also report on the number of participants who had put what they learned into practice in their teaching. The results are critically discussed in terms of three evaluation levels for judging the effectiveness of ETPD: satisfaction levels with ETPD; knowledge and skills gained; and changes in teaching practice. Design One The post-workshop evaluation for workshop design 1 used in our first workshop consisted of the standard university evaluation questions. There were 14 (N=14) participants in the first workshop. 9 (N=9) participants agreed to take part in the research. Percentage figures have been rounded to the nearest integer (Table 2). The workshops were evaluated very positively with respect to satisfaction levels with ETPD, content, delivery, and with respect to knowledge and skills gained. With respect to content, 78% (n=7) participants either strongly agreed or agreed that the workshop was intellectually stimulating and 89% (n=8) either strongly agreed or agreed that the workshop materials helped them to learn. In terms of participants’ experiences as a dependent variable we see that whilst 100% responded positively to the statement “The teaching staff showed an interest in my needs during this workshop”, only 44% (n=4) responded positively to the statement, “I received helpful feedback on how I was going in this workshop.” This finding might be explained by the fact that facilitators’ spent more time with some individuals than with others, which accords with the facilitators’ experience that there was a lack of teaching 388 Australasian Journal of Educational Technology, 2011, 27(3) presence during this workshop. Using “This workshop helped to deepen my understanding” as a proxy measure for achieving learning outcomes, 100% (n=9) of respondents either strongly agreed or agreed that the workshop had helped to deepen their understanding. However, the statement itself provides for something of a blunt measure even though it was taken from the university question bank. For example, we do not know how participants’ understanding deepened. Nor do we know whether participants understood some parts of the workshop better than others. This is something that we addressed by revising our evaluation instrument for the second iteration of the workshops. Table 2: Post-workshop evaluation for Workshop 1 (N=9) SA=Strongly agree; A=Agree; N=Neutral; D=Disagree; SD=Strongly disagree SA % A % N % D % SD % I had a clear idea of what was expected of me in this workshop 1 11 6 67 2 22 - - - - The workshop helped motivate me to learn 1 11 8 89 - - - - - - I found the workshop intellectually stimulating 2 22 5 56 2 22 - - - - The workshop materials helped me to learn. 2 22 6 67 1 11 - - - - The volume of work in this workshop was appropriate 1 11 6 67 1 11 1 11 - - The teaching staff showed an interest in my needs during this workshop 2 22 7 78 - - - - - - I received helpful feedback on how I was going in this workshop 3 33 1 11 5 56 - - - - The physical environment of the workshop helped me to learn - - 9 100 - - - - - - This workshop helped deepen my understanding 3 33 6 67 - - - - - - Overall I was satisfied with the quality of this workshop 2 22 7 78 - - - - - - Five (N=5) participants agreed to a follow-up interview during which we addressed the third dependent variable – putting learning into practice – through asking interviewees what they had done since the workshop. Only one participant 20% (n=1) had put what they had learned in practice. This participant had used a blog and wiki in an English language class to engage the students through requiring them to post their own e-journal and comment on others’ journal entries. There are two important points. First, one (n=1) respondent putting what they learned into practice raises the question whether the EPTD was in fact effective because, as we said earlier, ETPD must ultimately impact on teaching practice. However, the workshop was delivered in June and the interviews were carried out in mid September during semester two teaching. Therefore, one explanation for lack of implementation is that participants did not have time to revise their courses prior to the beginning of semester two. That said, we did not rule out that we needed to engage learners at a deeper level in order to improvement implementation rates. None of the participants identified local factors that had facilitated their use of Web 2.0 tools. Finally, three (n=3) interviewees identified barriers to putting what they had learned into practice. one (n=1) participant cited departmental processes for innovating as a barrier as a manager had to be consulted and approve the innovation. One (n=1) participant cited lack of time as a barrier to putting learning into practice. One (n=1) participant cited hardware issues – system settings preventing access to social bookmarking – as a barrier to using the tools. Doherty 389 Design Two The second iteration of our learning design was used in Workshops 2, 3, 4 and 5. As a result of an oversight on the part of administrators, a substantial number of participants at Workshop 2 were general staff members rather than educators and so we will not be reporting evaluation data for that workshop. We had seven (N=7) respondents for Workshop 3, six (N=6) respondents for Workshop number 4, and six (N=6) respondents for Workshop number 5. We are therefore reporting data for a total number of 19 (N=19) respondents. Due to what we assume to be a system error, one response was missing for “I found the workshop intellectually stimulating” and there are, therefore, only 18 responses for that question. Table 3: Post-workshop evaluations for Workshops 3, 4, 5 SA=Strongly agree; A=Agree; N=Neutral; D=Disagree; SD=Strongly disagree SA % A % N % D % SD % I had a clear idea of what was expected of me in this workshop 6 33 11 61 1 6 1 6 - - The workshop helped motivate me to learn 8 42 11 58 - - - - - - I found the workshop intellectually stimulating 3 17 12 66 3 17 - - - - The workshop materials helped me to learn. 4 21 15 79 - - - - - - The volume of work in this workshop was appropriate 1 5 16 84 2 11 - - - - The teaching staff showed an interest in my needs during this workshop 10 53 8 42 1 5 - - - - I received helpful feedback on how I was going in this workshop 5 26 12 63 2 11 - - - - The physical environment of the workshop helped me to learn 5 28 10 55 3 17 1 - - - I could now create and use a blog in my teaching 4 21 12 63 3 16 - - - - I could now create and use a wiki in my teaching 5 26 10 53 4 21 - - - - I could now use social bookmarking in my teaching 1 6 9 50 8 44 1 6 - - The class discussion helped me with my learning 3 16 13 68 3 16 - - - - The small group activities helped me with my learning 2 11 9 47 8 42 - - - - This workshop helped deepen my understanding 6 32 13 68 - - - - - - Overall I was satisfied with the quality of this workshop 10 53 9 47 - - - - - - The workshops were evaluated very positively with respect to satisfaction levels with ETPD, content, delivery and with respect to knowledge and skills gained. With respect to content, 83% (n=15) of participants either strongly agreed or agreed that the workshop was intellectually stimulating and 100% (n=19) either strongly agreed or agreed that the workshop materials helped them to learn. In terms of participants’ experiences as a dependent variable we see that 95% (n=18) responded positively to the statement “The teaching staff showed an interest in my needs during this workshop”, and 89% (n=17) responded positively to the statement, “I received helpful feedback on how I was going in this workshop.” The figure for feedback represents an improvement compared with the first workshop, suggesting that structuring the workshop around triggering interactions helped with time management and with facilitator-participant interactions. 84% of participants (n=16) agreed or strongly agreed that the class discussions helped them with their learning, whilst only 58% of participants (n=11) strongly agreed or 390 Australasian Journal of Educational Technology, 2011, 27(3) agreed that the small group activities helped them with their learning. The small group activities comprised of a series of paired discussions in which participants discussed how they might use the tools in their teaching. Participants were then asked to post their thoughts on the educational uses of the tools to the discussion board. The majority of discussion board postings showed evidence of thoughtfulness with respect to the potential use of the tools and, therefore, the figure of 58% appears anomalous. Using the pre-course evaluation as a baseline, the participants clearly achieved the desired learning outcomes. This means that in terms of the second variable we see that: 84% (n=16) of respondents felt prepared to use blogs in their teaching; 79% (n=15) of respondents felt ready to use wikis in their teaching; and 56% (n=10) of the respondents felt ready to use social bookmarking in their teaching. One explanation for the relatively high number of neutral responses for social bookmarking is that it was taught last in each workshop and time constraints meant that the topic was covered in a relatively hurried fashion. Another possible explanation is that participants did not see the usefulness of social bookmarking for their teaching and so did not engage with social bookmarking in the same way they engaged with blogs and wikis. With reference to the third dependent variable – putting learning into practice – we conducted six (n=6) interviews for Workshop number 3, two (n=2) interviews for Workshop number 4 and four (n=4) interviews for Workshop number 5. This gives us a total of 12 (n=12) interviews from the 19 (N=19) participants who agreed to take part in the research project. The seven (n=7) participants who did not take part did not respond to multiple email/ telephone requests to be interviewed. The interview process consisted of asking the participants set questions whilst providing an opportunity for participants to expand on their answers. The questions covered: use of the Web 2.0 tools since completing the workshop; environmental factors that had either facilitated or hindered their use of the tools; future goals for teaching with the tools; and perceived strengths and weaknesses of the workshop. Only 25% (n=3) of the interviewees had made use of the Web 2.0 tools in their teaching since completing the workshops. This rate is consistent with the literature (Prebble, et al., 2004). One participant had used a wiki to deliver content to a school outreach program; one participant had made use of all three Web 2.0 tools through including information on the tools in their course content for their students who were trainee teachers; one, a librarian, had been teaching others how to use social bookmarking as part of an information literacy course. Only one of the nine (n=9) interviewees who had not made use of the Web 2.0 tools in their teaching provided a clear explanation of how they intended to use the tools – a blog and a wiki – to facilitate collaborative learning. Two interviewees reported that they had decided that Web 2.0 tools would not be useful in their teaching. Enabling factors for those who had made use of the Web 2.0 tools included: technical support either from within their own department or from outside the department (n=1); teaching support at a departmental level for innovating with Web 2.0 tools (n=1); and personal motivation and attitude (n=1). All interviewees were asked about whether they had encountered any barriers to making use of Web 2.0 in their teaching. Four participants cited lack of time or being too busy. Design Three The third workshop design was used in Workshops 6, 7, 8, 9 and 10. Seven educators from workshop number 6 agreed to take part in the research. Six of these participants completed the post-workshop survey. Six participants from Workshop number 7 Doherty 391 agreed to take part in the research and all completed the post-workshop survey. We did not conduct any research for Workshop number 8 because an oversight by administrators meant that a number of non-educators were attending the course. Three educators from Workshop number 9 agreed to take part in the research and all completed the post-workshop survey. There were eight participants for Workshop number 10, six of whom agreed to take part in the research and completed the post- workshop evaluation. This gives a total number of respondents as 21 (N=21). Table 4: Post-workshop evaluations for Workshops 6, 7, 9 and 10 SA=Strongly agree; A=Agree; N=Neutral; D=Disagree; SD=Strongly disagree SA % A % N % D % SD % I had a clear idea of what was expected of me in this workshop 2 10 12 57 7 33 - - - - The workshop helped motivate me to learn 9 43 10 48 2 10 - - - - I found the workshop intellectually stimulating 7 33 11 52 3 14 - - - - The workshop materials helped me to learn. 4 19 15 71 2 10 - - - - This workshop helped deepen my understanding 7 33 12 57 2 10 - - - - The volume of work in this workshop was appropriate 5 24 11 52 2 10 3 14 - - The teaching staff showed an interest in my needs during this workshop 12 57 8 38 1 5 - - - - I received helpful feedback on how I was going in this workshop 9 43 9 43 2 10 1 5 - - The physical environment of the workshop helped me to learn 4 19 10 48 6 29 - - 1 5 The learning design project helped me to understand how to integrate Web 2.0 technologies into my teaching 6 29 10 48 5 23 - - - - Overall I was satisfied with the quality of this workshop 8 38 12 57 1 5 - - - - The workshops were evaluated very positively with respect to satisfaction levels with ETPD, content, delivery and with respect to knowledge and skills gained. With respect to content, 85% (n=18) participants either strongly agreed or agreed that the workshop was intellectually stimulating and 90% (n=19) either strongly agreed or agreed that the workshop materials helped them to learn. These results are comparable with those from the second series of workshops discussed above. In terms of participants’ experiences as a dependent variable we see that 95% (n=20) responded positively to the statement “The teaching staff showed an interest in my needs during this workshop”, and 86% (n=18) responded positively to the statement, “I received helpful feedback on how I was going in this workshop.” These results are comparable with those from the second series of workshops discussed above. Only 77% of respondents (n=16) responded positively to the statement, “The learning design project helped me to understand how to integrate Web 2.0 technologies into my teaching.” 23% (n=5) were neutral concerning the value of the learning design project. However, as we shall see below, on one interpretation of the data the learning design project did prepare educators to use the Web 2.0 tools in their teaching. Respondents (N=21) were asked which of the Web 2.0 tools they had used in their projects, and participants were able to respond in terms of one or more of the tools. Results are as follows. 392 Australasian Journal of Educational Technology, 2011, 27(3) Table 5: Post-workshop evaluations for Workshops 6, 7, 9 and 10 B=Blog; W=Wiki; SB=Social bookmarking; SN=Social networking B % W % SB % SN % Which of the Web 2.0 tools did you use in your project? 7 37 15 80 6 32 4 21 Finally we asked participants to indicate which Web 2.0 tools they felt able to use in their teaching. The results are as presented below. Table 6: Post-workshop evaluations for Workshops 6, 7, 9 and 10 B=Blog; W=Wiki; SB=Social bookmarking; SN=Social networking B % W % SB % SN % I could now make use of Web 2.0 tools in my teaching (check all that apply). 7 37 14 74 6 32 4 21 This means that with one exception the respondents who made use of the respective tools in their learning design project all felt able to use those tools in their teaching by the end of the workshop. Using the pre-course evaluation as a baseline and with respect to the third dependent variable – learning outcomes – this finding suggests that the learning design template prepared participants to use Web 2.0 tools in their teaching. The reason for this is that, with the one exception each time, the participants employed a tool in the learning design for which they indicated preparedness to develop their projects further using the tool in question. Thus, although 23% (n=5) of respondents were neutral concerning the value of the learning design, the learning design project does in fact appear to have prepared workshop participants to use the Web 2.0 tools in their teaching. However, it is also possible that those who were neutral regarding the value of the learning design project were prepared to use the tools, due to other aspects of the workshop such as demonstrations and /or facilitator support. Finally, compared with the second workshop design, we see the vast majority of participants reporting positively on their ability to use a particular tool. This suggests that taking a project-based approach to teaching these Web 2.0 tools enabled participants to make use of only those tools in which they were interested, thereby giving them more time to become familiar with their tool(s) of choice. We conducted a total of seven (N=7) follow-up interviews for Workshops 6, 7 and 10. We did not conduct any interviews for Workshop number 9 because none of those who had agreed to take part in the research responded to our email or telephone requests. The breakdown for the interviews is as follows: two (n=2) interviews were conducted for Workshop number 6; two (n=2) interviews were conducted for Workshop number 7; three (n=3) interviews were conducted for workshop number 10. During the interviews 43% (n=3) of the respondents indicated that they had used the Web 2.0 tools since the workshop. However, in reality only 29% (n=2) out of the seven (N=7) interviewees had actually made use of Web 2.0 tools in their teaching since completing the workshop as one of the interviewees who responded positively had actually set up course evaluations using an online survey tool. We are at a loss to explain this response. Whilst the number of interviews conducted is relatively small, the implementation rate for the third workshop design did improve as compared with workshop designs 1 and 2. Each of the two (n=2) respondents who had made use of the Web 2.0 tools in their teaching was able to explain what they had done with the tool. One participant had Doherty 393 successfully set up a national social network for nurses. Whilst actual participation rates for the wiki were low, the nurse was pleased to have mastered the technical skills required to establish the network. The same respondent was working through a design to incorporate a wiki into an undergraduate course as part of the student assessment. One participant had established a collaborative wiki assessment that required third year undergraduate pharmacy students to work in five different groups to produce a wiki page on their particular topic. The interviewee had also produced a marking rubric for the assessment and the assessment exercise had been delivered successfully. Of the five (n=5) participants who had not made use of the tools in their teaching one respondent provided an explanation concerning their intention to develop introductory course material; one participant was considering ways to help students to collaborate and interact online; one respondent articulated a potential project to use a Web 2.0 tool to help distance students to stay in touch with one another; one respondent indicated a desire to establish a social networking site to enable students to collaborate and share knowledge; and one respondent was in the process of setting up a social networking site for undergraduate students. Thus, whilst these participants had not put what they had learned into practice, they did articulate clear ways in which they might make use of the Web 2.0 tools in their teaching. This fact further supports our judgment that the learning design template better prepared participants to use the tools in their teaching, particularly given the fact that only one of the nine interviewees who had not made use of the Web 2.0 tools in their teaching in the previous workshop iteration provided a clear explanation of how they intended to use the tools to facilitate collaborative learning. Discussion Workshops using active learning techniques and carefully managed interactions can work in terms of delivering a quality learning experience in which participants achieve the desired learning outcomes. Furthermore, a project-based approach to delivering ETPD workshops appears to lead to deeper levels of engagement and to more participants being prepared to use the Web 2.0 tools purposefully in their teaching. Implementation rates across the three workshops were relatively low. This finding is in line with suggestions from the literature (Prebble, et al., 2004). However, interviewees for the third iteration of the workshop who had not implemented their learning designs were able to give a very clear explanation of what they intended to do with Web 2.0 tools in their teaching. This again suggests that the learning design approach – in conjunction with tactics to ensure active learning and triggering interactions – was more effective in bringing participants to understand how they might incorporate Web 2.0 tools purposefully in their teaching. Whilst implementation rates were low, it may be unrealistic to expect all participants to put learning into practice. For example, some participants may have attended simply to be exposed new ideas and these participants may have decided that Web 2.0 tools were not useful for their teaching. We found evidence that some participants decided that Web 2.0 was not something that they wanted to incorporate into their teaching with two (n=2) out of a total of 19 (N=19) interviewees for Workshops 3, 4, 5, 6, 7 and 10 saying that they had decided that the tools would not be useful. Secondly, extraneous factors such as lack of time, research pressure and lack of departmental support impacted negatively on participants’ putting their learning into practice. This suggests that larger issues to do with, for example, university teaching and learning 394 Australasian Journal of Educational Technology, 2011, 27(3) culture need to be considered. We are not addressing these issues in this paper but as a summary point the key question here has to do with the influence that academic developers and their managers can have on teaching policy and practice within universities. For example, can academic develop managers advise and advocate for increased time for teaching development? Implications for practice This study was carried out in an institute of higher education – specifically a university – in New Zealand and our findings should be understood in that context. Project- based workshops have been shown to work particularly well in terms of providing academics with new skills that they might use to deliver their teaching. Workshops are, therefore, an important form of professional development. However, workshops are not a professional development panacea. In particular, workshops do not result in high numbers of participants putting learning into practice. This means that staff developers need to offer alternative professional development opportunities that might more exactly meet the professional development needs of particular academics, whilst also addressing some of the barriers to ETPD, such as lack of time to engage in teaching development and the dominance of the research culture in research-intensive universities. Embedded professional development suggests itself as being particular useful to complement workshops (Prebble, et al., 2004; Timperley, et al., 2007). In a recent paper Whitehouse (2011) has drawn attention to the fact that professional development which is embedded in classroom practices can be effective in changing teaching beliefs and behaviours. Embedded professional development provides a context within which staff developers can attend to the specific needs and objectives of a particular group or particular individual in their own academic and teaching setting. Although there is some research supporting the effectiveness of embedded professional development, the evidence remains limited and initiatives in this area will need to be evaluated in order to demonstrate their effectiveness. Conclusions We have critically discussed three learning designs for ETPD workshops and presented evaluation data for the workshops. We suggested that whilst the refinements to our learning design did not result in an increase in workshop participants putting what they learned into practice, a project-based workshop approach did better prepare educators to use Web 2.0 tools in their teaching. Finally, we discussed alternative professional development models along with the fact that factors outside our control – e.g. lack of educator time – impact on whether or not workshop participants do actually implement what they learn. References Anderson, P. (2007). What is Web 2.0? Ideas, technologies and implications for education. Joint Information Systems Technology and Standards Watch. http://www.jisc.ac.uk/media/documents/techwatch/tsw0701b.pdf Atherton, J. S. (2005). Learning and teaching: Bloom's taxonomy. [viewed 30 Aug 2008, verified 17 May 2011] http://www.learningandteaching.info/learning/bloomtax.htm Doherty 395 Biggs, J. (2008). Aligning teaching for constructing learning. Focus, 16(1), 1-3. [verified 17 May 2011] http://learningandteaching.dal.ca/focus2/Vol16No1pp1-3.pdf Bloom, B. S. (1956). Taxonomy of educational objectives: Handbook I - The cognitive domain. New York: Longman. Chalmers, L. & Keown, P. (2006). Communities of practice and professional development. International Journal of Lifelong Education, 25(2), 139-156. DOI: 10.1080/02601370500510793 Clark, D. R. (2008). Learning domains or Bloom's Taxonomy. [viewed 30 Aug 2008, verified 18 May 2011] http://www.nwlink.com/~Donclark/hrd/bloom.html Collins, A., Joseph, D. & Bielaczyc, K. (2004). Design research: Theoretical and methodological issues. The Journal of Learning Sciences, 13(1), 15-42. Desimone, L. M. (2009). Improving impact studies of teachers’ professional development: Toward better conceptualizations and measures. Educational Researcher, 38(3), 181-199. Doherty, I. (2010a). Staff development workshops for teaching with Web 2.0 tools. In C. D. Maddux, D. Gibson & B. Dodge (Eds.), Research highlights in information technology and teacher education 2010 (pp. 61-70). Chesapeake, VA: SITE. Doherty, I. (2010b). A learning design for engaging academics with online professional development modules. Journal of Learning Design, 4(1), 1-14. http://www.jld.qut.edu.au/publications/vol4no1/DohertyJLDvol4no1_000.pdf Doherty, I., Blake, A. & Cooper, P. (2009). Staff development workshops: Web 2.0 for teaching and learning. Paper presented at the Society for Information Technology and Teacher Education 20th International Conference, Charleston, South Carolina, USA. Doherty, I. & Cooper, P. (2009). Educating educators in the purposeful use of Web 2.0 tools for teaching and learning. In Same places, different spaces. Proceedings ascilite Auckland 2009. http://www.ascilite.org.au/conferences/auckland09/procs/doherty.pdf Dunlap, J. C., Sobel, D. & Sands, D. I. (2007). Designing for deep and meaningful student-to- content interactions. Tech Trends, 51(4), 20-31. Fink, L. D. (2006). Active learning. [viewed 6 Jun 2008 at http://www.ou.edu/pii/tips/ideas/ model.html, not accessible 18 May 2011, try http://honolulu.hawaii.edu/intranet/ committees/FacDevCom/guidebk/teachtip/active.htm] Guskey, T. R. (2000). Evaluating professional development. Thousand Oaks, California: Corwin Press Inc. Herrington, J. & Oliver, R. (2000). An instructional design framework for authentic learning environments. Educational Technology Research & Development, 48(3), 23-48. Hutchins, H. (2003). Instructional immediacy and the seven principles: Strategies for facilitating online courses. Online Journal of Distance Learning Administration, 6(3). http://www.westga.edu/~distance/ojdla/fall63/hutchins63.html Jones, P. (2007). When a wiki is the way: Exploring the use of a wiki in a constructively aligned learning design. In ICT: Providing choices for learners and learning. Proceedings ascilite Singapore 2007. http://www.ascilite.org.au/conferences/singapore07/procs/jones-p.pdf Laurillard, D. (2008). The teacher as action researcher: Using technology to capture pedagogic form. Studies in Higher Education, 33(2), 139-154. Oh, E. & Reeves, T. (2008). Design research vs. instructional systems design: Implications for educational technologists. Paper presented at the World Conference on Educational Multimedia, Hypermedia and Telecommunications 2008, Vienna, Austria. Prebble, T., Hargraves, H., Leach, L., Naidoo, K., Suddaby, G. & Zepke, Z. (2004). Impact of student support services and academic development programmes on student outcomes in undergraduate tertiary study: A synthesis of the research. Auckland: Massey University College 396 Australasian Journal of Educational Technology, 2011, 27(3) of Education. [verified 18 May 2011] http://www.educationcounts.govt.nz/publications/tertiary_education/5519 Reeves, T. C. (2000). Enhancing the worth of instructional technology research through “design experiments” and other development research strategies. Paper presented at the Annual Meeting of the American Educational Research Association. http://it.coe.uga.edu/%7Etreeves/AERA2000Reeves.pdf Reeves, T. C. (2006). Design research from a technology perspective. In J. V. D. Akker, S. Gravemeijer, S. McKenny & N. Nieveen (Eds.), Educational design research (pp. 52-66). London: Routledge. Reeves, T. C., Herrington, J. & Oliver, R. (2002). Authentic activities and online learning. In Quality conversations. Proceedings of the 25th HERDSA Annual Conference. Perth, Western Australia, 7-10 July 2002. http://www.herdsa.org.au/wp- content/uploads/conference/2002/papers/Reeves.pdf Reeves, T. C., Herrington, J. & Oliver, R. (2005). Design research: A socially responsible approach to instructional technology research in higher education. Journal of Computing in Higher Education, 16(2), 96-115. http://it.coe.uga.edu/~treeves/EDIT9990/JCHEDesignResearch05.pdf Rourke, L., Anderson, T., Garrison, D. R. & Archer, W. (1999). Assessing social presence in asynchronous text-based computer conferencing. Journal of Distance Education/ Revue de l'Éducation à Distance, 14(2), 50-71. http://www.jofde.ca/index.php/jde/article/view/153/341 Sandoval, W. A. (2004). Developing learning theory by refining conjectures embodied in educational designs. Educational Psychologist, 39(4), 213-223. Schön, D. A. (1987). Educating the reflective practitioner. San Francisco: Jossey-Bass. Tan, S. C., Hu, C., Wong, S. K. & Wettasinghe, C. M. (2003). Teacher training on technology- enhanced instruction – a holistic approach. Educational Technology and Society, 6(1), 96-104. http://www.ifets.info/journals/6_1/tan.html Timperley, H., Wilson, A., Barrar, H. & Fung, I. (2007). Teacher professional learning and development: Best evidence synthesis. Wellington, New Zealand: Ministry of Education. http://www.educationcounts.govt.nz/publications/series/2515/15341 Wayne, A. J., Yoon, K. S., Zhu, P., Cronen, S. & Garet, M. S. (2008). Experimenting with teacher professional development: Motives and methods. Educational Researcher, 37(8), 469-479. Whitehouse, P. (2011). Networked teacher professional development: The case of Globaloria. Journal of Interactive Learning Research, 22(1), 139-165. http://www.editlib.org/p/31421 Author: Dr Iain Doherty Director, Learning Technology Unit, Faculty of Medical and Health Sciences The University of Auckland Email: i.doherty@auckland.ac.nz Web: http://www.fmhs.auckland.ac.nz/faculty/ staffct/staff_details.aspx?staffID=69646F68303031 Please cite as: Doherty, I. (2011). Evaluating the impact of educational technology professional development upon adoption of Web 2.0 tools in teaching. Australasian Journal of Educational Technology, 27(3), 381-396. http://www.ascilite.org.au/ajet/ajet27/doherty.html