peeraer.pdf Australasian Journal of Educational Technology 2012, 28(Special issue, 6), 1039-1056 The limits of programmed professional development on integration of information and communication technology in education Jef Peeraer Flemish Association for Development Cooperation and Technical Assistance, Vietnam Peter Van Petegem University of Antwerp In the framework of a development cooperation program on quality of education in Vietnam, a professional development trajectory for teacher educators on the use of information and communication technology (ICT) in education was developed and implemented over the course of a three-year program. We describe how the framework on Technological Pedagogical Content Knowledge inspired the development of the program. In a longitudinal panel study, we followed a group of 1191 teacher educators (the “total accumulated popluation”). We describe the trainees’ reaction to training and learning outcomes and we compare the use of ICT by educators who participated in the program with those who did not participate. Data was collected from self-reported questionnaires and focus group interviews. Comparison of the use of ICT for teaching over the course of the program shows that an ICT core group of enthusiasts made significant progress in integrating ICT into teaching practice. Exploratory multiple regression analysis highlights technological pedagogical knowledge and membership of a community of practice as significant factors for improved use of ICT. Nevertheless, the elements of the professional development program explain only 26.2% of variance. We discuss the extent to which programmed professional development can have an effect on integration of ICT in education and reflect on how to make training programs more effective. Introduction Over the last decade, more and more critical voices have been raised concerning the impact of information and communication technology (ICT) on education (Becker, 2000; Kirkup & Kirkwood, 2005; Selwyn, 2007). Uptake of ICT in teaching practice remains limited and not much change seems to have taken place in classrooms. Much research is therefore done on factors influencing or constraining the use of ICT for teaching practice. One important factor is the preparation of future teachers. Culp, Honey and Mandinach (2005) found that teachers’ professional development has been identified repeatedly as a top priority in education policies and is the most important aspect of ICT integration. As pre-service teachers have a significant role to play in the sustained application of ICT in schools, it is imperative that they be exposed to effective use of ICT in their training in teacher education institutions (TEIs) (Steketee, 2005). Nevertheless, in 1040 Australasian Journal of Educational Technology, 2012, 28(Special issue, 6) teacher education the use of ICT for teaching and learning is also generally low (Drent & Meelissen, 2008). Most teacher educators, even in developed countries, are ill- prepared to teach with the new ICTs (Russell et al. in Ng, Miao & Lee, 2010). As a result, pre-service teachers have limited exposure to effective use of ICT in their training. As argued by Lim, Chai and Churchill (2011), it is therefore imperative to build the capacity of TEIs to provide a meaningful context that allows pre-service teachers to critically examine their own pedagogical beliefs, and explore the application of ICT in a more constructivist learning environment. Lim et al. (2011) identify the professional development of teacher educators as an important strategic action for the capacity building of TEIs. In the framework of a development cooperation program on improving the quality of education in Vietnam, we designed a professional development program on integration of ICT in education for Vietnamese teacher educators. The specific objective of the program was to improve the use of ICT by teacher educators, to ensure that future teachers are more exposed to effective use of ICT for teaching and learning. Peeraer and Van Petegem (2011b) show that over a decade, an extended body of guidelines and policies has been put in place in Vietnam, addressing several aspects of integration of ICT in education. Furthermore, impressive progress on improving access to ICT has been made, as is shown in the ICT Development Index (International Telecommunication Union, 2009). A report of the South-East Asian Ministers of Education Organization (SEAMEO, 2010) mentions that the policy guidelines and ICT infrastructure and resources in schools may have provided the necessary and sufficient conditions for some schools to transform their ICT-mediated teaching and learning practices. In teacher education, findings of the authors (Peeraer & Petegem, 2011a, 2011b) show that access to ICT is indeed no longer a barrier. But, at the same time, the findings make clear that even though Vietnamese teacher educators have good basic ICT skills, they do not feel confident in using ICT, and ICT is used mostly to replace traditional teaching practices. In this context, a professional development program has been developed for five Vietnamese TEIs. These institutions were selected by the Ministry of Education and Training for participation in the program. The professional development program on integration of ICT was developed in the second half of 2008 and implemented in 2009-2010. Within the total accumulated population of 1191 teacher educators, 505 participated in the training program. Jung (2005) argues that a well-designed professional development program is essential to meet the demand of today’s educators who want to learn how to use ICT effectively for their teaching. In general, professional development programs on ICT integration are often criticised for their lack of effectiveness (Brinkerhoff, 2006; Triggs & John, 2004). Nevertheless, there are few published evaluations of programs on integration of ICT, and it can be argued that more research is needed to document the effects of teacher development efforts on classroom practices (McDougall, 2008). In what follows, we therefore describe the development of the professional development program for Vietnamese teacher educators which was inspired by the framework on Technological Pedagogical Content Knowledge (TPCK) of Koehler, Mishra and Yahya (2007) and the four-step knowledge construction process as described by Angeli and Valanides (2009). The purpose of our study is to evaluate the effectiveness of the professional development program and to explore the impact of the different elements of the program on the teaching practice of trainees. Explaining the variance between what occurred (or what did not occur) and the consequent impact should be part of any educational research project (Marshall & Cox, 2008). For evaluation of the training Peeraer and Van Petegem 1041 program, we use Kirkpatrick’s framework of four ‘levels’ of criteria. This framework has provided a straightforward system or language for talking about training outcomes, and the kinds of information that can be provided to assess the extent to which training programs have achieved certain objectives (Bates, 2004). In short, the model delineates four levels of training outcomes: reaction, learning, behaviour and results (Kirkpatrick & Kirkpatrick, 2006). The mixed methods that are applied to assess the professional development program are described in a methodology section. After presenting our findings, we discuss the implications for development of future professional development programs. A professional development program for Vietnamese teacher educators In 2008 a core group of 179 ICT enthusiasts (the “ICT core group”) was established from the five participating TEIs, consisting of self-selected teacher educators who were: 1) interested in the use of ICT in education, 2) motivated to learn about ICT, and 3) willing to share and collaborate with peer teacher educators. Although participation was voluntary, nevertheless, we ensured that the group contained a representation of different subjects being taught in the respective TEIs. A step-by-step professional development trajectory was developed, inspired by the four-step knowledge construction process as described by Angeli and Valanides (2009). This process reveals the trajectory of novices’ learning related to unknown and ill-defined tasks, such as designing integration of ICT in teaching and learning. As argued by Angeli and Valanides (2009), after gathering initial information related to instructional design, students need to engage in real-world authentic tasks; share, discuss and reflect with others to eliminate uncertainty; and finally, discuss with an expert. The program that was developed for the Vietnamese teacher educators addresses these elements in three major steps: input on TPCK; follow up practice; and additional engagement with the topic (see Figure 1). The kind of professional development required depends on the nature of the adoption targeted for ICT in the curriculum (Law, 2008). To initiate the development of the training program, we therefore followed the “Technology Mapping” methodology as described by Angeli and Valanides (2009). Technology mapping is a situated methodology where educators establish connections among the affordances of a tool (software), content and pedagogy. We listed a series of ICT applications and their educational purposes. In a discussion with a group of Vietnamese teacher educators and experts in the field of ICT integration, we ordered these ICT applications from less innovative to more innovative, reflecting stages of development of ICT use in education (Mioduser, Nachmias, Tubin & Forkosh-Baruch, 2003; Pilkington, 2008; UNESCO, 2004, 2005). The discussion led to a technology map representing teacher educators on a continuum ranging from educators who use ICT to replace and enhance teaching and learning, to educators who use ICT to innovate, and up to educators who use ICT to transform teaching and learning (see Appendix 1). Input on TPCK Technology mapping builds on an analytical theoretical framework developed by Koehler et al. (2007) in which they advocated increased attention to the complex interplay among technology, content, and pedagogy. They argued that technology integration in teaching and learning requires understanding the dynamic, transactional 1042 Australasian Journal of Educational Technology, 2012, 28(Special issue, 6) relationship among these three knowledge components. Based on the developed technology map, two training modules have been designed to address the interplay among technological, pedagogical and content knowledge, targeting the stage of ICT development where educators use ICT to enhance and innovate teaching and learning. In a module “ICT for Active Teaching and Learning”, seven generic ICT enhanced instructional designs were introduced where the focus was on student-centred teaching methodologies. This training has been organised for a mix of subject matter teacher educators (the ICT core group) in their respective TEIs. A module on “ICT for specific subjects” followed up on this initial module, explicitly addressing the interplay between technological and content knowledge of the participants. Different workshops for different subject matter groups in the ICT core group were organised off-campus. Experts from the respective TEIs were invited to share their insights and experience with using ICT for teaching and support of student learning in their respective subjects. The objective of both modules was for trainees to gather initial information related to instructional design. Follow-up practice Professional development goes beyond knowledge input (cognitive capacity). Koehler et al. (2007) emphasised that in a training program, trainees need to be engaged in rich design activities in order to understand the interrelationships among content, pedagogy, and technology. Therefore, in the second step, the teacher educators were engaged to carry out real-world authentic tasks, as in the knowledge construction process suggested by Angeli and Valanides (2009): they had to design a lesson plan of an ICT integrated lesson, a design activity in which they had to reflect on the interrelationships among content, pedagogy, and technology. Angeli and Valanides (2009) commented that as teachers become more expert in designing learning with technology, technology integration efforts will move away from teacher-directed practices to more learner-centred approaches. Guided by a lesson plan template, teacher educators had to prepare for a 50 minute teaching session. As an incentive, trainees who submitted a lesson plan received a 2GB USB drive. All teacher educators who submitted a lesson plan received feedback from an ICT in education expert. These lesson plans were shared with peers from the respective TEIs, and these peers were invited to observe, discuss, and reflect on the lesson. A lesson observation and assessment tool had been designed to facilitate peer reflection. To conclude the training program, all participants were invited to take a final test to reinforce the initial training on “ICT for Active Teaching and Learning”. All members of the initial ICT core group who completed the full training program received a certificate which allowed them to disseminate the input training to colleagues and pre-service students in their respective TEIs. Additional engagement Apart from cognitive abilities and practice, Law (2008) highlighted the importance of metacognitive ability as an autonomous learner to identify problems and knowledge gaps, to monitor and review one’s own professional learning, and to assess the extent to which the problems are resolved or targeted goals are achieved. Angeli and Valanides (2009) argued that educators must be trained in ways in which they themselves constitute part of a larger learning and professional community for the purpose of exchanging perspectives, resolving dilemmas, and confronting uncertainty Peeraer and Van Petegem 1043 in transforming classroom practice. Pedagogical experimentation and sharing of experiences in a professional network of innovators will set up a reinforcing, iterative cycle of learning in the different domains (Law, 2008). During a final workshop on “ICT for sharing and collaboration”, ICT core group members were therefore encouraged to set up online communities of practice (CoPs), to become involved in continued professional development, to participate in extra training and workshops, and to carry out action research on the topic of ICT integration. Research questions The first objective of our explorative research is to assess the impact of the professional development program on different outcome levels of professional development. For evaluation of the program, we used Kirkpatrick’s framework of four ‘levels’ of criteria. The distinction in the model between learning and behaviour has drawn increased attention to the importance of the learning transfer process in making training truly effective (Bates, 2004). For our research, we focused on level one of the evaluation model (Kirkpatrick & Kirkpatrick, 2006), measuring reaction to training; on level two, measuring quantifiable indicators of learning that have taken place during the course of the training; and on level three, measuring the extent to which knowledge and skills gained in training are applied (behaviour change) (Figure 1). The second objective is to obtain insights into the importance of the three elements of the professional development program, as described above, and their impact on behaviour change (level three outcomes) of the participants. Figure 1: Elements of the professional development program and their impact on behaviour change Methodology Studies and measurement tools to investigate the ICT integration level of higher education institutions are scarce (Akbulut, 2009). In this study, we wanted to monitor 1044 Australasian Journal of Educational Technology, 2012, 28(Special issue, 6) the reactions of participants to the professional development program (level 1), the impact of professional development on learning (level 2), and trainees’ use of ICT for teaching and support of student learning (level 3). Christensen and Knezek (2008) argued that competencies, defined in terms of behaviours, could be reasonably assessed by observation as well as by self-reporting. Typically, research conducted within a behaviourist perspective will use quantitative methods and questionnaires, designed to provide evidence at the points in time of program practices, features and outcomes (Marshall & Cox, 2008). In this study, we therefore used self-reporting questionnaires to assess respondents’ learning and use of ICT for teaching and support of student learning, complemented by focus group interviews with trainees in the program to monitor reaction to the training. We then used descriptive and explorative techniques to analyse the collected data. Data collection In this panel research, we followed the total accumulated population of 1191 teacher educators from the five participating TEIs over the entire course of the professional development program (Table 1). Of these, 505 teacher educators took part in at least the input workshops on TPCK. We collected quantitative baseline data for the total population on their use of ICT in teaching practice in the beginning of school year 2008-2009, and repeated this measurement at the beginning of 2010. One year later, the 505 teacher educators who participated to some extent in the program were once more invited to complete a self-reporting questionnaire on their use of ICT for teaching and support of student learning. At the same time, these trainees were required to answer additional questions assessing their knowledge on integration of ICT and their engagement with the topic. All data can be linked together on the level of the individual teacher educator as respondent codes were assigned from the start of the panel research. To complement the quantitative self-reported data, by the end of school year 2009- 2010, focus group interviews with ICT core group members took place. Focus group sessions in the five TEIs were organised to interview all of the 67 teacher educators who by that time had completed the full training program and were, as such, certified to provide similar training in turn to their colleagues. We used a semi-structured topic list to collect their reactions to the training program, as well as their perceptions on the elements of the program. Table 1: Chronological overview data collection Timing Study Tools Response rate September 2008 Baseline study Self-reporting questionnaire 1a 899 (75.5% of accumulated population of 1191) February 2010 Follow up #1 baseline study Self-reporting questionnaire 1b 933 (78.3% of accumulated population of 1191) June 2010 Focus group interviews Semi-structured topic list 67 (100% of all certified trainers) February 2011 Follow up #2 baseline study and assessment training program Self-reporting questionnaire 2 392 (77.6% of 505 trained teacher educators) Peeraer and Van Petegem 1045 Quantified variables self-reporting questionnaire Participation in professional development program Over the course of the program, data on participation in the different steps of the program has been collected through systematic administration of participation lists. This data allows singling out the teacher educators who participated in input on TPCK, those who also took the reinforcement test, and ICT core group members who were engaged in follow-up steps for the designed program. Additionally, in the questionnaire of 2011, there were extra questions on participation in other professional development initiatives, indicating additional engagement. Respondents were asked whether they had participated in other workshops, training or conferences; whether they had carried out action research on integration of ICT in education; and whether they were members of an online CoP on ICT in education. Learning Schmidt et al. (2009) developed and validated a self-reporting questionnaire to collect data on teachers’ self-assessment of the seven knowledge domains within the TPCK framework. For our research, we selected a 5-point Likert scale on technological pedagogical knowledge (TPK). TPK refers to teachers’ knowledge of how various technologies can be used in teaching, as well as their understanding that using technology may change the way that an individual teaches (Schmidt et al., 2009). Behaviour change The most robust evidence of ICT use enhancing students’ learning is from studies which focused on specific uses of ICT, and clearly identified the range and type of ICT use (Cox & Abbot in Marshall & Cox, 2008). To measure the use of ICT in teaching practice, we used an instrument developed by the authors (Peeraer & Van Petegem, 2012). They developed two sets of items that make up a unidimensional self-reporting measurement instrument which answers the need to analyse ICT-based innovations on a continuum, ranging from the assimilation level through the transition level, and up to the transformation level. The first set of questions comprised a 4-point Likert scale on educators’ self-reported use of ICT for teaching purposes. This set was used for data collection at the start of the training program (2008). The set consisted of items which gradually address more innovative ICT applications for teaching practice. High scores indicate a more regular use of ICT for teaching practice. For the follow-up surveys in 2010 and 2011, a second set of questions on educators’ perceived use of ICT for support of student learning was added to the questionnaire. As described by the authors (Peeraer & Van Petegem, 2012), this selection of items focuses on innovative uses of ICT by students as defined in Drent and Meelissen (2008), addressing generalisable ICT strategies (Quellmalz & Kozma, 2003), and different educational purposes (UNESCO, 2003). Reliability of the TPK questionnaire scale and the instrument measuring self-reported use of ICT for teaching and support of student learning was examined with Cronbach’s alpha. Both instruments showed satisfactory internal consistency with a Cronbach’s alpha value of .83 and .90 respectively. Access to ICT To control for institutional support, we also used self-reporting questionnaires to examine access to ICT at the level of individual teacher educators. Access to computers is measured using the list of ICT Core Indicators developed by the Partnership on 1046 Australasian Journal of Educational Technology, 2012, 28(Special issue, 6) Measuring ICT for Development (UNCTAD, 2005, 2007). Access to computers can vary: teacher educators may, or may not, have access to a computer and when they do have access this may be for their own individual use, or to share with others. In addition, the teacher educators were asked as well about access to a data projector for use in teaching practice. Analysis With regard to the first research aim, descriptive analyses of participation data and data on learning (level 2) and behaviour outcomes (level 3) were applied to draw a picture of the effectiveness of the professional development program. We compared the mean score in 2008 and in 2010 on the use of ICT by educators who participated in the program with those who did not participate. Reaction to training (level 1) was described based on the interview transcripts from the focus group interviews with trained ICT core group members in 2010. With regard to the second research aim, exploratory multiple regression analysis was applied on the questionnaire data of 2011 to analyse the influence of the described elements of the professional development program and measured at the level of the individual teacher educator on the use of ICT for teaching and support of student learning (level 3). A hierarchy of sets of independent variables was developed and tests were carried out on the significance of increments to R2 by means of the F ratio. The hierarchy of sets is an important part of the hypothesis statement (Cohen, 1968). In our research, we wanted to explore the importance of three major elements of the professional development program that was designed for Vietnamese teacher educators. For the analysis, we started with the variables on access to ICT (set 1), which was an indication of institutional support in the TEIs. These variables were expected to be relevant to the dependent variable, but had to be controlled for when measuring the contribution of the elements contained in the professional development program. After controlling for access to ICT, we wanted first to assess the importance of input on TPCK and learning and therefore included participation in the reinforcement test on “ICT for Active Teaching and Learning” (set 2). All respondents who took the reinforcement test had attended input training on TPCK. In addition, we included respondents’ TPK in the model (set 3). Second, we wanted to assess the net influence of engagement in follow-up practice by including in the analysis the dummy variable of membership in the ICT core group (set 4). These ICT core group members had been engaged in authentic design activities as well as expert and peer assessment. Third, we wanted to assess the net influence of additional engagement with integration of ICT in education by including dummy variables on participation in other workshops, research on integration of ICT and membership in online CoPs (set 5). Findings Participation In 2008, we started with an ICT core group of 179 teacher educators. This ICT core group participated in the first step of the training program and was encouraged to continue their learning trajectory by participation in the follow-up steps. Only 67 teacher educators (37.4%) of this original ICT core group completed all of the necessary steps to become expert practitioners on “ICT for Active Teaching and Learning” (see Table 2). In this regard, the requirement to teach the prepared ICT integrated lesson Peeraer and Van Petegem 1047 plan under observation and assessment of invited peers proved to be a critical selection criteria. Almost all ICT core group members participated in the training on “ICT for sharing and collaboration”. The 67 teacher educators who completed all of the necessary steps to become expert practitioners on “ICT for Active Teaching and Learning” were certified to disseminate the training program to colleagues in their TEIs. By the beginning of 2011, an extra 326 teacher educators had completed the first step of the program by participation in dissemination workshops, and 293 of them had taken the ICT for Active Teaching and Learning reinforcement test. None of these teacher educators had yet submitted an ICT integrated lesson plan or organised a peer review session in the context of the training program. For these trainees, there was no training organised on “ICT for sharing and collaboration”. Table 2: Program participation Step 2008-2010 (ICT core group) 2010-2011 (Dissem- ination) 2008-2011 (Total accumu- lated numbers) Training on ICT for Active Teaching and Learning (ATL)/ICT for specific subjects 179 326 505 ICT for Active Teaching and Learning reinforcement test 146 293 439 Development of an ICT integrated lesson plan 113 nil 113 Teaching practice under peer review 72 nil 72 Training on ICT for sharing and collaboration 166 nil 166 In what follows, we first describe the ICT core group’s reactions to the professional development program as well as the learning outcomes and behaviour changes (research aim 1). Thereafter, we explore how the different elements of the professional development program impacted on the use of ICT for teaching and support of student learning (research aim 2). Reactions to training, learning and behaviour changes Reactions to training (level 1) From the focus group interviews with those educators who completed the full training program, it was clear that this was a group of ICT enthusiasts who were keen to learn, as well as to share their knowledge, about integration of ICT in education. As one participant noted, for example: I am interested in taking part in whatever is difficult. I like applying ICT in my life and in teaching. I am satisfied when I can learn something. Friends and colleagues have commented that my command of ICT is very good. When I cannot do as expected, I feel disappointed with myself. (focus group in TEI 4) Most participants found the step by step professional development program very inspiring as it gave them an idea about how to apply ICT in teaching. One core group member commented that she really wanted to get access to ICT and to teach about it to other colleagues in her institution. When organising dissemination training for colleagues, ICT core group members observed an eagerness to learn about ICT. Teacher educators seemed to be very excited to take part in training sessions and appreciate expert input. For example, as one of them observed: 1048 Australasian Journal of Educational Technology, 2012, 28(Special issue, 6) After the training, I have different views about applying ICT in teaching; I learned from the specialists and could always ask for help when it came to difficulties. The step by step training is very good: I learned something from each step. (focus group in TEI 5) Most ICT core group members showed high appreciation for the input. As one stated: After the training, I gained insights into applying ICT in teaching. I really hope that we can improve the teaching quality and that we can continue progressing, even after the program finishes. (focus group in TEI 2) However, some interviewees realised that there was something more than just the input training, as exemplified by the statement from one of them: I don’t think we have fully applied what we were trained to do. It takes time. (focus group in TEI 2) Therefore, participants recommended building in more time for practice to improve skills on integration of ICT in teaching: I took part in the workshops, but I did not get a lot of practice. Therefore, it was very easy to forget. There was also a lot of information that was difficult to remember. If possible, we prefer to learn the theory before the practice and then have more time to practice. Learning is a continuous process, and it would be a waste of time if we do not have enough opportunity to practice. (focus group in TEI 3) In addition, the focus group interviewees identified some difficulties. The main obstacle to participating in the step by step program was the work overload resulting in lack of time. Some institutions assisted in time management of training sessions and follow-up activities, but most trainees complained that, after the input sessions, it was difficult to find the time to organise self-study on the tools that had been introduced. As noted by one of them: During the workshop I understood the material presented, but, after that, I did not have time to read and follow-up. The materials are useful, but we don’t really spend enough time studying them. (focus group in TEI 1) Most of the trainees also stated that they lacked time and opportunities to practice what they had learned. The two-day workshop did not prove to be sufficient to go beyond an introduction to the tools, and, apart from developing the required ICT integrated lesson plan, some core group members could not find the time to practice and apply what they had learned. In some cases, this resulted in decreased enthusiasm and motivation. Thus, one trainee mentioned: During the training I thought I knew how to apply ICT in my teaching, but, once back home, I could not do it. That’s when I felt discouraged. (focus group in TEI 2) Learning (level 2) By the end of the training program, all teacher educators who participated in it felt very confident in applying ICT in classroom teaching (see Table 3). More than 90% agreed or strongly agreed that they could choose technologies that enhance teaching and students’ learning, that they could adapt the use of technology to different teaching activities, and that they were thinking critically about their use of technology in the classroom. On average they fully agreed that the training program caused them to think more deeply about how technology could influence the teaching approaches that they had used in the classroom. Peeraer and Van Petegem 1049 Table 3: Learning: Descriptive statistics TPK (N = 392; 77.6% of trained teacher educators) M SD Strongly disagree (%) Disagree (%) Neither disagree nor agree (%) Agree (%) Strongly agree (%) I can choose technologies that enhance the teaching approaches for a lesson. 4.13 .467 0.0 0.3 4.3 77.3 18.1 I can choose technologies that enha- nce students’ learning for a lesson. 4.06 .489 0.0 0.5 7.4 77.3 14.8 The training program has caused me to think more deeply about how tech- nology could influence the teaching approaches I use in my classroom. 4.05 .485 0.0 0.5 7.9 77.8 13.8 I am thinking critically about how to use technology in my classroom. 4.06 .470 0.0 0.0 8.2 77.6 14.3 I can adapt the use of technologies that I am learning about to different teaching activities. 4.06 .473 0.0 0.3 7.4 78.1 14.3 Behaviour change (level 3) During the focus group interviews, different trainees stated that they were using new tools in their teaching practice. As one, remarked: I used Mind Mapping in my teaching practice. My students told me to keep on using it. (focus group in TEI 1) But these ICT enthusiasts also realised that it would take more time to make a real change. One interviewee commented as follows: After developing a lesson plan and reflection on teaching methodologies and integration of ICT, this approach needs to be really implemented, and not just “performed” to attract the students’ attention. (focus group in TEI 1) In Table 4, we compared the means for different uses of ICT for teaching practice at the start of the program (2008) and after finalisation of the program for the ICT core group (2010). In 2008 and 2010, we could compare the mean scores of the original ICT core group members with all of the other teacher educators who were not engaged in the initial training of trainers. Table 4: Behaviour changes: Use of ICT in 2008 and 2010, respectively, for ICT core group members and other teacher educators ICT core group 2008 - 2010 Other teacher educators TEIs 2008 - 2010Use of ICT for teaching purposes N Mean(2008) Mean (2010) Diff. N Mean (2008) Mean (2010) Diff. Production of documents 116 2.29 2.34 n.s. 517 1.94 1.97 n.s. Presentation 118 1.92 2.36 -.432*** 516 1.44 1.72 -.287*** Integration into subjects 118 1.54 1.81 -.271** 508 1.17 1.39 -.222*** Access offline information 116 1.90 1.91 n.s. 512 1.53 1.77 -.240*** Electronic communication 117 1.15 1.78 -.624*** 514 1.04 1.46 -.420*** Access online information 118 2.32 2.40 n.s. 505 1.86 2.14 -.279*** Classroom management 117 0.50 0.70 -.197* 511 0.46 0.61 -.155** Note: Significance paired differences paired samples test (2-tailed): * < .05, ** < .01, *** < .001 1050 Australasian Journal of Educational Technology, 2012, 28(Special issue, 6) The use of ICT to produce documents (like lesson plans produced with word processing) did not improve, and ICT core group members did not use ICT to access offline or online data more often. ICT core group members, as well as other teacher educators, used ICT more often for presentation purposes, for integration into subject teaching, and for electronic communication with students. ICT core group members were already using ICT more frequently and in more diverse ways at the start of the program in 2008. Nevertheless, the differences between the two measurements (for 2008 and 2010, respectively) were larger for the ICT core group members for certain types of ICT integration. Impact of various professional development program elements Hierarchical multiple regression analysis resulted in five models, each of which significantly explained a proportion of variance in the use of ICT for teaching and support of student learning (see Table 5). Table 5: Elements of the professional development program influencing the use of ICT for teaching and support of student learning: Standardised regression coefficients, proportion of explained variance and F change (N = 392; 77.6% of trained teacher educators) Program element Model1 Model 2 Model 3 Model 4 Model 5 Access to laptop .175*** .171** .138** .122** .105*Access to ICT Access to data projector .149** .145* .099* .094* .098* Workshop and reinforcement test - .103* n.s. n.s. n.s.Input on TPK (element 1) TPK - - .355*** .341*** .285*** Follow-up (element 2) ICT core group member - - - .123** n.s. Other input (workshops/training) - - - - n.s. Research on topic - - - - .099* Additional engagement (element 3) Online community of practice - - - - .244*** Adjusted R2 ‘Use of ICT in teaching practice’ .055 .063 .184 .197 .262 F change 12.421 9.812 23.067 20.154 18.387 df1 2 3 4 5 8 sig. F change .000 .000 .000 .000 .000 Notes: - not included in model; n.s. not significant; *p < .05, **p < .01, *** p < .001 The element of access to ICT alone explains only 5.5% of the variance, but the importance of having access to a computer and projector remains significant in all models. In all models, TPK has the most significant influence on the use of ICT for teaching and support of student learning (β from .355 to .285). All respondents who took the “ICT for Active Teaching and Learning” reinforcement test had been trained on “ICT for Active Teaching and Learning” during initial training for the core group or in dissemination workshops. Before adding TPK to the model, participation in this training and taking the reinforcement test as such has an influence on use of ICT. Membership in the original ICT core group proves to have some influence (β = .123), even when controlling for TPK. Research on the topic, as well as participation in an online community of practice, have a positive influence on ICT use in education. The final model explains 26.2% of variance in the dependent variable. The factors contributing the most to the explained variance in the final model remain TPK and Peeraer and Van Petegem 1051 participation in an online community of practice with standardised beta coefficients of .285 and .244 respectively. During the focus group interviews in 2010, ICT core group members highlighted that support from the education institution and the management team was the key to the success of the training program. To overcome time burdens and work overload, some TEIs assist with time management of the training program. A difference was observed between provisions of access to ICT. In one institution, a package of equipment provided in the context of the development cooperation program was supplemented with whole school access to teaching and learning aids, as mentioned during the focus group interviews by one teacher educator. In other institutions, trained teacher educators were not even aware about the availability of equipment for classroom teaching. In another institution, integration of ICT in education was taken very seriously. The TEI provided incentives for faculty to develop ICT integrated lessons and created an online platform to share these lessons on the institution’s website. In most institutions, the ICT core group continued to operate as a group of specialists on integration of ICT in teaching and learning, with the responsibility to reach out to colleagues and peers in different departments. Some core group members did set up online platforms to connect teachers in order to share with, collaborate with, and learn from each other. During the focus group interviews, some trainees recommended supporting more interaction among education institutions to disseminate the training program widely. In another institution, teaching staff were requested to share developed materials online. Different institutions had organised conferences on ICT, as well as additional training and dissemination workshops. Most training was perceived as supplementing the initial input provided, even though some interviewees shared some doubts about the effectiveness of all of the training. Some institutions provided financial incentives and reimbursement of transportation expenses to take part in training sessions. Out of 392 respondents who completed the final questionnaire in 2011, 171 (43.6%) took part in other ICT workshops, training or conferences; 117 (29.8%) carried out action research on the topic; and 140 (35.7%) were members of an online community of practice on integration of ICT. Discussion and conclusions Even though there is no intention for this study to generalise from the described professional development program, some lessons related to professional development can be learned. Only about one third of the initial ICT core group members (37.4%) completed all steps and followed up on administration. Of all of the teacher educators who participated in dissemination workshops, no one submitted a lesson plan or invited peers to observe their lesson. ICT core group members who did complete the full training program are truly enthusiastic; nevertheless, they realise that there is a need for more time to practise. Work overload, time constraints and limited institutional support mean that trainees often do not follow up on the training input provided to them. Nevertheless, the training program has been effective in the sense that, by the end of the program in 2011, all trainees who took part in the input training on TPCK felt very confident and knowledgeable about the interplay between technology and pedagogy. After completion of all steps of the program in 2010, ICT core group members’ use of ICT for teaching practice increased and improved significantly, especially in respect of the use of ICT for communication with students, 1052 Australasian Journal of Educational Technology, 2012, 28(Special issue, 6) for integration in subject teaching, and for presentation purposes. Even though other teacher educators who did not participate in the training program improved as well, the progress of the ICT core group members has been significantly larger. The findings of the impact assessment show that TPK is a significant factor contributing to the use of ICT for teaching and support of student learning. This indicates that the learning transfer process is effective. On the other hand, it is very clear that there is more to professional development than participation in a well- designed training program. Participation in input training and completion of a reinforcement test as such is not significant in explaining improved use of ICT. Also, the elements of the training program explain only a small proportion of the variance in the use of ICT for teaching and support of student learning. Moreover, contextual factors like access to ICT resources for integration in classroom teaching remain important, and it is clear that institutional support and additional individual engagement also are critical. In earlier research, the authors (Peeraer & Van Petegem, 2011a) have highlighted other important factors that influence the use of ICT for teaching in teacher education in Vietnam, such as teacher educators’ basic and advanced computer skills, ICT confidence, and the institution where they are teaching. As the purpose of the current study was to assess the importance of the elements of the designed training program, we did not include these factors in the regression analysis. The training program that was developed and implemented in five TEIs in Vietnam attracted a group of enthusiasts who volunteered to engage in a hands-on learning trajectory on integration of ICT in their teaching practice. Thus, the program engaged a group of motivated teacher educators who were eager to learn and share amongst peers. Even at the start of the program, they were using ICT more regularly than other teacher educators. It can be concluded that success in getting the most out of training is not so much about participation in workshops and training, but more about an individual’s drive and additional engagement. From the focus group interviews with ICT core group members, we understand that, to succeed, it takes an educator who is “interested in taking part in whatever is difficult”, and who is “satisfied when learning something new”. This attitude goes beyond knowledge and skills and underpins a metacognitive ability to identify problems and knowledge gaps, to monitor and review one’s own professional learning, and to assess the extent to which the problems resolved or targeted goals are achieved as an autonomous learner (Law, 2008). Our concern is that staff training and development achieves little if it is only for the few who volunteer. The same people turn up time after time, and it is a case of “preaching to the converted”. Ways must be found to provide training system wide (Latchem & Jung, 2010). As argued by Phelps, Graham and Kerr (2004), a metacognitive approach can also foster the formation of support structures and networks which could support educators’ learning beyond their involvement in the professional development initiative and, as such, it becomes a powerful vehicle to support change processes. CoPs receive special attention related to capacity development on integration of ICT in education. Using ICT to support CoPs in which practice can be shared as a basis for mutual professional learning seems to offer the best hope for providing teacher educators and teachers with opportunities for continuing development (Albion, Knezek & Adubra, 2011). Our findings show that participation in an online CoP is indeed a significant predictor for improved use of ICT for classroom teaching. Albion et al. (2011) acknowledge that there is still much to be learned about how best to initiate and support such learning Peeraer and Van Petegem 1053 communities and the change to the professional culture that is needed to facilitate their success. Our findings show that more than one third of participants in the professional development program participated in online CoPs. Trained teacher educators also took responsibility to involve their colleagues in professional development on integration of ICT in education. At the same time, ICT core group members showed a high appreciation for expert input from specialists. As argued by others (Barton & Haydn, 2006; Karagiorgi & Charalambous, 2006; Latchem & Jung, 2010), we believe that it may be best to combine programmed professional development addressing TPCK of teacher educators with incentives for additional engagement with the topic, especially as encouragement for ICT enthusiasts to exchange with and encourage peers. The Asia-Pacific region has the advantage of a relative youthful population, which makes the education sector more receptive to harnessing newer (and cheaper) forms of educational ICT (Lee, Hung & Cheah, 2008). Apart from supporting staff in providing access to ICT for integration in education, TEIs ideally support a culture of social and collaborative professional development to harness the full potential of their resources. References Akbulut, Y. (2009). Investigating underlying components of the ICT indicators measurement scale: The extended version. Journal of Educational Computing Research, 40(4), 405-427. http://dx.doi.org/10.2190/EC.40.4.b Albion, P., Knezek, D. & Adubra, E. (2011). Teacher professional development. Paper presented at the EDUsummIT 2011. http://edusummit.nl/res2011/calltoaction2011/briefpapers2011; also at http://eprints.usq.edu.au/19566/ Angeli, C. & Valanides, N. (2009). Epistemological and methodological issues for the conceptualization, development, and assessment of ICT-TPCK: Advances in technological pedagogical content knowledge (TPCK). Computers & Education, 52(1), 154-168. http://dx.doi.org/10.1016/j.compedu.2008.07.006 Barton, R. & Haydn, T. (2006). Trainee teachers' views on what helps them to use information and communication technology effectively in their subject teaching. Journal of Computer Assisted Learning, 22(4), 257-272. http://dx.doi.org/10.1111/j.1365-2729.2006.00175.x Bates, R. (2004). A critical analysis of evaluation practice: The Kirkpatrick model and the principle of beneficence. Evaluation and Program Planning, 27(3), 341-347. http://dx.doi.org/10.1016/j.evalprogplan.2004.04.011 Becker, H. J. (2000). Findings from the teaching, learning, and computing survey: Is Larry Cuban right? Education Policy Analysis Archives, 8(51). http://epaa.asu.edu/ojs/article/view/442/565 Brinkerhoff, J. (2006). Effects of a long-duration, professional development academy on technology skills, computer self-efficacy, and technology integration beliefs and practices. Journal of Research on Technology in Education, 39(1), 22-43. http://www.eric.ed.gov/ERIC WebPortal/contentdelivery/servlet/ERICServlet?accno=EJ768867 Christensen, R. & Knezek, G. (2008). Self-report measures and findings for information technology attitudes and competencies. In J. Voogt & G. Knezek (Eds.), International handbook of information technology in primary and secondary education (Vol. 20, pp. 349-365): Springer US. Cohen, J. (1968). Multiple regression as a general data-analytic system. Psychological Bulletin, 70(6), 426-443. http://psycnet.apa.org/doi/10.1037/h0026714 Culp, K. M., Honey, M. & Mandinach, E. (2005). A retrospective on twenty years of education technology policy. Journal of Educational Computing Research, 32(3), 279-307. http://dx.doi.org/10.2190/7W71-QVT2-PAP2-UDX7 1054 Australasian Journal of Educational Technology, 2012, 28(Special issue, 6) Drent, M. & Meelissen, M. (2008). Which factors obstruct or stimulate teacher educators to use ICT innovatively? Computers & Education, 51(1), 187-199. http://dx.doi.org/10.1016/j.compedu.2007.05.001 International Telecommunication Union (2009). Measuring the Information Society - The ICT Development Index. Geneva, Switzerland: International Telecommunication Union. http://www.itu.int/ITU-D/ict/publications/idi/material/2009/MIS2009_w5.pdf Jung, I. (2005). ICT-pedagogy integration in teacher training: Application cases worldwide. Educational Technology & Society, 8(2), 94-101. http://www.ifets.info/journals/8_2/8.pdf Karagiorgi, Y. & Charalambous, K. (2006). ICT in-service training and school practices: In search for the impact. Journal of Education for Teaching: International Research and Pedagogy, 32(4), 395- 411. http://dx.doi.org/10.1080/02607470600981995 Kirkpatrick, D. L. & Kirkpatrick, J. D. (2006). Evaluating training programs: The four levels. Berrett- Koehler. Kirkup, G. & Kirkwood, A. (2005). Information and communications technologies (ICT) in higher education teaching - a tale of gradualism rather than revolution. Learning, Media and Technology, 30(2), 185-199. http://dx.doi.org/10.1080/17439880500093810 Koehler, M. J., Mishra, P. & Yahya, K. (2007). Tracing the development of teacher knowledge in a design seminar: Integrating content, pedagogy and technology. Computers & Education, 49(3), 740-762. http://dx.doi.org/10.1016/j.compedu.2005.11.012 Latchem, C. R. & Jung, I. (2010). Distance and blended learning in Asia. New York: Routledge. Law, N. (2008). Teacher learning beyond knowledge for pedagogical innovations with ICT. In J. Voogt & G. Knezek (Eds.), International handbook of information technology in primary and secondary education (Vol. 20, pp. 425-434): Springer US. Lee, Y.-J., Hung, D. & Cheah, H.-M. (2008). IT and educational policy in the Asia-Pacific region. In J. Voogt & G. Knezek (Eds.), International handbook of information technology in primary and secondary education (Vol. 20, pp. 1119-1132). New York: Springer. Lim, C. P., Chai, C. S. & Churchill, D. (2011). A framework for developing pre-service teachers' competencies in using technologies to enhance teaching and learning. Educational Media International, 48(2), 69-83. http://dx.doi.org/10.1080/09523987.2011.576512 Marshall, G. & Cox, M. J. (2008). Research methods: Their design, applicability and reliability. In J. Voogt & G. Knezek (Eds.), International handbook of information technology in primary and secondary education (Vol. 20, pp. 983-1002). New York: Springer. McDougall, A. (2008). Models and practices in teacher education programs for teaching with and about IT. In J. Voogt & G. Knezek (Eds.), International handbook of information technology in primary and secondary education (Vol. 20, pp. 461-474). New York: Springer. Mioduser, D., Nachmias, R., Tubin, D. & Forkosh-Baruch, A. (2003). Analysis schema for the study of domains and levels of pedagogical innovation in schools using ICT. Education and Information Technologies, 8(1), 23-36. http://dx.doi.org/10.1023/A%3A1023922207476 Peeraer, J. & Van Petegem, P. (2011a). Information and communication technology in teacher education in an emerging developing country: Vietnam's baseline situation at the start of 'The Year of ICT'. Computers & Education, 56(4), 974-982. http://dx.doi.org/10.1016/j.compedu.2010.11.015 Peeraer, J. & Van Petegem, P. (2011b). Information and communication technology in teacher education in Vietnam: from policy to practice. Educational Research for Policy and Practice, 11(2), 89-103. http://dx.doi.org/10.1007/s10671-011-9106-9 Peeraer and Van Petegem 1055 Peeraer, J. & Van Petegem, P. (2012). Measuring integration of information and communication technology in education: An item response modeling approach. Computers & Education, 58(4), 1247-1259. http://dx.doi.org/10.1016/j.compedu.2011.12.015 Phelps, R., Graham, A. & Kerr, B. (2004). Teachers and ICT: exploring a metacognitive approach to professional development. Australasian Journal of Educational Technology, 20(1), 49-68. http://www.ascilite.org.au/ajet/ajet20/phelps.html Pilkington, R. M. (2008). Measuring the impact of information technology on students’ learning. In J. Voogt & G. Knezek (Eds.), International handbook of information technology in primary and secondary education (Vol. 20, pp. 1003-1018). New York: Springer. Quellmalz, E. S. & Kozma, R. (2003). Designing assessment of learning with technology. Assessment in Education: Principles, Policy and Practice, 10(3), 389-408. http://dx.doi.org/10.1080/0969594032000148208 Schmidt, D. A., Baran, E., Thompson, A. D., Mishra, P., Koehler, M. J. & Shin, T. S. (2009). Technological pedagogical content knowledge (TPACK): The development and validation of an assessment instrument for preservice teachers. Journal of Research on Technology in Education, 42(2), 123-149. http://www.iste.org/store/Product.aspx?ID=1133 SEAMEO (2010). Status of ICT integration in education in Southeast Asian countries. Bangkok: The Southeast Asian Ministers of Education Organization (SEAMEO). [verified 6 Aug 2012] http://www.icde.org/filestore/Resources/Reports/SEAMEO_ICT-Integration-Education2010.pdf Selwyn, N. (2007). The use of computer technology in university teaching and learning: a critical perspective. Journal of Computer Assisted Learning, 23(2), 83-94. http://dx.doi.org/10.1111/j.1365-2729.2006.00204.x Steketee, C. (2005). Integrating ICT as an integral teaching and learning tool into pre-service teacher training courses. Issues in Educational Research, 15(1), 101-113. http://www.iier.org.au/iier15/steketee.html Triggs, P. & John, P. (2004). From transaction to transformation: information and communication technology, professional development and the formation of communities of practice. Journal of Computer Assisted Learning, 20(6), 426-439. http://dx.doi.org/10.1111/j.1365-2729.2004.00101.x UNCTAD (2005). Core ICT indicators. http://new.unctad.org/upload/docs/Core%20ICT% 20Indicators_Eng.pdf UNCTAD (2007). Revisions and additions to the core list of ICT indicators. http://unstats.un.org/unsd/statcom/doc09/BG-ICTIndicators.pdf UNESCO (2003). Consultative workshop on performance indicators for ICT in education. Bangkok: UNESCO Asia and Pacific Regional Bureau for Education. http://www.unescobkk.org/ fileadmin/user_upload/ict/e-books/ICTindicators/ICTindicators.pdf UNESCO (2004). Integrating ICTs into education: Lessons learned. Bangkok: UNESCO Asia and Pacific Regional Bureau for Education. http://www.unescobkk.org/?id=1793 UNESCO (2005). Regional guidelines on teacher development for pedagogy-technology integration. Bangkok: UNESCO Asia and Pacific Regional Bureau for Education. http://unesdoc.unesco.org/images/0014/001405/140577e.pdf Ng, W.-K., Miao, F. & Lee, M. (2010). Capacity-building for ICT integration in education. Digital Review of Asia Pacific 2009-2010, 67-76. http://www.digital-review.org/uploads/files/pdf/ 2009-2010/capacity_building.pdf 1056 Australasian Journal of Educational Technology, 2012, 28(Special issue, 6) Appendix 1: Construct map integration of ICT in teaching and learning ↑ Increasing innovativeness of use of ICT for teaching and learning Educators Use of ICT applications … Educators who use ICT to transform teaching and learning practice. Classroom management systems for coaching and evaluation, web search assignments for problem based learning; Electronic communication with students and students communicating with others. Educators who use ICT to innovate teaching towards more student centred learning, integrated into existing curriculum. Integration into subject teaching (word processing for shared writing exercises, simulations and data processing by students); Student presentations and students creating multimedia products, students constructing and synthesising knowledge. Educators who use ICT to enhance teaching practice. Presentation software for enhancing, multimedia presentations; Accessing offline and online databases and information to prepare resources; Electronic practice and drill exercises for revision. Educators who use ICT to replace traditional teaching practice. Word processing for production of documents (preparation of lesson plans, handouts, slides, …); Presentation software for lecturing. ↓ Decreasing innovativeness of use of ICT for teaching and learning Authors: Jef Peeraer Thao Vien Guesthouse A3 rm 307, 1B Bac Son-Ngoc Ha, Hanoi, Vietnam Email: jef.peeraer@gmail.com Web: http://www.vvob.be/vietnam/ Jef Peeraer works for the Flemish Association for Development Cooperation and Technical Assistance (VVOB) as adviser on M&E and integration of Information and Communication Technology (ICT). The focus of his research is on education change processes in general and on integration of ICT in education in particular. He is supporting teacher education institutions in Vietnam in operationalising their vision on education innovation, and developed and implemented capacity building programs for educators in a variety of learning environments. Professor Dr Peter Van Petegem Venusstraat 35, 2000 Antwerp, Belgium Email: peter.vanpetegem@ua.ac.be Web: http://www.ua.ac.be/main.aspx?c=peter.vanpetegem Peter Van Petegem is full professor of education at the Institute for Education and Information Sciences of the University of Antwerp, Belgium. He is head of the research group Edubron (http://www.edubron.be/) that is focusing on diverse aspects of education such as performance indicators as a tool for school improvement, evaluation of educational innovations, policy making capacities of schools, school effectiveness, and environmental education. Please cite as: Peeraer, J. & Van Petegem, P. (2012). The limits of programmed professional development on integration of information and communication technology in education. In C. P. Lim & C. S. Chai (Eds), Building the ICT capacity of the next generation of teachers in Asia. Australasian Journal of Educational Technology, 28(Special issue, 6), 1039-1056. http://www.ascilite.org.au/ajet/ajet28/peeraer.html