wong.pdf Australasian Journal of Educational Technology 2011, 27(2), 361-379 Framing ICT implementation in a context of educational change: A structural equation modelling analysis Emily M. L. Wong Vine Education Consultancy, Hong Kong Sandy C. Li Hong Kong Baptist University Despite the common belief that information and communication technology (ICT) has the potential to support certain fundamental changes in learning, few have examined ICT implementation conceptually within a wider context of educational change. Methodologically, we are by and large limited to building simple models that accommodate only a single dependence relationship among variables. Framing ICT implementation as a process of interactions among pedagogical and organisational factors in bringing about changes in student learning, this article used data collected from 1076 teachers in 130 schools to construct a structural equation model (SEM), from which we are able to examine multiple interrelated dependence relationships in a single model. Results indicated that from teacher perspectives, the collegial capacity of ICT implementation strategies played a central and mediating role in effecting changes in student learning, of moving away from a teacher-centred approach to one that is more student-centred. Specifically, ICT brought about these changes in the context of establishing collegiality in fostering pedagogical innovations in schools. Implications for both researchers and practitioners are discussed. Introduction Despite frequent attempts to address educational changes and the roles of ICT in effecting changes in student learning, few have acknowledged that effective use of ICT needs to be embedded in a larger process of school change (Balanskat, Blamire & Kefala, 2006). One possible direction is to frame ICT implementation as a dynamic process in which the benefits of ICT on student learning is mediated by pedagogical and organisational factors in a school setting. Methodologically, many of the available statistical tools are limited to building simple models that accommodate only a single dependence relationship among variables. The purpose of the present study is to bridge these research gaps by developing models that conceptually integrate ICT implementation with changes in student learning within a context of managing change in schools. It was hoped that the model can capture the contextual variables of pedagogical and organisational factors, with a view to understanding how ICT acts as a lever to bring about change in student learning. With an ecological perspective, the conceptualisation goes beyond the boundary of simply identifying factors affecting the use of ICT in schools and takes a further step to scrutinise the interplay of the contextual factors (Wong & Li, 2008; Zhao & Frank, 2003). 362 Australasian Journal of Educational Technology, 2011, 27(2) The present study is exploratory in nature, as the research empirically tests alternative theoretical models that conceptualise and unfold the dynamics of ICT implementation in schools. It is also explanatory in nature, as the plausible model(s) seek to explain how ICT implementation strategies interact with organisational intervention and pedagogical intervention, to enhance changes in student learning from a teacher- centred approach to a student-centred approach). Specifically, the study addresses the following questions: 1. In a context of managing change in schools with ICT implementation, what are the factors directly affecting perceived changes in student learning from a teacher- centred approach to a student-centred approach? 2. How do these factors interact with one another to bring about changes in student learning? 3. How does ICT act as a lever in bringing about a change in student learning? Theoretical framework The theoretical framework has been informed mainly by the concept of viewing implementation dynamics as the interplay between organisational issues and personal concerns (Smylie, Miretzky & Konkol, 2004; Van der Vegt, Smyth & Vandenberghe, 2001), and the concept of viewing implementation from the perspective of the learning organisation (Fullan, 1993; Senge, 1990). At the same time, it also has adopted a situative perspective with the unit of analysis as the activity system to frame schoolwide ICT implementation. The situative perspective has been widely applied to knowing, thinking and learning (Brown, Collins & Duguid, 1989; Greeno, 1998; Lave & Wenger, 1991) and policy implementation (Spillane, Reiser & Todd, 2002). With its emphasis on the larger systems in which the principal, teachers, and students interact with each other, the situative perspective has guided us to understand schoolwide ICT implementation as a result of activity between pedagogical intervention and organisational intervention to achieve changes in student learning, with particular attention given to the often ignored social contexts and institutional culture in which teachers are situated (Windschitl & Sahl, 2002). Figure 1 helps to depict the theoretical framework. Implementation dynamics emerges from the elements interacting in the activity system. Guided by the theoretical underpinning that implementation is relevant to the interaction of implementation strategies, the changes achieved, and also the factors that influence these changes (Levin, 2001), we have proposed a theoretical model comprising ICT implementation strategies, perceived changes in student learning (outcome variable), and some key contextual factors. With the emerging trend of defining learning outcomes in terms of understanding and performance (Fullan, 1996), constructivism becomes the common lens that people adopt in examining the effectiveness of using ICT in teaching and learning. Based on the understanding that constructivism stresses the importance of each individual's autonomy as a thinker and the importance of the social context of learning, and thereby emphasising the importance of the learner's initiative (Tiene & Ingram, 2001), this paper has conceptualised perceived changes in pedagogy and learning as changes from a teacher-centred approach to a student-centred approach over a period of two years’ time, which is parallel to changes in classroom practices from a traditional approach to a constructivist approach. Wong and Li 363 Figure 1: Theoretical framework conceptualising ICT implementation in schools Sample and instrument Sample The working population was defined as the public primary and secondary schools in Hong Kong, and all of them were invited to participate in the survey. Out of the working population, a total of 1076 teachers from 130 schools were recruited on voluntary basis, which was about one tenth of the school population. The sample of recruited schools therefore was a convenience sample located in all 18 school districts in Hong Kong. Convenience sampling suffices to address the research questions posed, as the gist of the entire study is not to make generalisations over the entire population in Hong Kong, but to examine the dynamics of interplay of various constructs pertaining to ICT implementation and learning in schools. In terms of distribution according to finance types (government, direct subsidy scheme, and aided), the sample population closely resembled the true population. Primary schools made up 53% of the sample, and secondary schools 47%, the distribution of which was quite similar to the true population (60% primary schools and 40% secondary schools). To increase the representativeness, teachers in each school were selected from diversified subject domains including languages, mathematics, humanities, science, technology, art, and physical education. In each primary school, 10 teachers coming from specified subject Activity system: School ICT implementation strategies Changes in pedagogy Changes in learning School climate Government ICT policy Organisational intervention Pedagogical intervention School leadership External input Outcome variable 364 Australasian Journal of Educational Technology, 2011, 27(2) domains were selected by schools to fill out the questionnaires; and in each secondary school, 9 teachers from specified subject domains were selected by schools to fill out the questionnaire. Of the 1076 teachers, 56.3% were primary school teachers, while 43.7% were secondary school teachers. The sample consisted of relatively more female participants (61.8%). Ages ranged from 18-25 years to over 50 years, with the majority (35.4%) at the age of 31-40, and the minority (7.8%) over 50. Teaching experience of the sample was of the range of below 3 years to over 20 years. While the majority (39.1%) had 4-10 years of teaching experience, nearly half of the sample population (42.8%) had a teaching experience of over 10 years. Details are shown in Table 1. Table 1: Demographic data of the sample teachers Demographic Frequency Percent (%) 18-25 134 12.5 26-30 309 28.8 31-40 380 35.4 41-50 166 15.5 Over 50 84 7.8 Age Total 1073 100 0–3 years 195 18.1 4–10 years 421 39.1 11-15 years 209 19.4 16–20 years 97 9.0 Over 20 years 154 14.3 Teaching experience Total 1076 100 Male 411 38.2 Female 664 61.8 Gender Total 1075 100 Survey instrument Generating an initial set of items for the survey instrument is an important first step to attaining construct validity, and therefore we have given considerably more attention to the conceptual definition of the instrument. Deductive scale development has been a common approach for creating preliminary items to assess the constructs under examination, in which a theoretical foundation is expected to provide enough information to generate the initial set of items (Hinkin, 1998). With the approach of deductive scale development guided by a thorough literature review and expert opinions, we defined conceptually the initial set of items for the survey instrument on a theoretical basis that would indicate the content domains for each construct (see Table 3). As it was not possible to measure the complete domain of interest, we adopted the domain sampling theory, which aimed at generating a sample of items that adequately represented the construct under examination (Hinkin, 1998). The set of items was then assessed subjectively by experts to ensure content validity (Hair et al., 1998); Hinkin, 1998). A small-scale pilot test was then conducted to refine the instrument, mainly in terms of clarity in wording, before the questionnaire administration. The survey instrument for studying ICT implementation in schools consisted of the following six scales: (1) perceived transformational leadership, (2) perceived climate for collaboration and experimentation, (3) collegial capacity of school’s ICT implementation strategies, (4) perceived changes in teacher pedagogy, (5) perceived changes in student learning, and (6) perceived effectiveness of government ICT policy. Wong and Li 365 On average, each of the six scales was made up of about 6 items, with a total of 39 items (ultimately reduced to 33 items) on the whole, as shown in Table 3. All items were measured on a 4-point Likert scale ranging from “strongly disagree” to “strongly agree”. This type of ordinal scales measuring the intensity of feeling toward the item generates more information than dichotomous scoring, and it facilitates statistical analysis by more faithfully reflecting the individual differences on the attribute (Nunnally & Bernstein, 1994). On the other hand, statistical analyses of ordinal scales are usually based on a Pearson product-moment correlation (PPM) matrix that assumes the input data as variables of continuous scale. In fact, variables based on ordinal response modes from Likert scale measurements actually depart from the representation of a continuous scale (Joreskog & Sorbom, 2001). To overcome the limitation, the present study used a polychoric correlation matrix as the input matrix for confirmatory factor analysis and structural equation modelling (Flora & Curran, 2004; Joreskog & Sorbom, 2001). Assuming that there is a continuous variable underlying each ordinal variable, polychoric correlations are not correlations computed from actual scores but are rather estimated theoretical correlations of the underlying continuous variables (Joreskog & Sorbom, 2001). Regarding the issue of estimation methods in SEM, diagonally weighted least squares (DWLS) was used as it does not make assumptions about the distribution of the observed variables (Diamantopoulos & Siguaw, 2000; Flora & Curran, 2004), and it was shown to perform better than WLS in small sample sizes in terms of chi-square statistics, parameter estimates, and standard errors. Instrument validation Construct validity was a core issue here as the present study relied heavily on the use of questionnaires as the primary means of data collection (Hinkin, 1998), and the present study also aimed at theory development and testing. At the outset, while the Cronbach’s alpha reliability test identified the 6 constructs as internally consistent, a preliminary exploratory factor analysis (EFA) has found out that the factors of leadership and climate were not clearly separable. However, a more rigorous structural equation-based approach of confirmatory factor analysis (CFA) subsequently provided strong evidence in supporting the constructs as unidimensional in representing a single concept (Anderson & Gerbing, 1988). Of the CFA model, the composite reliability and variance-extracted for each construct provided further support of convergent validity (Hair et al., 1998) (Table 2). The chi- square difference test and its complementary assessment method as suggested by Anderson & Gerbing (1988) provided strong evidence of discriminant validity. Despite this, the variance extracted against squared correlation test as suggested by Fornell and Larcker (1981) found that “leadership against climate” as well as “changes in pedagogy against changes in learning” were not clearly separable from one another. The average variance extracted for these two pairs of constructs was smaller than their corresponding correlation (Table 4). On such considerations, modifications were subsequently made to the constructs for leadership, climate, changes in pedagogy, and changes in learning. With a total deletion of 9 items, 3 of which were deleted after conducting EFA, and 6 of which were deleted after conducting discriminant validity checks, the modified measure subsequently contained 33 items (see Table 3). 366 Australasian Journal of Educational Technology, 2011, 27(2) Table 2: Parameter estimates for the CFA measurement model of the 6 constructs (with values of the modified measure in parentheses) Constructs Indicators Factorloading t-value Composite reliability Variance extracted Ld1 .82(.82) 24.02(23.45) Ld2 .81(---) 25.82(---) Ld3 .74(.74) 26.98(26.29) Ld4 .56(.58) 13.13(14.26) Ld5 .61(.62) 17.84(18.18) Ld6 .80(.80) 35.48(32.18) Leadership Ld7 .73(.75) 25.91(25.67) .89(.89) .53(.54) Cm1/ Ld8 .80(.79) 34.58(35.46) Cm2 .83(---) 40.02(---) Cm3 .75(.84) 27.11(30.78) Cm4 .65(.73) 21.28(22.05) Climate Cm5 .64(.72) 18.24(18.78) .85(.85) .54(.59) Gp1 .78(.78) 26.65(27.37) Gp2 .83(.83) 34.44(35.80) Gp3 .68(.69) 18.60(19.39) Gp4 .66(.68) 19.03(21.15) Gp5 .85(.85) 34.77(36.47) Gp6 .68(.70) 18.07(19.25) Gp7 .81(.80) 29.36(29.45) Government ICT policy Gp8 .78(.77) 24.52(24.97) .92(.92) .58(.58) Is1 .66(.63) 17.05(16.23) Is3 .64(.64) 18.19(18.03) Is5 .71(.68) 19.45(18.60) Is6 .65(.64) 18.43(18.50) Is7 .69(.72) 17.90(20.31) ICT implement strategies Is8 .69(.72) 18.85(22.22) .83(.83) .45(.45) Pd1 .66(---) 15.62(---) Pd2 .58(---) 14.02(---) Pd3 .64(---) 16.53(---) Pd4 .60(.70) 15.23(15.61) Pd5 .57(.69) 13.73(14.04) Changes in pedagogy Pd6 .65(.81) 19.56(21.41) .79(.78) .38(.54) Ln1 .79(---) 33.33(---) Ln2 .87(.87) 37.55(36.50) Ln3 .83(.86) 38.01(37.42) Ln4 .77(.77) 29.15(28.69) Ln5 .72(.72) 27.53(26.55) Ln6 .81(.83) 34.56(33.77) Changes in learning Ln7 .80(.81) 28.86(28.62) .93(.92) .64(.66) The modified measure for the most part demonstrated unidimensionality, reliability, convergent validity (Table 2), and discriminant validity (Table 4). Results from the structural equation-based approach of CFA and discriminant validity checks indicated that the values all reached an acceptable level indicating construct validity. However, the Cronbach’s alpha for the construct of changes in pedagogy (.68) was slightly below the value of .70 as recommended for newly developed measures (Hinkin, 1998). In addition, the average variance extracted for the construct of ICT implementation strategies (.45) was also slightly below the recommended value of .50 (Fornell & Larcker, 1981). These limitations were, however, compensated by the validity of the scales strongly demonstrated in the dimensions of unidimensionality, composite reliability, convergent validity, and discriminant validity. Wong and Li 367 Table 3: Items of the questionnaire for the various scales Scale Item Questions Ld1 My educational beliefs are reflected in the school goals. Ld2 I feel comfortable to talk to the principal about school matters. Ld3 The principal trusts me with school matters. Ld4 Teachers have autonomy to make decisions relevant to their teaching. Ld5 The principal encourages me to experiment with new ideas in classroom practice. Ld6 My school provides adequate resources to support staff professional development. Leadership Ld7 The principal values staff professional development. Cm1 I have a sense of belonging to my school. Teachers in our school: Cm2 work in a collegial manner. Climate Cm3 are willing to experiment with new ideas in classroom practice. The initiative: Gp1 provides adequate professional development for teachers. Gp2 provides adequate technical support for schools. Gp3 enables schools to have autonomy in allocating or recruiting resources relevant to ICT. Gp4 provides adequate hardware facilities for schools. Gp5 provides adequate support on curriculum resources for teachers. Gp6 provides sound network infrastructure for schools. Gp7 enables students to learn more effectively. Government ICT policy Gp8 helps to strengthen teachers’ quality of teaching. Is1 My school has a mechanism to disseminate the experiences of using ICT resources for teaching and learning. Is2 My school has mobilised resources from external parties (e.g. parents/ alumni/ other schools/organisations) to help to implement ICT in teaching and learning. Is3 Teachers' opinions can be conveyed to ICT policy-making bodies effectively within school. Is4 Colleagues in my school exchange experiences of using ICT to enhance teaching and learning. I participate in sharing sessions for exchanging experiences (of using ICT to enhance teaching and learning) with: Is5 teachers from other schools. ICT implemen- tation strategies Is6 educators from tertiary institutions. Compared to the past two academic years, I am: Pd1 creating more opportunities for discussions to develop students' expressive and analytical abilities. Pd2 encouraging students to explore and to inquire in learning more. Changes in pedagogy Pd3 providing more opportunities for students to determine their learning activities Compared to students of the same level in the past two academic years, my students: Ln1 are more independent in their learning. Ln2 are more active in constructing knowledge. Ln3 are able to make better use of collaborative work to facilitate learning. Ln4 have more courage to express ideas in class. Ln5 are more motivated in their learning. Changes in learning Ln6 enjoy learning more. 368 Australasian Journal of Educational Technology, 2011, 27(2) Table 4: Discriminant validity checks by comparing the average variance extracted and the square of correlation Square of correlation (average variance extracted for the pair of constructs) Construct Climate Govt. ICTpolicy ICT strategies Changes in pedagogy Changes in learning Leadership .79 (.53, .54) .52 (.54, .61)* .073 (.53, .59) .062 (.54, .58)* .12 (.53, .44) .12 (.54, .44)* .18 (.53, .39) .10 (.54, .56)* .18 (.53, .64) .16 (.54, .66)* Climate .079 (.55, .59) .072 (.59, .59)* .15 (.55, .45) .11 (.58, .47)* .21 (.55, .39) .08 (.59, .56)* .23 (.55, .65) .16 (.58, .66)* Govt. ICT policy .09 (.59, .44) .084 (.59, .39) .049 (.59, .55)* .10 (.59, .65) .10 (.59, .66)* ICT strategies .24 (.44, .39) .19 (.47, .54)* .19 (.44, .64) .18 (.47, .66)* Changes in pedagogy .56 (.39, .64) .39 (.55, .66)* * Values for the modified measure Model building After validating the instrument, the next step is to find out a plausible model unfolding the ecological dynamics of various contextual factors in ICT implementation (Figure 1). A basic model was proposed and examined with SEM techniques using LISREL 8.54 (Joreskog & Sorbom, 2001). The analysis adopted the two-step approach for analysing the postulated model (Anderson & Gerbing, 1988), where a confirmatory measurement model was specified prior to the simultaneous estimation of the measurement and the structural model. In the process of exploring the most plausible model, a theory-driven approach of model comparison strategy was adopted, in which competing models were generated on the basis of alternative formulations of the underlying theory (Browne & Cudeck, 1993; Hair et al., 1998; MacCallum, 1995). In evaluating the alternative models, consideration was given to three aspects: (1) the overall fit measures based on a number of fit indices, namely root mean square error of approximation (RMSEA) (Browne & Cudeck, 1993), non-normed fit index (NNFI), comparative fit index (CFI) (Marsh, Balla, & Hau, 1996); (2) the component fit measures came from parameter estimates (Bollen, 1989), which included the squared multiple correlation (R2) for each pair of relationships and the t-value of the path coefficients; and (3) the model parsimony based on fit indices of the Akaike Information Criterion (AIC), the Consistent Akaike Information Criterion (CAIC), and the Expected Cross-Validation Index (ECVI) (Joreskog, 1993). The essence of the indices was to examine measures of fit that took the number of parameters in the model into account, thereby the fit of model was not necessarily improved as parameters were added to the model (Joreskog, 1993). Before proceeding to model comparisons, overall fit indices of individual models and component fit measures (parameter estimates and R2) were examined first to check whether any models would be rejected. In other words, overall fit and component fit evaluations were supposed to serve as a preliminary step to screen out unacceptable models. The acceptable models in terms of overall fit and component fit were then compared and evaluated in terms of parsimony indices that take fit as well as parsimony (in the sense of number of parameters) into account. Wong and Li 369 The basic model The basic model represented the hypothesis that the collegial capacity of ICT implementation strategies mediated the effect of organisational interventions on changes in classroom practices (pedagogy and learning). Specifically, “ICT implementation strategies” was proposed as having a direct effect on “changes in learning”, and an indirect effect through “changes in pedagogy” on “changes in learning”. At the same time, “ICT implementation strategies” was proposed as a variable that mediated the effect of “school climate” and “government ICT policy” on “changes in learning”, and “leadership” was proposed as an exogenous variable that related only to “school climate (Figure 2). Figure 2: The basic model examining the pattern of relationship among the constructs in ICT implementation Non-mediating model The non-mediating model was supposed to test against the basic model and other competitive models that were constructed on the basis of theoretical justifications, which supported the mediating effect of organisational variables on changes in student learning. Therefore, in the non-mediating model, all contextual variables were simply proposed as directly related to the outcome variable “changes in learning” (Figure 3). Climatic model The climatic model represented the hypothesis that the climate for collaboration and experimentation mediated the effect of organisational interventions on changes in classroom practices (pedagogy and learning). As a competitive model based on alternative hypothesis, it was proposed that a climate for collaboration and experimentation played a central role in the social ecology of ICT implementation by mediating the effect of other organisational interventions on changes in pedagogy and 0.67 (9.66) 0.79 (18.67) Changes in pedagogy (R2 = 0.29) 0.22 (5.86) 0.49 (6.81) 0.37 (9.07) Changes in learning (R2 = 0.52) ICT implementation strategies (R2 = 0.45) School leadership School climate (R2 = 0.64) Government ICT policy 0.67 (9.66) Note: Path coefficients are shown with t-values in parentheses 370 Australasian Journal of Educational Technology, 2011, 27(2) learning. Specifically, “school climate” was proposed as having a direct effect on “changes in learning”, and an indirect effect through “changes in pedagogy” on “changes in learning”. At the same time, “school climate” was proposed as a variable that mediated the effect of “leadership”, “ICT implementation strategies” and “government ICT policy” on “changes in learning” (Figure 4). Figure 3: The non-mediating model examining the pattern of relationships among the constructs in ICT implementation. Figure 4: The climatic model examining the pattern of relationships among the constructs in ICT implementation 0.12 (3.85) 0.19 (2.91) ICT implementation strategies School climate 0.13 (2.45) 0.60 (9.29) 0.06 (0.90) Changes in pedagogy Changes in learning (R2 = 0.48) School leadership Government ICT policy Notes: Path coefficients are shown with t-values in parentheses School leadership Changes in pedagogy (R2 = 0.23) Changes in learning (R2 = 0.50) 0.48 (7.93) 0.49 (8.88) 0.50 (9.80) 0.16 (3.38) 0.54 (7.65) School climate (R2 = 0.75) ICT implementation strategies Government ICT policy Notes: Path coefficients are shown with t-values in parentheses Wong and Li 371 External model The external model represented the hypothesis that the external input in the form of government ICT policy mediated the effect of organisational interventions on changes in classroom practices (pedagogy and learning). As a competitive model based on alternative hypothesis, it was proposed that the government ICT policy played a central role in ICT implementation by mediating the effect of other organisational interventions on changes in pedagogy and learning. Specifically, “government ICT policy” was proposed as having a direct effect on “changes in learning”, and an indirect effect through “changes in pedagogy” on “changes in learning”. At the same time, “government ICT policy” was proposed as a variable that mediated the effect of “climate” and “ICT implementation strategies” on “changes in learning”. Taking into account the essential role of transformational leadership on shaping school climate, “leadership” was proposed as an exogenous variable that related only to “school climate” (Figure 5). Figure 5: The external model examining the pattern of relationships among the constructs in ICT implementation Pedagogical model The pedagogical model represented the hypothesis that pedagogical intervention mediated the effect of organisational interventions on changes in learning. It has become clear through research evidence that instructional strategy underlying the use of ICT determines learning effectiveness (Honey et al., 2000; McCombs, 2000; Means, 1994; Mehlinger, 1995). As a competitive model based on alternative hypothesis, it was proposed that in the social ecology of ICT implementation, the effect of organisational interventions on changes in learning was mediated entirely by pedagogical intervention. Specifically, “changes in pedagogy” was proposed as the variable that mediated the effect of “climate”, “ICT implementation strategies”, and “government Note: Path coefficients are shown with t-values in parentheses 0.43 (8.02) 0.77 (7.04) Changes in pedagogy (R2 = 0.18) 0.43 (8.27) 0.59 (10.18) 0.30 (7.22) Changes in learning (R2 = 0.49) Government ICT policy (R2 = 0.35) School leadership School climate (R2 = 0.58) ICT implementation strategies 0.44 (7.66) 372 Australasian Journal of Educational Technology, 2011, 27(2) ICT policy” on “changes in learning”. Taking into account the essential role of transformational leadership on shaping school climate, “leadership” was proposed as an exogenous variable that related only to “school climate” (Figure 6). Figure 6: The pedagogical model examining the pattern of relationships among the constructs in ICT implementation Results Overall fit evaluation Table 5 shows the correlation matrix for the variables. Before proceeding to model comparison, overall fit indices of individual models were examined first to check whether any models would be rejected. It can be seen that nearly all the selected fit indices for the models fell within an acceptable range, except the chi-square values for the models and the NNFI (.94) for the external model (Table 6). As the chi-square statistic assumes that the model fits perfectly in the population, and is known to be sensitive to departures from large sample size, we therefore made reference to other indices such as RMSEA, NNFI and CFI as well. Overall, other than the external model, all models had index values indicating reasonably good fit of model data. Table 5: Correlation matrix for the 6 variables in the social ecology of ICT implementation 1 2 3 4 5 6 1. Leadership 1.00 2. Climate .73 1.00 3. Govt ICT policy .25 .27 1.00 4. ICT implementation strategies .37 .37 .31 1.00 5. Changes in pedagogy .31 .30 .23 .43 1.00 6. Changes in learning .41 .44 .32 .43 .62 1.00 1.36 (13.53) 0.81 (19.03) Changes in pedagogy (R2 = 0.41) 0.31 (6.90) 0.11 (3.65) 0.23 (6.43) Changes in learning (R2 = 0.73) ICT implementation strategies School leadership School climate (R2 = 0.61) Government ICT Policy Note: Path coefficients are shown with t-values in parentheses Wong and Li 373 Table 6: Indices comparing the goodness of fit and parsimony for alternative models Goodness of fit index Recommended good fit values Basic model Non-mediating model Climatic model External model Pedagogical model Satorra-Bentler chi- square --- 1597.98 1636.31 1689.54 1769.46 1714.38 Degree of freedom --- 488 480 486 488 487 Chi-square per degree of freedom <3.0 3.27 3.41 3.48 3.63 3.52 P-value for chi- square test >.05 .00 .00 .00 .00 .00 RMSEA <.06 .045 .047 .047 .049 .048 P-value for test of close fit, RMSEA <.05 >.05 1.00 .99 .96 .79 .93 NNFI >.95 .97 .98 .97 .94 .97 CFI >.95 .97 .98 .97 .95 .98 ECVI Smaller value 1.58 1.63 1.67 1.74 1.69 AIC Smaller value 1743.98 1798.31 1839.54 1915.46 1862.38 CAIC Smaller value 2182.54 2284.92 2290.11 2354.02 2306.94 Component fit evaluation Results of the parameter estimates were shown in Figures 2 to Figure 6, and Table 7. Upon examination, it can be seen from Table 7 that the non-mediating model had some unreasonable results. The magnitude of the path coefficients from “leadership” to “changes in learning” was very low (.07) with t-value (.89) less than the minimum value of 1.96, indicating that the corresponding parameter was not statistically significantly at the 5% level. In addition, specific R2 for the external model was relatively low, which indicated that the independent variables explained a small portion of variance in the endogenous variables concerned (Bentler, 2007, Bollen, 1989; Diamantopoulos & Siguaw, 2000; Goffin, 2007, Markland, 2009). Specifically, the effect of “government ICT policy” explained only 18% of variance in “changes in pedagogy”. Overall, results of parameter estimates supported the rejection of the non-mediating model and the external model. Model parsimony evaluation Before proceeding to model parsimony evaluation, the overall fit and component fit evaluations already screened out the external model and the non-mediating model. The other three models (basic model, climatic model, pedagogical model) had a reasonably good fit of model data in terms of RMSEA (<.06), NNFI (>.95), and CFI (>.95) (Bentler, 2007; Hu & Bentler, 1999). Magnitude and t-values of path coefficients (>1.96) as well as R2 (ranging from .23 to .75) for the models were also within the acceptable range (Bollen, 1989; Diamantopoulos & Siguaw, 2000, Markland, 2009). The three models were further compared with reference to selected parsimony indices (ECVI, AIC, and CAIC) that took fit as well as parsimony (in the sense of number of parameters) into account. From Table 6, it can be seen that the values of ECVI (1.58), AIC (1743.98), and CAIC (2182.54) all favoured the basic model, which had the lowest values among the models (Joreskog, 1993). Nevertheless, its magnitude of difference in terms of parsimony indices over the other models was not very significant. As the three competing models were supposed to be approximations to reality, it was worthwhile to investigate similarities among the seemingly different approximations to reality instead of finding the most plausible models. 374 Australasian Journal of Educational Technology, 2011, 27(2) Table 7: Parameter estimates for alternative models Model Structural equations R2 Unreasonable parameter estimates climate = .79 leadership .64 --- strategies = .37 climate + .22 policy .45 --- pedagogy = .67 strategies .29 --- Basic (strategic) learning = .49 pedagogy + .67 strategies .52 --- Non- mediating learning = .06 leadership + .19 climate + .12 policy + .13 strategies + .60 pedagogy .48 Leadership -> learning .06 (t value = .90) climate = .50 leadership + .40 strategies + .16 policy .75 --- pedagogy = .49 climate .23 --- Climatic learning = .54 pedagogy + .48 climate .50 --- climate = .77 leadership .58 --- policy = .58 climate + .43 strategies .32 --- pedagogy = .43 policy .18 “Government ICT policy” explained only 18% of variance in “changes in pedagogy” External learning = .59 pedagogy + .44 policy .48 --- climate = .81 leadership .61 --- pedagogy = .23 climate + .31 strategies + .11 policy .41 --- Pedagog- ical learn = 1.36 pedagogy .73 --- Interpretations Investigating similarities among plausible models The three plausible models shared similarities, from which we discovered some patterns that might give insights into the social ecology of ICT implementation in schools: 1. The models supported the proposition that changes in pedagogy mediated the effect of organisational intervention on changes in learning, though in different patterns. 2. Results of comparison did not give a unanimous answer on the pattern of the mediation effect with regard to changes in pedagogy. The basic model and the climatic model supported a partial mediation effect of changes in pedagogy. In other words, organisational interventions might directly affect changes in learning, or indirectly through changes in pedagogy to affect changes in learning. However, the pedagogical model supported a full mediation effect of changes in pedagogy, of which organisational interventions only indirectly affected changes in learning through changes in pedagogy. 3. The models provided support for the important role of transformational leadership on shaping the climate for collaboration and experimentation, in which the path coefficients from transformational leadership to school climate were high (ranging from .50 to .81). Despite the above differences, the similarities shared among the models of approximations to reality confirmed the critical role of pedagogical intervention in bringing about changes in learning as well as the critical role of transformational leadership in shaping school climate. Wong and Li 375 Discussion and conclusions Framing ICT implementation in a context of managing change in schools, this study has expanded the concept of ICT implementation from mere ICT use to ICT use in the context of pedagogical and organisational interventions. Drawing on the conceptualisation, we have developed a model for deepening our understanding of how ICT acts as a lever to bring about changes in student learning. The rejection of the non-mediating model revealed that, in the social ecology of ICT implementation in schools, it was likely that organisational interventions and pedagogical interventions interacted with each other to affect changes in student learning. Specifically, changes in pedagogy as perceived by teachers were found to be a mediating variable transferring the effect of organisational interventions to the outcome variable “perceived changes in student learning”. Of its own accord, “perceived changes in pedagogy” was found to be a significant predictor of perceived changes in student learning. The findings supported the important role of instructional strategy underlying the use of ICT in determining learning effectiveness. The rejection of the external model also gave us insights into the dynamics of organisational interventions. It highlighted the central role of internal organisational interventions, in comparison with external policy, in affecting changes in student learning. Notably, the collegial capacity of school’s ICT implementation strategies as perceived by teachers was identified as one of the possible focal points of linking up the organisational interventions to the outcome variable “changes in student learning”. Specifically, this study has shown that ICT was able to act as a lever to bring about changes in student learning in the context of establishing collegiality in fostering pedagogical innovations in schools (Figure 2, the basic model). While other organisational interventions were shown to have no direct impact on the outcome variable, they were capable of exerting indirect influences by shaping the collegial capacity of school’s ICT implementation strategies. A transformational leadership had no direct effect on classroom practices (changes in pedagogy and student learning), yet it exerted an indirect effect through shaping the school climate. The study has offered insights into the controversy over the relative importance of changes related to classroom practices and changes related to organisational factors. It has given empirical support to the proposition that attention to organisational changes is essential to changes in classroom practices (Leithwood, 1994). From the empirical results, schools are advised to place more emphasis on enhancing the collegial exchange and sharing of ICT experiences in order to harness the power of ICT in shifting learning from a teacher-centred to a student-centred approach. The significant path coefficient from changes in pedagogy to changes in learning has clear implication on the crucial role of teachers’ pedagogical approach in students’ learning outcome. It supported the postulation that ICT implementation strategies will exhibit significant effect on changes in learning outcome if there is a change effected by ICT on teachers’ pedagogy in classroom practices. The results have been consistent with previous empirical work, which have identified the impact of ICT as highly related to how teachers exploit it efficiently for pedagogical purposes (Balanskat et al., 2006). These empirical findings actually echo the contemporary concepts of educational change, of which ICT is seen as an enabler to reshape the delivery of instruction. As such, greater impact of ICT implementation is achieved if teachers strengthen ICT use with pedagogical strategies. 376 Australasian Journal of Educational Technology, 2011, 27(2) In addition, empirical evidence from the present study supported that a transformational leadership was highly influential in establishing a cohesive and proactive school climate that enhanced the collegial capacity of ICT implementation strategies. It alerts school administrators and policymakers to the positive effect of cultivating collegiality in schools upon enhancing changes in student learning, which at the same time also offers insights into the appropriate direction of education reform. It is worth pausing to consider whether to adopt a control strategy, or whether to adopt a commitment strategy that seeks to develop innovative working arrangements supporting teachers’ decision making, and increasing teachers’ engagement in the tasks of teaching (Rowen, 1990). Apart from that, an effective government ICT policy in enhancing professional development for teachers and in providing curriculum and resource support was also very important in facilitating effective ICT implementation in schools, though the impact was not as significant as that exerted by the school climate. In this connection, policymakers are urged to review the impact of adopting private sector practices to tighten control of educational process and the work of professionals, and to consider whether the dominant ideas of economic rationalism and managerialism will really increase or worsen educational outcomes (Mok & Welch, 2002). Limitations and suggestions With the convenience sampling, results of the present study were limited to producing a general pattern, but they were not supposed to produce generalisations for the whole Hong Kong school population. Despite the limitations of using teacher perceptions in assessing changes in teaching and learning, we believe perceived impacts are important to examine in school studies. Such an opinion-based approach has been regarded as informative and was used by a number of Nordic studies in investigating perceived impacts of ICT on student learning in non-traditional processes, such as team work, independent learning and critical thinking (Balanskat et al., 2006). Nevertheless, future studies may consider conducting longitudinal studies tracing actual changes in student learning by assessing actual performance of students at different junctures of time. However, we anticipate challenges in conducting these kinds of studies, as few education systems have formally adopted such new modes of assessment with greater emphasis upon measuring learning attitude and learning ability in terms of active construction of knowledge and collaborative work (Balanskat et al., 2006; Valli, Cooper & Frankes, 1997). Despite the large sample size of over 1076 teachers, the study was possibly limited by the common method variance since the measurement scales were based on a single method of questionnaire survey (Bagozzi, Yi & Phillips, 1991). To avoid the common method variance, future studies may also consider adopting multiple measures obtained with multiple methods. Hence, construct validation can be done with multitrait-multimethod (MTMM) matrix (Bagozzi et al., 1991). The present study aims at examining the pattern of relationships among the constructs rather than about the predictive ability of the constructs, and the analysis is at the teacher level. As pointed out by Kreft and De Leeuw (1998) and Raudenbush and Bryk (2002), if the school is regarded as a unit of study, the school-level variables may also play a role in contributing to the variation of the outcome variable. It is thus worthy to further explore the school effect on ICT implementation through the building of multi-level structural equation models. It would be interesting to find out whether the multi-level results obtained from each individual school are consistent with those in the SEM Wong and Li 377 analysis and how much variation found in the outcome variable can be attributed to different school-level variables. Acknowledgments We are grateful to the Research Grants Council for supporting the research project “Comparative study of IT implementation in education between Hong Kong and Singapore” (Competitive Earmarked Grant HKBU2034/02H), on which these analysis of ICT implementation of Hong Kong schools are based. References Anderson, J. C. & Gerbing, D. W. (1988). Structural equation modeling in practice: A review and recommended two-step approach. Pyschological Bulletin, 103(3), 411-423. Bagozzi, R. P., Yi, Y. & Phillips, L. W. (1991). Assessing construct validity in organisational research. Administrative Science Quarterly, 36(3), 421-458. Balanskat, A., Blamire, R. & Kefala, S. (2006). The ICT Impact Report: A review of studies of ICT impact on schools in Europe. [viewed 11 Apr 2011] http://ec.europa.eu/education/doc/reports/doc/ictimpact.pdf Bentler, P. M. (2007). On tests and indices for evaluating structural models. Personality and Individual Differences, 42(5), 825-829 (Special issue on Structural Equation Modeling). doi:10.1016/j.paid.2006.09.024 Bollen, K. A. (1989). Structural equations with latent variables. New York, Chichester: Wiley. Brown, J. S., Collins, A. & Duguid, P. (1989). Situated cognition and the culture of learning. Educational Researcher, 18(1), 32-42. Fornell, G. & Larcker, D. F. (1981). Evaluating structural equation models with unobservable variables and measurement error. Journal of Marketing Research, 18(1), 39-50. Fullan, M. (1992). Successful school improvement: The implementation perspective and beyond. Buckingham, Philadelphia: Open University Press. Fullan, M. (1996). Leadership for change. In K. Leithwood, J. Chapman, D. Corson, P. Hallinger & A. Hart (Eds.), International handbook of educational leadership and administration. Dordrecht, Boston, London: Kluwer Academic Publishers. Goffin, R. D. (2007). Assessing the adequacy of structural equation models: Golden rules and editorial policies. Personality and Individual Differences, 42(5), 831-839 (Special issue on Structural Equation Modeling). doi:10.1016/j.paid.2006.09.019 Hair, J. F., Anderson, R. E., Tatham, R. L. & Black, W. C. (1998). Multivariate data analysis. New Jersey: Prentice Hall, Upper Saddle River. Hinkin, T. R. (1998). A brief tutorial on the development of measures for use in survey questionnaires. Organizational Research Methods, 1(1), 104-121. Hu, L. T. & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6(1), 1-55. Joreskog, K. G. (1993). Testing structural equation models. In J. S. Long (Ed.), Testing structural equation models. Newbury Park, London, New Delhi: Sage. Joreskog, K. G. & Sorbom, D. (2001). LISREL 8: User's reference guide. Lincolnwood: Scientific Software International. 378 Australasian Journal of Educational Technology, 2011, 27(2) Kreft, I. & De Leeuw, J. (1998). Introducing multilevel modeling. London, Thousand Oaks: Sage. Lave, J. & Wenger, E. (1991). Situated learning: Legitimate peripheral participation. Cambridge, New York: Cambridge University Press. Leithwood, K. (1994). Leadership for school restructuring. Educational Administration Quarterly, 30(4), 498-518. Levin, B. (2001). Reforming education: From origins to outcomes. New York: RoutledgeFalmer. MacCallum, R. C. (1995). Model specification: Procedures, strategies, and related issues. In R. H. Hoyle (Ed.), Structural equation modeling: Concepts, issues, and applications. Thousand Oaks CA and London: Sage. Markland, D. (2007). The golden rule is that there are no golden rules: A commentary on Paul Barrett's recommendations for reporting model fit in structural equation modeling. Personality and Individual Differences, 42(5), 851-858 (Special issue on Structural Equation Modeling). doi:10.1016/j.paid.2006.09.023. Marsh, H. W., Balla, J. R. & Hau, K. T. (1996). An evaluation of incremental fit indices: A clarification of mathematical and empirical properties. In R. E. Schumacker (Ed.), Advanced structural equation modeling. Mahwah, New Jersey: Lawrence Erlbaum Associates. McCombs, B. L. (2000, 11-12 September). Assessing the role of educational technology in the teaching and learning process: A learner-centred perspective. Paper presented at the Secretary's Conference on Educational Technology 2000: Measuring Impacts and Shaping the Future, Alexandria, VA. Means, B. (1994). Introduction: Using technology to advance educational goals. In B. Means (Ed.), Technology and education reform. San Francisco: Jossey-Bass Publishers. Mehlinger, H. D. (1995). School reform in the information age. Bloomington: Center for Excellence in Education, Indiana University. Mok, J. K. H. & Welch, A. R. (2002). Economic rationalism, managerialism and structural reform in education. In J. K. H. Mok & D. K. K. Chan (Eds.), Globalization and education: The quest for quality education in Hong Kong. Hong Kong: Hong Kong University Press. Nunnally, J. C. & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). New York: McGraw-Hill. Raudenbush, S. W. & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods. Thousand Oaks, London, New Delhi: Sage. Rowen, B. (1990). Commitment and control: Alternative strategies for the organizational design of schools. Review of Research in Education, 16, 353-389 Senge, P. M. (1990). The fifth discipline: The art and practice of the learning organization. New York: Doubleday/Currency. Smylie, M. A., Miretzky, D. & Konkol, P. (2004). Rethinking teacher workforce development: A strategic human resource management perspective. In D. Miretzky (Ed.), Developing the teacher workforce. Chicago: NSSE. Spillane, J. P., Reiser, B. J. & Todd, R. (2002). Policy implementation and cognition: Reframing and refocusing implementation research. Review of Educational Research, 72(3), 387-431. Tiene, T. & Ingram, A. (2001). Exploring current issues in educational technology. Boston: McGraw- Hill. Valli, L., Cooper, D. & Frankes, L. (1997). Professional development schools and equity: A critical analysis of rhetoric and research. Review of Research in Education, 22, 251-304. Wong and Li 379 Windschitl, M. & Sahl, K. (2002). Tracing teachers' use of technology in a laptop computer school: The interplay of teacher beliefs, social dynamics, and institutional culture. American Educational Research Journal, 39(1), 165-205. Wong, E. M. L. & Li, S. C. (2008). Framing ICT Implementation in a context of educational change: A multilevel analysis. School Effectiveness and School Improvement, 19(1), 99-120. Zhao, Y. & Frank, K. A. (2003). Factors affecting technology use in schools: An ecological perspective. American Educational Research Journal, 40(4), 807-840. Authors: Emily Mei Ling Wong, Director, Vine Education Consultancy 13/F, Po On Commercial Building, 198 Nathan Road, Hong Kong Email: info@vineedu.com.hk Sandy C. Li (correspondence author), Associate Head and Associate Professor Department of Education Studies, Hong Kong Baptist University Kowloon Tong, Hong Kong. Email: sandyli@hkbu.edu.hk Please cite as: Wong, E. M. L. & Li, S. C. (2011). Framing ICT implementation in a context of educational change: A structural equation modelling analysis. Australasian Journal of Educational Technology, 27(2), 361-379. http://www.ascilite.org.au/ajet/ajet27/wong.html