Microsoft Word - v17n7.doc Readers are free to copy, display, and distribute this article, as long as the work is attributed to the author(s) and Education Policy Analysis Archives (Archivos Analíticos de Políticas Educativas), it is distributed for non-commercial purposes only, and no alteration or transformation is made in the work. More details of this Creative Commons license are available at http://creativecommons.org/licenses/by-nc-nd/2.5/. All other uses must be approved by the author(s) or EPAA/AAPE. EPAA/AAPE is published jointly by the Colleges of Education at Arizona State University and the University of South Florida. Articles are indexed by the Directory of Open Access Journals, H.W. Wilson & Co., and SCOPUS. EDUCATION POLICY ANALYSIS ARCHIVES English Editor: Sherman Dorn Spanish Editor: Gustavo Fischman College of Education Mary Lou Fulton College of Education University of South Florida Arizona State University Volume 17 Number 7 April 10, 2009 ISSN 1068–2341 Improving Low-Performing Schools through External Assistance: Lessons from Chicago and California Kara S. Finnigan University of Rochester Catherine Bitter Jennifer O’Day American Institutes for Research Citation: Finnigan, K. S., Bitter, C., & O’Day, J. (2009). Improving low-performing schools through external assistance: Lessons from Chicago and California. Education Policy Analysis Archives, 17(7). Retrieved [date] from http://epaa.asu.edu/epaa/v17n7/. Abstract This article describes the design and implementation of external support to low- performing schools using data from Chicago and California. Using the literature on external support, instructional capacity, and policy strength, the study gathered data from interviews, observations, document review, and surveys. The findings suggest that the model of assistance employed in both Chicago and California was inadequate to the task. While the policies examined demonstrate recognition that low-performing schools need additional capacity if they are to substantially improve student outcomes, external support providers used limited and haphazard approaches, and as a result, the support component had little influence on teaching and learning. In addition, because the external supports relied on a market-like support structure with few other mechanisms to ensure quality, and because there was limited quantity (intensity) of support, the benefit that external assistance might otherwise have provided was limited. This was particularly problematic for the lowest capacity schools, many of which experienced limited change despite increased educator effort and involvement of external providers. In essence, Education Policy Analysis Archives Vol. 17 No. 7 2 external assistance through these school accountability policies did little to improve educator and organizational performance. Keywords: accountability; educational policy; educational improvement La mejora de las escuelas de bajo desempeño a través de ayudas externas: Lecciones de Chicago y California Resumen En este artículo se describe el diseño y aplicación de ayudas externas a escuelas de bajo desempeño en Chicago y California. Este estudio recogió datos de entrevistas, observaciones, revisión de documentos, encuestas y literatura sobre la ayudas externas , capacidad de instrucción y fuerza política. Los resultados sugieren que el modelo de asistencia que usaron Chicago y California fue inadecuado. Si bien las políticas examinadas reconocieron que las escuelas de bajo rendimiento necesitan mas apoyo si quieren mejorar sustancialmente los resultados de los estudiantes, la ayuda externa fue muy limitada, usaba los métodos de manera azarosa, y consecuentemente tuvo poca influencia en la mejora de la enseñanza y el aprendizaje. Además, las ayudas externas emplearon estructuras de apoyo de cuasi- mercados con pocos mecanismos para garantizar la calidad, y dado que la cantidad (intensidad) de asistencia era limitada, los beneficios que las ayudas externas serian muy limitados. Esto fue particularmente problemático para las escuelas con menor capacidad, a pesar del aumento de los esfuerzos de los educador y la participación de proveedores externos. Básicamente, la ayuda exterior a través de estas políticas de rendición de cuentas (accountability) hicieron muy poco para mejorar la organización y el rendimiento escolar. Palabras clave: rendición de cuentas; política educativa; mejora de la educación. Through the No Child Left Behind Act of 2001 (NCLB), the federal government requires states to hold schools accountable for student performance. NCLB is based on an assumption that consequences will motivate school staff to perform at higher levels and focus attention on student outcomes. Once schools are identified as not making adequate yearly progress, districts must provide support and assistance to schools specifically in the areas of data analysis, identification and implementation of instructional strategies, and budget analysis, and states must “create and sustain a statewide system of support” (U. S. Department of Education, 2006, p. 25). The law requires that states allocate 4% of their Title 1 funds for school improvement (see Public Law 107–110, Title 1, Section 1003). This focus on district and state assistance suggests an additional underlying assumption: Low student performance is not solely related to will but also to school and educator capacity. In mandating external support to low-performing schools, NCLB offers a public statement about the need for districts and states to assist in school improvement. Unfortunately, few low- performing schools have received intensive assistance because of limited district and state capacity (Le Floch, Boyle, & Therriault, 2008; Olson, 2004; Richard, 2004; U. S. Department of Education, 2004). Furthermore, while recent research has focused on some aspects of NCLB implementation, such as assessment provisions and state capacity issues (e.g., Sunderman, 2008b), scant research exists on external assistance to low-performing schools. Improving Low-Performing Schools through External Assistance 3 This article seeks to inform the debate around NCLB by discussing the design and implementation of external support to low-performing schools under accountability policy sanctions based on an analysis of data from two studies: The Chicago School Probation Study and the Immediate Intervention/Underperforming Schools Program (II/USP) Evaluation in California (see Finnigan & O’Day, 2003, and O’Day & Bitter, 2003, respectively). The accountability policies in Chicago and California precede NCLB by several years and thus have been referred to as “first- generation” accountability policies because of their design similarities compared with the federal legislation (Mintrop & Trujillo, 2005). Both of these policies recognized the need to build capacity in identified low-performing schools and both provided resources to do so. In particular, the policies mandated to one degree or another that schools work with external agents to assist them in their improvement efforts. In some cases, schools benefited from the assistance of an experienced and qualified external support provider and attributed improvements to their work with this provider. However, the two studies in combination suggest a key finding: The support provided to low-performing schools in most cases was not sufficiently targeted, coherent, or intensive to influence instruction and student learning in a meaningful way. As a result, the schools’ responses to external support were quite variable and most schools realized at most a minimal benefit from this support. These findings have important implications for the design of school accountability policies to ensure that low-performing schools receive adequate and high quality assistance that will enable them to improve. Other jurisdictions can learn from the experiences in Chicago and California to better design and implement policies to build capacity in low-performing schools under NCLB or subsequent federal legislation. The following sections provide an overview of the conceptual framework and literature used in our analysis, a discussion of the policy contexts in California and Chicago, and a description of the methods and data sources for the two studies. We then discuss the major results through an analysis of the policy strength of the two external assistance policies and the monitoring of the quality of support by states and districts. Literature Despite the millions of dollars spent on external change agents in school reform, research on their effectiveness remains sparse and inconclusive. The conceptual framework for this article is informed by an analysis of the underlying assumptions of school accountability policies as well as the literature on instructional capacity and external assistance. We begin with a discussion of one of the core assumptions of these policies―that support must come from the outside―and describe the research to date on this type of support. Next, we discuss an additional assumption, that external support will lead to school improvement. In that section we provide conceptualization of instructional capacity to frame our discussion of the design and implementation of the external assistance provided to low-performing schools in these contexts. Finally, we describe the analytical lens we use to understand the policy strength of the external support component of these policies. External Support The inclusion of external assistance in the design of these policies suggests a key assumption: that low-performing schools need external assistance because they lack the capacity to improve on their own. External organizations can potentially bring new information, perspectives, and resources into school communities (Datnow & Honig, 2008; Huberman, 1995). These external Education Policy Analysis Archives Vol. 17 No. 7 4 groups may provide different types of assistance, including brokering or connecting communities to new ideas or practices, modeling new practices, facilitating dialogue and shared learning experiences around teaching and learning, and developing conceptual or practical tools (Honig & Ikemoto, 2008). However, scant research currently exists regarding the operation or impact of these types of providers in schools and districts (Datnow & Honig, 2008; Supovitz, 2008). Investigations like the Rand Change Agent Study (Berman & McLaughlin, 1978) found little impact of external change agents.1 The limited impact or effectiveness of these groups may be the result of support providers generally spending inadequate time in schools and lacking a strong vision for instructional improvement (Chimerine, Haslam, & Laguarda, 1994; Fullan, 1991). Similarly, the research on professional development suggests that support must be of high intensity and sufficient duration before it will have a measurable effect on practice (Desimone et al., 2002; Garet, Porter, Desimone, Birman, & Yoon, 2001; Hawley & Valli, 1998; Smylie, Bilcer, Greenberg, & Harris, 1998; Supovitz, & Turner, 2000). In fact, several studies suggest that the more intensive involvement of school “coaches” can play an important role in developing school capacity, even for schools identified as low performing (David, Kannapel, & McDiarmid, 2000; O’Day, Goertz, & Floden, 1995). The limited research, to date, suggests that not only the nature of assistance and target of support, but also the intensity and strategic approach (or lack thereof) matter. Instructional Capacity The policies in Chicago and California assumed that these external agents would plan and implement programs in the schools that would result in school improvement. Past research suggests that to improve school-level student performance, one must focus on individual teacher capacity, including teachers’ knowledge of content, pedagogical content knowledge, and general pedagogical knowledge (Shulman, 1986). An implication of this research is that external support providers seeking to improve student learning should begin by focusing on the knowledge of teachers. However, Cohen and Ball (1999) argue that considering teacher capacity in isolation is insufficient. Articulating a broader view of the instructional unit, these researchers argue that instruction (and thus instructional improvement) is “a function of the interaction among [teachers, students, and educational materials]… not the sole province of any single one, such as teachers’ knowledge or skill, or curriculum” (pp. 2–3; emphasis in original). In essence, Cohen and Ball’s examination of instructional improvement suggests that external support providers must move beyond developing teacher knowledge alone and instead leverage ways of strengthening all aspects of the instructional unit. In addition, because the instructional unit is nested within the school organization and influenced by it, interaction with and management of the environment are also integral to instructional effectiveness. Prior research in this area has focused on the importance of organizational capacity, often describing particular characteristics or components of the environment that are necessary to bring about school improvement (Levine & Lezotte, 1990; Newmann & Wehlage, 1995; O’Day, Goertz, & Floden, 1995; Purkey & Smith, 1983). For external support providers to initiate school improvement, they must take into account each of these levels (the instructional unit and the school environment). At the level of the instructional unit, teachers, materials, and students should be the primary foci, while important targets at the organizational level 1 Ten years later, McLaughlin (1987) found a greater impact as a result of more flexible agent-school relationships. Improving Low-Performing Schools through External Assistance 5 include coordination, professional norms, learning opportunities, resources, and the monitoring of student learning. Policy Strength Given the two policy assumptions discussed above (that external assistance is necessary and that the work of these external agents will bring about school improvement through capacity building), our analysis in this article examines the design and implementation of the external assistance provided through the school accountability policies in Chicago and California. Using a framework developed by Porter and his colleagues to examine curriculum policy (see Porter, Floden, Freeman, Schmidt, & Schwille, 1988), we consider the policy strength of the Chicago and California external assistance by identifying policy characteristics or aspects of implementation that are likely to lead to improved performance. Desimone (2002) argues that this “simple, but powerful framework” is useful in examining policy implementation and school reform (p. 438). Building on Porter et al.’s (1988) work, we examine the strength of the external assistance component of these accountability policies along four dimensions: specificity, consistency, authority, and power.2 Specificity refers to how narrowly defined a policy is. Porter et al. note that in curriculum policy, for example, broad standards are less specific than a more elaborated scope and sequence, which are less specific than a mandated curriculum and instructional materials. Consistency refers to the coherence among different policies and policy elements. In this context, one must consider both external coherence, or coherence of the support policy with other state or district policies, and internal coherence, or coherence of the reforms or approaches of the external support provider and other providers within the school context. Policies acquire their power from the resources committed to their enactment as well as from the constellation of rewards and sanctions accompanying them and the administrative authority of those involved. In our adaptation of Porter et al.’s framework, we argue that the resources allocated toward the implementation of the policy are an integral part of the policy’s power. Policy authority may derive from varying sources such as professional expertise, the perception thereof, and hierarchical positions of the key individuals. It is important to note that these policy attributes are not necessarily additive or even independent. In reality, consistency among policies may have a multiplicative rather than a merely additive effect on the strength of each. In addition, policy strength may derive more from achieving a proper balance of the components rather than high levels of each. Policy Contexts: Chicago and California In 1996, the Chicago Public Schools (CPS) made history by placing 71 elementary and 38 other low performing schools on probation, a judgment based on aggregate reading scores on the Iowa Test of Basic Skills (ITBS). Probation schools faced consequences including a decrease in autonomy and the threat of more severe sanctions, such as “reconstitution” or the process of removing all of the staff. The schools also received support from an External Partner, a Probation Manager, a Regional Education Officer, a Business Manager, and a District Facilitator. The district 2 Porter et al. use the term prescriptiveness, but we believe specificity more appropriately describes this dimension of policy strength, particularly given recent usage of prescriptiveness in the field of education to refer to pre-packaged curriculum or programs. Education Policy Analysis Archives Vol. 17 No. 7 6 fully funded the first year of external support, but in the second year the school assumed one-half the cost, and in the third and following years, the school was responsible for the entire amount. After a school was removed from probation, it was required to continue its relationship with a partner for one full year so that the supports were not withdrawn immediately. The External Partners were approved by the district and primarily university-based; that is, partners affiliated with local universities worked with the majority of the elementary schools. The district also assigned to each probation school an administrator that it judged had a proven track record of leading a school (though no clear criteria or definition were evident for these “successful” administrators) to serve as a Probation Manager and work directly with the principal. A few years after Chicago’s school probation policy was instituted, California state lawmakers established the Public Schools Accountability Act of 1999 (PSAA) with a new state measure called the Academic Performance Index (API)—a composite scale used to measure school performance. In that law was also a policy to support improvement in the state’s low-performing schools called the Immediate Intervention/Underperforming Schools Program (II/USP). This article focuses on II/USP, which provided funds to support schools’ efforts to improve but subjected recipient schools to future sanctions should that improvement not be realized after three or four years of such support. Participation was voluntary, and eligibility for II/USP involved a school’s ranking in the bottom half of scores statewide and failing to achieve annual growth expectations on the API.3 The funding included one year for planning and two to three years for implementation. Implementation was funded for each participating school using either state general funds or federal Comprehensive School Reform Demonstration (CSRD) program funds. The law required that schools hire a state-approved External Evaluator during the first (planning) year to assist them in developing their school improvement Action Plans. Four hundred thirty schools participated in II/USP in each of three cohorts (1999–2000, 2000–01, and 2001–02) in this study, for a total of 1290 schools.4 The Chicago and California policies had similar design components. Both targeted low- performing schools and located the problem of low performance within the schools. In addition, the Chicago and California policies involved the provision of external support. In both cases, schools selected and contracted with independent providers with only moderate constraints by the district and state, respectively. Directions to external support providers were minimal, and multiple approaches were acceptable on the assumption that this variation would better meet the diverse needs of schools and would lead to greater buy-in from school staff.5 Finally, in both places the policy targeted districts and schools serving primarily poor and minority populations, as documented by Diamond and Spillane (2004) and Mintrop (2004). 3 In the first year of the program, the statewide performance index (API) was not yet operational. Therefore, schools were eligible by scoring in the bottom half of the state’s schools for two consecutive years (1998 and 1999) on the statewide standardized exam (SAT-9). In addition, the state included all of the schools receiving funds through the Comprehensive School Reform Demonstration (CSRD) program. 4 A portion of these schools have since entered a subsequent program, the School Assistance and Intervention Team (SAIT) program, because they made insufficient growth during II/USP participation. In this program, schools work with an external, state-approved team for approximately three years to implement nine “Essential Program Components” (EPCs), a set of target areas for school improvement including, among others, the implementation of State Board-adopted curricula and the implementation of an assessment system to monitor student progress. In addition, California later implemented the High Priority Schools Grant Program (HPSGP), a similar program that focuses a higher level of funding to a smaller subset of the lowest performing schools. Approximately 290 II/USP schools later entered the HPSGP, thus receiving twice the dollar amount per pupil starting in the 2002–03 school year. 5 The two policies have different titles for those groups providing external support to schools. From this point forward, they will be referred to as “external support providers” generally and we use the language from each policy when discussing specifics in each context. Improving Low-Performing Schools through External Assistance 7 Despite these similarities, the policies differed in a number of important ways. While Chicago’s probation policy was a district-level policy, the II/USP was statewide and involved urban, rural, and suburban schools. In addition, the criteria used to determine eligibility differed, with Chicago (during the course of this study) judging aggregate performance based on a cut-off score and California incorporating school-level performance growth.6 Another distinction is the timing of assistance: California’s mandate regarding external assistance focused primarily on the planning stage of school improvement, and Chicago’s policy focused on both planning and implementation. Finally, California’s policy specified outcomes in multiple subject areas (including science, social studies, literacy and mathematics), while Chicago concentrated on reading scores. In spite of the differences, these two contexts provide important insights into the design and implementation of support within school accountability policies. Methods This section provides a brief description of the methods used in each study. The Chicago School Probation Study was conducted from 1999 through 2001 and focused on elementary schools on probationary status. To understand the nature, intensity, and quality of support provided to these schools under accountability policy sanctions, the study examined the two central components of the support system, the External Partners and Probation Managers. This article is based primarily on interviews and observations of 11 (of the 18) External Partners during this time period and interviews with a random sample of 16 (of the 53) Probation Managers. Other data sources include interviews with district administrators and facilitators, review of district documents, observations of support provider meetings and workshops, and interviews with a random sample of 15 principals at probation or post-probation elementary schools. Finally, the article used interviews and observations from case studies of ten schools that were placed on probation. (Half moved off probationary status and half remained on probation throughout the study.). California’s II/USP Evaluation Study occurred from 2001 through 2003. This article primarily draws on interviews with 21 school principals and more than 100 teachers at 21 schools in the study. In addition, the article uses interview data from the External Evaluators for 18 of the case study schools and district personnel in each of six districts associated with the case study schools. The information is supplemented by findings from a survey of teachers and principals in a stratified random sample of II/USP and matched comparison schools and survey data from 104 External Evaluators and 162 district personnel from districts with at least one II/USP school. In addition, a short follow-up study was completed in 2005 to examine longer term impacts and implementation of II/USP. The analysis for this article involved an examination of the nature and target of external assistance across these contexts, as well as the different aspects of the design and implementation of these policies relating to policy strength. The authors conducted an iterative analytical process of identifying and refining the emerging themes. 6 In Chicago at the time of the study, the district identified schools based on their aggregate reading scores on the ITBS. In the first year schools were placed on probation if less than 15% of their students in grades three through eight were reading at or above national norms (the cut-off of 15% has been raised during subsequent years of implementation). More recently, the district used the statewide test and included a component that focuses on growth in achievement. In California, eligibility was determined by the ranking of the school in comparison to other schools in the state, as well as by the growth (or lack thereof) in student achievement across years. In California, each school was assigned a yearly growth expectation for the API equivalent to 5% of the difference between their current API score and an overall state target. Education Policy Analysis Archives Vol. 17 No. 7 8 Results In this section we discuss our findings relating to the strength of the support component of these school accountability policies using the concept of policy strength discussed previously. Within our discussion, we describe the nature and targets of external assistance in terms of our conceptualization of instructional capacity. Finally, we link back to an overall policy assumption regarding the role of the district and state in providing support to low-performing schools―the belief that these agencies have the capacity to provide adequate and high quality support to individual schools. Strength of External Support Specificity. Specificity in this context refers to whether the policy provided explicit or detailed requirements regarding the assistance. Our findings suggest that these policies were highly specified about certain procedures as well as the expected outcomes, but neither policy was sufficiently specified about the nature and target of assistance to ensure that the external support providers developed coherent strategies to bring about school improvement. Both policies were highly specified in certain ways. For example, California’s policy prescribed activities during the planning year with respect to the processes the External Evaluator must use to conduct a needs assessment and develop an Action Plan for each school. The External Evaluator was required to analyze school data and gather information from parents and teachers. Both the California and Chicago policies were also specified in their emphasis on test scores―schools were expected to meet specific achievement goals in order to exit probation (in Chicago) or to avoid additional sanctions (in California). This specification oriented the external assistance providers’ work towards improving test scores. In both contexts, the policies for external support were vague by design in terms of the nature and targets of assistance. In theory, this lack of prescriptiveness allowed for individual matches between support providers and schools, and for improvement strategies tailored to the needs of the school. Indeed, a positive aspect of these policies is the recognition that schools differ, as do the nature of their problems. A school could choose a provider that had a similar reform approach (for example, a direct instruction approach versus constructivist approach), or that had expertise specific to the challenges at their school (for example, a high proportion of English Language Learners or a weak reading program). By allowing flexibility in the target and strategies for improvement, individualized improvement plans could be customized to the specific school needs. While positive in its intent, such vagueness in the nature and targets of assistance may have limited the policies’ overall effectiveness. For example, by relying on market forces to produce greater fit between schools and providers, the policies assumed that schools would select providers who would appropriately serve their individual needs. In reality, many schools did not have the capacity to know what services they needed or to determine who would provide them appropriately. In addition, schools typically did not know which providers had a record of success with schools like theirs. Thus, many schools chose a provider based on availability or location or a person whom they already knew, with whom they had a pre-existing relationship, or about whom they had heard from others. As a result, rather than creating better matches between providers and schools as intended, the lack of specificity around the nature of support often left schools with providers who were not well-suited to assist with their specific needs. Improving Low-Performing Schools through External Assistance 9 In addition, data from both studies indicate that the lack of specificity regarding the targets of support often resulted in a scattered rather than strategic orientation to improvement. The following discussion provides general details of what assistance looked like in these contexts; however, support providers varied in their particular designs. In both policy contexts, teachers were a primary focus. In Chicago, the external support providers worked with teachers individually or in group settings to increase their knowledge and skills or hone their classroom management and instructional practices. In California, the External Evaluators did not directly provide professional development or assistance in the first (planning) year, but the Evaluators targeted teachers’ knowledge and skills in their needs assessments and Action Plans. Nearly all of the external support providers also attended to instructional materials in some manner, including state or district- adopted curricular packages, lesson plan guides, “how-to” packets, and assessments. In addition, some of the external support providers focused their assistance at the organizational level. For example, they could have worked with a principal or a schoolwide team to strengthen leadership and to develop a shared vision and common goals; targeted specific organizational structures to facilitate teachers’ sharing ideas and practices; or brokered resources. Finally, in both California and Chicago external support providers worked with teachers to increase the use of data to monitor student achievement and to make instructional and organizational decisions in their classrooms. Our conceptualization of instructional capacity suggests that these individual and organizational level targets are appropriate; however, our data indicate that the assistance provided to schools was often scattered in its focus, not strategic. In other words, rather than targeting systematically and strategically the interaction of teachers and students working around particular content and the organizational support for such work, the providers typically focused on loosely connected elements of instruction or school environment. In addition, the target of assistance varied from provider to provider with no real guidance from the policy authority. This variation in target could be beneficial if the assistance was consistently aligned to the real needs of the schools or directed clearly toward the improvement of instruction and student learning; however, our data suggest otherwise. For example, in California, many schools rewrote, revised, or ignored the Action Plans developed in the planning year by the External Evaluator if their staff did not think the plan met their (often changing) needs. The variation in targets and design of support was compounded in California by a lack of specificity for the external support provider’s role in the implementation years of the policy. While the policy provided specific guidance concerning the work of the External Evaluator in the planning year (which, depending on the timing of funds, ranged from just a few months to nearly a whole school year), External Evaluators were neither required to assist the school nor monitor implementation beyond that first year. In fact, schools were initially discouraged from using the same External Evaluator during the planning and implementation years for fear that the External Evaluator’s work in the initial phase would be biased toward identifying strategies that matched their own programs rather than the school’s needs. Thus, again, the policy was designed to provide flexibility for schools in the use of their implementation funds. However, in the end, schools were left with little direction on how to implement their Action Plans and on whom to call for assistance in doing so. In fact, 77% of surveyed II/USP principals said they had received limited or no follow- up from their External Evaluators during implementation. This disconnect between the specificity in the planning year and years to follow likely hindered the implementation of plans significantly. The Evaluators themselves acknowledged this shortcoming: 63% of surveyed External Evaluators responded that confining the External Evaluator’s role to the first (planning) year of II/USP was not useful. In fact, the II/USP Evaluation found no clear connection between the quality of the External Evaluator’s work during the planning year and subsequent implementation of plans and growth in student outcomes. Education Policy Analysis Archives Vol. 17 No. 7 10 Our analysis of the specificity of the California and Chicago policies raises the question of which aspects of a policy should be highly specified and which should remain vague or unspecified. Both policies while narrowly defining the expected outcomes did not specify the focus of assistance. First, there was no explicit guidance to link planning to practice and changes in instruction. Second, little direction was provided in either policy context around the ways in which improvement goals set by schools and external assistance providers should directly relate to instruction. Thus, plans and strategies often did not tie into a coherent set of instructional goals (see discussion of consistency and coherence below). In fact, the external support providers’ strategies rarely involved coherent planning or targeted implementation and, as such, the external strategies frequently deposited an additional layer of reforms in these schools. In Chicago, most external support providers offered in- service training to teachers that emphasized generic pedagogical and management strategies, rather than strategies linked to particular content areas and to goals or standards in those areas. Furthermore, most of these providers did not emphasize (or assist teachers and administrators in developing) instructional coherence as a key goal (see more details below). As a result, the support to low-performing schools in Chicago through this policy was not likely to result in deep instructional improvement. Similarly, as mentioned above, a lack of specificity in the external support provider’s role regarding implementation of Action Plans in California resulted in little connection between the external support providers’ work in the planning year and actual changes in instruction and student outcomes.7 Even though External Evaluators had the opportunity to provide insights about ways to improve instruction across subject areas during the planning year, their insights were only meaningful if the school followed through on actions to address them. The ability to follow through was hindered by the division between planning and implementation. In some cases, specificity was infused through another entity. For example, some districts in California required their schools to use the same highly specified curricula or instructional materials. These policies provided more specific guidance in terms of instructional goals and improvement strategies, and indeed, we found that those schools that were most successful in implementing a coherent instructional program had strategies that were at least specified enough to get all teachers on the same page. Unfortunately, the districts’ more prescriptive guidance sometimes conflicted with reforms put in place at the schools through II/USP. We discuss the effect of this specificity on the policy’s consistency in more detail in the following section. Consistency. Coherence at the school level is important to the improvement of low- performing schools as it ensures clear goals and fosters the alignment of school strategies and resources to these goals (Newmann, Smith, Allensworth, & Bryk, 2001). We found that only four (of 10) case study schools in the Chicago study and eight (of 21) case study schools in California had coherent instructional programs. In California we found a strong relationship in our case study schools between the presence of a coherent instructional program and improvements in student achievement outcomes—that is, those schools with more coherent programs also demonstrated greater and more consistent gains in student test scores. In a few cases, the coherence appeared to develop, as intended, through the needs assessment and planning process of II/USP, with help from the External Evaluator. However, in other cases this coherence had clearly been forged in spite of rather than because of the work of the external provider. Both studies found that the external support providers’ programs were often considered add-ons to the schools’ core work and were rarely integrated into the schools’ overall goals and reform efforts. This add-on nature of support resulted in a lack of internal consistency around the reforms or approaches implemented at the 7 In contrast, in the SAIT program, the SAIT providers were required to organize their work around a set of nine “essential components” for school improvement, set out by the state. These components increased the specificity of the SAIT’s work in both the planning and implementation phases. Improving Low-Performing Schools through External Assistance 11 schools. In other words, the support often did not become part of an integrated, institutionalized improvement strategy. Our findings suggest that this dimension of the policies was weak for the following reasons: the lack of fit between external support provider and schools; the fragmentation between planning and implementation; the lack of coordination among multiple agents in these schools, and other conflicting policies implemented in these contexts. In fact, inconsistency or lack of coherence in support providers’ work was a substantial design and implementation issue in both California and Chicago and limited the coherence of the resultant improvement strategies and instructional reforms. Though not explicitly written in the legislation, California’s policy attempted to foster instructional coherence within the schools through the planning year; the external support provider was expected to facilitate the development of a coordinated plan for improvement during this year. In Chicago, coherence was not explicitly addressed but was promoted indirectly through the school improvement plans that probation schools were required to submit and Probation Managers were required to monitor. In several ways, the lack of integration discussed above resulted from the policies’ limited specificity around instruction, as described in the previous section. For example, one school in Chicago with a constructivist approach to learning hired an external support provider associated with the Direct Instruction program because of the strong reputation of the program director. Since the school intended to continue with its constructivist learning philosophy, the external support provider’s program was in conflict with the school’s core program and agreed-upon philosophy of teaching and learning. Direct Instruction was not adopted programmatically, although the provider continued as the school’s official External Partner. The conflict was exacerbated by the fact that the external support provider appeared to do little to understand and adapt its assistance to meet the individual needs of this school. The school’s performance levels improved rather quickly, but the improvements seemed to have nothing to do with the policy’s external assistance, which school personnel viewed as peripheral to the changes that led to improved performance. In California, the policy provided little guidance with regard to the instructional content focus of the Action Plan. Thus, in several cases, external support providers developed plans for improvement that included a range of disconnected strategies. For example, in one rural middle school, the Action Plan included 35 strategies for teaching, learning, and staff development, while seven strategies focused on governance. This laundry list of actions to address a long list of identified barriers was a challenge for staff to implement and was therefore essentially ignored. In California, consistency of the policy was also weakened by the disconnect between the planning and the implementation years, as described earlier. The lack of a specified support component in the implementation years weakened the coherence of both the support and the resultant improvement strategies. Since the planning process was divorced from implementation, external support providers generally lacked a commitment to the school and to the implementation of the plan. This aspect of California’s policy design limited schools from following through with their school improvement plans in coherent ways. Beyond the official external support provider associated with these policies, multiple other agents often assisted these schools. Communication among these various agents was minimal, leaving little opportunity for these multiple groups to coordinate their efforts and assist the school in a coherent way. When various groups involved in the same effort have contradictory messages, each group’s effectiveness is severely limited (Hatch, 1998). In this scenario, the existence of multiple groups heightened the probability that school staff would be forced to contend with conflicting messages about school improvement and, as a result, opportunities for real improvement would be duplicated, contradicted, or lost. The school improvement efforts of these external support providers frequently became peripheral or irrelevant, especially when they were the newest partner to work with these schools. Education Policy Analysis Archives Vol. 17 No. 7 12 On a similar note, to foster coherence at the school level, a policy must be explicitly designed to engender that coherence. To do so, the policy itself needs to be consistent with other existing policies to avoid pulling schools in multiple directions. In both California and Chicago such policy coherence was limited. Specifically, district policies sometimes hindered the ability of schools to utilize fully the external assistance they received, particularly when those district policies were specified in ways that contradicted the assistance provided through the II/USP or probation programs. For example, in California, two case study schools as part of their participation in II/USP (CSRD) adopted whole school reform models that emphasized the development of the whole child and incorporated art as a means of doing so. However, these models contradicted the subsequently adopted scripted reading and mathematics programs for their districts. Unable to ignore the district requirements, the schools pulled away from the reform model, though it remained in the school as a source of incoherence and distraction rather than as a comprehensive reform design. Similarly, in Chicago, the district began recommending that low-performing schools use structured reading lessons provided by the central office. In part, this district policy was a response to the recognition that schools needed additional guidance with respect to instruction and was intended to provide schools with instructional tools. Unfortunately, these structured lessons were sometimes in conflict with the External Partners’ approach to reading, resulting in confusion regarding how to align these disparate strategies. The district also instituted a policy for schools that received additional federal class-size reduction funds through the Targeted Assistance Program, requiring that students be instructed using only grade-level reading materials during the literacy block. This policy was in direct conflict with that of some external partners’ programs (e.g., Success for All) that focused on flexible grouping across classrooms and use of literacy materials at the student’s instructional level rather than only on a uniform grade level. Even where the external partners’ programs appeared to be improving student literacy performance, schools were forced to abandon them in favor of the uniform, grade-level approach. These types of contradictions arise when the multiple policies to which schools are subject are not aligned or when key stakeholders, such as the district in California’s case, are not involved in designing the policy. When examining support between the district and external organizations discussed by Supovitz (2008), both Chicago and California exhibited fragmentation rather than the melding. While the previous discussion focuses on coherence and consistency at the school level, inconsistencies across the state (California) and district (Chicago) also existed. For example, in California, the external support providers’ skills and work were not consistent across II/USP schools. As discussed previously, the targets of the providers varied considerably, with some focusing on instructional materials, and others focusing on professional development. This variation was intentional, to foster reforms more tailored to individual schools’ needs. However, in both contexts, there was little to no networking or training to ensure that external support providers were consistent, resulting in a wide range of experiences among schools in the planning year, and a broad set of reforms implemented sporadically across the state.8 We also found a range of expertise and commitment of the external support providers to these low-performing schools. In both contexts, an approved list of external support providers was developed but the selection process did not ensure a minimum level of quality, and little to no monitoring of the providers’ work occurred as they assisted schools. As mentioned previously, the support providers were mostly selected based on prior relationships rather than a fit with the school’s needs or evidence of the quality of support to be provided. 8 In California, this problem has been mitigated somewhat in the SAIT process, since all external providers attend a common training and have a common template to use to plan strategies for improvement. Improving Low-Performing Schools through External Assistance 13 Power. The Chicago and California policies manifested their power through large financial commitments and threatened sanctions to participating schools. Despite these large financial commitments in terms of aggregate dollars for the program, our findings suggest that the policies’ power was limited by the low intensity and limited resources associated with the external support providers’ work; limited implementation of the consequences associated with the policies; and a low level of administrative authority. These limitations in power lowered the salience of the policy among school staff and limited the ability of the policies to influence classroom instruction. Significant funding was allocated toward both policies. In Chicago, the district provided the funding in full for the External Partners in the first year of probation and paid for half of the support during the second year. By the third year the district no longer paid for the assistance; the schools were required to incur these costs. Many schools began using federal Comprehensive School Reform Demonstration Program (CSRD) grants, which they received through the district, to pay for their portion of the support after the first year on probation. The district also paid the small stipend to Probation Managers every year. In California, schools were allocated a planning grant of $50,000 in the first year to be used for planning activities and hiring the External Evaluator. During each of the subsequent implementation years, schools received $200 per pupil. Funds were provided by the state and included federal CSRD funds, and schools were expected to allocate $200 per pupil to match the II/USP grants (typically using Title I funds). The research indicates that external support must be of high intensity if it is to improve educator and organizational capacity (Hawley & Valli, 1998; Smylie, et al., 1998; Supovitz & Turner, 2000). In these low capacity schools, the external support providers were often stretched too thin to provide the necessary support to teachers or principals to generate the level of improvement needed. All external support providers in Chicago reported observing teachers in their classrooms, but this assistance varied in quality and was generally of low intensity⎯approximately one or two days a week for the whole school for External Partners and two to three days a month for Probation Managers’ work with the principals. The number of teachers receiving direct assistance from external support providers was difficult to determine through our interviews and observations and seemed to change on a regular basis. Many External Partners spent approximately one day a week at a school focusing on multiple tasks during that time, including observing teachers in classrooms, providing feedback to teachers, and distributing materials. In some cases, more intensive support was provided to a small number of staff. An exception to this low level of intensity was evident in one of the Chicago schools. In this school the external support provider’s program involved the addition of an on-site staff person to provide continuous support and professional development to teachers, in conjunction with a coherent literacy program in which the on-site coach received intensive training. This staff literacy coach was in frequent contact with consultants from the external support provider. Over time, the school developed a system in which teachers at every grade level met several times a week to plan, and during one of these planning sessions they were required to meet with the on-site support person to discuss curriculum and instructional issues or challenges. This type of institutionalized support and school-level capacity building resulting from the partnership with the external support provider was rare. Instead, many support providers in Chicago continued to reflect the conventional model of fragmented, one-shot sessions and trainings during half-day and full-day workshops rather than intensive and ongoing support. In California, some of the External Evaluators spent only a few days in total at the schools. In addition, some providers were spread too thin across multiple schools to adequately identify needs among teachers and the effective and comprehensive strategies necessary to address them. At one school, staff said that the External Evaluator “took the money and ran.” According to the teachers, the Evaluator did what was required by law but did not actually incorporate any feedback Education Policy Analysis Archives Vol. 17 No. 7 14 from teachers, using primarily documents that were already written to prepare the Action Plan. While this was an extreme case, these teachers’ complaints of seeing too little of the External Evaluator were by no means unique among participating schools. At another school, for example, three out of four teachers interviewed would have liked to have seen more of the External Evaluating group. “They leave and don’t come back,” one teacher said. The teachers also reported that the External Evaluators were not in the classroom enough when they were at the school. Another aspect of policy power lies in the incentive mechanisms incorporated into implementation. These incentives may be either positive (rewards) or negative (sanctions). While Chicago and California included incentive components in their accountability policies, the data indicated low salience or motivational power associated with these incentives in both contexts. In Chicago, the threat of sanctions was not the only policy consequence. Indeed, identification as a probation school (note the punitive name: probation) was itself considered a sanction by most school personnel. The more severe sanction outlined in the policy, however, was the threat of reconstitution or re-engineering of schools that did not move out of the probation status, both of which involved removal of school staff. During our study, elementary schools that had been on probation four or five years were unclear about the consequences they faced if they did not reach the performance targets. In 1996 seven CPS high schools were reconstituted, but no elementary schools had been reconstituted through this policy at the time of the study. Uncertainty, a general sense that there was little the district could actually do to them, and the lack of positive rewards severely limited the salience of the policy consequences.9 This finding is similar to that in Mintrop’s (2004) study of the response to probation in Kentucky and Maryland, in which educators disregarded the threat of sanctions contending that they were not credible responses. This is not to say that there was no motivational impact to probation in Chicago. Certainly, probation brought increased oversight from the district and the stigma of the probation label. While these were not the severe sanctions that were to result from the policy, these negative incentives proved rather strong in practice for many school staff in Chicago who believed that their professionalism was being called into question by their peers at other schools or in the broader community (Finnigan & Gross, 2007). Similar reactions to sanctions were evident in California. While school staff generally recognized that the policy opened their school to sanctions, few expected the state to implement the most severe sanctions (e.g., school takeover or closure, or the firing of staff or administrators). Less than half of the surveyed teachers participating in II/USP expected that these severe sanctions would occur, while a higher percentage of teachers expected less severe sanctions such as a public hearing or the assignment of an assistance team. Like Chicago’s probation policy, however, the II/USP did initially hold a negative stigma due to the label of underperforming and attribution of underperformance to the school and staff. The power of these sanctions was not universal in Chicago and California and, in fact, the effect of these dissipated over time as schools began to see that nothing concrete happened as a result of extended identification. For example, in California, over half of the case study schools later viewed participation more positively as an opportunity for change. At the time of the Chicago study, a number of schools had been on probation for five or more years, and the district was only beginning to discuss potential sanctions if they did not show improvement. The consequences for these types of schools that lingered on accountability sanctions were not addressed in either context and have not yet been addressed under NCLB as schools remain under the final restructuring stage (Center on Education Policy, 2008). 9 We are not suggesting that the sanctions were appropriate, but are noting that the power of the policy decreases without rewards or sanctions for those involved. Improving Low-Performing Schools through External Assistance 15 At the support level, the policies were rather weak in terms of the authority of support providers to ensure implementation of improvement strategies. In both contexts, the external support providers had no line authority; that is, they had no supervisory power over the schools. They could not force the schools to implement their programs, hire or fire teachers or administrators, or require that certain teachers receive their assistance. Nor did the relevant school bureaucracy use its administrative authority to press principals toward more effective implementation of the providers’ assistance. The only providers that had any degree of administrative authority were the Probation Managers in Chicago, who monitored the schools’ implementation of their improvement plans. The monitoring role was only vaguely defined, however, and few Probation Managers believed that their responsibilities extended beyond principal mentoring.10 Authority. For support policies associated with school accountability programs to produce their desired effects on schools, school personnel must perceive and accept the external support providers as knowledgeable experts—that is, as specialists who could help the schools diagnose problems and plan for school improvement. Since the external support providers did not have administrative power, they had to rely on professional authority. Unfortunately, the variable quality of assistance threatened the ability of these groups to win the professional confidence of audiences at the school sites. Our findings suggest that the vendor relationship between the external support providers and schools, inadequate selection processes, concerns about the quality or appropriateness of the providers’ support, and the resistance of school staff and subsequent compliance resulted in the low professional authority of the external support providers in most of these schools. The external support providers with the highest level of bureaucratic authority and varied levels of professional authority were the Probation Managers in Chicago. Probation Managers worked directly with the principals, mentoring them in a variety of areas. In some schools, the Probation Managers’ high levels of professional authority were linked to their own experiences in the Chicago Public Schools, although in some cases the probation school principals complained that these administrators came from different types of schools and, therefore, had limited understanding of the situation at hand. While there were instances of the external support providers commanding high levels of professional authority among teachers, our findings suggest that this was less evident for the reasons described in the following paragraphs. The support providers in both California and Chicago served in a vendor relationship, where schools hired them for a specific, limited service, rather than joining the schools in a partnership arrangement. This vendor relationship meant that their ability to carry professional authority at the school site was even more critical in maintaining their status (and funding) as the school’s external support provider. While some teachers in both contexts discussed the benefit of the policies’ external support components in terms of the extra tools they received, opportunities to try something new, training, or new perspectives, a frequent complaint in a few Chicago schools was teachers’ feeling that their own professional authority was diminished by the requirement to have someone from the outside teach them how to teach. As one teacher said, “[They’re telling us] ‘you guys don’t know what you are doing so we have to have some outsiders come in here and tell you what to do.’ [These are] people that have never been in the classroom or have been in it 20 years ago and they’ll tell you what to do.” This scenario was observed in a California school where teachers 10 The power of the policy in California has increased over time through the SAIT process. Monitoring by state officials is an important component of this process with specified expectations for monitoring meetings and reports, as well as requirements with regards to implementation benchmarks. This increased level of administrative authority over the schools strengthened the power of this policy. Education Policy Analysis Archives Vol. 17 No. 7 16 complained that the External Evaluator focused too much on the negative and did not acknowledge the work and skills of teachers who had been struggling with the identified issues all along. Inadequate selection processes further hindered the appropriate matching of providers with schools and thus lowered the professional authority of the support providers. In most schools the staff did not attempt to thoroughly review their programs and data, identify the core problems in their schools, and select an appropriate partner based on these analyses. Instead, many schools made decisions to select particular partners based on personal connections or previous relationships. Similarly, decisions in California did not seem to result from a clear diagnosis of the problem and appropriate external support provider; however, this was primarily due to time restrictions and the limited pool of providers. Several schools reported that it was difficult to locate an available, approved provider. Resultant concerns regarding the quality of assistance provided and the extent to which it was adapted to local circumstances further diminished the professional authority of the providers. Low professional authority in Chicago was associated with the perception that the external support providers’ intervention was not clearly defined. As a result, school staff did not fully understand what expertise or services they were buying and even how to differentiate among support providers when determining which to select. Low professional authority in both California and Chicago was linked to concerns within the schools regarding the capacity of the external support providers to offer the promised assistance. In some instances external support providers were forced to hire additional staff using lower standards to ensure that they could work with the high number of schools they were assisting. Furthermore, both Chicago and California experienced a high degree of turnover in support providers, resulting in a lack of continuity for schools. Finally, in both contexts, school staff complained about the quality of the assistance.11 The fact that the district and state did not have rigorous selection processes, nor did they monitor the assistance provided by these groups, further weakened the professional authority. Finally, in both California and Chicago, professional authority was called into question when the support providers proposed strategies, made recommendations, or promoted their particular programs without identifying ways in which these should be adapted to local contexts. This finding is in clear contrast to Honig & Ikemoto’s (2008) finding that the external provider in their study continuously revisited or altered their activities based on local conditions and needs. In Chicago, staff in several schools questioned whether the support providers understood the extreme circumstances they faced in these schools and communities. Similarly, in California, school staff in rural settings questioned whether their External Evaluators understood their unique, rural context. This criticism was linked to the low level of intensity of the support (discussed previously) because school staff believed that these individuals simply were not at their schools enough to understand their schools. Overall, in the Chicago and California cases where the professional authority of the external support providers was high and the benefit of what they were offering was clear, school staff were likely to implement their programs or recommendations. If the support providers were not viewed as legitimate—in other words, if the professional authority of these providers was low—school staff were likely to resist, adapt, or simply not implement the programs or recommendations the providers offered. In some of the schools, teachers and principals did not buy in to the external 11 The SAIT process in California provides an interesting contrast to these scenarios. A combination of factors in the SAIT process served to increase the professional authority of the external support providers. First, the state implemented more rigorous selection processes for providers to ensure they had experience turning around low- performing schools. Also, county offices of education could serve as SAITs. Several respondents noted that the county was very familiar with the schools they served, and therefore could more adequately understand and address their challenges. Finally, the process for SAIT involved more rigorous training required by the state. Improving Low-Performing Schools through External Assistance 17 support providers’ programs, while in others they primarily reacted to the mandatory nature of the policy and associated stigma of being identified as a low-performing school. Regardless of the reason, their resulting resistance meant that the programs were never fully integrated into their classrooms and schools. For example, in one II/USP case study school in California where a subgroup of teachers resisted implementing a whole school reform model recommended by the External Evaluator, the school made participation in this model and the associated external support optional. As a result, he reform did not result in a universal, coherent strategy through the school. As one teacher in Chicago described, “[The university-based external support provider] has a program that they use in all the schools they work with. [The provider] was brought in with no input from the faculty. I don’t remember being asked about it. Maybe we were but I don’t think so. It worked out for me because I did things like that before but others may do things differently. They were forced to change.” This teacher does not explicitly discuss the resistance of teachers but provides insights into the lens teachers used to determine whether they accepted the external support provider’s approach that had to do, in part, with the actual instructional strategies (and whether or not they were already doing it) and, in part on the process for determining who would provide assistance as a result of this policy. In Chicago, perhaps linked to this resistance, as well as the mandatory nature of probation and external support, most schools focused on compliance. Principals played a critical role in conveying their own views of the professional authority of the external support providers and overall policy. For example, in one Chicago school the principal viewed neither the policy itself nor the support component as legitimate. As a result, his school became an example of minimal compliance with the External Partner requirement. Rather than valuing what the support provider could offer in terms of professional expertise, he had the provider focus on a very narrow aspect of the school’s program: the retrofitting of the school’s wiring for technology. The lack of specificity in the policy with regard to focusing the providers’ work on instructional support contributed to this principal’s adaptation of the support mechanism. As mentioned previously, another of the Chicago case study schools chose an external support provider with a philosophy of learning and instruction that contradicted the school’s philosophy, illustrating in a different way the limited value placed on the role of the External Partner in the school’s improvement efforts. Interestingly, at least minimal compliance occurred across the board in Chicago in spite of the fact that there were no clear consequences resulting from non-compliance. Limited Monitoring of External Support Quality A key problem in both Chicago and California was the limited capacity of the district and state, respectively, to directly assist the large number of low-performing schools. More recently this same problem has been found in response to NCLB as a result of low levels of funding and increased (and shifting) responsibilities of these agencies in addressing the problem of low performance (Sunderman & Orfield, 2008). According to the U. S. Department of Education (2004), districts, and not states were most often providing assistance to low-performing schools, yet most districts were unable to provide the resource-intensive support necessary because of their own limited capacity. In Chicago and California, limited capacity resulted in a reliance on external support providers to bring about school improvement. Therefore, it was of utmost importance that these providers were carefully selected and monitored to ensure that they provided high quality assistance. Unfortunately, in these contexts district (Chicago) and state (California) administrators had limited capacity to oversee external support providers, with little screening of providers or monitoring and evaluation of assistance occurring. While California did improve the rigor of the External Evaluator Education Policy Analysis Archives Vol. 17 No. 7 18 selection process over time, they did not implement a process to directly monitor or evaluate individual support providers’ work. As one teacher said, “[The state] needs to look at [the External Evaluators] much more closely. In some cases there are people on the list to be consultants who are not at all able to carry out that function.” In other words, while the schools faced sanctions for low performance, there were no sanctions applied to external support providers if they offered inadequate or poor quality assistance. The capacity of state and district agencies to even provide this most basic oversight of external support providers, not to mention the provision of direct assistance to their low-performing schools that is assumed by NCLB, must be addressed for schools to receive adequate support and to ensure that these policies address the central assumption that low performance is linked to school capacity. Conclusions The Chicago and California policies represented important advancements with regard to building the capacity of educators in low-performing schools. However, these studies suggest fundamental problems in the design and implementation of school improvement support mechanisms that must be addressed if this support is to truly benefit the recipient schools. Weak policy strength and limited monitoring of external providers’ quality resulted in the support component of these policies having little influence on teaching and learning. While some low- performing schools benefited from the external assistance, perhaps as a result of the higher levels of initial capacity within the school (Gwynne & Easton, 2001), the majority continued to struggle despite additional assistance. The limited benefit of this support may not be surprising given the decades of problems and neglect that many schools faced. The implementation of both policies relied on market mechanisms to control the quality of external support providers. However, most selections of partners had little to do with the quality (or even the fit) of their programs. Combined with the variability in the nature of support, this fact meant that a large proportion of schools did not receive appropriate or high quality support targeted toward their needs. External support providers’ assistance should be integrated into a comprehensive, strategic approach to school improvement linked to individual school problems. Furthermore, at the heart of school improvement is instructional change; therefore, the external support providers should be required to demonstrate a link between their programs and instructional improvements. Schools must be provided with more information before selecting partners to allow them to make better selections (although the capacity of school staff to identify the root of their performance problems remains an issue). In addition, the district or state must evaluate the assistance of the support providers to ensure that they not only meet the needs of schools they serve, but also are held accountable for the implementation and outcomes of their work. In both California and Chicago, the external support providers were not affected in any way if the schools they assisted continued to perform at low levels: They were not removed from the list of approved agencies, nor even oftentimes the school itself.12 While these policies target the problem of low performance inside the school, the root of low performance rarely is located in the school alone. Instead, many school problems are related to larger, systemic issues and policies at the district and state levels. Low-performing schools in both Chicago and California were dealing with decades of problems that were beyond the schools’ 12 While California did remove some providers when preparing a revised list for later cohorts, providers were chosen based on their applications rather than on an analysis of the work and success they had with prior II/USP schools. Improving Low-Performing Schools through External Assistance 19 control. For example, these schools typically faced greater turnover in staff and on average had less experienced school personnel than higher performing schools (Allen, 2005). As Mintrop (2004) describes, these are challenging work environments. Successfully turning around low-performing schools requires an examination of both the problems relating to the internal functioning of the school organization, as well as systemic factors (policies and practices within and outside of the school system) contributing to low school performance. Finally, the strength of this policy is closely linked to the resources available for supporting schools. As McLaughlin (1987) argues, a combination of pressure and support is critical. Yet, our studies indicate that the support components were so weakened that only the pressure was having an effect, albeit limited, on school staff in Chicago. Without the high levels of intensity required for the support to have a marked effect on school performance, the support component of these policies was simply not adequate. This was particularly problematic for the lowest capacity schools, many of which experienced little change in performance despite the assistance they received. In essence, simply requiring assistance does little to improve educator and organizational performance. Whatever its original design, a policy is transformed as agents at multiple levels of the system interpret, adapt, and act (Elmore, 1987; Spillane, 2000; Weatherly & Lipsky, 1977). In Chicago and California, these policies were designed to improve schools through professionals in the field and were a promising way of addressing the limited capacity of the district and state by relying on outside experts and encouraging the investment of these experts in turning around low-performing schools. However, these attempts met with minimal success, in part because individuals at all levels of the support system lacked the capacity to ensure that the appropriate assistance was provided through means such as effective selection processes for external support providers and strong monitoring and evaluating of the assistance provided. To date, NCLB implementation has not focused on this critical area of school, district, and state capacity-building (Sunderman, 2008a). The findings from Chicago and California are important to our nation’s understanding of what is required of schools, district, states, and external agencies in the improvement of low- performing schools. While NCLB, like the Chicago and California policies, requires support from external groups to assist low-performing schools in school improvement, this support will likely have little impact on teaching and learning unless the support component is more clearly specified to focus on instruction; a more rigorous screening process exists at the school or district level to ensure a coherent fit between school needs and support provider programs; substantial resources are available (perhaps targeted toward those schools with the greatest needs) to allow for the necessary intensity of support at the school site; monitoring of the support providers occurs to lessen the variation in the quality of support across schools; and district and state practices that contribute to persistent low performance in some schools are addressed and corrected. In sum, our findings suggest that improving low-performing schools will only occur after a focused diagnosis of school needs, matching of these needs with particular external support providers, and intensive and sustained support by these groups. Rather than just brokering this assistance, states and districts must take a more active role in building school-level capacity and monitoring external support provider quality; otherwise, the support component of NCLB will likely have little impact on school performance. Education Policy Analysis Archives Vol. 17 No. 7 20 References Allen, M. (2005). Eight questions on teacher recruitment and retention: What does the research say? Denver, CO: Education Commission of the States. Berman, P., & McLaughlin, M. (1978). Federal programs supporting educational change, volume VII: Factors affecting implementation and continuation. Santa Monica: RAND. Center on Education Policy. (2008). A call to restructure restructuring: Lessons from the No Child Left Behind Act in five states. Washington D. C.: Author. Chimerine, C. B., Haslam, M. B., & Laguarda, K. G. (1994). Third-year evaluation of the nine site program improvement initiative. Washington, DC: Policy Studies Associates. Cohen, D. K., & Ball, D. L. (1999). Instruction, capacity, and improvement (CPRE Research Report Series No. RR-43). Philadelphia: Consortium for Policy Research in Education. Datnow, A., & Honig, M. I. (2008). Introduction to the special issue on scaling up teaching and learning improvement in urban districts: The promises and pitfalls of external assistance providers. Peabody Journal of Education, 83, 323–327. David, J. L., Kannapel, P. J., & McDiarmid, P. (2000). The influence of distinguished educators on school improvement: A study of Kentucky’s school intervention program. Lexington, KY: Partnership for Kentucky’s Schools. Desimone, L. (2002). How can comprehensive school reform models be successfully implemented? Review of Educational Research, 72(3), 433–480. Desimone, L., Porter, A. C., Garet, M., Suk Yoon, K., & Birman, B. (2002). Does professional development change teachers’ instruction? Results from a three-year study. Educational Evaluation and Policy Analysis, 24(2), 81–112. Diamond, J. B., & Spillane, J. P. (2004). High-stakes accountability in urban elementary schools; Challenging or reproducing inequality? Teachers College Record, 106(6), 1145–1176. Elmore, R. (1987). Reform and the culture of authority in schools. Educational Administration Quarterly, 23(4), 60–78. Finnigan, K. S., & Gross, B. (2007). Do accountability policy sanctions influence teacher motivation? Lessons from Chicago’s low-performing schools. American Educational Research Journal, 44(3), 594–629. Finnigan, K., & O’Day, J. (2003). External support to schools on probation: Getting a leg up? Philadelphia, PA: Consortium for Policy Research in Education and Consortium on Chicago School Research. Improving Low-Performing Schools through External Assistance 21 Fullan, M. (1991). The new meaning of educational change (2nd Ed.). New York: Teachers College Press. Garet, M. S., Porter, A. C., Desimone, L., Birman, B. F., & Suk Yoon, K. (2001). What makes professional development effective? An analysis of a national sample of teachers. American Educational Research Journal, 38(3), 915–945. Gwynne, J., & Easton, J. Q. (2001, April). Probation, organizational capacity, and student achievement in Chicago elementary schools. Paper presented at the annual meeting of the American Educational Research Association, Seattle. Hatch, T. (1998). The differences in theory that matter in the practice of school improvement. American Educational Research Journal, 35(1), 3–31. Hawley, W. D., & Valli, L. (1998). Guide to the National Partnership for Excellence and Accountability in Teaching (NPEAT). Washington, DC: ERIC Clearinghouse on Teaching and Teacher Education, American Association of Colleges for Teacher Education. Honig, M. I., & Ikemoto, G. S. (2008). Adaptive assistance for learning improvement efforts: The case of the Institute for Learning. Peabody Journal of Education, 83, 328–363. Huberman, M. (1995). Professional careers and professional development: Some intersections. In T. R. Guskey & M. Huberman (Eds.), Professional development in education: New paradigms and practices (pp. 193–224). New York: Teachers College Press. Le Floch, K., Boyle, A., & Therriault, S. B. (2008). Help wanted: State capacity for school improvement. Washington D. C.: American Institutes for Research. Levine, D. & Lezotte, L. (1990). Unusually effective schools: A review and analysis of research and practice. Madison, WI: National Center for Effective Schools Research and Development. McLaughlin, M. (1987). Learning from experience: Lessons from policy implementation. Educational Evaluation and Policy Analysis, 9(2), 171–178. Mintrop, H. (2004). School on probation: How accountability works (and doesn’t work). New York: Teachers College Press. Mintrop, H., & Trujillo, T. (2005). Corrective action in low performing schools: Lessons for NCLB implementation systems from first-generation accountability systems. Education Policy Analysis Archives, 13(48). Retrieved January 8, 2009, from http://epaa.asu.edu/epaa/v13n48/. Newmann, F. M., Smith, B., Allensworth, E., & Bryk, A. (2001). School instructional program coherence: Benefits and challenges. Chicago: Consortium on Chicago School Research. Newmann, F. M., & Wehlage, G. G. (1995). Successful school restructuring: A report to the public and educators by the Center on Organization and Restructuring of Schools. Madison, WI: Education Policy Analysis Archives Vol. 17 No. 7 22 Center on the Organization and Restructuring of Schools, University of Wisconsin- Madison. O’Day, J., & Bitter, C. (2003). Evaluation study of the immediate intervention/underperforming schools program and the high achieving/improving schools program of the Public Schools Accountability Act of 1999. Palo Alto, CA: American Institutes for Research. O’Day, J., Goertz, M. E., & Floden, R. F. (1995). Building capacity for education reform (Policy Brief RB-18). New Brunswick, NJ: Consortium for Policy Research in Education. Olson, L. (2004, January 28). States train sights on school districts for interventions. Education Week, 23(20), 1, 21. Porter, A., Floden, R., Freeman, D., Schmidt, W., & Schwille, J. (1988). Content determinants in elementary school mathematics. D. Grouws & T. Cooney (eds.), Perspectives on research on effective mathematics teaching, volume 1 (pp. 96–113). Reston, VA: National Council of Teachers of Mathematics. Purkey, S., & Smith, M. (1983). Effective schools: A review. Elementary School Journal, 83(4), 427–452. Richard, A. (2004, January 7). States unable to help all struggling schools. Education Week, 23(16), 19, 23. Shulman, L. S. (1986). Those who understand: Knowledge growth in teaching. Educational Researcher, 15(2), 4–14. Smylie, M. A., Bilcer, D. K., Greenberg, R. C., & Harris, R. L. (1998, April). Urban teacher professional development: A portrait of practice from Chicago. Paper presented at the annual meeting of the American Educational Research Association, San Diego, CA. Spillane, J. (2000). Cognition and policy implementation: District policymakers and the reform of mathematics education. Cognition and Instruction, 18(2), 141–179. Sunderman, G. L. (2008a). Conclusion: Accountability after five years. In G. L. Sunderman (Ed.), Holding NCLB accountable: Achieving accountability, equity, & school reform (pp. 223–225). Thousand Oaks, CA: Corwin Press. Sunderman, G. L. (Ed.). (2008b). Holding NCLB accountable: Achieving accountability, equity, & school reform. Thousand Oaks, CA: Corwin Press. Sunderman, G. L., & Orfield, G. (2008). Massive responsibilities and limited resources: The state response to NCLB. In G. L. Sunderman (Ed.), Holding NCLB accountable: Achieving accountability, equity, & school reform (pp. 121–136). Thousand Oaks, CA: Corwin Press. Supovitz, J. (2008). Melding internal and external support for school improvement: How the district role changes when working closely with external instructional support providers. Peabody Journal of Education, 83, 459–478. Improving Low-Performing Schools through External Assistance 23 Supovitz, J. A., & Turner, J. (2000). The influence of standards-based reform on classroom practices and culture. Journal of Research on Science Teaching, 37(9), 963–980. U. S. Department of Education. (2006). LEA and school improvement: Non-regulatory guidance. Washington, DC: Author. Retrieved March 14, 2009, from http://www.ed.gov/policy/elsec/guid/schoolimprovementguid.pdf. U. S. Department of Education. (2004). Evaluation of Title 1 accountability systems and school improvement efforts (TASSIE): First-year findings. Washington, DC: Author. Retrieved March 14, 2009, from http://policyweb.sri.com/cep/publications/TASSIE_Year1Report_01-02.pdf. Weatherly, R., & Lipsky, M. (1977). Street-level bureaucrats and institutional innovation: Implementing special education reform. Harvard Educational Review, 47(2), 171–197. Education Policy Analysis Archives Vol. 17 No. 7 24 About the Author Kara S. Finnegan University of Rochester Catherine Bitter American Institutes for Research Jennifer O’Day American Institutes for Research Email: kfinnigan@warner.rochester.edu Kara S. Finnigan is an Assistant Professor of Education Policy at the University of Rochester’s Warner School of Education. Her research focuses on low-performing schools and high-stakes accountability, principal leadership, teacher motivation, inter-district choice, and charter schools. She was project manager for the Chicago School Probation Study. Catherine Bitter is a Senior Research Analyst at the American Institutes for Research. Her work has focused on K-12 school reform, accountability, literacy instruction, and district reform. She managed the study of the Immediate Intervention/Underperforming Schools Program, conducted for the California Department of Education. Jennifer O’Day is a Managing Research Scientist and policy analyst in the Education Program of the American Institutes for Research. Her recent research has centered on accountability policies and school improvement efforts, particularly in high poverty, low- performing schools. She was Principal Investigator/Project Director for both the Chicago and California studies. Improving Low-Performing Schools through External Assistance 25 EDUCATION POLICY ANALYSIS ARCHIVES http://epaa.asu.edu Editor: Sherman Dorn, University of South Florida Production Assistant: Chris Murrell, Arizona State University Editorial Assistant: Judy Castillo, University of South Florida General questions about appropriateness of topics or particular articles may be addressed to the Editor, Sherman Dorn, epaa-editor@shermandorn.com. Editorial Board Noga Admon Jessica Allen Cheryl Aman Michael W. Apple David C. Berliner Damian Betebenner Robert Bickel Robert Bifulco Anne Black Henry Braun Nick Burbules Marisa Cannata Casey Cobb Arnold Danzig Linda Darling-Hammond Chad d'Entremont John Diamond Amy Garrett Dikkers Tara Donohue Gunapala Edirisooriya Camille Farrington Gustavo Fischman Chris Frey Richard Garlikov Misty Ginicola Gene V Glass Harvey Goldstein Jake Gross Hee Kyung Hong Aimee Howley Craig B. Howley William Hunter Jaekyung Lee Benjamin Levin Jennifer Lloyd Sarah Lubienski Les McLean Roslyn Arlin Mickelson Heinrich Mintrop Shereeza Mohammed Michele Moses Sharon L. Nichols Sean Reardon A.G. Rud Ben Superfine Cally Waite John Weathers Kevin Welner Ed Wiley Terrence G. Wiley Kyo Yamashiro Stuart Yeh Education Policy Analysis Archives Vol. 17 No. 7 26 EDUCATION POLICY ANALYSIS ARCHIVES http://epaa.asu.edu New Scholar Board English Language Articles 2007–2009 Wendy Chi Corinna Crane Jenny DeMonte Craig Esposito Timothy Ford Samara Foster Melissa L. Freeman Kimberly Howard Nils Kauffman Felicia Sanders Kenzo Sung Tina Trujillo Larisa Warhol Improving Low-Performing Schools through External Assistance 27 Archivos Analíticos de Políticas Educativas http://epaa.asu.edu Editor Gustavo E. Fischman Arizona State University Asistentes editoriales: Rafael O. Serrano (ASU) & Lucia Terra (UBC) Hugo Aboites UAM-Xochimilco, México Armando Alcántara Santuario CESU, México Claudio Almonacid Avila UMCE, Chile Dalila Andrade de Oliveira UFMG, Brasil Alejandra Birgin FLACSO-UBA, Argentina Sigfredo Chiroque IPP, Perú Mariano Fernández Enguita Universidad de Salamanca. España Gaudêncio Frigotto UERJ, Brasil Roberto Leher UFRJ, Brasil Nilma Lino Gomes UFMG, Brasil Pia Lindquist Wong CSUS, USA María Loreto Egaña PIIE, Chile Alma Maldonado University of Arizona, USA José Felipe Martínez Fernández UCLA, USA Imanol Ordorika IIE-UNAM, México Vanilda Paiva UERJ, Brasil Miguel A. Pereyra Universidad de Granada, España Mónica Pini UNSAM, Argentina Romualdo Portella de Oliveira Universidade de São Paulo, Brasil Paula Razquin UNESCO, Francia José Ignacio Rivas Flores Universidad de Málaga, España Diana Rhoten SSRC, USA José Gimeno Sacristán Universidad de Valencia, España Daniel Schugurensky UT-OISE Canadá Susan Street CIESAS Occidente,México Nelly P. Stromquist USC, USA Daniel Suárez LPP-UBA, Argentina Antonio Teodoro Universidade Lusófona, Lisboa Jurjo Torres Santomé Universidad de la Coruña, España Lílian do Valle UERJ, Brasil