Microsoft Word - v13n40.doc Readers are free to copy, display, and distribute this article, as long as the work is attributed to the author(s) and Education Policy Analysis Archives, it is distributed for non- commercial purposes only, and no alteration or transformation is made in the work. More details of this Creative Commons license are available at http://creativecommons.org/licenses/by-nc-nd/2.5/. All other uses must be approved by the author(s) or EPAA. EPAA is published jointly by the Colleges of Education at Arizona State University and the University of South Florida. Articles are indexed by H.W. Wilson & Co. Send formal remarks to EPAA Commentary Editor Casey Cobb (casey.cobb@uconn.edu) and errata requests to Sherman Dorn (epaa-editor@shermandorn.com). EDUCATION POLICY ANALYSIS ARCHIVES A peer-reviewed scholarly journal Editor: Sherman Dorn College of Education University of South Florida Volume 13 Number 40 September 28, 2005 ISSN 1068–2341 A Forced March for Failing Schools: Lessons from the New York City Chancellor's District1 Deinya Phenix, Dorothy Siegel, Ariel Zaltsman, Norm Fruchter New York University Citation: Phenix, D., Siegel, D., Zaltsman, A., & Fruchter, N. (2005). A forced march for failing schools: Lessons from the New York City Chancellor’s District. Education Policy Analysis Archives, 13(40). Retrieved [date] from http://epaa.asu.edu/epaa/v13n40/. Abstract In the mid-nineties, the New York City Schools Chancellor created a citywide improvement zone to take over a significant proportion of the city’s lowest performing schools whose local community school districts had failed to improve them. This “Chancellor’s District” defined centralized management, rather than local control, as the critical variable necessary to initiate, enforce and ensure the implementation of school improvement. This large-scale intervention involved both a governance change and a set of capacity-building interventions presumably unavailable under local sub-district control. Our study retrospectively examined the origins, structure and components of the Chancellor’s District, and analyzed the characteristics and outcomes of the elementary schools mandated to receive these interventions. Our longitudinal analysis compared Chancellor’s District schools to New York City’s other state-identified low performing schools, based on a school- level panel of performance, demographic, human resource, and expenditure data 1 Partial support for the research leading to this article was provided by the J.P. Morgan Chase Foundation. The authors would also like to thank Bree Picower, Margaret Murphy, Ben Kennedy, Natasha Pchelintseva and Cameron Cole for their work on earlier formations of the analysis. We also thank Richard Arum, Amy Ellen Schwartz, Leanna Stiefel, Dae Yeop Kim, Hella Bel Hadj Amor, Colin Chellman, Patrice Iatarola and three anonymous reviewers for their comments on an earlier version of this paper. Education Policy Analysis Archives Vol. 13 No. 40 2 collected from district Annual School Report Cards and School Based Expenditure Reports from 1998–99 through 2001–02. The results suggest that the Chancellor’s District intervention improved these schools’ instructional capacity and academic outcomes, both relative to where these schools would have been and relative to comparable schools. Keywords: school reform; low performing schools; accountability; district intervention Introduction This article analyzes the results of the Chancellor’s District, an initiative created to accelerate their improvement by remove state-identified low-performing schools from their local district authorities, imposing a uniform curriculum, intensive professional development, reduced class size, extended time and other reforms. The seven-year Chancellor’s District initiative represents both an unprecedented intervention into New York City school governance and a major challenge to several reigning theories about the relationship between centralized administration and local school change. Consider, first, how the Chancellor’s District departed from the New York City school system’s governance norms. From 1969 to 2003, New York City’s public elementary and middle schools were governed by 32 decentralized community school districts (hereafter sub-districts), administered by locally elected school boards and their appointees, the community superintendents. These sub-districts were quite large, averaging more than 20,000 students, with several of the largest districts enrolling more than 40,000 students. Many of these sub-districts would have ranked among the 50 largest school systems in the country had they been independent jurisdictions. During their thirty-four years of relative autonomy, these sub-districts developed diverse, and differentially effective, patterns of operation. Consistently high performance characterized schools in some sub-districts, while poor management and dismal student outcomes plagued schools in others. Though the grim correlations among race, poverty and student achievement that characterize most urban districts have also persisted in New York City, individual school outcomes varied widely, both across and within the community school sub-districts. Academic performance was especially poor, and particularly highly correlated with indicators of race and poverty, in those sub-districts whose governance was marked by patterns of corruption, patronage and, most importantly, a consistent failure to focus on improving teaching and learning. The school system’s central administration, governed by an appointed citywide board of education and a chief administrative officer (the Chancellor), had possessed the authority to remove failing schools from their community school sub-districts since the city system was decentralized in 1969. But that power remained unexercised for almost three decades until 1996, when the reigning Chancellor created a new, geographically non-contiguous sub-district, and imposed the same improvement regimen on each school. The Chancellor’s District became a new, non-geographic improvement zone that eventually removed some 58 elementary and middle schools from local sub- district control. This effort to remove failing schools from their sub-district jurisdictions in order to improve them was a radical change in New York City school governance. From the onset of decentralization, central leadership had bemoaned sub-district failure but had refused to intervene, either to force sub- districts to take steps to improve their schools or to take failing schools away from local sub-district control. The Chancellor’s assertion of the power to take over failing schools, and his creation of a new A Forced March for Failing Schools 3 district to force-feed their improvement represents an historic departure from three decades of unobstructed local sub-district control. This article describes the origins, structure, and components of the Chancellor’s District, and details our analysis of whether these particularly low performing schools were improved more than other state-identified low performing schools. To understand the nature of the intervention that the Chancellor’s District represented, we analyzed administrative documents, including budget allocation memoranda, and conducted numerous interviews with state and city administrators. To evaluate the impact of the Chancellor’s District as an intervention, we conducted a longitudinal analysis that compared the academic performance of Chancellor’s District schools to New York City’s other state identified, low performing (SURR) schools. The next section situates the Chancellor’s District initiative in reform theory and reform efforts, and the following section details the results of interviews with district officials about how this special district worked as an intervention. The subsequent sections detail the quantitative data and methods used for this study, including its limitations, and changes in the Chancellor’s District schools and their implications about the district’s overall effectiveness. The Chancellor’s District initiative ended in July 2003, with the implementation of a system- wide restructuring policy that reorganized the entire New York City school system. The 32 elementary and middle schools in the Chancellor’s District were transferred back to their local sub- districts, which were themselves subsumed into a new regional structure under the Chancellor’s direct control. Thus the Chancellor’s District initiative is now history, and each of the new administrative regions is now responsible for improving its failing schools. But the extent to which the Chancellor’s District initiative succeeded in improving student outcomes, particularly in the failing elementary schools whose outcomes we examined, directly challenges the reigning theories that link school improvement to decentralization, and has important implications for the variety of school- and district-level improvement efforts underway in urban districts across the country. Big Bureaucracy, District Capacity and School Improvement The Chancellor’s District initiative is unique among recent large-scale reform efforts. Given the scale and complexity of the New York City system, the Chancellor’s District initiative is akin to state takeover efforts of poorly performing districts. Because the Chancellor is responsible for more schools (currently over 1,300) than many state chiefs, the Chancellor’s administrative relationship to the 32 sub-districts was comparable to state commissioners’ relationships to their local school districts. Moreover, as most states’ takeover efforts of local districts have been for financial mismanagement rather than instructional failure, few state takeovers have targeted as many schools for restructuring, redesign and instructional improvement as the Chancellor’s District effort. The Chancellor’s District initiative also poses a strong challenge to three important arguments in the research about the relationship between district administration and school change. Historically, many researchers and critics have inveighed against the effects of district size and the resulting bureaucracies, contending that large urban systems have become ungovernable and impervious to reform efforts (Domanico, 1994). Seymour Sarason’s (1996) classic analysis maintained that big-city schools are “insulated,” “encapsulated,” and in other ways immune from hierarchically imposed efforts to alter dysfunctional practice at the school level. In another classic study, Weick (1976) argued that the “loose coupling” within the various layers of complex urban systems stymies the efforts of centralized interventions to produce changes in school practices that might lead to school improvement. Education Policy Analysis Archives Vol. 13 No. 40 4 This analysis of the inevitable barriers to change that scale and hierarchical complexity impose may not necessarily imply reform from below or one-school-at-a-time change. But the critique’s prognosis of the likely effectiveness of centralized administrative efforts to drive change is quite bleak. The Chancellor’s District’s forced-march efforts to improve the schools taken from their local sub-districts challenge this critical tradition. Second, the Chancellor’s District initiative challenges several influential currents of recent reform theory that link the necessity for decentralization with the need to provide maximum autonomy at the school level to achieve successful schools. In Politics, Markets and America’s Schools, John Chubb and Terry Moe (1990) argued that the key characteristic that distinguishes academically effective private schools from less effective public schools is the extent of autonomy at the school level. Chubb and Moe’s influential arguments stressed the inevitability of bureaucratization and consequent poor school performance unless schools are severed from district control and governed by market principles. Recent theoretical efforts to establish the primacy of the school, rather than the district, as the locus of improvement have not been monopolized by conservative scholars or market advocates. The Cross City Campaign for Urban School Reform, an advocacy organization composed of city reform groups, education advocates and parent activists, published an influential report (Hallet, 1995) that urged radical decentralization, to the school level, of all essential instructional and administrative functions, leaving school districts with only vestigial governing roles. The authors of Reinventing Central Office argued that urban school districts had consistently failed to implement effective improvement efforts, and characterized their administrations as retarding forces that stifled school-based reform efforts. A third recent and influential reform stream stresses the necessity for bottom-up or school- by-school reform efforts. Several national reform consortia, such as the Coalition of Essential Schools, the School Development Program, the Accelerated Schools project and the New American Schools, all focus on the need to generate individual school improvement through the implementation of replicable programs. This reform stream was elevated into national prominence through federal legislation, the Comprehensive School Reform Demonstration program, popularly known as the Obey–Porter Act of 1997, which has allocated more than $300 million annually for grants to individual schools to implement supposedly research-validated improvement models. The role of the district in initiating, coordinating, or supporting these school-based efforts was, at best, subordinated to the role of the intermediary organizations marketing the particular models or, at worst, essentially untheorized (Bodilly, 2001). These reform currents that target individual schools as key improvement sites have begun to be challenged by efforts to define the school district as the necessary locus of capacity-building initiatives. Ascher, Fruchter, and Ikeda (1999), for example, argued that the local district “is the critical actor that can encourage or retard the school’s development of the necessary capacity for self-improvement” (Ascher, Fruchter, & Ikeda, 1999, p. 43). The Council of Great City Schools’ Foundations for Success (2002) analyzed the efforts of three urban districts to improve student academic performance, and to narrow the achievement gap between white students and students of color. The Annenberg Institute for School Reform has created School Communities that Work: A National Task Force on the Future of Urban Districts, to help districts restructure themselves into effective support systems focused on improving instruction. The University of Pittsburgh’s Learning Research and Development Center has created the Institute for Learning to help urban districts reorganize and improve their capacities to help their schools, and themselves, become continuous learning organizations (Snipes, Doolittle, & Herlihy, 2002). But in 1996, the Chancellor was bucking several traditions of reform theory when he took over schools whose local sub-districts had failed to improve them. His theory of action defined A Forced March for Failing Schools 5 centralized management, rather than decentralized local control, as the critical variable necessary to initiate, enforce and ensure the implementation of school improvement. Defining the core issues as the ability to mobilize the political will and instructional capacity necessary to improve schools, he asserted that the central administration could mandate the policies, implement the procedures and provide the resources necessary to transform failing schools. Given this premise, the Chancellor’s District involved both a governance change and a coherent set of capacity-building interventions presumably unavailable to low-performing schools under local sub-district control. The Design of the Chancellor’s District Since 1989, the New York State Education Department (SED) has used the Schools Under Registration Review (SURR) process to identify low-performing schools, and place them on a list of schools under registration review. SED requires SURR schools to create a comprehensive education plan, and can order chronically low-performing schools to undergo school redesign. Schools that fail to improve may have their registration revoked, which means they are effectively closed—the ultimate sanction of the SURR process. In October 1995, the New York State Education Commissioner informed New York City’s Schools Chancellor that he would revoke the registrations of sixteen chronically low-performing New York City schools that had long languished on the SURR list if their student performance did not improve by June 1997. In response, the Chancellor met with the schools’ local sub-district superintendents, community school board members, principals, and parent leaders, and notified them that he was requiring the schools to develop and implement instructional improvement plans. In half the schools, he removed the principals and mandated comprehensive school redesign. Unsatisfied with the sub-district and school responses to his actions, the Chancellor decided, in February 1996, to intervene directly in schools in which, in his determination, the local sub- districts had “fail[ed] to demonstrate the capacity to redesign failing organizations” (New York City Board of Education, 1999b, p. 1). On his recommendation, the New York City Board of Education created the Chancellor’s District, with the mission “to develop and expand central, district, and local school capacity to transform failing school organizations into redesigned and revitalized schools that meet high educational standards for students” (New York City Board of Education, n.d., p. vi). He immediately transferred ten schools into the new sub-district. In the seven years of its operation, the Chancellor’s District included 58 elementary and middle schools.2 These schools entered the District in annual cohorts of various sizes. By the end of the 2002–03 school year, the District had closed eleven schools and returned fifteen to their home sub-districts. The Chancellor’s District has taken over schools from every New York City borough except Staten Island, with a disproportionate number from the Bronx. There were 32 elementary and middle schools in the District at the time of its dissolution in June 2003. The 1999–00 school year was a seminal year for the Chancellor’s District. Between 1996 and 1999, the district had taken in only a small number of schools. But in 1999, after an extensive review of the patterns of failure across the city’s low-performing SURR schools, the Chancellor decided to take 37 more of the city’s lowest performing schools into the Chancellor’s District, and imposed a new, highly structured improvement plan, A Model of Excellence, on all the District’s schools. He also increased staff capacity in all the SURR schools, especially those in the Chancellor’s District. 2 Throughout its existence, the Chancellor’s District also included ten high schools, which had already been under more centralized control. These schools are not included in the analysis here. Education Policy Analysis Archives Vol. 13 No. 40 6 His goal was to remove every New York City school from the SURR list within two years (New York City Board of Education, 1999b). Table 1 Schools in the New York City Chancellor's District (CD), 1996–2003 Academic Year Entered the CD Returned to Home District Closed 1996–97 9 0 0 1997–98 3 0 0 1998–99 0 2 0 1999–00 37 0 2 2000–01 1 8 5 2001–02 5 5 3 2002–03 3 — 1 Total 58 15 11 Sources: New York State Education Department; New York City Board of Education City officials divided the city’s SURR schools into three groups. Category 1 schools were those assessed at highest risk of continued failure. Category 2 schools were at the next highest risk of failure. Category 3 schools were schools that were improving enough to become candidates for removal from the SURR list in the following year. Ten of the elementary and middle schools were identified as Category 1 schools had already been placed in the Chancellor’s District in previous years. Thirty-seven more Category 1 schools were added to the Chancellor’s District for the 1999–00 school year. Though the remaining five Category 1 schools exhibited the same pattern of failure as the schools in the Chancellor’s District, they were allowed to remain in their local districts because their districts engaged in other major reform initiatives and the Chancellor decided that their sub- districts had the capacity to support their schools’ improvement plans. The superintendents of these districts met monthly with the Chancellor’s District Supervising Superintendent to coordinate implementation of the intervention. Thus, in 1999–00, the Chancellor’s District consisted of 47 of the city’s SURR schools deemed to be the lowest performing elementary and middle schools. The district was sub-divided into four regions, each with its own instructional superintendent. In that same year, Chancellor’s District schools began implementing the new Model of Excellence. Class size was reduced throughout the district. A maximum of 20 students were mandated for kindergarten through grade 3, and 25 students for grades 4 through 8. Instructional time was increased by extending both the school day and the school year. The school day was lengthened to 20 minutes longer than in other elementary and middle schools in New York City. The school calendar was also extended by one week. Instructional time was further enhanced by developing after-school programs, implemented in each Chancellor’s District school through a schedule of activities that extended the school day to 6 pm. The after-school program was designed to “enhance and enrich daily learning” (New York City Board of Education, 2001a, p. 2). Tutoring was offered from 3 to 4 pm in small group settings for those students in grades 3–5 who required extra reading or math assistance. A prescribed instructional program, a mandated daily schedule and a required curriculum were imposed throughout the district. In elementary schools, the schedule mandated two daily 90- minute literacy blocks, the first using Success for All, and the second using the Balanced Literacy A Forced March for Failing Schools 7 program. The daily schedule also included a 60-minute math block, using the Trailblazers math program; and a 30-minute skills block, alternating between math and literacy skills. Science and social studies were each taught once per week. Because the time devoted to literacy instruction in the elementary school schedule was almost three times that assigned to math, the Chancellor’s District was perceived as concentrating on literacy skills improvement at the elementary level much more intensively than on math. The district also provided intensive professional development. Each school was assigned at least four on-site staff developers focused on English Language Arts, mathematics, technology and Success for All. Extra time was provided for professional development designed to be intensive, systematic, structured and aligned with the curriculum. Each school was provided with an on-site teacher center staffed by a teacher specialist who offered additional coaching and professional development. Assessments, integrated into the Success for All and Trailblazers programs, were designed to provide regular feedback to classroom teachers. In kindergarten through grade 3, the schools used New York City’s Early Childhood Literacy Assessment System (ECLAS) to assess and improve literacy growth. Specially developed benchmark assessments in reading and mathematics were used to assess the performance of students from grades 3 on. Table 2 Model of Excellence Components, NYC Low-Performing Schools Component Chancellor's District schools (N=49) Other SURR schools (N=53) Reduced class size Extended school day and year After-school program Mandated instructional program Mandated daily schedule Mandated curriculum Number of on site staff developers 4 2 Extra time for staff development Prescribed staff development Teacher center and teacher specialist Student assessment program Additional supervisory/district support Category 2 and 3 schools (“other SURR schools”) received fewer intervention components than schools in the Chancellor’s District, as illustrated in Table 2. Central authorities allocated additional funds, initially $20,000,000 (New York City Board of Education, 1999a), targeted to implement the specific interventions prescribed in the Model of Excellence in the Chancellor’s District schools. By 2000–01, when the Chancellor’s District model was fully implemented, the District’s schools spent an average of $2,400 more per student than the other SURR schools.3 Most of the increased spending represented increased teacher costs, including two programs designed to attract certified teachers to the Chancellor’s District schools. Certified teachers who 3 According to the New York City Board of Education School Based Expenditure Reports, Chancellor’s District elementary and middle schools in 2000-01 spent an average of $13,150 per student. Other SURR schools spent an average of $10,744. This compares to an overall average New York City per student expenditure of $9,679 for elementary and middle schools. Education Policy Analysis Archives Vol. 13 No. 40 8 chose to work in Extended Time Schools (ETS) received 15% additional pay in exchange for additional work. The ETS program, developed in collaboration with the United Federation of Teachers, was implemented in 1999–00 in all but two Chancellor’s District schools. Certified, experienced private school teachers who chose to teach in the Chancellor’s District received $10,000 bonuses (Iatarola, 2001). The Chancellor’s District also introduced several policies to improve the qualifications, quality, preparation, and stability of the leadership and staff in Chancellor’s District schools. Most of the Chancellor’s District schools were assigned new principals. Additional assistant principals were also assigned, and both principals and assistant principals received professional development focused on how to supervise implementation of the instructional plan. The District, through its four instructional regions, provided additional intensive instructional and supervisory support to the schools’ leadership and staff. On-site professional development specialists, including three full-time instructional specialists (one each in literacy, mathematics and technology), a Success for All facilitator, and a teacher center specialist, provided consistent, intensive, highly structured professional development for all teachers in the district’s elementary schools. Teachers attended a one-week professional development program every August, in addition to the training they received on citywide staff development days. Furthermore, many ineffective teachers were removed from the Chancellor’s District schools. According to one official, the Chancellor’s District absorbed the cost of approximately two dozen teachers’ salaries until their cases were adjudicated, rather than allow them to remain in the classroom. Finally, the Chancellor introduced two important teacher incentive initiatives for all SURR schools, including those in the Chancellor’s District. The immediate impetus was the state mandate that as of September 1, 2000, only certified teachers would be assigned to SURR schools. In exchange for agreeing to work in SURR or other hard-to-staff schools for a full year, teachers received grants of up to $3,400 from the Teachers for Tomorrow program, which they could use to repay educational loans or meet other qualified educational expenses. Beginning in the 1999–00 school year, candidates for teacher positions who lacked traditional certification could participate in a new alternative certification program called the New York City Teaching Fellows Program. This program met the cost of a master’s degree in education and provided training during the summer, as well as mentors during the school year. New York State gave New York City Teaching Fellows provisional certification to allow them to teach while completing their degrees. Most Teaching Fellows were placed in the Chancellor’s District or other SURR schools. SURR schools often had a disproportionate number of full-time special education students, the result of previous sub-district placement decisions. City policy for all SURR schools, including Chancellor’s District Schools, was to examine the number of special education students and reduce disproportionate placements. As these program descriptions indicate, the Chancellor’s District mounted a comprehensive effort to improve the poorly performing schools that had been removed from their sub-district jurisdiction. The next section describes how we assessed the effectiveness of the Chancellor’s District’s efforts to improve these schools. Methods In addition to our analysis of New York City Board of Education budgets and interview data described above, we conducted a longitudinal analysis that compared the academic performance of A Forced March for Failing Schools 9 Chancellor’s District elementary schools to New York City’s other state identified, low performing (SURR) schools. We constructed a school-level panel, based on data collected from Annual School Report Cards and School Based Expenditure Reports from 1998–99 through 2001–02. These two administrative databases offer a wealth of information on student demographics, teacher credentials and experience, school organizational characteristics, and expenditures for all schools in the New York City school system. Our chief outcome variable was schools’ fourth-grade academic performance, expressed as average scale scores, the percent of students meeting the standard (Levels 3 and 4), and the percent far below the standard (Level 1) on the state’s English Language Arts (hereafter reading) and Mathematics exams. During the years observed, fourth grade test results were the primary criteria that the State Education Department used to determine elementary schools’ accountability status. We also examined differences in student, school, and teacher characteristics, as well as general education expenditures for Chancellor’s District schools, other SURR schools, and the citywide average, across all four years. Tables reporting changes over time on these variables are presented in the section below. Tables reporting cross-sectional means, standard deviations, and ranges for each variable analyzed are available from the authors. Study Sample As we indicate above, the Chancellor’s District as an intervention evolved over time, culminating in the implementation of the Model of Excellence in the 1999–00 school year. This evolution in design, and the extent of the changes in the Chancellor’s District as an intervention across time, posed analytical challenges for our evaluation. Not only were different schools in—or out—of the Chancellor’s District at different times, but the instructional regimen imposed on those schools also varied across time. Our solution was to focus our analysis on the elementary schools that were state identified as low performing (SURR) in 1998–99 and entered the Chancellor’s District in 1999–00 and thus received the full intervention described in the previous section. We use their 1998–99 data as a baseline. As a comparison group, we use the other elementary schools on the SURR list in 1998–99. Because the number of middle schools that were on the SURR list in 1998–99 was too small to support an appropriate statistical analysis, our study focuses on elementary schools only. For the purpose of this analysis, elementary schools are all schools that included a fourth grade, regardless of their overall grade configuration.4 Category 1 schools outside the Chancellor’s District, described above, were not included in the analysis. Table 3 Sample Schools By Status, 1998–2002 School Status 1998–99 1999–00 2000–01 2001–02 Chancellor's District schools Open — 25 25 24 Closed — 0 0 1 Other SURR schools Open 50 25 25 23 Closed 0 0 0 2 4 This includes nine schools spanning Kindergarten through Eighth Grade. Education Policy Analysis Archives Vol. 13 No. 40 10 As Table 3 shows, half the elementary schools that were on the SURR list in 1998–99 entered the Chancellor’s District in 1999–00. Our univariate analysis presents data for the Chancellor’s District elementary schools that remained open for the four school years from 1998– 99, the pre-implementation or baseline year, through 2001–02, and compares changes in performance and other variables in those schools to changes in other SURR schools.5 We present the citywide averages for elementary schools as an overall benchmark. Regression Models To assess the effect of the Chancellor’s District intervention on school performance in the context of other potential causal factors, we developed regression models that include controls for student characteristics (e.g., school-level English proficiency, poverty, attendance, student demographics), as well as school-level characteristics (e.g., school size, expenditures, certain teacher characteristics). Our dependent variables are school-level fourth-grade reading and math performance.6 Our basic analysis is outlined in the equation: Pst = a + ß1Sst + ß2CDst + ß3Tt + SCHs + est (1) where Pst is the reading or math performance of the school s in year t, Sst is a vector of student and school characteristics, including school size, and CDst is the Chancellor’s District dummy that takes a value of “1” for the Chancellor’s District schools for years 1999–00, 2000–01 and 2001–02, and “0” for the baseline year 1998–99. The comparison SURR schools take “0” for all years. Tt is a vector of year dummies, and SCH is a school fixed effect, essentially a vector of school dummies measuring the effects of unobserved or unmeasured time-invariant school characteristics, such as school culture, leadership, and other school-based factors affecting the implementation of the various SURR and Chancellor’s District components. est is an error term with the usual properties. To correct for heteroskedasticity, we employ robust standard errors. In order to estimate and control for the effect of additional resources on schools, we estimate a fuller model: Pst = a + ß1Sst + ß2CDst + ß3Tt + ß4TCHst + ß5Rst +SCHs + est (2) where TCHst is a vector of human resources, including teacher characteristics and the number of teachers per 100 students, and Rst represents monetary resources (non-teacher per student 5 Schools that were closed during this time period were excluded from the univariate analysis because their inclusion draws the 1998-99 averages downward—making the Chancellor’s District appear artificially successful in the later years. Conversely, closed schools are included in the regression analysis, described below, because their exclusion, as demonstrated by performing the analysis with and without these schools, would bias the Chancellor’s District coefficients upward. 6 Our models are based on ordinary least squares (OLS) analysis. A case could be made that, given the great variety in schools’ sizes, it would be more appropriate to employ weighted least squares (WLS) models and weight our estimates by school enrollment. However, we compared the findings that we report in this paper with those of a WLS regression analysis and found that, in spite of some differences in the size of some coefficients, the conclusions remained the same. Thus, and because our focus is the Chancellor’s District’s impact on schools’ – rather than estimating the student population’s performance, we rely on the OLS results for our assessment. A Forced March for Failing Schools 11 expenditures). We analyze model (2) in reference to model (1), anticipating a change in the coefficient ß2 once resources are introduced to the model. We included school fixed effects in the model to allow a difference-in-difference specification; that is, we take into account how schools differ from one another, and we estimate the impact of the Chancellor’s District over and above the general differences. This methodology offers a precise estimate of Chancellor’s District improvement, averaged across all four years, compared to improvement in other SURR schools, the closest comparison group. SURR schools were also the beneficiaries of additional intervention and support. While these interventions were not as intensive as the many interventions of the Chancellor’s District, SURR schools thus present a moving target against which we measure the Chancellor’s District performance. Measurable Impact The goal of the Chancellor’s District was to increase the instructional capacity and the academic outcomes of the failing schools the district had incorporated. To assess whether, and to what extent, the Chancellor’s District achieved its goal, we compare the Chancellor’s District elementary schools to the other SURR elementary schools—as well as to all New York City elementary schools—on school-level characteristics and fourth grade academic performance in the 1998–99 baseline year. We examine changes in Chancellor’s District, other SURR and the average New York City elementary schools between the 1998–99 and 2001–02 school years, after the target schools had spent three years in the Chancellor’s District. Finally, we report the results of our regression analyses that compare the academic performance in the Chancellor’s District schools to the other SURR schools, while controlling for factors other than the Chancellor’s District interventions, across all years. Table 4 Student Demographics in Sample and All NYC Schools, 1998–99 Demographic characteristics Chancellor's District Schools (N=25) Other SURR Schools (N=25) All NYC schools (N=666) % White 0.8 0.9 17.1 % Black 54.1 56.0 35.6 % Hispanic 43.2 41.6 36.9 % Asian/other 1.9 1.5 10.4 % Limited English proficient 16.9 15.4 14.7 % Recent immigrant 4.4 3.8 7.1 % Free lunch eligible 91.6 93.0 74.7 % Full time special education 8.0** 12.1 5.8 % In this school entire year 90.6** 87.7 91.5 % Days students attended 87.8 88.4 91.0 % Referrals to special education 4.3 4.2 3.6 % Part time special education 6.0 6.2 6.4 Student enrollment 715.9 760.1 795.9 * p < .10; ** p < .05; *** p < .01. Education Policy Analysis Archives Vol. 13 No. 40 12 Chancellor’s District and other SURR schools and the citywide average, 1998–99. In the 1998–99 baseline year, Chancellor’s District elementary schools and all other SURR elementary schools differed considerably from the average New York City elementary school across a variety of student and school characteristics. As Table 4 shows, both the Chancellor’s District schools and other SURR schools were somewhat smaller, much less white, considerably poorer, and had more special education students, but fewer immigrant students, than the average New York City elementary school. The Chancellor’s District schools and other SURR schools were quite similar to each other, with the important exceptions of the percent of students in full time special education and the percent of students who remained in their school for the entire year. The other SURR schools had proportionally more students in full-time special education than the Chancellor’s District schools, and more than twice as many as the average New York City elementary school (12.1% vs. 5.8%). In addition, other SURR schools had a significantly lower proportion of students who remained in the school for the entire year than the Chancellor’s District schools; the latter’s percentage was much closer to the citywide average. Thus, Chancellor’s District schools had significantly fewer students who moved in or out of the school during the school year than the other SURR schools. In 1998–99, Chancellor’s District Schools and other SURR schools also differed considerably from the average New York City elementary school in teacher resources and school expenditures. (See Table 5.) The Chancellor’s District schools had the lowest level of teacher resources in the city—lower percentages of fully licensed and experienced teachers—and the least stable teaching force. Furthermore, Chancellor’s District schools spent less than other SURR schools, in terms of both teacher expenditures and total per student expenditures. As Table 5 indicates, the student performance outcomes of all Chancellor’s District schools and other SURR schools were considerably below the citywide average in the 1998–99 baseline year. Table 5 Mean teacher characteristics and school expenditures, 1998–99 Characteristics CD (N=25) Other SURR (N=25) All schools (N=666) % Licensed teachers 67.1* 72.6 81.5 % Taught 2+ yrs in this school 42.6 48.6 59.8 % Taught 5 or more years 49.1* 54.1 59.3 % Teachers with masters degrees 69.0 71.1 77.1 Teachers per 100 students 6.7 7.2 6.4 Per student expenditures $7,792.80** $8,537.20 $7,554.00 Per student spending on teachers $3,357.20** $3,777.30 $3,509.20 Non-teacher spending per student $4,435.70 $4,759.90 $4,044.70 * p < .10; ** p < .05; *** p < .01. Expenditures are per student, for general education and part time special education students. Table 6 indicates that student performance in the Chancellor’s District schools did not differ much from performance in the other SURR schools. Chancellor’s District schools had a slightly lower percentage of students meeting the standard on the fourth grade reading test than the other SURR schools, and a slightly higher average scale score on the fourth grade math tests. In both cases, the differences between the average scores were marginally significant. However, both reading A Forced March for Failing Schools 13 and math performance in the Chancellor’s District and other SURR schools were considerably below the average performance of all the city’s elementary schools. Table 6 Mean fourth grade test scores, NYC, 1998–99 Characteristic CD schools (N=25) Other SURR (N = 25) All NYC (N=666) Mean reading scale score 606.5 607.5 628.0 % meeting reading std. 12.2* 15.2 33.3 % far below reading std. 38.6 38.5 20.8 Mean math scale score 614.2* 609.0 636.2 % meeting math std. 27.6 23.6 50.7 % far below math std. 34.0 37.7 18.5 * p < .10; ** p < .05; *** p < .01. Thus, in the 1998–99 school year, Chancellor’s District and other SURR schools had much higher levels of student need, lower levels of teacher resources, and poorer student performance than the average elementary school in the New York City system. This pattern of high student need, poor teacher resources and poor student performance is what the Chancellor targeted for improvement through the takeover of failing schools and the imposition of the Model of Excellence. Changes in Chancellor’s District and other SURR schools, 1998–99 to 2001–02 Student demographics in Chancellor’s District schools and other SURR schools remained fairly constant from the 1998–99 baseline year through the 2001–02 school years, with several important exceptions. (See Table 7.) During this period, the overall student population declined in both groups of schools, by 10% in Chancellor’s District schools and 7% in other SURR schools. The proportion of special education students declined as well. In 1998–99, the average percentage of students in full time special education in Chancellor’s District schools (8.1%) was higher than the citywide average (5.8%). By 2001–02, this percentage had decreased to 4.8%, very similar to the average New York City school (4.6%). Other SURR schools experienced an even greater decline in the percentage of their students in full time special education—from 12.4% to 7.8%. However, even with that decline, the percentage of students in full time special education in the other SURR schools in 2002 was still much higher than the citywide average and the Chancellor’s District average. The difference between the percentage of full time special education students in Chancellor’s District and other SURR schools was highly significant in both 1998–99 and 2001–02. The percentage of students referred for special education evaluation in Chancellor’s District schools also declined by 1.3 percentage points, from 4.3% to 3.0%, between 1998–99 and 2001–02. By comparison, the referral rate in other SURR schools increased by 1.8 percentage points, from 4.0% to 5.8%. The difference between the changes in the two groups was highly significant.47 The citywide referral rate also increased, from 3.6% in 1998–99 to 4.1% in 2001–02. The proportion of students who were English language learners declined in both Chancellor’s District (4.9 percentage points) and other SURR schools (4.2 percentage points), compared to a citywide decline of almost three percentage points. Education Policy Analysis Archives Vol. 13 No. 40 14 Table 7 Change in mean student and school characteristics, 1998–99 to 2001–02 Pre- intervention Intervention Difference 1998–99 1999–00 2000–01 2001–02 1999–2002 Characteristic CD SURR CD SURR CD SURR CD SURR CD SURR % White 0.8 0.9 0.8 1.0 0.8 1.2 0.8* 1.3 0.0*** 0.4 % Black 55.1 55.9 54.5 56.3 54.6 56.0 54.3 55.6 -0.8 -0.3 % Hispanic 42.2 41.6 42.5 40.9 42.4 41.2 42.7 41.3 0.5 -0.3 % Asian/other 1.9 1.6 2.3 1.8 2.2 1.7 2.3 1.8 0.4 0.3 % Limited English 16.5 15.1 14.9 13.3 13.0 11.9 11.6 10.9 -4.9 -4.2 % Recent immigrant 4.4 3.8 4.1 3.3 3.7 3.3 4.0 3.6 -0.4 -0.1 % Free lunch eligible 91.6 92.8 89.3 91.8 87.2 90.0 87.2 90.0 -4.4 -2.8 % Full time Special Ed 8.1** 12.4 7.9* 11.5 6.7 9.4 4.8** 7.8 -3.3 -4.6 % In school for year 90.5** 87.3 91.7** 89.4 91.3*** 88.3 90.6*** 87.1 0.1 -0.2 % Attendance 87.9 88.3 88.5** 89.4 89.3* 90.0 90.2 90.4 2.3 2.1 % Special Ed. referrals 4.3 4.0 6.2 5.9 4.8 5.2 3.0*** 5.8 -1.3*** 1.8 % Part time special Ed 6.1 6.2 6.3 6.1 5.8 5.7 5.0 5.6 -1.1 -0.7 Mean enrollment 700.5 750.7 667.8 722.0 660.6 713.0 631.9 696.6 -68.7 -54.1 Differences between CD and other SURR schools: * p < .10; ** p < .05; *** p < .01. The most dramatic changes in the Chancellor’s District and other SURR schools occurred in resource provision. Table 8 shows a considerable improvement in the teacher resources of the other SURR schools, and an even more remarkable increase in the resources in Chancellor’s District schools. The formerly under-resourced Chancellor’s District schools were the beneficiaries of large increases in the number, quality, and stability of their teaching staffs. The Chancellor’s District schools also benefited from major increases in funding; their per student spending increased by $5,713 from 1998–99 to 2001–02, compared to an increase of $2,667 per student in other SURR schools during the same period. By contrast, the average New York City school saw a smaller $2,234 increase in per student expenditures. The additional costs associated with the Chancellor’s District’s elementary schools reflect the implementation cost of the Model of Excellence in the Chancellor’s District schools. Most of this increased expenditure was for teachers. The implementation of the Model of Excellence in Chancellor’s District elementary schools not only reduced class size, but also provided at least four on-site staff developers in each school. Moreover, Chancellor’s District school expenditures also involved the cost of absorbing the salaries of ineffective teachers, as well as the 15% salary differential for the additional extended time hours that teachers worked. These efforts brought Chancellor’s District schools, which had been lowest in expenditures on teachers, well above all other schools. In the 1998–99 school year, there were 6.7 teachers for every 100 students in Chancellor’s District schools. This ratio increased by 1.9 teachers, to 8.6 teachers per 100 students in the 2001–02 school year. By contrast, there were 7.1 teachers for every 100 students in other SURR schools in 1998–99, but that ratio increased by only 0.6—to 7.7 teachers—in 2001–02. The increase in the number of teachers per 100 students in Chancellor’s District schools, probably a reflection of reduced class size and the increase in staff developers, was highly significant, compared to the increase in the number of teachers per 100 students in other SURR schools, as well as to the much smaller increase in the citywide average. A Forced March for Failing Schools 15 Table 8 Change in mean teacher characteristics and school expenditures, 1998–99 to 2001–02 Pre-intervention Intervention Difference 1998–99 1999–00 2000–01 2001–02 1999–2002 Characteristic CD SURR CD SURR CD SURR CD SURR CD SURR % Licensed teachers 67.2* 73.0 71.5 69.2 91.1 86.8 93.4* 89.7 26.2** 16.7 % Taught 2+ years in this school 42.2* 50.9 43.0* 50.5 45.6** 53.0 54.8** 62.6 12.6 11.7 % Taught 5 or more years 49.0 54 49.0 47.5 45.2 44.1 42.8 44 -6.3 -10.0 % Teachers with masters degrees 68.6 70.9 70.6 67.7 69.2 68.0 70.7 70.6 2.0 -0.3 Teachers per 100 students 6.7 7.1 7.9 8.1 9.2 8.8 8.6** 7.7 1.9** 0.6 Per pupil spending on teachers ($) 3,346*** 3,751 4,713*** 4,165 5,995*** 4,962 6,431*** 4,970 3,085*** 1,219 Per pupil spending ($) 7,808* 8,495 9,792 9,689 12,344*** 11,033 13,520*** 11,162 5,713*** 2,667 Differences between CD and other SURR schools: * p < .10; ** p < .05; *** p < .01. Expenditures are per student, for general education and PT special education students. Differences are calculated only for schools that existed in both 1998–99 and 2001–2002. In 1998–99, the percentage of licensed teachers in Chancellor’s District schools (67.2%) was significantly lower than in other SURR schools (73.0%). By 2001–02, the two groups’ relative positions reversed, and the percentage of licensed teachers in Chancellor’s District schools (93.4%) was significantly higher than in other SURR schools (89.7%). Chancellor’s District schools increased their licensed teachers by 26.2 percentage points in this three-year period, while other SURR schools increased their licensed teachers by 16.7 percentage points. The increase in the percentage of licensed teachers in Chancellor’s District schools was highly significant, compared to the increase in the percentage of licensed teachers in other SURR schools, as well to the citywide average. A third area of improvement was in the stability of the teaching staff. In 1998–99, only 42.2% of teachers in Chancellor’s District schools, compared to 50.9% in other SURR schools, had been in their school for two or more years. While this statistic rose by 12.6 percentage points from 1998–99 to 2001–02 in Chancellor’s District schools, it rose by a similar amount (11.7 percentage points) in other SURR schools. Although both Chancellor’s District and other SURR elementary schools experienced improvements in their overall funding and expenditure on teacher resources throughout the period, improvements in the Chancellor’s District schools were greater than in the other SURR schools. By 2001–02, Chancellor’s District schools’ total spending and their spending on teachers were much greater than other SURR schools, and Chancellor’s District schools had a higher number of teachers per student and a higher percentage of fully licensed teachers. This situation contrasted sharply with what had prevailed four years earlier. Education Policy Analysis Archives Vol. 13 No. 40 16 There were also considerable changes in academic performance in the Chancellor’s District and other SURR schools from 1998–99 to 2001–02. As Table 9 shows, within those four school years, most of New York City’s SURR schools improved sufficiently to be removed from the state’s SURR list, a considerable achievement. Fifty-six percent of Chancellor’s District schools and 60% of other SURR schools were removed from the SURR list, a similar pace of improvement for both groups of schools. Table 9 Change in SURR status in New York City’s SURR schools, 1998–99 to 2001–02 CD schools Other SURR schools Number Percent Number Percent Closed 1 4 2 8 Removed from the SURR list 14 56 15 60 Still on the SURR list 10 40 8 32 Total 25 100 25 100 Source: New York State Education Department Table 10 indicates that the percentage of fourth grade students in Chancellor’s District schools meeting the state’s reading standard increased significantly more than the percentage of fourth grade students in other SURR schools. In 1998–99, a lower percentage of students met the reading standard in Chancellor’s District schools (12.3%) than in other SURR schools (15.3%). But by 2001–02, the two groups’ relative positions reversed; more students met the reading standard in Chancellor’s District schools (30%) than in other SURR schools (27.2%). The 18 percentage point improvement in the scores of Chancellor’s District schools is particularly strong. The citywide average for elementary schools across those years increased 14 percentage points—from 33.4% of fourth grade students meeting the state’s reading standard in 1998–99 to 47.8% in 2001–02. (See Figure 5.) Table 10 Change in mean fourth grade reading and math results, 1998–99 to 2001–02 Pre- intervention Intervention Difference 1998–99 1999–00 2000–01 2001–02 1999–2002 Characteristic CD SURR CD SURR CD SURR CD SURR CD SURR Mean reading scale score 606.8 607.4 613.2 614.5 619.6 615.5 627.8 626.7 21.0 19.3 % meeting reading std. 12.3 15.3 21.5 22.1 26.9** 22.6 30.0 27.2 17.7** 11.9 % far below reading std. 38.3 39.0 32.6 34.3 26.3** 32.1 21.7 21.8 -16.6 -17.2 Mean math scale score 614.3* 609.4 610.7 612.1 618.7 617.8 626.0 623.5 11.8 14.1 % meeting math std. 27.8 23.6 22.0 23.9 30.6 29.8 38.0 34.1 10.2 10.5 % far below math std. 33.8 37.3 35.0 32.9 28.3 30 16.8 19.7 -17.0 -17.7 Differences between CD and other SURR schools: * p < .10; ** p < .05; *** p < .01. Between the 1998–99 and 2001–02 school years, there were no significant differences in the change in math scores in Chancellor’s District schools compared to other SURR schools. However, the pattern of change in math performance is more complex. While both groups improved over these years, Chancellor’s District performance, which was significantly higher than the performance of SURR schools in 1998–99, declined in 1999–00 and recovered over the next two years. Math A Forced March for Failing Schools 17 performance in other SURR schools, by contrast, had a continuously positive upward trajectory. The end result is that the math difference between the two groups remained essentially the same. The findings of our univariate analyses suggest that the Chancellor’s District schools improved their students’ reading skills more than the other SURR schools. They also show that, although both groups of schools experienced important improvements in the number and quality of their teaching staff and their expenditure levels, these changes were much more pronounced in Chancellor’s District schools than in other SURR schools. However, the univariate analysis does not provide insight into how much of this improvement can be attributed to the Chancellor’s District intervention as opposed to the possible effects of other factors, such as changes in the composition of the schools’ student populations. It is particularly important to control for student characteristics such as special education or limited English proficiency, because the proportions of students in these categories were sharply reduced across the years of the analysis. Similarly, given the dramatic improvement in funding, teacher to student ratios and teacher quality, it is important to determine how much of the change in student performance can be attributed to improved funding and teacher resources. To examine these issues while disentangling the effects of the different factors involved, the next section presents the regression analyses carried out to determine if the patterns in school-level performance remain after controlling for differences in student, school and teacher characteristics, as well as school expenditures. Regression Analysis Table 11 displays the estimated differences in academic performance, controlling for student, school, and teacher characteristics, as well as per-student expenditures, between Chancellor’s District schools and other SURR schools. A positive coefficient in the regressions estimating effects on the percent of students scoring at or above the state reading and math standards (Levels 3 and 4) indicates a positive association with student performance. Conversely, a positive coefficient in the regressions predicting the percent far below the state reading and math standards indicates a negative association with student performance. The coefficients suggest that, when we control for student and school characteristics, student performance on the fourth grade state reading test is significantly better in Chancellor’s District schools than in other SURR schools. This is reflected in the higher average percent of students scoring at or above the standard, and in the lower percent of students scoring far below the standard. The influence of the other variables varies across models, depending on whether the model controls for resources or not. Moreover, the coefficient on the year dummies is in many cases significant, and becomes larger in the more recent years, reflecting an overall pattern of increasing achievement in student performance across the entire sample of schools. The Chancellor’s District’s effect on student performance on the fourth grade math test is not as encouraging. As the regression table indicates, Chancellor’s District schools do not differ significantly from other SURR schools on the percent of students scoring at Levels 3 and 4, or at Level 1. The failure to significantly improve math scores in the Chancellor’s District may be a direct result of the much more intensive curricular and scheduling focus on improving reading skills. Or the reading skills, and scores, of the Chancellor’s District students may have improved at the expense of their math scores. Education Policy Analysis Archives Vol. 13 No. 40 18 Table 11 Effects on fourth grade academic performance, without (Model 1) and with (Model 2) resources % meeting reading standard % reading far below standard % meeting math standard % far below math standard Characteristic Model 1 Model 2 Model 1 Model 2 Model 1 Model 2 Model 1 Model 2 Chancellor's District 5.706*** (2.043) 5.898** (2.140) -4.299* (2.308) -3.944* (2.334) -1.842 (3.066) -2.314 (3.159) 1.615 (2.736) 2.273 (2.948) Student & school % Black -1.826 (1.517) -2.850* (1.607) -0.338 (1.483) 0.128 (1.566) 0.080 (1.814) -1.133 (1.718) -0.791 (2.064) -0.024 (2.224) % Hispanic -2.396 (1.529) -3.325** (1.608) 0.028 (1.516) 0.512 (1.555) -0.820 (1.885) -1.888 (1.798) -0.456 (2.089) 0.148 2.220) % Asian or other -1.150 (1.711) -2.550 (1.783) 0.608 (1.826) 1.129 (1.933) -0.004 (2.260) -1.803 (2.179) 0.350 (2.231) 1.198 (2.458) % Limited English 0.214 (0.260) 0.307 (0.285) -0.008 (0.277) -0.135 (0.307) -0.083 (0.326) 0.073 (0.353) 0.222 (0.290) 0.133 (0.312) % Recent immigrant -1.228* (0.636) -1.259** (0.629) 0.704 (0.610) 0.821 (0.625) 0.408 (0.782) 0.374 (0.776) -0.091 (0.799) 0.040 (0.772) % Free lunch eligible -0.043 (0.095) -0.077 (0.095) -0.132 (0.100) -0.136 (0.107) -0.302** (0.141) -0.288* (0.151) 0.114 (0.137) 0.087 (0.140) % Full-time special ed. -0.092 (0.210) -0.061 (0.213) 0.706*** (0.223) 0.639*** (0.240) -0.453* (0.262) -0.428 (0.282) 0.539** (0.226) 0.520** (0.234) % In school entire year -0.083 (0.218) -0.073 (0.228) 0.039 (0.227) 0.075 (0.238) 0.007 (0.306) 0.055 (0.306) 0.2404 (0.272) 0.301 (0.270) % Attendance -0.043 (0.737) -0.232 (0.715) 0.166 (0.771) 0.511 (0.795) 0.116 (1.001) -0.290 (0.941) -0.083 (0.967) 0.364 (0.907) % Special ed. referrals -0.146 (0.265) -0.050 (0.279) -0.624* (0.344) -0.708** (0.349) 0.190 (0.356) 0.273 (0.357) -0.235 (0.314) -0.384 (0.316) % Part-time special ed. -0.175 (0.521) -0.435 (0.553) -0.111 (0.561) -0.124 (0.588) 0.044 (0.612) -0.074 (0.621) 0.844 (0.653) 1.266* (0.652) Students -3.390 (5.469) 6.984 (7.729) -4.818 (6.797) -15.263 (11.053) -12.721* (7.093) 0.697 (10.701) 11.197* (5.867) -9.215 (8.894) Year 4.454** 4.200** -1.184 -1.067 -4.116* -4.927** -0.812 1.051 2000 (-1.75) (-1.906) (-1.991) (-2.061) (-2.191) (-2.44) (-2.213) (-2.386) 7.392*** 4.312 -6.203*** -2.488 2.114 -4.203 -4.281 2.847 2001 (-2.056) (-2.932) (-2.207) (-2.899) (-2.721) (-3.382) (-2.598) (-3.029) 11.248** 7.523** -13.03*** -8.825*** 6.973** 0.289 13.286*** -6.068 2002 (-2.633) (-3.414) (-2.503) (-3.221) (-3.266) (-3.986) (-3.236) (-3.75) A Forced March for Failing Schools 19 % meeting reading standard % reading far below standard % meeting math standard % far below math standard Characteristic Model 1 Model 2 Model 1 Model 2 Model 1 Model 2 Model 1 Model 2 Resources 0.052 -0.108 0.155 -0.102 % Licensed teachers (-0.101) (-0.111) (-0.125) (-0.12) 0.106** -0.028 0.155** -0.055 % Taught 2+ yrs in school (-0.05) (-0.068) (-0.067) (-0.071) 0.083 -0.176 0.207 % Taught 5+ years 0.008 (-0.115) (-0.122) (-0.144) (-0.148) -0.199* 0.111 -0.119 0.176 % with masters degrees (-0.109) (-0.136) (-0.147) (-0.136) 0.043 0.181 -1.05 -0.784 Teachers per 100 students (-0.617) (-0.701) (-0.721) (-0.753) 1.226 -1.597* 0.721 -2.178***Non-teacher expenditures (-0.77) (-0.85) (-0.953) (-0.792) 239.5 347.6** 46.1 -22.5 85.6 207.4 52.1 -42.9 Constant (-159.7) (-165.1) (-154.0) (-162.3) (-186. 8) (-176.5) (-213. 6) (-221.4) N 197 195 197 195 197 195 197 195 R-squared 0.70 0.72 0.69 0.71 0.64 0.68 0.67 0.70 F-stat (Resources) 1.69 (-0.129) 1.02 (-0.42) 2.31** (-0.04) 3.14** (-0.01) F-statistic for school fixed effects 2.044** (-0.001) 2.251*** (0.000) 1.793** (-0.005) 1.818** (-0.004) 2.043** (-0.001) 2.297*** (0.000) 1.434* (-0.056) 1.706** (-0.010) * p < .10; ** p < .05; *** p < .01. Equally important, Model 2 assesses the effect of resources on reading and math performance. Teacher stability has a positive effect on the percent of students meeting the standard, suggesting the importance of a well-supported teaching staff. Non-teacher expenditures have a negative effect on the percent far below the standard, suggesting the positive effect of general resources. These results are intuitive. However, the fact that the effects of the other resource variables are not consistent across models, and the resource variables are jointly significant only in the two math models presented (F=2.31 and 3.14, probably for different reasons) reflect a generally weak, counterintuitive relationship between resources and performance. This finding is consistent with the reverse causality problem in education production, in which resources tend to be negatively correlated with performance, due to the high correlation between categorical expenditures, such as title 1 funds, and student need—which in turn are negatively correlated with performance. In comparing Model 1—a basic model in which teacher characteristics and expenditures are excluded—with a more complete model controlling for teacher characteristics and expenditures, we Education Policy Analysis Archives Vol. 13 No. 40 20 find that the results are essentially the same. In theory, in Model 1 the Chancellor’s District coefficient could be inflated by the district’s huge resource advantage versus other SURR schools. If that were the case, we would expect the coefficients for Chancellor’s District dummy—and other variables—to be radically different (i.e., smaller) in Model 2, when we control for resources and teacher characteristics. That Model 2 shows a substantively unchanged Chancellor’s District coefficient suggests that the Chancellor’s District effect was not simply a matter of increased funding and teacher characteristics. The positive effect of the Chancellor’s District may be tied to enhanced administrative support, more efficient use of instructional resources, and other factors. However, without detailed implementation data, it is impossible to disentangle which parts of the intervention were more causal. Overall, these regression results reiterate the univariate findings in school-level performance—on average, the Chancellor’s District schools performed significantly higher in reading performance during the years these schools were under the centralized improvement regimen described above, but did not show much progress in math. The positive regression coefficients in reading suggest a significant improvement for the Chancellor’s District schools, relative to where these schools would have been and relative to comparable schools. These results are consistent when resources are added to the models. Revisiting the twin goals of our analysis, we assess whether the Chancellor’s District intervention increased schools’ instructional capacity and academic outcomes. Across the 1998–99 through 2001–02 school years, the Chancellor’s District schools sustained higher student stability rates, increased teacher resources, and substantially increased per student expenditures, compared to both other SURR schools and the citywide average. Moreover, holding student characteristics, teacher characteristics and expenditures constant, the Chancellor’s District schools increased their fourth grade reading performance by considerably more than the other SURR schools. This finding suggests that, at the elementary school level, the Chancellor’s District as an intervention succeeded in improving the reading outcomes, though not the math outcomes, of its schools and students. Conclusion The Chancellor’s District, as a unique initiative in centrally-driven school improvement, represents a signal intervention into New York City school governance and administration. When the Chancellor, in 1996, invoked a previously unexercised power to take failing elementary and middle schools from their sub-district jurisdictions, he did what no other New York City schools chief had ever attempted. He proceeded to create a special, non-geographic district that eventually encompassed 58 failing schools, and developed a series of organizational, curricular, instructional, and personnel interventions, mandated for all the district’s schools, to jump-start their improvement. Thus the Chancellor’s District effort represents an historic departure from three decades of central school system tolerance of local sub-district instructional failure. The Chancellor’s District initiative challenges several traditions of policy analysis about the relationship between district administration and school change. Its theory of change counters reigning theories about the stultifying weight of urban education bureaucracies, the inability of loosely coupled systems to sustain centrally-driven change, and the dichotomy between what bureaucratic systems impose and the autonomy successful schools require. Thus the Chancellor’s District effort may represent a return to more traditional notions of centralized management, or a harbinger of the newly emerging emphasis on the district as the necessary locus of school change. But where it departs from this debate is as a demonstration that a top down approach, if attached to A Forced March for Failing Schools 21 resources and a set of appropriate reforms rather than the mere fact of greater centralization, could yield positive results. How the Chancellor’s District initiative is ultimately assessed in the history of urban education reform depends primarily on the outcomes of the effort. Our findings suggest two categories of results. First, our univariate analysis demonstrates that the Chancellor’s District intervention significantly increased teacher resources and per-student expenditure across the district’s schools, and significantly increased the percentage of students meeting the standard on the fourth grade state reading tests, compared to the outcomes of other SURR schools. Second, our regression analyses demonstrate that when the Chancellor’s District schools are compared to the other SURR schools (the schools most similar to those in the Chancellor’s District) and when the analyses control for student and school characteristics, teacher resources and per- student expenditures, the Chancellor’s District schools do significantly better than other SURR schools in reading, but not in math. But given that the major curricular, instructional and organizational interventions of the Chancellor’s District focused intensively on improving student literacy, these outcomes suggest that the Chancellor’s District had begun to achieve one of its primary student achievement goals. The eventual impact of these gains in math performance is yet to be determined. It is important to note that the district’s upward curve in reading outcomes still left the Chancellor’s District schools quite far below the citywide average, though the initiative was clearly narrowing the gap. It is also important that the Chancellor’s District initiative we evaluated represents only those three academic years of effort, from 1999–00 to 2001–02, in which the components of the Model of Excellence were implemented in the district’s schools. Had the initiative not been terminated in 2003, would the upward curve of reading achievement have continued to rise? Would the math achievement that began to accelerate in 1999–00 have continued upward? Our data do not allow us to speculate. Both the Chancellor’s District and other SURR schools seem to have benefited from increases in teacher resources as well as overall expenditures. The Chancellor’s District schools received significantly more resources than the other SURR schools, which in turn received significantly more than the city schools as a whole. But when we control for the effects of teacher resources and per student expenditures, the Chancellor’s District’s elementary schools still perform significantly better than the other SURR schools in reading. Thus, something was working to improve outcomes in the Chancellor’s District schools that is not explained by increases in teacher resources or school-level expenditures. We cannot define what that something is, other than to point to the set of components that comprised the Chancellor’s District intervention. Because our evaluation was retrospective, we cannot specify what components of the intervention helped to produce the reading gains our findings demonstrate. Future research, perhaps benefiting from more intensive implementation data, can make the connection between gains for particularly successful or unsuccessful schools and the specific levels of interventions imposed. But it is important to reiterate that the Chancellor’s District took over some of the city’s least well-resourced schools serving the city’s poorest and lowest performing students. By developing, mandating and implementing a comprehensive set of organizational, curricular, instructional and personnel changes, the Chancellor’s District significantly improved the reading outcomes of the students in those schools, in three years of focused effort. This is not a small accomplishment. Whether the additional resources expended, in both teacher resources and per student expenditures, were ultimately worth the extent of improved achievement the Chancellor’s District initiative generated, is a complex but essential question that our subsequent research will attempt to answer. Education Policy Analysis Archives Vol. 13 No. 40 22 References Ascher, C., Fruchter, N., & Ikeda, K. (1999). Schools in context: Final report to the New York State Education Department 1997–1998, An analysis of SURR schools and their districts. New York: Institute for Education and Social Policy. Bodilly, S. J. (2001). New American Schools’ concept of break the mold designs: How designs evolved and why. Santa Monica, CA: RAND, MR–1288–NAS. Chubb, J. E. & Moe, T. M. (1990). Politics, markets and America’s schools. Washington, D.C.: The Brookings Institution. Domanico, R. (1994). Undoing the failure of large school systems: Policy options for school autonomy. Journal of Negro Education, 63, 19–27. Hallett, A. (Ed.). (1995). Reinventing central office: A primer for successful schools. Chicago: Cross City Campaign for Urban School Reform. Iatarola, P. (2001). IESP policy brief: Distributing teacher quality equitably: The case of New York City. New York: Institute for Education and Social Policy. Iatarola, P., & Fruchter, N. (2004). District effectiveness: A study of investment strategies in New York City public schools and districts. Educational Policy, 18, 491–512. Mintrop, H. (2003). The limits of sanctions in low-performing schools: A study of Maryland and Kentucky schools on probation. Education Policy Analysis Archives, 11(3). Retrieved September 27, 2005, from http://epaa.asu.edu/epaa/v11n3/. New York City Board of Education. (1998–2001a). Annual school reports 1998–2001. New York: New York City Board of Education. New York City Board of Education. (1998–2001b). School-based expenditure reports, 1998– 2001. New York: New York City Board of Education. New York City Board of Education. (1999a). Budget and operations review memorandum #1 FY00. New York: New York City Board of Education. New York City Board of Education. (1999b). Chancellor’s District: A model of excellence for extended time schools, 1999–2000. New York: New York City Board of Education. New York City Board of Education. (2000). Flash report #1. Analyses of performance of extended-time and non-extended time SURR schools. New York: New York City Board of Education. New York City Board of Education. (2001a). Chancellor’s District: A model of excellence: 2001– 2002. New York: New York City Board of Education. New York City Board of Education. (2001b). Fact sheet I, Teachers of tomorrow program, 2000– 2001. New York: New York City Board of Education. A Forced March for Failing Schools 23 New York City Board of Education. (2002). Flash report #7. Year two analyses of performance of extended-time and non-extended time SURR schools. New York: New York City Board of Education. New York City Board of Education. (n.d.). Corrective action plan: A citywide implementation framework for redesign schools. New York: New York City Board of Education. Resnick, L.B. & Glennan, T. K. (2002). Leadership for learning: A theory of action for urban school districts. In A.M. Hightower, M.S. Knapp, J.A. Marsh & M.W. McLaughlin (Eds.). School districts and instructional renewal (pp.160–172). New York: Teachers College Press. Sarason, S. (1996). Revisiting the culture of the school and the problem of change. New York: Teachers College Press. Snipes, J., Doolittle, F., & Herlihy, C. (2002). Foundations for success: Case studies of how urban school systems improve student achievement. New York: MDRC for the Council of the Great City Schools. Weick, K. E. (1976). Educational organizations as loosely coupled systems. Administrative Science Quarterly, 21, 1–19. Education Policy Analysis Archives Vol. 13 No. 40 24 About the Authors Deinya Phenix Dorothy Siegel Ariel Zaltsman Norm Fruchter New York University, Steinhardt School of Education Email: deinya.phenix@nyu.edu Deinya Phenix is a Research Scientist at the New York University Institute for Education and Social Policy. Her research interests include urban education policy and social geography. Dorothy Siegel is a Senior Project Director at the New York University Institute for Education and Social Policy. She is currently facilitating the development of a citywide inclusion program for children with Autism Spectrum Disorders in New York City. Ariel Zaltsman is a Doctoral Candidate in Public Administration at the R.F. Wagner School of Public Service, NYU. His research interests program evaluation, performance-based budgeting, and social policy. Norm Fruchter is Clinical Professor of Education Policy at New York University’s Steinhardt School of Education and Director of the Institute for Education and Social Policy. He is co-author of Choosing Equality: The Case for Democratic Schooling and Hard Lessons: Public Schools and Privatization. A Forced March for Failing Schools 25 EDUCATION POLICY ANALYSIS ARCHIVES http://epaa.asu.edu Editor: Sherman Dorn, University of South Florida Production Assistant: Chris Murrell, Arizona State University General questions about appropriateness of topics or particular articles may be addressed to the Editor, Sherman Dorn, epaa-editor@shermandorn.com. Editorial Board Michael W. Apple University of Wisconsin David C. Berliner Arizona State University Greg Camilli Rutgers University Casey Cobb University of Connecticut Linda Darling-Hammond Stanford University Mark E. Fetler California Commission on Teacher Credentialing Gustavo E. Fischman Arizona State Univeristy Richard Garlikov Birmingham, Alabama Gene V Glass Arizona State University Thomas F. Green Syracuse University Aimee Howley Ohio University Craig B. Howley Appalachia Educational Laboratory William Hunter University of Ontario Institute of Technology Patricia Fey Jarvis Seattle, Washington Daniel Kallós Umeå University Benjamin Levin University of Manitoba Thomas Mauhs-Pugh Green Mountain College Les McLean University of Toronto Heinrich Mintrop University of California, Berkeley Michele Moses Arizona State University Anthony G. Rud Jr. Purdue University Michael Scriven Western Michigan University Terrence G. Wiley Arizona State University John Willinsky University of British Columbia Education Policy Analysis Archives Vol. 13 No. 40 26 EDUCATION POLICY ANALYSIS ARCHIVES English-language Graduate-Student Editorial Board Noga Admon New York University Jessica Allen University of Colorado Cheryl Aman University of British Columbia Anne Black University of Connecticut Marisa Burian-Fitzgerald Michigan State University Chad d'Entremont Teachers College Columbia University Carol Da Silva Harvard University Tara Donahue Michigan State University Camille Farrington University of Illinois Chicago Chris Frey Indiana University Amy Garrett Dikkers University of Minnesota Misty Ginicola Yale University Jake Gross Indiana University Hee Kyung Hong Loyola University Chicago Jennifer Lloyd University of British Columbia Heather Lord Yale University Shereeza Mohammed Florida Atlantic University Ben Superfine University of Michigan John Weathers University of Pennsylvania Kyo Yamashiro University of California Los Angeles A Forced March for Failing Schools 27 Archivos Analíticos de Políticas Educativas Associate Editors Gustavo E. Fischman & Pablo Gentili Arizona State University & Universidade do Estado do Rio de Janeiro Founding Associate Editor for Spanish Language (1998—2003) Roberto Rodríguez Gómez Editorial Board Hugo Aboites Universidad Autónoma Metropolitana-Xochimilco Adrián Acosta Universidad de Guadalajara México Claudio Almonacid Avila Universidad Metropolitana de Ciencias de la Educación, Chile Dalila Andrade de Oliveira Universidade Federal de Minas Gerais, Belo Horizonte, Brasil Alejandra Birgin Ministerio de Educación, Argentina Teresa Bracho Centro de Investigación y Docencia Económica-CIDE Alejandro Canales Universidad Nacional Autónoma de México Ursula Casanova Arizona State University, Tempe, Arizona Sigfredo Chiroque Instituto de Pedagogía Popular, Perú Erwin Epstein Loyola University, Chicago, Illinois Mariano Fernández Enguita Universidad de Salamanca. España Gaudêncio Frigotto Universidade Estadual do Rio de Janeiro, Brasil Rollin Kent Universidad Autónoma de Puebla. Puebla, México Walter Kohan Universidade Estadual do Rio de Janeiro, Brasil Roberto Leher Universidade Estadual do Rio de Janeiro, Brasil Daniel C. Levy University at Albany, SUNY, Albany, New York Nilma Limo Gomes Universidade Federal de Minas Gerais, Belo Horizonte Pia Lindquist Wong California State University, Sacramento, California María Loreto Egaña Programa Interdisciplinario de Investigación en Educación Mariano Narodowski Universidad Torcuato Di Tella, Argentina Iolanda de Oliveira Universidade Federal Fluminense, Brasil Grover Pango Foro Latinoamericano de Políticas Educativas, Perú Vanilda Paiva Universidade Estadual Do Rio De Janeiro, Brasil Miguel Pereira Catedratico Universidad de Granada, España Angel Ignacio Pérez Gómez Universidad de Málaga Mónica Pini Universidad Nacional de San Martin, Argentina Romualdo Portella do Oliveira Universidade de São Paulo Diana Rhoten Social Science Research Council, New York, New York José Gimeno Sacristán Universidad de Valencia, España Daniel Schugurensky Ontario Institute for Studies in Education, Canada Susan Street Centro de Investigaciones y Estudios Superiores en Antropologia Social Occidente, Guadalajara, México Nelly P. Stromquist University of Southern California, Los Angeles, California Daniel Suarez Laboratorio de Politicas Publicas-Universidad de Buenos Aires, Argentina Antonio Teodoro Universidade Lusófona Lisboa, Carlos A. Torres UCLA Jurjo Torres Santomé Universidad de la Coruña, España