Microsoft Word - EPAA_GraueDelaneyKarch_Jan24_fnl.docx SPECIAL ISSUE Value-Added: What America’s Policymakers Need to Know and Understand education policy analysis archives A peer-reviewed, independent, open access, multilingual journal Arizona State University Volume 21 Number 8 January 30th, 2013 ISSN 1068-2341 Ecologies of Education Quality M. Elizabeth Graue, Katherine K. Delaney, Anne S. Karch University of Wisconsin – Madison United States of America Citation: Graue, M.E., Delaney, K.K., & Karch, A.S. (2012). Ecologies of education quality. Education Policy Analysis Archives, 21(8). This article is part of EPAA/AAPE’s Special Issue on Value- Added: What America’s Policymakers Need to Know and Understand, Guest Edited by Dr. Audrey Amrein- Beardsley and Assistant Guest Editors Dr. Clarin Collins, Dr. Sarah Polasky, and Ed Sloat. Retrieved [date], from http://epaa.asu.edu/ojs/article/view/1163. Abstract: Accountability in education has prompted policy makers and practitioners to focus on data use for instructional and organizational decision-making. The popular media have seized on Value Added (VA) measures as a key type of data use for reforming U.S. schools. Among education researchers, however, there are both critics and proponents. We examined data use by the district leaders and staff members of 12 schools in a large urban district, with attention to the role VA metrics play in their decisions and their conceptions of themselves. VA is only one of many types of data that can be used to portray quality. While there was a soft relationship between VA and classroom quality measured by CLASS, we found that understanding the schools’ contexts, particularly the use of resources and the coherence of actions to improve student achievement, greatly enhanced the power of our descriptions. As a result, we suggest that policies promoting multidimensional approaches to quality will better capture the complexity of education. Keywords: value added, quality, ecological framework, data use, reform Ecologías de calidad educativa. epaa aape E ducation Policy A nalysis A rchives V ol. 21 N o. 8 SPECIAL ISSUE 2 Resumen: programas de responsabilidad educativa (accountability) han impulsado el “uso de información” (data use) entre responsables de implementar políticas y educadores profesionales tanto para decidir sobre cuestiones de instrucción como en decisiones sobre organización. Los medios de comunicación se han enfocado en los modelos de Valor Añadido (MVA) como un tipo clave de “uso de información” para reformar las escuelas estadounidenses. Entre los investigadores en educación, sin embargo, hay tanto críticos y como proponentes de MVAs. Examinamos el “uso de información” por directores, profesionales en 12 escuelas en un distrito urbano de gran tamaño, especialmente el papel que las medidas de MVAs tuvieron en sus decisiones y sus concepciones sobre sí mismos. MVAs es sólo uno de los muchos tipos de “usos de información” que pueden ser utilizados para indiicar calidad. Si bien hay una relación moderada entre MVA y calidad de la enseñanza medidas por CLASS, encontramos que la comprensión de los contextos escolares, especialmente el uso de los recursos y la coherencia de las acciones para mejorar el logro de los estudiantes, aumentó en gran medida el poder de nuestras descripciones. Sugerimos que las políticas que promueven enfoques multidimensionales de calidad captan mejor la complejidad de los sistemas educativos. Palabras clave: modelos de valor agregado; calidad; marco ecológico; uso de información; reforma. Ecologias de qualidade educativa. Resumo: Os programas de A responsabilidade educacionais (accountability) estão empurrando o "uso da informação" (uso de dados) aos responsáveis pela implementação de políticas e educadores nos processos de toma de decisões sobre questões educacionais e organizacionais. Os meios de comunicação têm-se centrado em modelos de valor agregado (MVA) como a questão chave no "uso de informação" para reformar as escolas americanas. Entre os pesquisadores de educação, no entanto, existem tanto críticos como defensores do MVA. Examinamos o "uso de informação" por diretores, profissionais de 12 escolas em um distrito escolar urbano de grande, tamanho especialmente o papel que tiverem as medidas de MVA em suas decisões e suas percepções sobre si mesmos. MVA é apenas um dos vários tipos de "uso de informações" que pode ser usado para indicar qualidade. Embora haja uma relação moderada entre MVA e qualidade da educação, medida por CLASS, encontramos que a compreensão do contexto escolar, especialmente o uso de recursos e da coerência das ações para melhorar o desempenho dos alunos, incremento muito o poder de nossas descrições. Nós sugerimos que as políticas que promovem a qualidade de enfoques multidimensionais de melhor captar a complexidade dos sistemas de ensino. Palavras-chave: modelos de valor agregado, qualidade, estrutura ecológica, o uso de informações; reforma. Introduction Yes, Virginia, you can have it all. We can get high equity and high student achievement. But we will have to go at it differently, very differently (Tucker, 2012). Public policymakers have developed what might seem like an insatiable appetite for quality indicators in education. Vast amounts of money have been invested in assessments, data warehouses, the development of analytical models, and curriculum alignment that should allow the public, policymakers and researchers to track the efficacy of schools in promoting achievement. Although no one would argue against education quality, and therefore the need to measure it, there is strong disagreement about how to enact it in practice, or even what quality means (Elliot, 2011; Zaslow, Tout, & Halle, 2011). E cologies of education quality 3 For some, quality is a matter of investment, identified through expenditures (Heck, 2007; Leak & Farkas, 2011). For others, quality is attained through regulatory mechanisms—teacher certification, years of experience, or class size, (Darling-Hammond, 1997; Konstantopoulos & Chung, 2011). A third group puts their money on the processes of the classroom, focusing on teacher-student interaction (Howes & Pianta, 2011; Pianta, Belsky, Houts, & Morrison, 2007). Still others frame quality in terms of test scores—the outputs yielded from schooling (Aaronson, Barrow, & Sander, 2007; Rivkin, Hanushek, & Kain, 2005). Linked to a commitment to accountability, the drive for quality has been pushed by methodological advances that translate complex constructs into technical tools (Rockoff, 2004). Value added modeling is a highly visible conception of quality, imported from economics and designed to identify effects of educational practice (Meyer, 1997). While a prominent feature of the quality debate, value added modeling is only one dimension of this complex construct. Building on work in early childhood education, we are informed by a multidimensional conceptualization of quality, one that highlights the uses for which a measure is designed (Zaslow, Tout, & Halle, 2011): The purpose for measuring quality is critical to selecting specific measures. If the goal is for overall quality improvement, a broad measure may be most appropriate, whereas if the goal is to improve practice in a specific domain, a measure focusing in depth on a particular aspect of the environment may be more appropriate (p. 393). Recognizing that quality is conceptualized differently by different audiences, for different purposes, we provide an analysis of quality in context. To do this, we examine how two distinct quality metrics—value-added and classroom quality—are promoted in local ecologies and often provide contrasting pictures as to why schools achieve success. We interpret these two measures of quality within social practices, examining how the conception of a high-, mid-, or low- quality school is facilitated and constrained through systems of belief, resources, and value. Specifically, we explore the research question: What is the role of local context in conceptions of quality such as value- added and classroom quality? How do these concepts play out in practice? Literature Review Discussions of quality in education have focused on three aspects of practice with specific attention to teacher quality: inputs, outputs, and process. Input approaches specify elements that are deemed necessary to set the scene for high quality teaching: per pupil expenditures, class size, or teacher credentials, education level and degrees earned. From an input perspective, quality is related to investments, and the characteristics of individuals that impact the education process (Pianta, et al., 2007; Rice, 2003). Recently, the outputs of teaching, as measures of effectiveness, have become targets of attention from an accountability perspective. Proximal measures of outputs explore student achievement/learning, with the expectation of improved outcomes are the result of high-quality teaching. For example, student promotion, higher grades, or decreased special education referrals could serve as proximal indicators of effective teaching. Distal measures of effectiveness examine students’ experiences later in life. Retention in later grades, graduation rates, incarceration rates, and adult income have all been used to analyze the long-term effectiveness of educational programs (e.g., Chetty, Friedman, & Rockoff, 2011; Reynolds, Temple, Robertson, & Mann, 2002). Attainment can be used as both a proximal and a distal measure, as can student growth or value-added (VA), given sufficient quality and quantity of data. The attainment perspective takes a snapshot of student achievement at a single point in time, as initially required by No Child Left Behind (NCLB; e.g., Schwartz, Schmitt, & Lose, 2012). In contrast, examining student growth is E ducation Policy A nalysis A rchives V ol. 21 N o. 8 SPECIAL ISSUE 4 designed to assess the contributions, or the VA, of particular schools or teachers (Hanushek & Rivkin, 2010; Harris, 2009). Similar to statistical models used in quasi-experimental evaluation studies, VA analyses may use existing information that includes all students in a district or state across multiple years, allowing population analysis over time (Braun, 2005; Rivers & Sanders, 2002). Proponents of value-added models (VAMs) argue that thoughtfully constructed VAMs can account for test scale errors, bias in test construction or administration, student background or participation, and/or classroom treatments (Ballou, Sanders, & Wright, 2004; Harris, 2011; VARC: Methodology, n.d.). VAM advocates conceptualize their contribution as enhancing our understanding of how teachers or schools support student achievement, above and beyond what one would expect from non-school factors (Hanushek & Rivkin, 2010). At the teacher level, value-added modeling is thought to fine-tune that estimate by parsing out the relative contributions of individual teachers to a student’s learning (Chetty, Friedman, & Rockoff, 2011; Rivers & Sanders, 2002). The use of VA as an output measure in education has become an area of intense interest in education policy and practice over the past three decades, bolstered by the push to examine educational practices using economic concepts (e.g., return on investment or cost effectiveness; Yeh & Ritter, 2009) and by competition for federal education dollars that emphasize the use of data- based decision making (e.g., Race to the Top, n.d.; Spillane, 2012). School leaders began using VAMs in the early 1990s when Tennessee launched the Tennessee Value-Added Assessment System (TVAAS), a statewide program monitoring teachers’ effectiveness as measured by a value-added model (Sanders & Horn, 1998). Over the next two decades, large city school districts (e.g., Boston, Dallas, Milwaukee) and entire states (e.g., Florida, North Carolina, Texas) adopted VAMs as ways to measure teacher quality (Braun, 2005; Harris, 2011). Highly public debates about teacher VA have appeared in The New York Times (Can a Few Year’s Data, 2012) and the Los Angeles Times (Grading the Teachers, n.d.). Publication of individual teachers’ VA scores in public databases has sparked discussion regarding the purpose and utility of these scores among education policy pundits, lawmakers, citizens, and bloggers (e.g. Gates, 2012; Wiggins, 2012; Winters, 2012). Skeptics of VAMs argue that we should be cautious in adopting the models, particularly if the aim is to use them to make comparisons between teachers or for individual personnel decisions because their reliability and validty are suspect (e.g., Au, 2007; Corcoran, 2010; Darling-Hammond, Amrein-Beardsely, Haertel, & Rothstein, 2012; Hill, Kapitula, & Umland, 2010). There is concern about how VAMs rely on standardized tests, which may not be up to the task of individual teacher evaluation (Amrein-Beardsley, 2009; Papay, 2010). There are issues of data quality in systems that are only now creating data warehouses capable of tracking data appropriately and various kinds of bias make modeling problematic (Briggs & Domingue, 2011; Rothstein, 2009). Furthermore, there is concern about the uneven track record VA has in education systems. For example, results have been inconsistent across models; the same teacher is shown as adding value for one model and not adding value in another model (Amrein-Beardsley, 2008; Newton, Darling-Hammond, Haertel, & Thomas, 2010; Papay, 2010). Even within the same model, teacher VA designations can be volatile, with wide swings from year to year (Amrein-Beardsley & Collins, 2012; Sass, 2008). It has proven difficult to disentangle teacher’s VA rom student characteristics because students are assigned in systematic ways to particular teachers and result in serious disincentives for teachers of students at risk (Rothstein, 2009). The highly technical nature of VAMs and the proprietary nature of their development have closed off review by anyone who is not an econometrician (Amrein-Beardsley, 2009). Finally, there are concerns about the validity of VAMs, particularly in using VAMs for making policy or personnel decisions (Amrein-Beardsley & Collins, 2012; Kupermintz, 2003) E cologies of education quality 5 Each of these approaches to quality frames it as an individual phenomenon, through a single teacher. We suggest a novel approach to quality, one that conceptualizes it from an ecological perspective. We have chosen this approach because it makes more complex the concept of classroom quality. In the space between quality inputs and outputs, some scholars have argued that we should examine the quality of student-teacher interaction because children’s learning and development are shaped within social and academic interactions in classrooms. It is through this cooperative encounter that instruction from the teacher turns into learning by the student (Stuhlman & Pianta, 2009). Through analysis of these classroom interactions, researchers have articulated elements of practice that are linked to student behaviors and outcomes (Hamre, Pianta, Mashburn, & Downer, 2007; Howes et al., 2008; NICHD, 2004; Wilson, Pianta, & Stuhlman, 2007). These elements of practice can form the basis for understanding and evaluating levels of classroom quality (LaParo, Pianta, & Stuhlman, 2004). This ecological approach draws our attention to the ways that metrics like VA variously attend to specific school contexts. By taking an ecological approach we push back on the assumption that VA alone can tell policy makers what they need to know about meeting the achievement needs of students. While VA accounts for student achievement mainly through demographic factors, an ecological approach allows us to consider the influence of context specific histories, relationships, and practices. Situating VA in relation to a school’s administrative history and practice, the resource allocation investments made by the community, or special funding sources, adds depth and breadth to our understanding of the many processes contributing to student achievement and classroom quality. The ecological perspective initially emerged as a theory of human development. Urie Bronfenbrenner (1977) first suggested that human development occurs within nested systems - some closer to a child and some further away - but all influencing her individual development. Experiences across these nested layers influence who each child is over time. While the ecological nested systems theory remains the most popular, Bronfenbrenner’s theory evolved as his own research and work developed, and he came to see his nested model as too static. Rather, he argued in 1995, development should be considered through process and context: a People Process Context Time (PPCT) theory. With PPCT, Bronfenbrenner asserted that meaningful research in development had to be grounded both in contexts and in their attending processes. PPCT theory provides a rich conceptual model of the kinds of data researchers and policy makers should consider when we seek to understand the development of a child, a student, or even a school context (Tudge, Mokrova, Hatfield, & Karnik, 2009). Students live in multiple systems - the classroom, the school, the community, families and home - that interact to influence their development (Bronfenbrenner, 1977). In addition, development is also affected by system actors, the temporal context and the processes experienced (Bronfenbrenner, 1995). From a policy perspective, with students nested within these ecological structures, the various systems, actors and resources that shape their academic development are important for consideration. We assert that by examining quality, including VA, within ecologies, we are more likely to understand how institutions and communities are resources for development and ultimately, student learning and achievement. We recognize that the teacher-student interactions that produce achievement come to be within particular social, educational, historical, and cultural contexts. These interactions result in certain processes of engagement. As such, we argue that an ecological view can provide a more nuanced understanding of how individual’s efforts interact to create education quality. This builds on the recent work on process and outcome quality, illustrating how school ecologies create opportunities and barriers for individual success, locating quality within systems and classrooms (e.g., O’Connor & McCartney, 2007). We believe this ecological model has the potential to deepen the E ducation Policy A nalysis A rchives V ol. 21 N o. 8 SPECIAL ISSUE 6 understanding VA provides, locating it within exploration of the how and why of student achievement. Methods As part of a larger study of a district’s implementation of a data use reform, we designed a mixed methods study that explored how the district worked to enhance the quality of decision making by building data-use capacity. This paper comes out of that project, focusing on a sample of schools to understand the relations among different metrics of quality within local education contexts. This ecologically based approach has been used productively in other studies considering practice in local contexts (Graue, Hatch, Rao, & Oen, 2007; Graue, Rauscher, & Sherfinski, 2009). The site of this study was a large, Midwestern, urban school district with a long history of low student achievement. At the time of this study, the district served over 80,000 students, 84 percent of whom were minorities (compared to a state average of 24 percent). Nearly 79 percent of the district student population was considered economically disadvantaged (compared to a state rate of just over 37 percent) and 20 percent had special needs (compared to 14 percent at the state level) (State electronic data base, 2012) When we began our study, the district had failed to make adequate yearly progress in math and reading for the prior six years and had been designated a District Identified For Improvement (DIFI) under NCLB. In response, the state department of education issued a demand for corrective action requiring the district to make specific reforms to improve student achievement. Our study took place within the context of a district-wide data use reform effort that provided professional development to district leaders on use of student data for instructional improvement. While data use was the impetus for the initial study, this paper focuses on conceptions of quality. Calculation of Value-Added Scores The VA model used by this district calculates growth in math and reading on the standards based state test, controlling for the previous years’ scores and student demographics, and correcting for measurement error in year 1 scores. Classroom effects are estimated separately for each grade, subject, and year, as are within-school and across-school variation in classroom effects. Univariate and multivariate shrinkage estimations are used to obtain more precise estimates, especially for classrooms with small class size. Univariate shrinkage assumes that teacher effects are independently distributed across the entire district. Multivariate shrinkage assumes that teacher effects are correlated within a school. With improved methods of student-teacher links, more precise estimates are obtained in later years. The district translated VA data into a 2x2 model to present static attainment in relation to change over time for the district and for individual schools. Through this 2x2 model, schools were classified into four quadrants based upon their attainment and VA (see Figure 1). Where schools fell on this quadrant chart became an important designation among staff within the district. While schools were not overtly sanctioned due to low VA scores, social pressure was paired with certain VA labels. In professional development sessions that we observed, some school staff would tout their high VA/high attainment status, while those from low/low schools remained silent. In addition, schools were required to document their quadrant placement within their School Improvement Plans. E cologies of education quality 7 Quadrant 3 Quadrant 1 Quadrant 4 Quadrant 2 Figure 1. District defined value added quadrants We used the quadrant model to select the sample for this project given its importance in the district. This allowed us to explore schools that varied in attainment and growth, using their quadrant classification as a quality metric. In addition to considering VA/attainment quadrant, we took into account contextual elements that shape practice, including a range of school types, from suburban ring to inner city, with varying levels of student poverty, English language learners and special needs students. Table 1 shows sample demographics in relation to the broader district, arranged in order of VA quadrant. ⇐ L ow A tt ai nm en t H ig h⇒ ⇐Low Value-Added High⇒ E ducation Policy A nalysis A rchives V ol. 21 N o. 8 SPECIAL ISSUE 8 Table 1 School Sample, 2009-2010 School* District Mulberry Fieldrush Wildwood Palmetto Nutmeg Myrtle Grades 3-8 PreK- 8 PreK - 5 PreK - 5 PreK-5 PreK-8 Total students 85,376 403 533 217 520 371 331 VA Quadrant Reading N/A 1 1 1 1 1 2 VA Quadrant Math N/A 1 1 1 1 3 2 % Prof/Adv in Math/Reading** 49/59 95/96 82/72 94/92 62/71 71/68 37/21 % Economically disadvantaged 77 38 28 31 94 42 93 % English proficient 91 99 99 99 84 100 100 % African American 57 53 9 6 55 47 98 % Latino 23 6 17 16 35 5 0 % White 15 36 71 76 8 43 2 % Asian 5 5 1 1 1 4 0 % Native American 1 0 2 2 1 1 0 % with Disability 19 4 13 7 23 23 24 *All names for sites and persons are pseudonyms. ** Percentage of students scoring Proficient and Advanced on state test in grades 3-5 or 3-8 in 2008-09 data; Source: State school information website. E cologies of education quality 9 Table 1 (Cont’d.) School Sample, 2009-2010 *All names for sites and persons are pseudonyms. ** Percentage of students scoring Proficient and Advanced on state test in grades 3-5 or 3-8 in 2008-09 data; Source: State school information website. Recording School-level Practices Once we had purposively selected our sample schools, the district research office contacted the principals, who forwarded our invitation to teachers. At each site, we invited all grade 3-5 teachers to participate in observations and interviews, with 81 percent participation overall (range 60 to 100 percent) for a total of 47. Table 2 shows overall participation at each school site. School* District Birch Acacia Hazel Sycamore Beech Hickory Grades PreK - 8 PreK - 5 PreK - 5 PreK- 5 PreK-5 PreK-8 Total students 85,376 459 247 368 305 353 325 VA Quadrant Reading N/A 3 n/a 4 4 4 4 VA Quadrant Math N/A 1 3 2 4 4 4 % Prof/Adv in Math/Reading** 49/59 82/73 51/70 51/47 61/46 47/38 35/19 % Economically disadvantaged 77 92 92 91 92 94 92 % English proficient 91 84 43 100 100 100 100 % African American 57 83 1 88 95 96 97 % Latino 23 1 99 5 1 1 0 % White 15 3 0 5 3 2 3 % Asian 5 13 0 1 1 1 0 % Native American 1 0 0 0 0 0 0 % with Disability 19 21 9 25 13 23 24 E ducation Policy A nalysis A rchives V ol. 21 N o. 8 SPECIAL ISSUE 10 Table 2 Participants and Participation School Teachers Invited Teachers Participated Percentage Participation Acacia 4 3 75% Beech 4 3 75% Birch 5 4 80% Fieldrush 6 5 83% Hazel 6 5 83% Hickory 4 3 75% Mulberry 5 5 100% Myrtle 5 4 80% Nutmeg 5 4 80% Palmetto 5 3 60% Sycamore 6 5 83% Wildwood 3 3 100% Total 58 47 81% Our school visits lasted two to three days during the spring of 2010. We conducted semi- structured interviews with each principal, with volunteers from the learning team (a school level leadership group charged with data analysis) and the broader teaching staff. Interview questions focused on the school and classroom practices related to data use and individuals’ understandings of VA. Questions also worked to link contextual issues in the district and the school to the specific practice in classrooms and work to promote student attainment. We provide an example of the interview protocol in Appendix A. Interviews ranged in length from 40 to 120 minutes and were audio taped and transcribed for analysis. We collected relevant documents and artifacts, including worksheets, curriculum guides, and school and district reports. We also obtained school-level test score and demographic data from the state’s public database. Pairs of observers visited the classroom of each participating teacher for two hours scheduled to coincide with typical instructional practice. While one observer took descriptive field notes that detailed instructional environment and interaction, the other rated classroom quality using the Classroom Assessment Scoring System (CLASS; see Pianta, LaParo, & Hamre, 2008). In each classroom, the rater observed and coded classroom quality across four twenty-minute cycles of instruction with CLASS. These observations generated brief field notes that the observer used to assign ratings of 1-7 on each of the CLASS dimensions, referring to detailed rubrics in the CLASS manual. Ratings are categorized at three levels: 1-2 = low, 3-5 = midrange, 6-7 = high. Average scores were tabulated across four cycles and CLASS dimension and domain scores were calculated by both classroom and school. Process Quality: The Classroom Assessment Scoring System The CLASS provides a common metric for understanding classroom quality, based on the relationships between student outcomes and the quality of teacher-child interactions (For more information on CLASS, see http://www.teachstone.org/research-and-evidence/research-summary- 2/). As a result of its use for observations in thousands of classrooms, CLASS reliably and validly describes the components of classroom interaction that are linked empirically to later student outcomes, pre-K through grade twelve (Hamre & Pianta, 2005; LaParo et al., 2004; Pianta et al., E cologies of education quality 11 2007; Stuhlman & Pianta, 2009). To ensure valid inferences, observers must be trained and tested by a CLASS certified trainer so that they reliably use the CLASS framework to rate classroom quality. All members of our research team were certified CLASS observers. The constructs employed by CLASS are described in Table 3. Average CLASS scores, and their range, by school are presented in Table 4. Table 3 Domains and Dimensions of CLASS Emotional Support Classroom Organization Instructional Support Positive Climate The enjoyment and emotional connection that teachers have with students and the nature of peer interactions Behavior Management How well teachers monitor, prevent, and redirect behaviors Concept Development Measures how teachers promote higher order thinking and problem solving going beyond fact and recall activities with children Negative Climate Reflects negativity such as anger, hostility, or aggression expressed by teachers and/or students in the classroom Productivity Considers how effectively teachers manage time and create classroom routines that maximize learning time in the classroom Quality of Feedback Considers how teachers extend student learning through their responses and participation in activities Regard for Student Perspectives Captures the degree to which teachers’ interactions with students and classroom activities place an emphasis on students’ interests, motivations, and points of view Instructional Learning Formats Focuses on how teachers engage students in activities and facilitate activities so that student learning is maximized Language Modeling Reflects the extent to which teachers facilitate and encourage student language Teacher Sensitivity Reflects teachers’ responsivity to individual students’ academic and emotional needs E ducation Policy A nalysis A rchives V ol. 21 N o. 8 SPECIAL ISSUE 12 Table 4 CLASS Scores and Range by School School Emotional Support Classroom Organization Instructional Support School CLASS Total 5.76 5.92 4.63 5.48 Mulberry Average Range 5.5 to 6.0 5.42 to 6.42 3.50 to 5.63 5.25 to 5.80 5.59 5.18 3.88 4.93 Fieldrush Average Range 4.69 to 6.44 3.75 to 5.92 2.69 to 5.13 3.75 to 5.97 5.19 5.22 3.78 4.84 Wildwood Average Range 4.75 to 5.94 4.50 to 5.92 2.83 to 4.44 4.10 to 5.29 4.92 5.36 4.20 4.86 Palmetto Average Range 4.38 to 5.63 4.58 to 6.17 3.06 to 5.17 4.19 to 5.65 5.40 5.25 3.80 4.90 Nutmeg Average Range 4.81 to 5.94 4.42 to 6.42 3.08 to 4.88 4.18 to 5.87 5.28 5.00 3.60 4.71 Myrtle Average Range 4.38 to 6.13 4.67 to 5.50 2.25 to 5.56 3.95 to 5.86 4.88 5.48 3.36 4.75 Birch Average Range 4.31 to 5.38 4.67 to 6.33 2.75 to 4.06 4.03 to 5.11 5.56 5.67 3.31 4.94 Acacia Average Range 5.38 to 5.94 5.50 to 5.75 2.88 to 3.92 4.75 to 5.05 4.85 4.75 2.85 4.31 Hazel Average Range 4.06 to 5.69 3.33 to 6.25 2.33 to 3.75 3.53 to 5.52 5.61 5.17 3.50 4.83 Sycamore Average Range 5.19 to 5.94 4.33 to 5.75 2.75 to 4.25 4.38 to 5.20 4.13 4.46 2.90 4.02 Beech Average Range 3.63 to 4.50 2.75 to 6.17 2.0 to 3.69 2.88 to 5.04 3.60 3.78 2.48 3.31 Hickory Average Range 2.44 to 4.94 2.58 to 4.83 1.5 to 3.69 2.51 to 4.61 Data Analysis Our analysis brought together teacher and school VA scores, CLASS ratings, notes from classroom observations, transcripts of interviews with school staff, and school documents. The specific work for this paper began with the examination of the CLASS ratings of quality in relation to school VA. Based on the assumption that classroom quality comes to be in the context of school ecologies, we explored the relationship between output and process quality. We conducted a linear correlation analysis, calculating the Pearson correlation coefficient. We used the following variables in our analysis: school-wide three-year n weighted VA scores in math and reading, average CLASS ratings across sampled classrooms at both the domain and aggregate levels, and the percent of students eligible for free and reduced lunch. These metrics were chosen as different aspects of quality, and we included socioeconomic status given its relationship to resources in school contexts and student achievement. This approach had both inductive and deductive elements. We observed that the variation within schools could be as great as the variation between schools. It was therefore important to closely examine patterns in administrative practice, teacher beliefs, types of classroom interaction, and school attainment. In contrast, we also worked from the ratings to the fieldnotes and interviews, exploring likenesses and differences in high, midrange, and low rated classrooms. For each school, E cologies of education quality 13 our field notes were used as a crosscheck for CLASS ratings. We read the CLASS ratings in relation to the full fieldnotes, working to illustrate the ratings in practice. We looked for confirming and disconfirming evidence for the patterns we identified. Supported by the qualitative data analysis program NVivo, the research team analyzed data sources using a shared set of codes. These codes reflected the assumptions that guided the research as well as concepts that were new to the project. We shared memos (Graue & Walsh, 1998; Maxwell, 1996) that detailed analytical conceptualizations that linked coding categories and illustrated crosscutting themes (Saldaña, 2009). With a mixed methods approach it is important to be clear about how we made research inferences. We took a strong interpretive approach (Graue et al, 2007; Graue et al., 2009), with attention to how the local school context set the stage for both school and classroom quality. We relied on the notions of trustworthiness and transferability, where the readers made connections from the case to their own knowledge and experience, evaluated by the degree to which the findings appeared to resonate with the reader from their reading of the existing literature and the presentation of the case relative to the literature (Lincoln & Guba, 1985). Given our interpretive approach, we are clear about the limitations of such a study. As a strength, we carefully assembled a bricolage of data sources related to education quality and examined them in relation to one another and in specific local contexts. We recognize that our sampling techniques limited the inferences made from our interpretations; the results presented are not population inferences. Instead we conceptualize them as quality in context, presenting carefully considered examples to enrich our thinking about quality. Results Our analyses focused on two levels. The first level examined the relationship between outcome and process measures of quality within our 12-school sample at both the school and classroom levels. This analysis is presented through correlations of the quality measures (CLASS and VA) in relation to the percentage of students in poverty in each school. The second level described four schools that exemplified how school ecologies (i.e., their histories, community resources, and administrative supports) provide a context for better understanding “quality.” Correlation of VA and CLASS for Schools and Teachers Given our interpretive approach, our attention to statistical significance is slightly different than if we were arguing for strong population inferences. With the small number of schools and teachers, we lack statistical power, but more importantly we see the correlation coefficients as rough indicators of shared variation, analogous to a scatterplot. We consider it in relation to the case study data to analyze how these two measures of quality are related in these specific contexts. We will note however that with the small sample the statistical significance indicated in the tables could be thought of as more pattern than noise. At the school level, these two measures of quality, VA and CLASS, were positively correlated (see Table 5). The CLASS Instructional Support dimension, which describes how teachers extend student thinking, provide feedback and support language, is most strongly related to VA, both in a single year and with a three-year average. This is in keeping with other research that has linked instructional support to school outcomes (Hamre & Pianta, 2005; LaParo et al, 2004; Pianta et al, 2007). Both VA and CLASS, as measures of outcome and process quality, are negatively related to the density of students living in poverty in a school (see Table 5). This, too, is congruent with previous research that finds that students in low-income schools receive lower quality instruction and as a result, have lower achievement (Adamson & Darling-Hammond, 2012; Burchinal et al., 2011). E ducation Policy A nalysis A rchives V ol. 21 N o. 8 SPECIAL ISSUE 14 Table 5 Correlation Between School VA and CLASS VA/Rdg 1 year VA/Math 1 year VA/Rdg 3 year average VA/Math 3 year average % of Students Economically Disadvantaged Emotional Support .42 .32 .53 .38 - .48 Classroom Organization .54 .41 .51 .58* - .36 Instructional Support .65* .63* .71* .64* - .59* CLASS Total .55 .43 .58 .56 - .50 % of Students Economically Disadvantaged -0.81** - .53 - .73* - .35 -- * p < .10; ** p <.05 At the level of an individual classroom, the relation between classroom quality (CLASS) and teacher quality (VA) was weaker than at the school level (see Table 6). Statistically this makes sense—more data points reduce the error—but from a practical point of view, this finding is problematic. If VA and CLASS are both measures of quality, and if policy decisions, like teacher evaluation, hinge on these measures, they should be similar rather than different to be valid indicators. A focus on VA alone could miss important information about education quality. We argue, therefore, that policy makers should consider, as we do in the next section, what other factors may be influencing classroom quality. Table 6 Correlation Between Classroom VA and CLASS Math VA 1 year Reading VA 1 year Math VA 3 year Reading VA 3 year Emotional Support (E.S.) .23 .22 .25* .41*** Classroom Organization (C.O.) .19 .24 .11 .37* Instructional Support (I.S.) .28* .29* .20 .40*** CLASS Total .22 .23 .14 .40*** * p < .10; ** p < .05; *** p <.001 E cologies of education quality 15 Broadening the Focus Ecologically While VA pays some attention to context through its inclusion of student demographic data, it is a statistical consideration, limited to what can be portrayed in the model. We wondered how a more nuanced view of context might help us understand how quality comes to be locally and, therefore, how VA reflects or obscures the contribution of local ecology to schooling. We turned to the development of descriptive portrayals of the schools, relying on documents, observations, and interviews, to provide a window of the local context, and to better understand how this related to the quantitative data we had gathered. The kind of description that would provide insight into context would have been unwieldy with all twelve schools in our sample, so we have narrowed our descriptive focus to four sites. We describe four schools that represent different VA quadrants, profiles, and communities to explore how school VA translates into particular school practices and further, how they map onto individual teacher measures of quality. Table 7 provides a longitudinal look at school VA quadrant placements over an 8-year period. Mulberry consistently occupies Quadrant 1 and Hickory predominantly Quadrant 4. Sycamore more often sits in Quadrant 2, while Nutmeg goes back and forth between Quadrants 1 and 3. Table 7 VA Quadrants Over Time for Four Sample Schools School 2001-02 2002-03 2003-04 2004-05 2005-06 2006-07 2007-08 2008-09 Reading 1 1 1 1 1 1 1 1 Mulberry School Math 1 1 1 1 1 1 1 1 Reading 3 1 3 1 1 1 1 3 Nutmeg School Math 3 1 3 1 1 3 1 3 Reading 1 1 3 2 2 2 4 2 Sycamore School Math 1 1 3 4 4 4 4 2 Reading 4 2 4 4 4 2 4 4 Hickory School Math 2 4 4 4 4 2 4 4 In the following sections we describe each school context and the ecological qualities that shape students’ and teachers’ experiences. As an overview, we illustrate the CLASS-VA relationship at the four schools at the individual teacher level. If classroom quality (measured by CLASS) and teacher quality (measured by VA) represented the same construct, we would expect a positive linear display. This graph, which represents individual teachers’ scores, is less linear than might be expected for measures that are used for high stakes decisions. E ducation Policy A nalysis A rchives V ol. 21 N o. 8 SPECIAL ISSUE 16 Figure 2. CLASS Instructional Support and VA by Teacher The Contexts for Quality Mulberry: Overflowing assets A high VA/high attainment school (Quadrant 1), Mulberry was overflowing with assets. The students at this selective Grade 3-8 school for the talented and gifted were the highest performing in the district. The school served few students with disabilities and had a higher proportion of white students (36 percent) than the district overall (15 percent). Table 8 School, CLASS, and VA Data for Mulberry School School CLASS Scores Emotional Support 5.8 Classroom Organization 5.9 Instructional Support 4.6 Key Demographics % eligible FRL 38 % Prof/Adv in Math 95 % Prof/Adv in Reading 96 VA Quadrant 2008-09 Math: 1 Reading: 1 Mulberry families were more affluent than their district peers, and the language of economic capital came up when a teacher described how parents saw the school: I think that parents consider this school an investment . . . these parents are investing so much time and effort, and they know that it will pay off in the end. And their investment and privilege is taken seriously [Mulberry teacher]. A sense of exceptionalism pervaded every aspect of life at Mulberry. The enriched educational experience at Mulberry, beyond the means of most district elementary schools, was possible because many Mulberry families could contribute (both financially and with their time) E cologies of education quality 17 and because teachers worked with the students to raise funds. The school’s website showcased its assets, programs and events by providing links to media coverage of the school and actively soliciting donations well beyond what the typical bake sale would bring in. One page suggested donations ranging from $100, to secure a bus for a field trip, to $40,000, to fund a full-time teacher of the arts. The physical plant was also a treasure. One of the oldest school buildings in the district, it had been lovingly restored and maintained. The classrooms were beautifully equipped with a variety of high tech teaching tools and teachers were well-trained in their use. In a district that struggled to provide any instruction outside the classroom core, education at Mulberry was multidimensional. Supplementing the art and music specialists from the district, Mulberry raised funds to hire a theater teacher and all students took tap dancing from a dance instructor. Not content to bring the world into the school, teachers took students on a variety of learning trips to destinations ranging from the state capital to Europe. According to one teacher, Mulberry was a place that had “all of those wonderful programs...that complete the whole child.” The teaching staff and their instructional expertise were assets as well. A dedicated and talented teaching staff viewed themselves as educators who pushed their students to reach their potential. Five world languages were taught and teachers could draw on a vast network of parent volunteers to help with almost any task and it was expected that all parents followed through on school work at home. Classroom quality was high at Mulberry, with uniformly strong CLASS ratings mirroring the positive VA ratings. Mr. Minsky, a 4th grade teacher, exemplified the wealth of teaching expertise at Mulberry. In sessions that we observed, Mr. Minsky drew on the considerable resources available in his classroom to build his students’ knowledge. For example, he activated students’ unique knowledge (referring to a trip that a student took), and worked to clarify their learning by making connections (referring to earlier books they had read or discussions they had had). It would be easy to say that Mulberry is a model school that could be held up as an example for reform. If the ecological context of Mulberry could be replicated across this district, it is a good bet that achievement would go up, teacher satisfaction would improve, and children’s futures would not seem so bleak. But was it the instructional context that added value to these students’ education? Or was the narrative more complex, an example of Holiday & Herzog’s lyrics: “Them that’s got shall have. Them that’s not, shall lose...” (1939). What made education work so well at Mulberry was a combination of the school’s tremendous resources and how those resources were used in a coherent way by staff working together toward the same goal. They had little resistance to work against and every advantage working for them. With parents supporting and contributing to their efforts, the students and staff at Mulberry enjoyed an embarrassment of riches. Given every advantage, it is not surprising that this school has had high VA scores year after year. Assuming that other schools could somehow follow the lead of Mulberry through instructional reform, however, ignores the very different circumstances in which other schools in this district exist. Hickory: A school all tattered and torn Unlike Mulberry, Hickory was not a school with either a high VA or a high attainment score, nor did they have overflowing assets. Instead, Hickory was a more common example of a school in this district: struggling under the weight of the burdens carried by its students and families. The school served a predominantly African American and low-income population and is nested deep within one of the poorest zip codes in the United States. E ducation Policy A nalysis A rchives V ol. 21 N o. 8 SPECIAL ISSUE 18 Table 9 School, CLASS, and VA Data for Hickory School School CLASS Scores Emotional Support 3.6 Classroom Organization 3.8 Instructional Support 2.5 Key Demographics % eligible FRL 92 % Prof/Adv in Math 35 % Prof/Adv in Reading 19 VA Quadrant 2008-09 Math: 4 Reading: 4 Hickory stood as a stark contrast to Mulberry, with some of the lowest CLASS scores from our sample (especially in Instructional Support) and low VA/low attainment scores. It was a school on the brink. Whereas the entrance to Mulberry was stately, the entrance to Hickory was a large, plain, locked black double door with a buzzer. Inside, the barren green walls and dusty floors gave the building an institutional feel. While there were provisions for security, they did not seem to be used; the security guard’s desk was unoccupied. What the worn public face of Hickory did not show was that the school received supplementary materials due to the extreme poverty of its students and their failing test scores. For example, the school received funding for staff to provide supplementary services to students. However, each additional staff member represented a different initiative at work in the school: a math teacher for the math grant; a reading coach for the literacy initiative. As each initiative was uncoordinated with others, they seemed to take on a hit or miss approach. These adults walked the halls, pulling their students from classrooms for supplemental instruction, regardless of the desires of the classroom teacher. These supplementary staff and the classroom teachers had little to do with each other beyond the fact that they served the same students. According to classroom teachers, they failed to provide cohesive support for the students that needed it most. Whereas Mulberry often successfully relied on families to raise money for technology, Hickory received grants for Smart boards in each classroom and for a central computer lab to support student learning. However, due to a lack of funds to train and support teachers or provide a teacher for the computer lab, many of these new tools were not integrated into instruction. During an observation of a classroom teacher at Hickory, one of our researchers noticed a Smart board that was entirely hidden behind stacks of old books. When she later asked the classroom teacher about the Smart board, she was told that the teacher had never been given support in using the board, so she couldn’t use it. Apparently this was the case in many classrooms through the school. Compared to Mulberry teachers, who only got stronger with strategic and concerted use of their resources, Hickory staff was too tired or overwhelmed to take up new opportunities and materials directed to their school. According to many of the teachers that we observed, the school experienced a hit and run approach to goods and services: a grant or organization would allocate resources at Hickory and then move on to another school. Hickory staff was left to contend with these contributions without training or support. The inability of Hickory’s teachers’ to activate and utilize these resources, and be supported in their use, minimized their benefit. This may have hurt schools like Hickory in another way as well. Being the beneficiaries of material goods, but not ongoing support, schools like Hickory ran the risk of looking ungrateful and unmotivated to use the goods that they were given. These patterns were visible across a variety of data. The CLASS scores at Hickory were the lowest in our sample and illustrated a typical image of a low VA/low attainment school. An E cologies of education quality 19 instructional example of the lack of coordination of practice at Hickory was seen in Ms. Connor’s fifth grade classroom. The beginning of the day was beset with interruptions. Teaching started and stopped around PA announcements or phone calls from the office; more than 10 of her students were pulled out for various services. The frustration on Ms. Connor’s face quickly became apparent as she had to repeatedly retrace and repeat the content she was reviewing. While the pull out system allowed students to get remedial and intervention services, it put those students behind what was happening in their classrooms. What might have been a well- designed lesson became a disjointed instructional experience for these students. Even students who were not pulled out were affected, as the forward motion of the lesson slowed to a crawl each time a specialist came in to either pull or return a student. With this system in play, intended learning was disrupted by many interruptions. The Hickory community worked in a challenging context—a rundown facility, children in poverty, a history of low performance. These issues had brought supplementary resources from various groups, but the inability to coordinate the structure and assets of the school sustained the problems that limited practices at Hickory. What the principal said was often in conflict with what the Learning Team and teachers shared. Likewise, the school leaders and teachers maintained a narrow view of their duties, only focusing on their own responsibilities. The principal, for his part, was primarily concerned with his administrative role – what happened in the classrooms was the purview of the teachers. When asked to reflect on their practices, both the principal and the Learning Team pointed fingers at other—the principal believed the Learning Team needed to be more focused, whereas the Learning Team thought the teachers needed to be better qualified and the principal more hands on in actualizing resources. Lastly, the teachers blamed the students’ home lives for their poor achievement levels. Given their focus on passing the blame, the staff could not leverage the resources available to develop a coherent plan to improve achievement. Nutmeg: Running hard to stay in place Unlike Hickory, Nutmeg Elementary School appeared to have many advantages. Bordering an affluent suburb, the neighborhood was predominantly white and middle class, largely populated by teachers, firefighters and police who are required to live within the city limits. The school was surrounded by a sizeable lawn and large areas dedicated to student play. The school building was a rambling, 1960’s open concept school. The long hallways were clean, and the walls a riot of color with posters, bulletin boards and student work. Outside each room was a small white board with the day’s plan in terms of concepts, tests, curriculum, and technology. Despite these outwards markers of a school primed to provide its students with advantages, Nutmeg was in fact a low VA/high attainment school (see Table 10). In addition, the range of classroom quality was wide at Nutmeg, with some teachers scoring very high on the CLASS and others in the mediocre range. Table 10 School, CLASS, and VA Data for Nutmeg School School CLASS Scores Emotional Support 5.4 Classroom Organization 5.3 Instructional Support 3.8 Key Demographics % eligible FRL 42 % Prof/Adv in Math 71 % Prof/Adv in Reading 68 VA Quadrant 2008-09 Math: 3 Reading: 3 E ducation Policy A nalysis A rchives V ol. 21 N o. 8 SPECIAL ISSUE 20 One of Nutmeg’s main assets was the student population, which was somewhat similar to that of high performing Mulberry School. The two schools had about the same proportion of low- income students, both well below the district average. The percentages of students of each race were about the same as well. In addition, the school had a resourceful, high energy principal who gave clear instructions (such as requiring the teachers to write their day’s plan on the whiteboard outside the classroom doors) and walked the halls, checking in with students and teachers to make sure that these instructions were being carried out. However, unlike Mulberry, which pulled students from across the city, Nutmeg was primarily a neighborhood school. In fact, as a magnet school for gifted and talented, Mulberry pulled the best students away from schools like Nutmeg. The principal, Mr. Reagan, expressed frustration at how the magnet schools skimmed off “his” students after they finished second grade: I've been here five years now—and I know that from grades 3-5 I've probably lost a good 20-25 students over the course of the time. Those are my proficient and advanced students. So I could just imagine what my [state test] and Benchmark scores would look like had those third, fourth, and fifth graders stayed here [Nutmeg Principal interview]. Losing 4-5 students per year was clearly painful for the principal, although it amounted to just 1 percent of his students annually. However, for the principal, this was a main reason that their school was not shown to be adding value like the District wanted. The loss of students to more “prestigious” schools was a professional affront for a principal who felt that his school had much to offer and that his achievement scores were suffering as a result. As we analyzed the practices at Nutmeg, we realized that Mr. Reagan was primarily focused on the school’s image to the outside world. At the school level, the strong procedural leadership of the principal gave the school an outward direction. Mr. Reagan gave many examples of how the school staff had been ahead of the curve on the most recent district reform initiatives because of their participation in nearly every pilot opportunity that the district offered. While much of Mr. Reagan’s focus was on managing the outward perceptions of his school, he was also in tune with his staff in terms of their expectations of him. In answer to our question, “What would you like the principal to know...?” nearly all of the teachers answered, “Oh, he knows.” As one teacher told us: He's very supportive of whatever we need. Whatever I need, he's pretty much willing to provide it, whether it's assistance in a discipline situation or—he's just aware. He's involved; it's not like he's hiding out in the office, and he's aware of what students are in the building, and he deals with the issues [5th grade teacher, Focus Group]. Mr. Reagan was protective of the teaching staff, questioning the logic of our research project and conveying teachers’ concerns to us before we entered classrooms. He was a school leader, but not an instructional leader. Nutmeg staff saw themselves as the hardworking but under-appreciated stepchild. As a low VA/high attainment school, staff were concerned that their identity as high achievers was questioned because they were not adding value. With good test scores and strong data practices, it was irritating that they were not recognized as an outstanding school. That mismatch made it easy to focus on the attainment scores as the favored way they identified themselves. It was additionally irritating, noted school staff, that lower achieving schools, such as Hickory, were the recipients of resources that these teacher felt they could use to improve their instruction. In addition to losing students to other schools, teachers noted that Nutmeg would be losing its class size reduction funding for the coming year, so K-3 class size would rise, and the librarian would be forced to return to the classroom. Special classes were being eliminated, and along with them, coordinated planning time. E cologies of education quality 21 As a group, Nutmeg teachers were clearly suffering from “reform fatigue.” Their principal, eager to capture the attention to District level staff, signed the school up for every reform being piloted, and staff, while impressed with Mr. Reagan’s confidence in their abilities, sometimes had trouble choosing which reform to pay attention to: If literacy centers are going to be it, they need to say that's going to be our focus for next year and provide in-servicing and training over the summer, where you can hit the ground running, and do it. And that's the focus this year, and not keep changing what's going on. I mean, it's such a reactive district now: anything comes out now, it's, “Okay, we've got to change; we've got to do something else.” And they have to throw something else out. It seems like it's just so reactive that nothing really stays in place and is given a chance to show whether it's good or bad [5th grade teacher, Focus Group]. The changing outside reforms and district agenda were indicative of what drove the education engine at Nutmeg—the perspectives of outsiders. Overall, the pressure placed on Nutmeg to reach standards had redirected attention from the internal—what the staff intuitively knew they needed to be doing to improve their school, to the external—what District administrators, and Mr. Reagan, were telling them they should be doing. Sycamore: All oars in the water but paddling upstream A sprawling mid-century style school building, Sycamore was located in a predominantly working class, African American neighborhood. The entrance to the school was light and airy with an area devoted to family materials – from upcoming neighborhood events to resources for free healthcare and dental services for children. At the beginning of the school day parents and children moved quickly down hallways and into the gym, which doubled as the cafeteria and was teeming with children eating breakfast. While Sycamore seemed to be a low VA/low attainment school like Hickory, serving an overwhelmingly low income, African American population, the schools could not have been more different. Unlike Hickory, Sycamore had some of the highest CLASS scores in our sample (see Table 11). Table 11 School, CLASS, and VA Data for Sycamore School School CLASS Scores Emotional Support 5.6 Classroom Organization 5.2 Instructional Support 3.5 Key Demographics % eligible FRL 92 % Prof/Adv in Math 51 % Prof/Adv in Reading 47 VA Quadrant 2008-09 Math: 2 Reading: 2 The principal was absent the morning of our visit, away at an administrative meeting, but had prepared her staff in advance for our visit. In her stead, the Curriculum Generalist (an administrative team member) maintained an easy sense of order, using a walkie-talkie, and carrying a clipboard. Mothers with infants and toddlers sat at tables with their school-age children and talked to them as they ate breakfast in the cafeteria before school started. Across from the cafeteria, the office was open and buzzing. The secretaries laughed with parents who dropped off notes or let them know of a child needing to leave early. The energy level in the school was high, happy, and contagious. E ducation Policy A nalysis A rchives V ol. 21 N o. 8 SPECIAL ISSUE 22 This entrance scene paints a picture of the overwhelmingly positive and family-oriented spirit of Sycamore. From this description, or its proximity to a largely white and affluent inner-ring suburb, one might think that this was a well-resourced, suburban school. Instead, Sycamore struggled daily to meet its students’ academic needs. Students were bussed to Sycamore from across the district, most coming from neighborhoods within the city that are among the poorest in the nation. Consistently moving back and forth between being high VA/low attainment (Quadrant 2) and low VA/low attainment (Quadrant 4), Sycamore was labeled a success one year and a failure the next (see Table 7). Though achievement was elusive, the mission of the school – to serve children and their families – could be seen in the friendly interactions among families and staff and the instructional practices aimed at high achievement. Sycamore staff spoke with one voice about the goals of the school – they wanted to do everything within their power to make their students successful. The coherent nature of the school staff, mission and goals was at the core of everything we saw at Sycamore. Each interview portrayed an image of a healthy school culture united in a common mission of student success. Interestingly, gains in one area – either math or literacy – were often not sustained when gains were made in the other. It became clear that as attention shifted to areas of need, teachers were able to promote gains in one area but that this often lead to losses in another. These trade-offs in VA scores point to the largest roadblock to success at Sycamore: an overall lack of resources. Not doing poorly enough to garner extra resources and also not doing well enough to overcome obstacles to student success, Sycamore sat on a bubble in the district. In some ways, more failing students would have secured them greater support. But this ran contrary to the strong school mission and goal of achievement for all students that was at the heart of Sycamore School. Because resources were limited at Sycamore, chart paper and chalkboards were the state of the art classroom technology, class sizes were larger than the other schools in our sample, and no specials were available to students. In general, school staff seemed to be paddling against a current that daily threatened to sweep their students away. It didn’t matter that they all had their oars in and were rowing together – they were still using a rowboat when to make headway they needed a motor. While teachers actively channeled their limited resources towards academic achievement within a context of developing respectful and ethical citizens, it wasn’t enough. For Sycamore, the disconnect between the overwhelming unity of the staff and mission and the stark reality of a lack of resources kept them from accomplishing the goals necessary to add enough value to raise their students’ academic achievement. Several characteristics of Sycamore were unique. At this year-round school, the rhythm of the school was different from the others in our sample. The school’s distributed leadership style valued the professional knowledge and abilities of staff and its active engagement of families. More than any other school in our sample, Sycamore had a large number of African American teachers and staff, many in leadership positions. Families were omnipresent at Sycamore. Families could be seen in every part of the school, in the cafeteria and in classrooms, allowing mothers who worked the third shift to see their children during the school day. Teachers seemed to view these visits as a matter of course and an opportunity to greet and chat with the parents, rather than an intrusion on their instructional time. These positive attributes met structural forces in resource allocation that limited the ability to translate these assets into achievement. At Sycamore, we found an interesting example of how classroom quality can be disturbed in a school united in its purpose, but fragile in its delivery. On the morning we observed Ms. Driver’s third grade class, we found an inviting, spacious classroom that hummed with activity at the start of E cologies of education quality 23 the day. The intellectual quality gained momentum as the morning proceeded and the children worked in literature groups interpreting complex texts. Whole group instruction went smoothly with this group of students and the level of discussion was high. At 10:00 AM, Ms. Driver was called to an IEP meeting. A paraprofessional took her place, continuing the literature lesson. The group immediately began to unravel and the lesson fell apart. Children began to walk around the classroom, and we saw a human version of herding cats. Ms. Driver returned after 25 minutes; the children immediately coalesced, regaining focus and interest in their work. Table 12 illustrates how support for learning broke down when the teacher was pulled for other duties. Observation cycles involved twenty minutes of observation followed by ten minutes of rating. Table 12 CLASS Scores for Teacher and her Paraprofessional Substitute Teacher for each 20 minute scoring segment Emotional Support Classroom Organization Instructional Support Total Ms. Driver 4.40 5.33 2.33 4.40 Ms. Driver 5.80 5.67 6.00 5.80 Paraprofessional 3.20 2.00 2.00 3.20 Ms. Driver 5.80 6.00 5.67 5.80 Sycamore provides a window to the challenges of systemic reform in an impoverished context. In many ways, the staff had embraced the idea of whole school reform. They had a coherence of message and a collaborative spirit that allowed them to work together toward the goal of improving achievement. But given the system of rewards and punishments in current education policy, their school still struggled with pulling together the resources needed to move forward. In addition, Sycamore’s VA scores, more consistently low than not, obscured the school’s potential. Behind the VA scores we saw shifting attainment, with momentary success in supporting achievement until the focus of effort was moved to something else. Though both Hickory and Sycamore had low VA scores, an ecological perspective gave us a window on important differences in how staff at these two schools negotiated unique challenges. Discussion It should be no surprise that policy makers have chosen to gloss over the long, difficult endeavor to identify the elements of effective teaching, and move straight to an emphasis on effective teachers leaving what makes some teachers more effective than others as an unsolved mystery (Gamoran, 2011, p. 202). The current focus on quality in education, nurtured in a context of accountability, has sharpened attention to the measurement of teaching practice and its outcomes (Cuban, 2007). In this context we undertook a study to examine different metrics of quality with attention to local enactment. We were convinced that situating quality in local sites of teaching and learning would help us illuminate how context mediates the process and outcomes of instruction. We found two elements important for understanding local practice. The first, resources, is often seen as the Holy Grail of education (Cohen, Raudenbush, & Ball, 2003). Resources may be E ducation Policy A nalysis A rchives V ol. 21 N o. 8 SPECIAL ISSUE 24 intellectual, material, environmental, or social assets that can be used in education. The second element, coherence, describes the consistency of goals, purposes, programs, and staffing (Madda, Halverson, & Gomez, 2007). Our analysis indicated that there is not a linear relationship among quality and neither resources nor coherence. Mulberry, a school with the highest quality ratings on either of our two measures, had abundant resources across cultural, social, and economic capital and strong coherence. That, however, was the only simple linear relationship that we could find. When we considered resources input by the district, the lowest rated schools also had more resources poured into them than those whose quality was low overall, but often drifted towards the middle. While this might seem appropriate from an equity perspective on resource allocation, we found Cohen and colleagues’ assertion quite compelling: The instructional effects of conventional resources depend on their usability, their use by the agents of instruction, and the environments in which they work. When added conventional resources appear to directly affect learning, it is because they are usable, because teachers and students know how to use them, and because environments enabled or did not impede their use...If these ideas are correct, then when added resources lie outside the range of teachers’ and students’ knowledge, norms, and incentives, they will have no discernable effect (Cohen et al., 2003, p. 132 ). This was borne out in our analysis. In schools like Mulberry, resources were available and activated along every dimension imaginable. In a district with crumbling buildings and perennial struggles to find funding for technology, programs and staff, Mulberry was a well-equipped education palace. Mulberry’s identity as a school for the talented and gifted framed every aspect of work and provided a unifying theme for practice. Teachers, students, and families enriched an educational environment that included supplemental trips and activities, and highly engaging pedagogy. In contrast, schools like Hickory were housed in lifeless buildings with a sense of despair. Serving some of the most vulnerable students in the city, these schools were more densely staffed and resourced, but the assets were only marginally related to instruction. With low profile leadership and without a mission, the staff found it difficult to use the resources they had. The dusty Smart board hidden behind piles of books is a metaphor for this school’s struggles – it was an investment in a school in need but not enough of an investment to allow it to be used. The main work of Hickory staff was to avoid losing the education game, keeping students in the building and out of trouble. Schools like Nutmeg or Sycamore had mixed resources and coherence. Nutmeg was a well- run school with a history of data use and neighborhood resources. The principal was focused on external audiences and less on his role as instructional leader. As a result, staff invested much energy and attention in communication of standards and activities, but there was less coherence in how or why they were doing what they were doing. Sycamore had the opposite issue. The school had a clear mission, and all teachers and staff spoke from the same script in terms of their goals for their students and the ways they worked. Unfortunately, they were not failing enough to garner the additional resources given to schools like Hickory. So they struggled to stay up to date in their practice, relying on time proven tools like chart paper and bulletin boards. Their classes were large, and there were fewer support staff. We had the impression, from our time at Sycamore, that just a few more resources and a little more support would have made a huge difference to their overall attainment of their students and VA scores of their teachers. The ecological dimensions of resources and coherence worked together to set a context for quality in each of the school communities we studied. Rather than capturing a characteristic or capacity of individual teachers, we argue that the metrics of quality are part of a broader community E cologies of education quality 25 narrative in which not all participants have equal opportunities. These histories helped us understand schools like Hickory and Sycamore that do not fit a simple relationship between VA and classroom quality at the school or classroom level. This study’s focus on context echoes research by scholars using social network analyses to understand the efficacy of school reforms. Rather than assuming that teachers are individual agents whose capacity is a matter of skill, will, and knowledge, social network analyses argue that resources are activated in social contexts. This makes an individual approach to quality, reflected in VA, problematic: Rather than presuming that estimating a teacher’s value-added in student achievement is the best indicator to use in selecting teachers into the profession, a social capital approach views teaching less as a solo practice and more as a practice that evolves within a broader social and organizational contexts (Penuel, Frank, Sun, & Kim, 2012, p. 198). Individual teachers are actors in the creation of educational quality but it is also true that the resources in the context such as funding, leadership, and collaboration are actors as well (Darling-Hammond et al, 2012). Quality in context is the punch line of this paper. Comparing two metrics of quality—VA and CLASS scores—within the context of the capacity of schools to enhance that quality, provides a window to the importance of ecologies in education. It provides a deeper sense of how teacher or school quality, as measured by VA, is nested within classroom or school quality, as measured by CLASS. It also highlights how student outcomes are made possible through classroom interaction and processes that are supported or undermined by contextual factors. Both types of quality are supported within local contexts and through engagements and interactions that make only certain things possible. Without this ecological view of quality, VA data alone tells us little about how to improve student achievement. Though the calculation of VA purports to control for demographic characteristics, there is more to context than student characteristics. A contextual look at quality provides us with a more nuanced understanding of the ways that institutions, in addition to individuals, create educational opportunities that translate into achievement. References Aaronson, D., Barrow, L., & Sander, W. (2007). Teachers and Student Achievement in the Chicago Public High Schools. Journal of Labor Economics, 25(1), 95–135. Adamson, F., & Darling-Hammond, L. (2012). Funding Disparities and the Inequitable Distribution of Teachers: Evaluating Sources and Solutions. Education Policy Analysis Archives, 20, 37. Retrieved from http://epaa.asu.edu/ojs/article/view/1053 Amrein-Beardsley, A. (2008). Methodological concerns about the education value-added assessment system. Educational Researcher, 37(2), 65-75. Amrein-Beardsley, A. (2009). Value-Added tests: Buyer be aware. Educational Leadership 67(3), 38-42. Amrein-Beardsley, A., & Collins, C. (2012). The SAS Education Value-Added Assessment System (SAS® EVAAS®) in the Houston Independent School District (HISD): Intended and Unintended Consequences. Education Policy Analysis Archives, 20, 1–31. Retrieved from http://epaa.asu.edu/ojs/article/view/1096 Au, W. (2007). High-stakes testing and curricular control: A qualitative metasynthesis. Educational Researcher, 36(5), 258-267. E ducation Policy A nalysis A rchives V ol. 21 N o. 8 SPECIAL ISSUE 26 Baker, E., Barton, P. E., Darling-Hammond, L., Haertel, E., Ladd, H. F., Linn, R. L., Ravitch, D., et al. (2010). Problems with the use of student test scores to evaluate teachers. Washington, D.C.: Economic Policy Institute (278). Ballou, D., Sanders, W., & Wright, P. (2004). Controlling for Student Background in Value-Added Assessment of Teachers. Journal of Educational and Behavioral Statistics, 29(1), 37–65. Retrieved from http://www.jstor.org/stable/3701306 Braun, H. I. (2005). Using Student Progress To Evaluate Teachers  : A Primer on Value-Added Models (pp. 1– 20). Princeton, New Jersey. Retrieved from www.ets.org/research/pic Briggs, D., & Domingue, B. (2011). Due diligence and the evaluation of teachers: A review of the value-added analysis underlying the effectiveness rankings of Los Angeles Unified School District teachers by the Los Angeles Times. Boulder, CO: National Education Policy Center. Retrieved June 30, 2012, from http://nepc.colorado.edu/publication/due-diligence Bronfenbrenner, U. (1977). Toward an experimental ecology of human development. American Psychologist, 32, 513-531. Bronfenbrenner, U. (1995). Developmental ecology through space and time: A future perspective. In P. Moen, G. H. Elder, Jr., and K. Luscher (Eds.), Examining lives in context: Perspectives on the ecology of human development. (pp. 619-647). Washington, D.C. American Psychological Association. Burchinal, M., McCartney, K., Steinberg, L., Crosnoe, R., Friedman, S. L., McLoyd, V., & Pianta, R. (2011). Examining the Black-White achievement gap among low-income children using the NICHD study of early child care and youth development. Child development, 82(5), 1404–20. doi:10.1111/j.1467-8624.2011.01620.x Campbell, F. A., & Ramey, C. T. (1994). Effects of early intervention on intellectual and academic achievement: A follow-up study of children from low-income families. Child Development, 65(2), 684-698. Can a few years’ data reveal bad teachers? (2012, January 12). The New York Times. Retrieved from http://www.nytimes.com/roomfordebate/2012/01/16/can-a-few-years-data-reveal-bad- teachers Chetty, R., Friedman, J. N., & Rockoff, J. E. (2011). The long-term impacts of teachers: Teacher Value-Added and student outcomes in adulthood. Cambridge, Massachusetts. Retrieved from http://www.nber.org/papers/w17699 Cohen, D. K., Raudenbush, S. W., & Ball, D. L. (2003). Resources, instruction, and research. Educational Evaluation and Policy Analysis, 25(2), pp. 119-142. Corcoran, S. P. (2010). Can teachers be evaluated by their students’ test scores  ? Should they be? The use of Value- Added measures of teacher effectiveness in policy and practice (pp. 1–46). Annenberg Institute for School Reform. Providence, Rhode Island. Cuban, L. (2007). Hugging the middle: Teaching in an era of testing and accountability. Education Policy Analysis Archives. Darling-Hammond, L. (1997). Doing what matters most: Investing in quality teaching. New York: National Committee on Teaching and America’s Future. Darling-Hammond, L., Amrein-Beardsley, A., Haertel, E., & Rothstein, J. (2012). Evaluating teacher evaluation. Phi Delta Kappan, 93(6), 8–15. PDK International. Retrieved from http://www.kappanmagazine.org/content/93/6/8.short Elliott, J. (2011). Are performance indicators educational quality indicators? In J. Elliot, Ed. Reconstructing teacher education (51-64). London: Routledge. Gamoran, A. (2012). Improving teacher quality. Incentives are not enough. In S. Kelly Assessing teacher quality. Understanding teacher effects on instruction and achievement. p. 201-214. New York: Teachers College Press. E cologies of education quality 27 Gates, B. (2012, February 23). Shame is not the solution. The New York Times, p. A27. Grading the teachers: Value-added analysis (n.d.). Los Angeles Times. Retrieved from http://www.latimes.com/news/local/teachers-investigation/ Graue, M. E., Hatch, K., Rao, K., & Oen, D. (2007). The wisdom of class size reduction. American Educational Research Journal, 44(2), 670–700. doi:10.3102/0002831207306755 Graue, E., Rauscher, E., & Sherfinski, M. (2009). The Synergy of Class Size Reduction and Classroom Quality. The Elementary School Journal, 110(2), 178–201. doi:10.1086/605772 Graue, M. E., & Walsh, D. J. (1998). Studying children in context : Theories, methods, and ethics. Thousand Oaks, Calif.: Sage Publications. Hamre, B. K., & Pianta, R. C. (2005). Can instructional and emotional support in the first-grade classroom make a difference for children at risk of school failure? Child development, 76(5), 949–67. doi:10.1111/j.1467-8624.2005.00889.x Hamre, B. K., Pianta, R. C., Mashburn, A. J., & Downer, J. T. (2007). Building a science of classrooms: Application of the CLASS framework in over 4,000 U.S. early childhood and elementary classrooms. New York: Foundation for Child Development. doi:http://www.fcd- us.org/resources/building-science-classrooms-application-class-framework-over-4000-us- early-childhood-and- e?destination=resources%2Fsearch%3Ftopic%3D0%26authors%3DHamre%252C%2520Pi anta%252C%2520Mashburn%26keywords%3D Hanushek, E. A. (2003). The failure of input-based schooling policies. The Economic Journal, 113(485, Features), F64–F98. Retrieved from http://www.jstor.org.ezproxy.library.wisc.edu/stable/3590139 Hanushek, E. A. & Rivkin, S. G. (2010). Generalizations about using value-added measures of teacher quality. American Economic Review, 100(2): 267–71. doi:10.1257/aer.100.2.267 Harris, D. N. (2009). Teacher value-added: Don’t end the search before it starts. Journal of Policy Analysis and Management, 28(4), 692-712. doi:10.1002/pam Harris, D. N. (2011). Value-added measures in education: What every educator needs to know. Cambridge, Massachusetts: Harvard Education Press. Heck, R. H. (2007). Examining the relationship between teacher quality as an organizational property of schools and students’ achievement and growth rates. Educational Administration Quarterly, 43(4), 399–432. doi:10.1177/0013161X07306452 Herzog, A & Holiday, B. (1939). God bless the child. On Billie Holiday’s greatest hits [CD] London: Geffen Records. (1967). Hill, H. C., Kapitula, L., & Umland, K. (2010). A Validity Argument Approach to Evaluating Teacher Value-Added Scores. American Educational Research Journal, 48(3), 794–831. doi:10.3102/0002831210387916 Howes, C., Burchinal, M., Pianta, R., Bryant, D., Early, D., Clifford, R., and Barbarin, O. (2008). Ready to learn? Children's pre-academic achievement in pre-kindergarten programs. Early Childhood Research Quarterly, 23(1), 27-50. doi:DOI: 10.1016/j.ecresq.2007.05.002 Howes, C. & Pianta, R.C. (2011). Foundations for teaching excellence: Connecting early childhood quality rating, professional development and competency systems in states. Baltimore, MD: Paul H. Brookes Publishing. Kennedy, M. (2008). Sorting out teacher quality. Phi Delta Kappan, 59-63. Konstantopoulos, S., & Chung, V. (2010). The Persistence of Teacher Effects in Elementary Grades. American Educational Research Journal, 48(2), 361–386. doi:10.3102/0002831210382888 Kupermintz, H. (2003). Teacher Effects and Teacher Effectiveness : A Validity Investigation of the Tennessee Value Added Assessment System. Educational Evaluation and Policy Analysis, 25(3), 287–298. E ducation Policy A nalysis A rchives V ol. 21 N o. 8 SPECIAL ISSUE 28 La Paro, K. M., Pianta, R. C., & Stuhlman, M. (2004). The Classroom Assessment Scoring System: Findings from the prekindergarten year. The Elementary School Journal, 104(5), 409. doi: 10.1086/499760. Leak, J. A., & Farkas, G. (2011). Effects of Teacher Credentials, Coursework, and Certification on Student Achievement in. Educational Evaluation and Policy Analysis, 26, 237-257. Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry. Beverly Hills, Calif.: Sage Publications. Madda, C. L., Halverson, R. R., & Gomez, L. M. (2007). Exploring coherence as an organizational resource for carrying out reform initiatives. Teachers College Record, 109(8), 1957–1979. Massell, D. (2001). The theory and practice of using data to build capacity: State and local strategies and their effects. In S. H. Fuhrman (Ed.), From the capitol to the classroom: Standards-based reform in the states. One hundredth yearbook of the National Society for the Study of Education (pp. 148–169). Chicago: National Society for the Study of Education (NSSE) and University of Chicago Press. Maxwell, J. A. (1996). Qualitative research design: An interactive approach. Thousand Oaks, CA: Sage Publications. Meyer, R. (1997). Value-added indicators of school performance: A primer. Economics of Education Review, 16(3), 283–301. doi:10.1016/S0272-7757(96)00081-7 Newton, X., Darling-Hammond, L., Haertel, E., and Thomas, E. (2010) Value-Added modeling of teacher effectiveness: An exploration of stability across models and contexts. Educational Policy Analysis Archives, 18 (23). Retrieved from http://epaa.asu.edu/ojs/article/view/810 National Institute for Child Health and Human Development (2004). Does class size in first grade relate to children's academic and social performance or observed classroom processes? Developmental Psychology, 40(5), 651. O’Connor, E,, & McCartney, K. (2007). Examining teacher-child relationships and achievement as part of an ecological model of development. American Educational Research Journal 44(2), 340- 369. Papay, J. P. (2010). Different tests, different answers: The stability of teacher ValueAdded estimates across outcome measures. American Educational Research Journal 48 (1) p. 163-193. Doi: 10.3102/0002831210362589 Penuel, W. R., Frank, K.A., Sun, M., & Kim, C.M. (2012) Teachers’ social capital and the implementations of schoolwide reforms. In S. Kelly Assessing teacher quality. Understanding teacher effects on instruction and achievement. p. 183-200. New York: Teachers College Press. Pianta, R. C., Belsky, J., Houts, R., & Morrison, F. (2007). Opportunities to learn in America’s elementary classrooms. Science, 315(10), 1795-1796. Pianta, R. C., LaParo, K. M., & Hamre, B. K. (2008). Classroom Assessment Scoring System manual K-3. Baltimore, Maryland: Paul H. Brookes Publishing Company. Race to the Top Fund (n.d.). Retrieved from the U.S. Department of Education website, http://www2.ed.gov/programs/racetothetop/index.html Reynolds, A. J., Temple, J. A., Robertson, D. L., & Mann, E. A. (2002). Age 21 cost-benefit analysis of the Title I Chicago Child-Parent Centers, 24(4), 267-303. Rice, J. K. (2003). Teacher quality: Understanding the effectiveness of teacher attributes. Washington, DC: Economic Policy Institute. Rivers, J. C., & Sanders, W. L. (2002). Teacher quality and equity in educational opportunity: Findings and policy implications. In W. M. Evers, & L. T. Izumi (Eds.), Teacher quality (pp. 13-24). Stanford, CA: Hoover Institution Press. Rivkin, B. Y. , Hanushek, E. A., & Kain, J. E. (2005). Teachers, Schools, and Academic Achievement. Econometrica, 73(2), 417–458. E cologies of education quality 29 Rockoff, J. E. (2004). The Impact of Individual Teachers on Student Achievement : Evidence from Panel Data. The American Economic Review, 94(2), 247–252. Rothstein, J. (2009). Student sorting and bias in Value-Added estimation: Selection on observables and unobservables. Education Finance and Policy, 4(4), 537-571. doi:10.1162/edfp.2009.4.4.537 Saldaña, J. (2009). The coding manual for qualitative researchers. Thousand Oaks, CA: Sage. Sanders, W. L., & Horn, S. P. (1998). Research Findings from the Tennessee Value-Added Assessment System ( TVAAS ) Database : Implications for Educational Evaluation and Research. Journal of Personnel Evaluation in Education, 12(3), 247–256. doi:10.1023/A:1008067210518 Sass, T. (2008). The stability of value-added measures of teacher quality and implications for teacher compensation policy. Washington, DC: CALDER. Schwartz, R. M., Schmitt, M. C., & Lose, M. K. (2012). Effects of Teacher-Student Ratio in Response to Intervention Approaches. The Elementary School Journal, 112(4), 547–567. doi:10.1086/664490 Spillane, J. P. (2012). Data in Practice: Conceptualizing the Data-Based Decision-Making Phenomena. American Journal of Education, 118(2), 113–141. Stuhlman, M.W., & Pianta, R.C. (2009). Profiles of educational quality in first grade. The Elementary School Journal, 109(4), 323-342. Tucker, M. (2012). Equity and quality: Is it possible to get both? Education Week, doi http://blogs.edweek.org/edweek/top_performers/2012/05/equity_and_quality_is_it_possi ble_to_get_both.html?qs=education+quality Tudge, J., Mokrova, I., Hatfield, B., & Karnik, R. B. (2009). Uses and Misuses of Bronfenbrenner’s Bioecological Theory of Human Development. Journal of Family Theory and Review, 1(December), 198–210. VARC: Methodology. (n.d.). Retrieved June 21, 2012 from the Value Added Research Center, WCER website, http://varc.wceruw.org/methodology.php Wiggins, G. (2012, May 6). Value added—Why it makes me angry (Or: a good idea gone bad, again, in education) [Web log message]. Retrieved from http://grantwiggins.wordpress.com/2012/05/06/value-added-why-its-use-makes-me-angry- or-good-idea-gone-bad/ Wilson, H. K., Pianta, R. C., & Stuhlman, M. (2007). Experiences in First Grade : The Role of Classroom Climate and Functional Risk in the Development of Social Competencies, The Elementary School Journal, 108(2), 81–96. Winters, M. A. (2012). Teachers matter: Rethinking how public schools identify, reward, and retain great educators. Lanham, Maryland: Rowman and Littlefield Publishers. Yeh, S. S., & Ritter, J. (2009). The cost-effectiveness of replacing the bottom quartile of novice teachers through value-added teacher assessment. Journal of Education Finance, 34(4), 426–51. Retrieved from http://search.ebscohost.com/login.aspx?direct=true&db=eoh&AN=1048443&site=ehost- live Zaslow, M., Tout, K., & Halle, T. (2011). Differing purposes for measuring quality in early childhood settings. Aligning purpose with procedures. In M. Zaslow, I. Martinez-Beck, K. Tout, & T. Halle (Eds.), Quality measurement in early childhood settings (p. 389-410). Baltimore: Paul Brookes. E ducation Policy A nalysis A rchives V ol. 21 N o. 8 SPECIAL ISSUE 30 Appendix A Learning Team Focus Group Interview Date: ________ Participants: _______________________________________ Interviewer: _____ 1 Tell me a little bit about your school and the community it serves. 2 Student achievement is a big focus in education currently. How would you describe this school’s student achievement? a What is your school doing to address student achievement? 3 The district has, like many other districts, put resources into enhancing the ability to use data at all levels of the system. What is happening at the district level to support data use? a How is your school being supported in this effort? b What data have been provided and how do you use them? 4 At this school, what assessments do you use to generate data? a How are these assessments related to your curriculum? 5 There are many ways to measure how a school is doing. One increasingly popular way is Value Added. As a Learning Team, how would you describe Value Added? a How does the district use VA? b How do you use VA in your work supporting school staff? 6 Policymakers and the public have been calling for classroom measures of quality. Value Added is one of the things bandied about as a tool. What other elements do you consider when you think about classroom quality? a How do you use these elements when you work as instructional leaders, supporting teacher development? 7 Is there anything else I should ask that would be important for me to know? E cologies of education quality 31 About the Authors M. Elizabeth Graue University of Wisconsin, Madison graue@education.wisc.edu M. Elizabeth Graue is Sorenson Professor, Department of Curriculum & Instruction, University of Wisconsin Madison. Her research interests include early childhood policy & practice, home-school relations, assessment, and qualitative research methods. Kate K. Delaney University of Wisconsin, Madison Kate K. Delaney is a PhD candidate at the University of Wisconsin, Madison. Her research interests include preK policy, curriculum, urban education, and culturally relevant pedagogy. Anne Sontag Karch University of Wisconsin, Madison Anne Sontag Karch is a PhD candidate at the University of Wisconsin, Madison. Her areas of interest are school reform and teacher professional development. About the Guest Editor and Assistant Guest Editors Guest Editor Dr. Audrey Amrein-Beardsley Arizona State University audrey.beardsley@asu.edu Dr. Amrein-Beardsley is currently an Associate Professor in the Mary Lou Fulton Teachers College at Arizona State University. Audrey's research interests include educational policy, research methods, and more specifically, high-stakes tests and value-added measurements and systems. In addition, she researches aspects of teacher quality and teacher education. She is also the creator and host of a show titled Inside the Academy during which she interviews some of the top educational researchers in the academy. For more information please see: http://insidetheacademy.asu.edu. Assistant Guest Editor Dr. Clarin Collins Virginia G. Piper Charitable Trust clarin.collins@asu.edu Clarin Collins recently completed her Ph.D. in Educational Policy and Evaluation from Arizona State University, with an emphasis in research methods. Via her dissertation, she examined teachers' understandings of and experiences with the SAS Education Value-Added Assessment System (EVAAS) in the Houston Independent School District where it is used to evaluate teacher effectiveness. Clarin is currently a Research and Evaluation Officer at the Virginia G. Piper Charitable Trust in Phoenix. Assistant Guest Editor Dr. Sarah Polasky Arizona State University E ducation Policy A nalysis A rchives V ol. 21 N o. 8 SPECIAL ISSUE 32 sarah.polasky@asu.edu Dr. Sarah Polasky is the Value-Added Specialist for the Arizona Ready-for-Rigor Project, a Teacher Incentive Fund Grant, within the Mary Lou Fulton Teachers College. Her current research interests include the development and implementation of value-added measurements and systems using high- stakes test data, assessment in early childhood education, the use of alternative achievement (e.g., district benchmarks, formative assessments) and non-achievement (i.e., developmental) data for value-added analysis, as well as the impact of socioemotional and neurological development of young children on their short- and long-term academic achievement. Assistant Guest Editor Edward F. Sloat Mary Lou Fulton Teachers College, Arizona State University; Dysart Unified School District, Surprise, Arizona esloat@asu.edu Mr. Sloat is currently employed as the Director of Research and Accountability at Dysart Unified School District located in Surprise, Arizona and a doctoral student in the Leadership and Innovation Program within the Mary Lou Fulton Teachers College, Arizona State University. Mr. Sloat has served as Deputy Associate Superintendent for Research and Evaluation within the Arizona Department of Education, the Director of Research, Planning, and Assessment for the Peoria (Arizona) Unified School District, and as Director of Research and Assessment at the Glendale (Arizona) Elementary School District. He regularly contributes to state technical and policy working/advisory groups concerning assessment design and accountability systems and is past President of the Arizona Education Research Organization. Mr. Sloat holds a Master’s Degree in Applied Economics from the University of Arizona, concentrating in econometric methods and management information systems. His academic interests focus on value-added modeling, education accountability and evaluation systems, data- driven instructional planning, applications of measurement theory, and research methods. SPECIAL ISSUE Value-Added: What America’s Policymakers Need to Know and Understand education policy analysis archives Volume 21 Number 8 January 30th, 2013 ISSN 1068-2341 Readers are free to copy, display, and distribute this article, as long as the work is attributed to the author(s) and Education Policy Analysis Archives, it is distributed for non- commercial purposes only, and no alteration or transformation is made in the work. More details of this Creative Commons license are available at http://creativecommons.org/licenses/by-nc-sa/3.0/. All other uses must be approved by the author(s) or EPAA. EPAA is published by the Mary Lou Fulton Institute and Graduate School of Education at Arizona State University Articles are indexed in CIRC (Clasificación Integrada de Revistas Científicas, Spain), DIALNET (Spain), Directory of Open Access Journals, EBSCO Education Research Complete, ERIC, Education Full Text (H.W. Wilson), QUALIS A2 (Brazil), SCImago Journal Rank; SCOPUS, SOCOLAR (China). E cologies of education quality 33 Please contribute commentaries at http://epaa.info/wordpress/ and send errata notes to Gustavo E. Fischman fischman@asu.edu Join EPAA’s Facebook community at https://www.facebook.com/EPAAAAPE and Twitter feed @epaa_aape. E ducation Policy A nalysis A rchives V ol. 21 N o. 8 SPECIAL ISSUE 34 education policy analysis archives editorial board Editor Gustavo E. Fischman (Arizona State University) Associate Editors: David R. Garcia (Arizona State University), Stephen Lawton (Arizona State University) Rick Mintrop, (University of California, Berkeley) Jeanne M. Powers (Arizona State University) Jessica Allen University of Colorado, Boulder Christopher Lubienski University of Illinois, Urbana- Champaign Gary Anderson New York University Sarah Lubienski University of Illinois, Urbana- Champaign Michael W. Apple University of Wisconsin, Madison Samuel R. Lucas University of California, Berkeley Angela Arzubiaga Arizona State University Maria Martinez-Coslo University of Texas, Arlington David C. Berliner Arizona State University William Mathis University of Colorado, Boulder Robert Bickel Marshall University Tristan McCowan Institute of Education, London Henry Braun Boston College Heinrich Mintrop University of California, Berkeley Eric Camburn University of Wisconsin, Madison Michele S. Moses University of Colorado, Boulder Wendy C. Chi* University of Colorado, Boulder Julianne Moss University of Melbourne Casey Cobb University of Connecticut Sharon Nichols University of Texas, San Antonio Arnold Danzig Arizona State University Noga O'Connor University of Iowa Antonia Darder University of Illinois, Urbana- Champaign João Paraskveva University of Massachusetts, Dartmouth Linda Darling-Hammond Stanford University Laurence Parker University of Illinois, Urbana- Champaign Chad d'Entremont Strategies for Children Susan L. Robertson Bristol University John Diamond Harvard University John Rogers University of California, Los Angeles Tara Donahue Learning Point Associates A. G. Rud Purdue University Sherman Dorn University of South Florida Felicia C. Sanders The Pennsylvania State University Christopher Joseph Frey Bowling Green State University Janelle Scott University of California, Berkeley Melissa Lynn Freeman* Adams State College Kimberly Scott Arizona State University Amy Garrett Dikkers University of Minnesota Dorothy Shipps Baruch College/CUNY Gene V Glass Arizona State University Maria Teresa Tatto Michigan State University Ronald Glass University of California, Santa Cruz Larisa Warhol University of Connecticut Harvey Goldstein Bristol University Cally Waite Social Science Research Council Jacob P. K. Gross Indiana University John Weathers University of Colorado, Colorado Springs Eric M. Haas WestEd Kevin Welner University of Colorado, Boulder Kimberly Joy Howard* University of Southern California Ed Wiley University of Colorado, Boulder Aimee Howley Ohio University Terrence G. Wiley Arizona State University Craig Howley Ohio University John Willinsky Stanford University Steve Klees University of Maryland Kyo Yamashiro University of California, Los Angeles Jaekyung Lee SUNY Buffalo * Members of the New Scholars Board E cologies of education quality 35 archivos analíticos de políticas educativas consejo editorial Editor: Gustavo E. Fischman (Arizona State University) Editores. Asociados Alejandro Canales (UNAM) y Jesús Romero Morante (Universidad de Cantabria) Armando Alcántara Santuario Instituto de Investigaciones sobre la Universidad y la Educación, UNAM México Fanni Muñoz Pontificia Universidad Católica de Perú Claudio Almonacid Universidad Metropolitana de Ciencias de la Educación, Chile Imanol Ordorika Instituto de Investigaciones Economicas – UNAM, México Pilar Arnaiz Sánchez Universidad de Murcia, España Maria Cristina Parra Sandoval Universidad de Zulia, Venezuela Xavier Besalú Costa Universitat de Girona, España Miguel A. Pereyra Universidad de Granada, España Jose Joaquin Brunner Universidad Diego Portales, Chile Monica Pini Universidad Nacional de San Martín, Argentina Damián Canales Sánchez Instituto Nacional para la Evaluación de la Educación, México Paula Razquin UNESCO, Francia María Caridad García Universidad Católica del Norte, Chile Ignacio Rivas Flores Universidad de Málaga, España Raimundo Cuesta Fernández IES Fray Luis de León, España Daniel Schugurensky Universidad de Toronto-Ontario Institute of Studies in Education, Canadá Marco Antonio Delgado Fuentes Universidad Iberoamericana, México Orlando Pulido Chaves Universidad Pedagógica Nacional, Colombia Inés Dussel FLACSO, Argentina José Gregorio Rodríguez Universidad Nacional de Colombia Rafael Feito Alonso Universidad Complutense de Madrid, España Miriam Rodríguez Vargas Universidad Autónoma de Tamaulipas, México Pedro Flores Crespo Universidad Iberoamericana, México Mario Rueda Beltrán Instituto de Investigaciones sobre la Universidad y la Educación, UNAM México Verónica García Martínez Universidad Juárez Autónoma de Tabasco, México José Luis San Fabián Maroto Universidad de Oviedo, España Francisco F. García Pérez Universidad de Sevilla, España Yengny Marisol Silva Laya Universidad Iberoamericana, México Edna Luna Serrano Universidad Autónoma de Baja California, México Aida Terrón Bañuelos Universidad de Oviedo, España Alma Maldonado Departamento de Investigaciones Educativas, Centro de Investigación y de Estudios Avanzados, México Jurjo Torres Santomé Universidad de la Coruña, España Alejandro Márquez Jiménez Instituto de Investigaciones sobre la Universidad y la Educación, UNAM México Antoni Verger Planells University of Amsterdam, Holanda José Felipe Martínez Fernández University of California Los Angeles, USA Mario Yapu Universidad Para la Investigación Estratégica, Bolivia E ducation Policy A nalysis A rchives V ol. 21 N o. 8 SPECIAL ISSUE 36 arquivos analíticos de políticas educativas conselho editorial Editor: Gustavo E. Fischman (Arizona State University) Editores Associados: Rosa Maria Bueno Fisher e Luis A. Gandin (Universidade Federal do Rio Grande do Sul) Dalila Andrade de Oliveira Universidade Federal de Minas Gerais, Brasil Jefferson Mainardes Universidade Estadual de Ponta Grossa, Brasil Paulo Carrano Universidade Federal Fluminense, Brasil Luciano Mendes de Faria Filho Universidade Federal de Minas Gerais, Brasil Alicia Maria Catalano de Bonamino Pontificia Universidade Católica-Rio, Brasil Lia Raquel Moreira Oliveira Universidade do Minho, Portugal Fabiana de Amorim Marcello Universidade Luterana do Brasil, Canoas, Brasil Belmira Oliveira Bueno Universidade de São Paulo, Brasil Alexandre Fernandez Vaz Universidade Federal de Santa Catarina, Brasil António Teodoro Universidade Lusófona, Portugal Gaudêncio Frigotto Universidade do Estado do Rio de Janeiro, Brasil Pia L. Wong California State University Sacramento, U.S.A Alfredo M Gomes Universidade Federal de Pernambuco, Brasil Sandra Regina Sales Universidade Federal Rural do Rio de Janeiro, Brasil Petronilha Beatriz Gonçalves e Silva Universidade Federal de São Carlos, Brasil Elba Siqueira Sá Barreto Fundação Carlos Chagas, Brasil Nadja Herman Pontificia Universidade Católica –Rio Grande do Sul, Brasil Manuela Terrasêca Universidade do Porto, Portugal José Machado Pais Instituto de Ciências Sociais da Universidade de Lisboa, Portugal Robert Verhine Universidade Federal da Bahia, Brasil Wenceslao Machado de Oliveira Jr. Universidade Estadual de Campinas, Brasil Antônio A. S. Zuin Universidade Federal de São Carlos, Brasil