Microsoft Word - FramingEPAAIssue_fnl.docx Journal website: http://epaa.asu.edu/ojs/ Manuscript received: 8/1/2013 Facebook: /EPAAA Revisions received: 10/1/2013 Twitter: @epaa_aape Accepted: 11/1/2013 SPECIAL ISSUE Value-Added: What America’s Policymakers Need to Know and Understand education policy analysis archives A peer-reviewed, independent, open access, multilingual journal Arizona State University Volume 21 Number 4 January 28th, 2013 ISSN 1068-2341 Value-Added Model (VAM) Research for Educational Policy: Framing the Issue Audrey Amrein-Beardsley Clarin Collins Sarah A. Polasky Arizona State University Edward F. Sloat Dysart Unified School District, Surprise, Arizona United States of America Citation: Amrein-Beardsley, A., Collins, C., Polasky, S. A., & Sloat, E. F. (2013). Value-Added Model (VAM) Research for Educational Policy: Framing the Issue. Education Policy Analysis Archives, 21(4). This article is part of EPAA/AAPE’s Special Issue on Value-Added: What America’s Policymakers Need to Know and Understand, Guest Edited by Dr. Audrey Amrein-Beardsley and Assistant Editors Dr. Clarin Collins, Dr. Sarah Polasky, and Ed Sloat. http://dx.doi.org/10.14507/epaa.v21n4.2013 Abstract: In this manuscript, the guest editors of the EPAA Special Issue on “Value-Added Model (VAM) Research for Educational Policy” (1) introduce the background and policy context surrounding the increased use of VAMs for teacher evaluation and accountability purposes across the United States; (2) summarize the five research papers and one research-based commentary that were peer-reviewed and selected for inclusion in this special issue; and (3) discuss the relevance of epaa aape E ducation P olicy A nalysis A rchives V ol. 21 N o. 4 SPECIAL ISSUE 2 the papers both individually and collectively. Their importance is discussed in terms of each paper’s contribution to the general research on this topic and each paper’s potential to inform educational policy. In addition, the papers reflect our shared thinking about VAMs, VAM output, and the inference-based decisions for which VAMs are increasingly being used. Keywords: Value-added; teacher evaluation; teacher accountability. La investigación de los Modelos de Valor Agregado (MVA) para las políticas educativas: Delimitando la discusión. Resumen: En este artículo, los editores invitados de este número especial de EPAA/AAPE sobre la investigación de los Modelos de Valor Agregado (MVA) para las políticas educativas: (1) presentan los antecedentes y el contexto político que rodean la utilización de MVA en la evaluación de los docentes y rendición de cuentas en los Estados Unidos, (2) un resumen de los cinco trabajos de investigación y el comentarios que fueron seleccionados para su inclusión en este número especial, y (3) analizamos la pertinencia de los documentos, tanto individual como colectivamente. Su importancia se discuten en términos de la contribución de cada documento para la investigación general sobre este tema y el potencial de cada artículo para informar la política educativa. Además, los documentos reflejan nuestro pensamiento sobre la producción de MVA y las decisiones basadas en MVA son cada vez más utilizados. Palabras clave: modelos de valor agregado; evaluación docente; responsabilidad docente. Pesquisa dos Modelos de Valor Adicionado (MVA) para as políticas de educação: Delimitação da discussão. Resumo: Neste artigo, os editores convidados desta edição especial da EPAA/AAPE sobre a Pesquisa dos Modelos de Valor Adicionado (MVA) para as políticas de educação: (1) Apresentam o contexto político em torno do uso dos MVA na avaliação de professores e de responsabilização nos Estados Unidos, (2) um resumo dos cinco trabalhos de pesquisa e o comentário que foram selecionados para inclusão nesta edição especial, e (3) analisar a relevância dos trabalhos selecionados tanto individualmente como coletivamente. Sua importância é discutida em termos da contribuição de cada artigo para a pesquisa geral sobre este tema e as potencialidades de cada artigo para informar a política educacional. Além disso, os artigos refletem o nosso pensamento sobre a produção de decisões e MVA, e como são cada vez mais utilizados. Palavras-chave: valor agregado, modelos de avaliação de professores, de responsabilidade do professor. Introduction Historically, throughout the United States, public education agencies have used localized approaches for evaluating teachers and making determinations about teacher effectiveness. With few exceptions (e.g., the state of Tennessee), teacher evaluation efforts have been traditionally governed and developed by school districts under the guises of district control. Accordingly, school districts have not been encouraged or incentivized to conform to any particular teacher accountability frameworks. Now, however, evaluating teachers using value-added models (VAMs)1 has become a matter of federal and state education policy, as well as a matter of federal and state educational 1 While Student Growth Models (SGMs) are being used more often than Value-Added Models (VAMs) at the state-level (Collins & Amrein-Beardsley, under review), VAM is the more popularly used term. As such, we refer to VAMs throughout this manuscript in their most general form, acknowledging the fact that there are distinct differences among and between specific VAMs and SGMs, as well as their model specifications, the methodologies and statistics used, and the assumptions upon which they are based. Value-added model (VAM) research for educational policy: Framing the issue 3 urgency and resolve (Corcoran, 2010; Stumbo & McWalters, 2011; U. S. Department of Education, 2009a). Encouraged by over $350 million in federal funds through President Obama’s Race to the Top (RttT) competition, states are exploring methods to capture the value a teacher adds to student learning from one year to the next (i.e., a teacher’s value-added). To date, 18 states, the District of Columbia (D.C.), and 16 school districts across the country have won RttT funding to support these efforts (U. S. Department of Education [USDOE], 2012a, 2012b). As a result, education agencies are increasingly developing teacher accountability systems based in large part on measures of academic growth that can be attributed to teachers’ effectiveness (USDOE, 2009b). Additionally, 44 states and D.C. have applied for No Child Left Behind (NCLB) waivers (Philips, 2012), excusing them from NCLB’s prior goal that 100% of the students in their public schools would be academically proficient by the year 2014. In exchange for these pardons, these states have also agreed to adopt stronger teacher accountability mechanisms, again based in large part on the growth demonstrated by students as measured via VAMs. By definition, VAMs are designed to isolate and measure teachers’ contributions to student learning and achievement on large-scaled standardized tests as groups of students move from one grade level to the next. Statisticians measure value-added by mathematically calculating the “value” a teacher “adds to” or “detracts from” student achievement scores over time, and as compared to teachers with “similar” students. Purportedly, VAMs allow for richer analyses of achievement data by tracking student learning trajectories from the time they enter a classroom to the time they leave. In addition, VAMs have arguably improved upon the educational measurement systems previously used for test-based accountability (see, for example, Capitol Hill Briefing, 2011; Harris, 2011). As such, it makes sense that VAM output be used as an integral component of contemporary teacher accountability policies. To this end, states are increasingly incorporating VAM components within their teacher evaluation frameworks. But just because it makes sense to do this does not mean it works. That said, it is important to examine whether integrating VAM output as part of teacher accountability policies works in the ways theorized (e.g., by those securing state and federal contracts to conduct this work, by the federal government via initiatives such as RttT and NCLB waivers). Furthermore, it is critical to examine whether VAMs work well enough to make highly consequential decisions about teachers (i.e., publishing teacher names and their VAM scores, using VAM output as a significant factor in decisions such as teacher tenure, merit pay, or continuation of employment). Calls for all types of research on VAMs are urgent and pertinent especially as policy development continues to focus on VAMs with unwarranted levels of certainty and conviction (see, for example, Schafer, Lissitz, Zhu, Zhang, Hou, & Li, 2012). To this end, five manuscripts and one commentary are featured in this special issue of Education Policy Analysis Archives (EPAA). Collectively, the authors present evidence-based arguments about VAMs and their use in local, state, and national policy contexts. In their own unique ways, with unique methods of inquiry, the authors advance our thinking about the proper use and role of VAMs for educational policy. Special Issue Summaries Today, EPAA features Diana Pullin’s (Boston College) research paper on Legal Issues in the Use of Student Test Scores and Value-Added Models (VAM) to Determine Educational Quality. Pullin addresses the changing legal landscape associated with policy-based, high-stakes, teacher evaluation systems using VAMs. She argues that when policy-based evaluation systems explicitly require use of score-based measures, the data quality standards of those measures may be subjected to critical E ducation P olicy A nalysis A rchives V ol. 21 N o. 4 SPECIAL ISSUE 4 examination. In addition, given the conflicting perspectives currently held by researchers regarding VAMs, plaintiffs in legal proceedings have begun to argue that such methods may be insufficient or otherwise inadequate to discriminate levels of professional quality in support of high-stakes employment decisions. As a result, courts may burden education agencies by requiring them to provide substantive evidence of reliability and validity. Technical and inferential limitations of VAM- based evaluation ratings may also raise legal considerations pertaining to substantive due process and equal protection issues, infringement on individual civil rights, and issues related to commercial liability (e.g., test design). In a related article also today, EPAA features The Legal Consequences of Mandating High Stakes Decisions Based on Low Quality Information: Teacher Evaluation in the Race-to-the-Top Era authored by Bruce Baker (Rutgers, The State University of New Jersey), Joseph Oluwole (Montclair State University), and Preston Green (The Pennsylvania State University). Contextualized in a review of district and state policies related to teacher evaluation and general human resource decision-making, Baker, Oluwole, and Green provide many reasons to question the usability of student growth and VAMs for such purposes. They also, like Pullin, focus on identifying potential legal ramifications and other consequential outcomes of these policies and practices if such considerations are not taken seriously. Tomorrow, EPAA will feature an in-depth analysis conducted by Nicole Kersting and Mei- Kuang Chen (University of Arizona) and James Stigler (University of California, Los Angeles) titled Value-Added Teacher Estimates as Part of Teacher Evaluations: Exploring the Effects of Data and Model Specifications on the Stability of Teacher Value-Added Scores. Kersting, Chen, and Stigler investigate the extent to which three VAM specifications impact the overall stability of value-added output (with implications for validity). The three specifications they examine include methodological variations in (1) accounting for students’ academic and other background variables, (2) using single or multiple cohorts of students to measure teacher value-added, and (3) sample size specifications and their related standards of error, all of which ultimately impact VAM output and teacher-level classifications. Like others, they find issues with stability (Koedel & Betts, 2005; Schochet & Chiang, 2010), sample sizes (Lockwood, McCaffrey, & Sass, 2008; McCaffrey, Lockwood, Koretz, & Hamilton, 2003; Nelson, 2011), and other data and model specifications (Harris, Sass, & Semykina, 2012; Papay, 2011). The paper adds important evidence to support the argument that before VAM data can be used for consequential purposes, such issues must be addressed and taken much more seriously than they currently are. On Wednesday, EPAA will feature Ecologies of Education Quality authored by Elizabeth Graue, Katherine Delaney, and Anne Karch (University of Wisconsin, Madison). In this qualitative piece, Graue, Delaney, and Karch confront the challenging task of dissecting the influence of school and community contexts on outcome measures of educational quality in general. Using value-added and observational data together, including both teacher and school level data from four school sites in one metropolitan area, they contrast how these evaluative measures are operationalized within the contexts in which the measurements are constructed, interpreted, and used. The authors provide readers with an in-depth review of the complexities, noting specifically the relative inadequacies of the outcome measures used to represent all aspects of teacher quality. They ultimately argue that even with the most sophisticated technical controls, VAMs cannot adequately capture or control for all of the contextual variables involved. On Thursday, EPAA will feature Sentinels Guarding the Grail: Value-Added Measurement and the Quest for Education Reform authored by Rachael Gabriel (University of Connecticut) and Jessica Nina Lester (Washington State University). Gabriel and Lester present another qualitative analysis, this time analyzing a series of state-level policy discussions surrounding the use of value-added data, specifically derived via the Tennessee Value-Added Assessment System (TVAAS). Using discourse Value-added model (VAM) research for educational policy: Framing the issue 5 analysis methods, the authors demonstrate how value-added is presented and received among both the public and policymakers as scientific, objective, accurate, and efficient. In addition, Gabriel and Lester highlight the numerous concerns and cautions issued by educational researchers and critics of VAMs (e.g., lack of reliability and validity, bias, errors, and inherent problems and overreliance on achievement tests), and they evidence how these issues are often overlooked and dismissed at multiple policy-levels. On Friday, EPAA will end the special issue featuring a commentary authored by Moshe Adler (Columbia University) titled “Findings vs. Interpretation in ‘The Long-Term Impacts of Teachers’ by Chetty et al.” Here, Adler critiques the acclaimed, overly publicized, and hotly contested study of New York City teachers’ value-added and long-term impacts that was conducted by Chetty, Friedman, and Rockoff (2011). Specifically, Adler critically highlights the omissions, fallacies, and misrepresentations made by the authors, all of which resulted in unwarranted national and international attention and misled policymakers and the public alike, especially in terms of the value and potential (yet exaggerated) powers of VAMs. Adler also makes the case that had Chetty et al.’s work been peer reviewed prior to its public release (see also Lowrey, 2012), it likely never would have had the impact it did given the serious methodological and other shortcomings highlighted here and elsewhere (Ballou, 2012; Ravitch, 2012; Winerip, 2012). In sum, the papers featured as part of the EPAA special issue on VAMs will highlight the ways in which education agencies across the United States are increasingly aligning their policy interests and dependencies on VAMs; that is, the use of VAMs to assess teacher quality and effectiveness as well as to hold teachers accountable. Yet while a growing volume of published research has revealed substantive methodological and inferential concerns when evaluating teacher effectiveness using VAM approaches (see, for example, Amrein-Beardsley, 2008; Corcoran, 2010; Darling-Hammond, Amrein-Beardsley, Haertel, & Rothstein, 2012; Muijs, 2006; McCaffrey, 2003; Papay, 2012; Raudenbush, 2004; Rivkin, 2007; Rothstein, 2009; Rubin, Stuart, & Zanutto, 2004; Scherrer, 2011; Schochet & Chiang, 2010; Zeis, Waronska, & Fuller, 2009), this dialogue is unfortunately taking place among academic researchers and not between academic researchers and policymakers. The translation of research to practice and policy development has yet to be realized in significant and impactful ways. Much more effort needs to be placed on sharing these results with policymakers (see, for example, Capitol Hill Briefing, 2011). Indeed, the goal of this EPAA collection is to present a series of research-based studies that are peer-reviewed, but that are also freely and openly accessible to all, including those at multiple policy levels. In addition, because EPAA facilitates providing policymakers direct access to research and research-based recommendations, in this case about a widely (and wildly) popular educational policy, hopefully this too will help to further inform our collective thinking about VAMs. Policymakers and others outside of academia must begin to better understand the methods, model specifications, and assumptions being made, as well as the appropriate interpretations and inference- based uses of VAM output. Framing the Issue With this in mind, we present next the major and minor themes prevalent across papers. We do this here particularly so that policymakers and others might be better equipped to delve further into the very real issues and concerns that come along with the adoption and implementation of VAMs. First, all of the papers trace their lineage back to the 2009 federal RttT initiative, which represented a critical transition point in public education policy. This popularized VAMs as the E ducation P olicy A nalysis A rchives V ol. 21 N o. 4 SPECIAL ISSUE 6 policy tools pitched for stronger accountability purposes and purposes of radical educational reform. This widespread and rapid acceptance is an underlying theme across the papers presented in this special issue. Second, all papers featured in this special issue are applicable to educational policymakers, particularly as all of the contributing researchers are focused on the same topic in their own diverse ways. While they each use unique methods of inquiry, the underlying purpose is to help others better understand and make sense of VAMs’ intended and unintended effects. As well, the individual and collective papers should help others to better ascertain whether VAMs indeed work in the ways theorized. Unique contributions here come from the qualitative pieces offered by Graue et al. and Gabriel and Lester. These are unique in that it is rare that qualitative research is published on this topic. Also distinctive are the law-based pieces put forth by Pullin and Baker et al. These are distinctive because we are only just beginning to understand the legal ramifications that might come along with VAM use in general, but more importantly for highly consequential decision-making purposes. In some cases, we are already beginning to witness the impact that VAM use (and abuse) can have on the personal and professional lives of public school teachers (Amrein-Beardsley & Collins, 2012). Also of great importance is Kersting et al.’s scholarly contribution. Via their examination of the methodological issues often associated with VAM-based ratings, they add more rich evidence in support of similar concerns about model stability, the substantial errors that are inhibiting VAM practicality, and how model specifications can compromise overall levels of validity. As well, Adler adds to our thinking about how technical details matter, especially in a case in which VAM-based findings may have resulted in erroneous conclusions about the long-term instructional effects of teachers. Alder highlights the potential for these judgments to improperly lodge themselves within prevalent policy ethos, especially if research-based findings are publicized prior to being peer- reviewed by the researchers best equipped to judge the merits of such research-based studies. A third perspective positions the papers within a larger conceptual validation study. That is, findings from each combine to form a larger conceptual framework for assessing the suitability of VAMs as tools for making consequential decisions about teachers and their effectiveness. In this regard, the stability of VAM ratings are examined by Kersting et al., while Baker et al. question the wisdom of employing pre-specified weights to VAM components. In addition, Graue et al. look at resource allocation and cultural coherence as unmeasured construct-relevant factors impacting student learning, and Gabriel and Lester highlight the absence of critical reflection on the part of policymakers regarding VAM-based measures. Pullin advances legal consequences emanating from the methodological problems associated with VAMs, and Adler’s examination of Chetty et al.’s (2011) study reveals that threats to validity may be internal, due to the improper application and interpretation of methodological approaches. Collectively these papers provide a more nuanced, multi-faceted, view of the VAMs currently situated within the larger policy context of stronger accountability, and also as currently situated as America’s ideal mechanisms to promote meaningful educational reform. Conclusion A close read of these articles reveals the tension existing between the policy and research communities regarding VAM and the evaluation of teacher effectiveness and educational quality. On one hand, policymakers throughout the country are increasingly embedding score-based (VAM) approaches within educational evaluation and accountability systems. On the other hand, social science researchers are increasingly questioning the methodological, technical, and inferential Value-added model (VAM) research for educational policy: Framing the issue 7 attributes of these same VAM approaches. Gabriel & Lester use the phrase sentinel of trust to reflect the degree to which policymakers have come to accept VAM as an objective, reliable, and valid measure of teacher quality. At the same time, they note how the same audience ignores the technical and methodological issues examined by some of the papers here and elsewhere. This is because of what Graue et al. call policymaker’s “insatiable appetite[s]” (p. 2) for quality, objective indicators in education. It is in this context that these research-based papers are presented to readers, individually and collectively, as these papers stand to “add value” to the literature regarding educational policy, high-stakes accountability, and teacher evaluation in general. Specifically, these papers stand to “add value” in terms of how policymakers, their affiliates, and others might more easily access a series of diverse, research-based contributions about how we might more wisely proceed in terms of thinking about VAMs and VAM-based use. References Amrein-Beardsley, A. (2008). Methodological concerns about the Education Value-Added Assessment System (EVAAS). Educational Researcher, 37(2), 65-75. doi: 10.3102/0013189X08316420 Amrein-Beardsley, A., & Collins, C. (2012). The SAS Education Value-Added Assessment System (SAS EVAAS) in the Houston Independent School District (HISD): Intended and unintended consequences. Education Policy Analysis Archives, 20, 1-31. Retrieved from http://epaa.asu.edu/ojs/article/view/1096 Ballou, D. (2012). Review of “The Long-Term Impacts of Teachers: Teacher Value-Added and Student Outcomes in Adulthood.” Boulder, CO: National Education Policy Center. Retrieved from http://nepc.colorado.edu/thinktank/review-long-term-impacts. Capitol Hill Briefing. (2011, September 14). Getting teacher evaluation right: A challenge for policy makers. A briefing by E. Haertel, J. Rothstein, A. Amrein-Beardsley, and L. Darling-Hammond. Washington DC: Dirksen Senate Office Building (research in brief). Retrieved from http://www.aera.net/Default.aspx?id=12856 Chetty, R., Friedman, J. N., & Rockoff, J.E. (2011, December). The long-term impacts of teachers: Teacher value-added and student outcomes in adulthood. Cambridge, MA: National Bureau of Economic Research. Retrieved from http://www.nber.org/papers/w17699 Collins, C. & Amrein-Beardsley, A. (under review). Putting growth and value-added models on the map: A national overview. Teachers College Record. Corcoran, S. P. (2010, September). Can teachers be evaluated by their student’s test scores? Should they be? The use of value-added measures of teacher effectiveness in policy and practice. Providence, RI: Annenberg Institute for School Reform at Brown University. Retrieved from http://annenberginstitute.org/publication/can-teachers-be-evaluated-their- students%E2%80%99-test-scores-should-they-be-use-value-added-mea Darling-Hammond, L., Amrein-Beardsley, A., Haertel, E., & Rothstein, J. (2012). Evaluating teacher evaluation. Phi Delta Kappan, 93(6), 8-15. Harris, D. N. (2011). Value-added measures in education: What every educator needs to know. Cambridge, MA: Harvard Education Press. Harris, D. N., Sass, T. R., & Semykina, A. (2012). Value-added models and the measurement of teacher productivity. Washington D.C.: The Urban Institute. Retrieved from http://www.urban.org/publications/1001508.html Koedel, C., & Betts, J. R. (2007). Re-examining the role of teacher quality in the educational E ducation P olicy A nalysis A rchives V ol. 21 N o. 4 SPECIAL ISSUE 8 production function. Working Paper No. 2007-03. Nashville, TN: National Center on Performance Initiatives. Retrieved from http://econpapers.repec.org/paper/umcwpaper/0708.htm Lockwood, J. R., McCaffrey, D. F., & Sass, T. R. (2008). The intertemporal stability, of teacher effect estimates. Paper presented at the National Conference on Value-Added Modeling. Sponsored by the Wisconsin Center for Education Research (WCER), Madison, WI. Retrieved from http://www.wcer.wisc.edu/news/events/VAM%20Conference%20Final%20Papers/Interte mporalStability_McCaffreySassLockwood.pdf Lowrey, A. (2012, January 6). Big study links good teachers to lasting gain. The New York Times. Retrieved from http://www.nytimes.com/2012/01/06/education/big-study-links-good- teachers-to-lasting-gain.html?emc=eta1#comments McCaffrey, D. F. (2003). Evaluating value-added models of teacher accountability. Santa Monica, CA: RAND Corporation. Retrieved from www.rand.org/pubs/monographs/2004/RAND_MG158.pdf McCaffrey, D. F., Lockwood, J. R., Koretz, D. M., & Hamilton, L. S. (2003). Evaluating value-added models for teacher accountability. Santa Monica, CA: RAND Corporation. Retrieved from www.rand.org/pubs/monographs/2004/RAND_MG158.pdf Muijs, D. (2006). Measuring teacher effectiveness: Some methodological reflection. Educational Research and Evaluation, 12(1), 53-74. doi:10.1080/13803610500392236 Nelson, F. H. (2011, April). A guide for developing growth models for teacher development and evaluation. Paper presented at the Annual Conference of the American Educational Research Association (AERA), New Orleans, LA. Papay, J. P. (2011). Different tests, different answers. American Educational Research Journal, 48(1), 163- 193. doi:10.3102/0002831210362589 Papay, J. P. (2012). Refocusing the debate: Assessing the purposes and tools of teacher evaluation. Harvard Educational Review, 82(1), 123-141. Philips, R. H. (2012, September 10). No child left behind waiver sought by Alabama, six other states. Alabama Press Register. Retrieved from http://blog.al.com/live/2012/09/no_child_left_behind_waiver_so.html Raudenbush, S. W. (2004). What are value-added models estimating and what does this imply for statistical practice? Journal of Educational and Behavioral Statistics, 29(1), 121-129. doi:10.3102/10769986029001121 Ravitch, D. (2012, January 17). What value did the Chetty study add? Education Week [Blog entry]. Retrieved from http://blogs.edweek.org/edweek/Bridging- Differences/2012/01/dear_deborah_just_days_ago.html Rivkin, S. G. (2007, November). Value-added analysis and education policy. Washington, DC: National Center for Analysis of Longitudinal Data in Education Research. Urban Institute. Retrieved from http://www.urban.org/UploadedPDF/411577_value-added_analysis.pdf Rothstein, J. (2009, January 11). Student sorting and bias in value-added estimation: Selection on observables and unobservables. Cambridge, MA: The National Bureau of Economic Research. Retrieved from http://www.nber.org/papers/w14607 Rubin, D. B., Stuart, E. A., & Zanutto, E. L. (2004). A potential outcomes view of value-added assessment in education. Journal of Educational and Behavioral Statistics, 29(1), 103-116. doi:10.3102/10769986029001103 Schafer, W. D., Lissitz, R. W., Zhu, X., Zhang, Y., Hou, X., & Li, Y. (2012). Evaluating teachers and schools using student growth models. Practical Assessment, Research & Evaluation, 17(17), 1-21. Scherrer, J. (2011). Measuring teaching using value-added modeling: The imperfect panacea. NASSP Bulletin, 95(2), 122-140. doi:10.1177/0192636511410052 Value-added model (VAM) research for educational policy: Framing the issue 9 Schochet, P. Z., & Chiang, H. S. (2010, July). Error rates in measuring teacher and school performance based on student test score gains. U. S. Department of Education. Retrieved from http://ies.ed.gov/ncee/pubs/20104004/ Stumbo, C., & McWalters, P. (2010). Measuring effectiveness: What will it take? Educational Leadership, 68(4), 10-15. U. S. Department of Education. (2009a). Race to the top application for Phase 2 funding: CFDA Number: 84.395A. Washington, DC. Retrieved from http://www07.grants.gov/search/search.do?&mode=VIEW&flag2006=false&oppId=5028 7 U. S. Department of Education. (2009b). Race to the Top program: Executive summary. Washington, DC: Retrieved from http://www2.ed.gov/programs/racetothetop/executive- summary.pdf U. S. Department of Education. (2012a). Awards. Washington DC: Retrieved from http://www2.ed.gov/programs/racetothetop/awards.html U. S. Department of Education. (2012b). Education department announces 16 winners of Race to the Top – district competition. Washington DC: Retrieved from http://www.ed.gov/news/press-releases/education-department-announces-16-winners- race-top-district-competition Winerip, M. (2012, January 15). Study on teacher value uses data from before teach-to-test era. The New York Times. Retrieved from http://www.nytimes.com/2012/01/16/us/study-on- teacher-value-uses-data-from-before-teach-to-test-era.html?emc=eta1 Zeis, C., Waronska, A. K., & Fuller, R. (2009). Value-added program assessment using nationally standardized tests: Insights into internal validity issues. Journal of Business and Economics, 9(1), 114-127. E ducation P olicy A nalysis A rchives V ol. 21 N o. 4 SPECIAL ISSUE 10 About the Guest Editors Audrey Amrein-Beardsley (Editor) Affiliation: Associate Professor, Arizona State University, Mary Lou Fulton Teachers College Email: audrey.beardsley@asu.edu Audrey Amrein-Beardsley, Ph.D., is currently an Associate Professor in the Mary Lou Fulton Teachers College at Arizona State University. Her research interests include educational policy, research methods, and more specifically, high-stakes tests and value-added measurements and systems. She was also recently named as one of the top 121 edu-scholars in the nation, honored for being a university-based academic who is contributing most substantially to public debates about the nation's educational system. She is also the creator and host of a show titled Inside the Academy during which she interviews some of the top educational researchers in the academy. For more information please see: http://insidetheacademy.asu.edu. Clarin Collins (Assistant Editor) Affiliation: Graduate, Arizona State University, Mary Lou Fulton Teachers College Email: clarin.collins@asu.edu Clarin Collins, Ph.D., recently graduated from the Educational Policy and Evaluation program in the Mary Lou Fulton Teachers College at Arizona State University. For her dissertation study she analyzed teachers’ understanding of and experiences with the SAS Educational Value-Added Assessment System (SAS® EVAAS®) in the Houston Independent School District where SAS® EVAAS® is currently used to evaluate teachers with high-stakes consequences. Her research interests include national and local policy implementation at the classroom level, teacher influences on policymaking and implementation, and education evaluation and accountability systems. Sarah A. Polasky (Assistant Editor) Affiliation: Assistant Research Professor, Arizona State University, Mary Lou Fulton Teachers College Email: sarah.polasky@asu.edu Sarah A. Polasky, Ph.D. works as the Value-Added Specialist on the Arizona Ready-for-Rigor Project, a federal Teacher Incentive Fund grant awarded to the Mary Lou Fulton Teachers College at Arizona State University in 2010. She works to support the implementation of value- added evaluation systems in partnering districts and schools, as well as support the betterment of their evaluation and data systems (e.g., by introducing and including new and unique measures into such systems). Her related research interests include assessment systems in early childhood education, the use of alternative achievement tests (e.g., district benchmarks, formative assessments) and non-achievement (i.e., developmental) data for value-added systems in general, and the impact of socioemotional and neurological development of young children on their academic achievement and growth over time. Edward F. Sloat (Assistant Editor) Affiliation: Director of Research and Accountability, Dysart Unified School District, Surprise, Value-added model (VAM) research for educational policy: Framing the issue 11 AZ; and Doctoral Student, Leadership and Innovation, Arizona State University, Mary Lou Fulton Teachers College Email: esloat@asu.edu Edward Sloat, M.Ed. is currently employed as the Director of Research and Accountability at Dysart Unified School District located in Surprise, Arizona. He is also a doctoral student in the Leadership and Innovation Program within the Mary Lou Fulton Teachers College at Arizona State University. He has previously served in research, evaluation, and assessment leadership positions at the Arizona Department of Education, Peoria (Arizona) Unified School District, and Glendale (Arizona) Elementary School District. He regularly contributes to state technical and policy working/advisory groups concerning assessment design and accountability systems and is past president of the Arizona Education Research Organization. His academic interests focus on value-added modeling, education accountability and evaluation systems, data-driven instructional planning, applications of measurement theory, and research methods. education policy analysis archives Volume 21 Number 4 January 28th, 2013 ISSN 1068-2341 Readers are free to copy, display, and distribute this article, as long as the work is attributed to the author(s) and Education Policy Analysis Archives, it is distributed for non- commercial purposes only, and no alteration or transformation is made in the work. More details of this Creative Commons license are available at http://creativecommons.org/licenses/by-nc-sa/3.0/. All other uses must be approved by the author(s) or EPAA. EPAA is published by the Mary Lou Fulton Institute and Graduate School of Education at Arizona State University Articles are indexed in CIRC (Clasificación Integrada de Revistas Científicas, Spain), DIALNET (Spain), Directory of Open Access Journals, EBSCO Education Research Complete, ERIC, Education Full Text (H.W. Wilson), QUALIS A2 (Brazil), SCImago Journal Rank; SCOPUS, SOCOLAR (China). Please contribute commentaries at http://epaa.info/wordpress/ and send errata notes to Gustavo E. Fischman fischman@asu.edu Join EPAA’s Facebook community at https://www.facebook.com/EPAAAAPE and Twitter feed @epaa_aape. E ducation P olicy A nalysis A rchives V ol. 21 N o. 4 SPECIAL ISSUE 12 education policy analysis archives editorial board Editor Gustavo E. Fischman (Arizona State University) Associate Editors: David R. Garcia (Arizona State University), Stephen Lawton (Arizona State University) Rick Mintrop, (University of California, Berkeley) Jeanne M. Powers (Arizona State University) Jessica Allen University of Colorado, Boulder Christopher Lubienski University of Illinois, Urbana- Champaign Gary Anderson New York University Sarah Lubienski University of Illinois, Urbana- Champaign Michael W. Apple University of Wisconsin, Madison Samuel R. Lucas University of California, Berkeley Angela Arzubiaga Arizona State University Maria Martinez-Coslo University of Texas, Arlington David C. Berliner Arizona State University William Mathis University of Colorado, Boulder Robert Bickel Marshall University Tristan McCowan Institute of Education, London Henry Braun Boston College Heinrich Mintrop University of California, Berkeley Eric Camburn University of Wisconsin, Madison Michele S. Moses University of Colorado, Boulder Wendy C. Chi* University of Colorado, Boulder Julianne Moss University of Melbourne Casey Cobb University of Connecticut Sharon Nichols University of Texas, San Antonio Arnold Danzig Arizona State University Noga O'Connor University of Iowa Antonia Darder University of Illinois, Urbana- Champaign João Paraskveva University of Massachusetts, Dartmouth Linda Darling-Hammond Stanford University Laurence Parker University of Illinois, Urbana- Champaign Chad d'Entremont Strategies for Children Susan L. Robertson Bristol University John Diamond Harvard University John Rogers University of California, Los Angeles Tara Donahue Learning Point Associates A. G. Rud Purdue University Sherman Dorn University of South Florida Felicia C. Sanders The Pennsylvania State University Christopher Joseph Frey Bowling Green State University Janelle Scott University of California, Berkeley Melissa Lynn Freeman* Adams State College Kimberly Scott Arizona State University Amy Garrett Dikkers University of Minnesota Dorothy Shipps Baruch College/CUNY Gene V Glass Arizona State University Maria Teresa Tatto Michigan State University Ronald Glass University of California, Santa Cruz Larisa Warhol University of Connecticut Harvey Goldstein Bristol University Cally Waite Social Science Research Council Jacob P. K. Gross Indiana University John Weathers University of Colorado, Colorado Springs Eric M. Haas WestEd Kevin Welner University of Colorado, Boulder Kimberly Joy Howard* University of Southern California Ed Wiley University of Colorado, Boulder Aimee Howley Ohio University Terrence G. Wiley Arizona State University Craig Howley Ohio University John Willinsky Stanford University Steve Klees University of Maryland Kyo Yamashiro University of California, Los Angeles Jaekyung Lee SUNY Buffalo * Members of the New Scholars Board Value-added model (VAM) research for educational policy: Framing the issue 13 archivos analíticos de políticas educativas consejo editorial Editor: Gustavo E. Fischman (Arizona State University) Editores. Asociados Alejandro Canales (UNAM) y Jesús Romero Morante (Universidad de Cantabria) Armando Alcántara Santuario Instituto de Investigaciones sobre la Universidad y la Educación, UNAM México Fanni Muñoz Pontificia Universidad Católica de Perú Claudio Almonacid Universidad Metropolitana de Ciencias de la Educación, Chile Imanol Ordorika Instituto de Investigaciones Economicas – UNAM, México Pilar Arnaiz Sánchez Universidad de Murcia, España Maria Cristina Parra Sandoval Universidad de Zulia, Venezuela Xavier Besalú Costa Universitat de Girona, España Miguel A. Pereyra Universidad de Granada, España Jose Joaquin Brunner Universidad Diego Portales, Chile Monica Pini Universidad Nacional de San Martín, Argentina Damián Canales Sánchez Instituto Nacional para la Evaluación de la Educación, México Paula Razquin UNESCO, Francia María Caridad García Universidad Católica del Norte, Chile Ignacio Rivas Flores Universidad de Málaga, España Raimundo Cuesta Fernández IES Fray Luis de León, España Daniel Schugurensky Universidad de Toronto-Ontario Institute of Studies in Education, Canadá Marco Antonio Delgado Fuentes Universidad Iberoamericana, México Orlando Pulido Chaves Universidad Pedagógica Nacional, Colombia Inés Dussel FLACSO, Argentina José Gregorio Rodríguez Universidad Nacional de Colombia Rafael Feito Alonso Universidad Complutense de Madrid, España Miriam Rodríguez Vargas Universidad Autónoma de Tamaulipas, México Pedro Flores Crespo Universidad Iberoamericana, México Mario Rueda Beltrán Instituto de Investigaciones sobre la Universidad y la Educación, UNAM México Verónica García Martínez Universidad Juárez Autónoma de Tabasco, México José Luis San Fabián Maroto Universidad de Oviedo, España Francisco F. García Pérez Universidad de Sevilla, España Yengny Marisol Silva Laya Universidad Iberoamericana, México Edna Luna Serrano Universidad Autónoma de Baja California, México Aida Terrón Bañuelos Universidad de Oviedo, España Alma Maldonado Departamento de Investigaciones Educativas, Centro de Investigación y de Estudios Avanzados, México Jurjo Torres Santomé Universidad de la Coruña, España Alejandro Márquez Jiménez Instituto de Investigaciones sobre la Universidad y la Educación, UNAM México Antoni Verger Planells University of Amsterdam, Holanda José Felipe Martínez Fernández University of California Los Angeles, USA Mario Yapu Universidad Para la Investigación Estratégica, Bolivia E ducation P olicy A nalysis A rchives V ol. 21 N o. 4 SPECIAL ISSUE 14 arquivos analíticos de políticas educativas conselho editorial Editor: Gustavo E. Fischman (Arizona State University) Editores Associados: Rosa Maria Bueno Fisher e Luis A. Gandin (Universidade Federal do Rio Grande do Sul) Dalila Andrade de Oliveira Universidade Federal de Minas Gerais, Brasil Jefferson Mainardes Universidade Estadual de Ponta Grossa, Brasil Paulo Carrano Universidade Federal Fluminense, Brasil Luciano Mendes de Faria Filho Universidade Federal de Minas Gerais, Brasil Alicia Maria Catalano de Bonamino Pontificia Universidade Católica-Rio, Brasil Lia Raquel Moreira Oliveira Universidade do Minho, Portugal Fabiana de Amorim Marcello Universidade Luterana do Brasil, Canoas, Brasil Belmira Oliveira Bueno Universidade de São Paulo, Brasil Alexandre Fernandez Vaz Universidade Federal de Santa Catarina, Brasil António Teodoro Universidade Lusófona, Portugal Gaudêncio Frigotto Universidade do Estado do Rio de Janeiro, Brasil Pia L. Wong California State University Sacramento, U.S.A Alfredo M Gomes Universidade Federal de Pernambuco, Brasil Sandra Regina Sales Universidade Federal Rural do Rio de Janeiro, Brasil Petronilha Beatriz Gonçalves e Silva Universidade Federal de São Carlos, Brasil Elba Siqueira Sá Barreto Fundação Carlos Chagas, Brasil Nadja Herman Pontificia Universidade Católica –Rio Grande do Sul, Brasil Manuela Terrasêca Universidade do Porto, Portugal José Machado Pais Instituto de Ciências Sociais da Universidade de Lisboa, Portugal Robert Verhine Universidade Federal da Bahia, Brasil Wenceslao Machado de Oliveira Jr. Universidade Estadual de Campinas, Brasil Antônio A. S. Zuin Universidade Federal de São Carlos, Brasil