Guzman, N. L. (2016, August 31). Review of Rethinking value-added models in education. Critical perspectives on tests and assessment-based accountability by A. Amrein-Beardsley. Education Review, 23. http://dx.doi.org/10.14507/er.v23.1940 August 31, 2016 ISSN 1094-5296 Amrein-Beardsley, A. (2014). Rethinking value-added models in education. Critical perspectives on tests and assessment-based accountability. New York: Routledge. Pp. xxi + 251 ISBN: 978-0-415-82012-7 Reviewed by Natalia L. Guzman University of Maryland United States U.S. educational policy, particularly under No Child Left Behind (NCLB, 2001), and more recently, under the federal initiative Race to the Top (RttT, 2011), has propelled students and teachers towards increased accountability through the use of different analytical instruments. In the case of teachers, several states adopted value-added models (VAMs) as a method to measure how much teachers contribute or add value to student learning and achievement on standardized tests (Murphy, 2012). As the recently released American Education Research Association’s (AERA, 2015) statement on VAMs testifies, questions about the credibility of these methods rose not only from the academic community, but also from practitioners and policymakers. For example, Harris and Herrington (2015) stress the need to understand how educators make sense of VAMs. Ballou and Springer (2015) draw attention to problems in the design, implementation, and data collection of valued- http://dx.doi.org/10.14507/er.v23.1940 Education Review /Reseñas Educativas 2 added data. The American Statistical Association (2014) points out the difficulty of drawing valid statistical inferences about teacher quality from students test scores. In this context, Rethinking Value-added Models in Education makes a significant contribution to understanding the limitations and consequences of using VAMs in the American public education system. Throughout the book, author Audrey Amrein-Beardsley synthesizes and critiques numerous studies and cases from both academic and popular outlets. The main themes that organize the content of book involve the development, implementation, consequences, and future of valued-added methods for teacher accountability: 1) the use of social engineering in American educational policy; 2) the negative impact on the human factor in schools; 3) the acceptance of unquestioned theoretical and methodological assumptions in VAMs; and 4) the availability of conventional alternatives and solutions to a newly created problem. The book’s most prominent theme, the use of social engineering in American educational policy, emerges in the introductory chapters of the book. The author argues that U.S. educational policy is predicated on the concept of social engineering—a powerful instrument that influences attitudes and social behaviors to promote the achievement of idealized political ends. In the case of American educational policy, the origins and development of VAMs is connected to the goal of improving student achievement and solving the problem of America’s failing public school system. Part I of the book offers a historical context and a political perspective of how VAMs were installed in the U.S. education system. According to Amrein- Beardsley, the development of VAMs can be traced to the federal report A Nation at Risk (U.S. Department of Education, 1983). This report serves as a paradigmatic case of social engineering because it has instilled fear about the poor performance of U.S. schools in contrast to other industrialized countries. The author develops a theory, which she calls Measure and Punish (M&P) Theory of Change, that works by holding districts, schools, teachers, and students accountable for performance on large-scale standardized tests so that “administrators would supervise the schools better, teachers would teach better, and students would learn more, particularly in the nation’s lowest performing schools” (p. 10). The human factor involved in the implementation of VAMs emerges as a prominent theme in Chapter 2. The author argues against the use of VAMs in schools and connects her theory of social engineering with the negative effects of using VAMs to evaluate teachers. For example, the author claims that the M&P Theory of Change influences attitudes and social behaviors of policymakers and leaders of private corporate organizations, who seek to elevate failing public schools by holding schools, teachers and students accountable for performance on large-scale standardized achievement tests. Amrein- Beardsley uses powerful examples of research- based accounts of how VAMs affected teachers and school districts, important aspects of the human factor involved in the implementation of these models. One of these examples reports on the SAS® Education Value-Added Assessment System (EVASS®) in the Houston Independent School District (HISD). The author’s detailed analysis of termination of four teachers in HISD concludes that VAMs, as they are currently used in practice, have serious problems with reliability, validity, and bias, affecting their fairness and transparency. Chapter 3 addresses the acceptance of unquestioned theoretical and methodological assumptions in VAMs. The author reexamines the history, model specifics, and use of the three most widely adopted models, the SAS® EVASS®, Value-Added Research Center (VARC), and Student Growth Percentile (SGP) models. Although all these models are used for educational accountability, they have different goals and objectives and their developers range from a private corporation to Review of Rethinking value-added models by N. L. Guzman 3 non-profit organizations with connections to universities. In Parts II and III, the author rebuts the assumptions that are held as rationales and justifications for the use of VAMs. Some of these assumptions place a great deal of importance on the human factor in schools. For instance, one widespread unquestioned assumption is that teachers are the most important factors that impact student learning. However, it was surprising to learn about research showing that the teacher effect is about 10% to 20% of the variance in test scores. Although teachers have an important role in the classroom, their influence explains only small fraction of how students perform in standardized tests. Moreover, the teacher effect does not supersede the effects of student-level and out-of-school influences. This reader appreciated the opportunity to learn about research that directly questions similar statistical and methodological assumptions in a way that was highly accessible, surprisingly, since discussions about VAM methodology tends to be highly technical. For example, it was surprising to find out about research that shows a weak correlation or strength of numerical relationships between one year’s VAM-based estimates to the next one (i.e., 0.3 ≤ r ≤0.4; Brophy, 1973; McCaffrey et al., 2009, among others). In other words, the reliability of VAM estimates is very low since teachers’ evaluations are not constant year after year. A teacher can be identified as outstanding one year but ineffective the next year. This lack of reliability brings numerous questions about the validity, transparence and fairness of using VAMs in education. The most significant part of Amrein-Beardsley’s argument is that, as researchers, we know the shortcomings related to VAMs; yet, these analytical models are still being inappropriately used. The book closes with an exploration of some traditional and conventional alternatives to VAMs. One alternative is to employ multiple measures and more holistic evaluation systems instead of relying solely on VAMs. Additional measures may include “teachers’ supervisor and/or peer observation scores, student and parent satisfaction indicators, and student work samples” (p. 187). The virtue of this proposal is that it contextualizes teacher evaluation, offering multiple perspectives of the complexity of teaching, and it engages different members of the school community, bringing in the voices of teacher colleagues, parents and/or students. Overall, this book offers one of the most comprehensive critiques of what we know about VAMs in the American public education system. The author contextualizes her critique to added-value methods in education within a larger socio-political discussion that revisits the history and evolution of teacher accountability in the US. The book incorporates studies from academic sources as well as summarizes cases from popular outlets such as newspapers and blogs. This author presents all this information using nontechnical language, which makes it suitable for the general public as well as academic readers. Another major contribution of this book is that it gives voice to the teachers and school administrators that were affected by VAMs, an aspect that has not yet been thoroughly researched. However, Amrein-Beardsley’s book falls short in a few areas. For example, the exhaustive treatment of assumptions in VAMs makes certain parts of the book difficult to follow. Parts II and Part III would benefit from including an annotated bibliography at the end of each chapter to summarize the sources and the main points of the chapters. This addition would be especially helpful for the academic reader who is interested in conducting research in VAMs. A more detailed discussion about alternatives and solutions to VAMs would potentially be beneficial. For example, although the book gives valuable recommendations about returning to conventional solutions in the United States, the author might also touch on how other countries deal with teacher Education Review /Reseñas Educativas 4 accountability. Amrein-Beardsley’s reference to Finland at the end of the chapter is interesting but too brief to draw any conclusions about other alternatives. Finally, the book offers a strong one-sided criticism that does not leave any space for more complex arguments about VAMs, so readers looking for a balanced approach to the topic should look elsewhere. Despite these criticisms, Rethinking value-added models in education is a valuable resource to anyone desiring to better understand the use of valued-added models in teacher evaluation. Amrein-Beardsley’s book is of great interest to teachers, policymakers, and scholars who seek a comprehensive critical overview of VAMs in the U.S. education system. Teachers can also benefit by reading this volume; they may gain more insight into how VAMs were designed and are applied in their schools as well as learn more about the technical aspects of these models. Policymakers and educational researchers who seek fairer and more inclusive ways of teacher evaluation may find the proposals in the book inspiring and useful as they engage in discussions about finding alternative ways to conduct teacher assessments. Finally, graduate students in education and other researchers interested in value-added methods in education can gain access to one of the most up to date, comprehensive resource on VAMs. To conclude, this book definitely delivers a much needed strong case against VAMs in public education, and has the potential to open a new cycle in the discussion about teacher assessment by inviting us to seek and demand more sustainable and fairer assessment alternatives. References American Education Research Association [AERA] Council. (2015). AERA statement on use of value-added models (VAM) for the evaluation of educators and educator preparation programs. Educational Researcher, XX(X), 1-5. doi:10.3102/0013189X15618385 American Statistical Association [ASA]. (2014). ASA statement on using value-added models for educational assessment. Alexandria, VA: Author. Retrieved from https://www.amstat.org/policy/pdfs/ASA_VAM_Statement.pd Ballou, D., & Springer, M. G. (2015). Using Student Test Scores to Measure Teacher Performance Some Problems in the Design and Implementation of Evaluation Systems. Educational Researcher, 44(2), 77-86. Brophy, J. E. (1973). Stability of teacher effectiveness. American Educational Research Journal, 10(3), 245- 252. Harris, D. N., & Herrington, C. D. (2015). Editors’ Introduction: The Use of Teacher Value Added Measures in Schools New Evidence, Unanswered Questions, and Future Prospects. Educational Researcher, 44(2), 71-76. McCaffrey, D. F., Sass, T. R., Lockwood, J. R., & Mihaly, K. (2009). The intertemporal variability of teacher effect estimates. Education Finance and Policy, 4(4), 572-606. Murphy, D. (2012). Where is the Value in Value-Added Modeling? Pearson White Papers. No Child Left Behind [NCLB]. (2001). No Child Left Behind. Title III: Language instruction for limited English proficient and immigrant students. 107th Congress, 1st Session, December 13, 2001. Washington, DC: George Washington University, National Clearinghouse for Bilingual Education. Race to the Top [RttT] (2011). Race to the Top Act of 2011, S. 844-112th Congress. Retrieved from https://www.govtrack.us/congress/bills/112/s844 https://www.amstat.org/policy/pdfs/ASA_VAM_Statement.pd https://www.govtrack.us/congress/bills/112/s844 Review of Rethinking value-added models by N. L. Guzman 5 About the Reviewer Natalia L. Guzman is a doctoral student in Second Language Education and Culture in the College of Education at the University of Maryland. Her academic interests include foreign and second language education, classroom-based research, and teacher/student assessment. Natalia has taught Spanish for several years in Argentina and the United States. Education Review/Reseñas Educativas/Resenhas Educativas is supported by the edXchange initiative’s Scholarly Communications Group at the Mary Lou Fulton Teachers College, Arizona State University. Copyright is retained by the first or sole author, who grants right of first publication to the Education Review. Readers are free to copy, display, and distribute this article, as long as the work is attributed to the author(s) and Education Review, it is distributed for non-commercial purposes only, and no alteration or transformation is made in the work. More details of this Creative Commons license are available at http://creativecommons.org/licenses/by-nc-sa/3.0/. All other uses must be approved by the author(s) or Education Review. Education Review is published by the Scholarly Communications Group of the Mary Lou Fulton Teachers College, Arizona State University. Please contribute reviews at http://www.edrev.info/contribute.html. Connect with Education Review on Facebook (https://www.facebook.com/pages/Education- Review/178358222192644) and on Twitter @EducReview http://www.edrev.info/contribute.html https://www.facebook.com/pages/Education-Review/178358222192644 https://www.facebook.com/pages/Education-Review/178358222192644