Microsoft Word - v13n37.doc EDUCATION POLICY ANALYSIS ARCHIVES A peer-reviewed scholarly journal Editor: Sherman Dorn College of Education University of South Florida Copyright is retained by the first or sole author, who grants right of first publication to the Education Policy Analysis Archives. EPAA is published jointly by the Colleges of Education at Arizona State University and the University of South Florida. Articles are indexed in the Directory of Open Access Journals (www.doaj.org) and by H.W. Wilson & Co. Volume 13 Number 37 August 30, 2005 ISSN 1068–2341 Response to "What Do Klein et al. Tell Us About Test Scores in Texas?"1 Stephen P. Klein, Laura S. Hamilton, Daniel F. McCaffrey, Brian M. Stecher RAND Citation: Klein, S. P., Hamilton, L. S., McCaffrey, D. F., & Stecher, B. M. (2005, August 30). Response to "What do Klein et al. tell us about test scores in Texas?" Education Policy Analysis Archives, 13(37). Retrieved [date] from http://epaa.asu.edu/epaa/v13n37/. We have reviewed the article by Toenjes (2005). Below we summarize our responses. First, Toenjes incorrectly describes the focus of our study. As we note in our paper, the findings from the 20-school analysis merely triggered and supplemented our statewide analyses. Most of our article examined the stark discontinuity between TAAS and NAEP trends for the entire state of Texas (namely, the meteoric increase in scores and the narrowing of the gap between racial/ethnic groups on TAAS not being reflected on NAEP). The huge discrepancy between NAEP and state test scores in Texas is well-documented, it is still present, and it has been described in other studies (Haney, 2000; McCombs et al., 2005; Peterson & Hess, 2005). Toenjes’ comments about the representativeness of the 20 schools are irrelevant to our statewide analyses, and that analysis was the focus of our paper. Second, Toenjes appears to have misunderstood the purpose of our 20-schools analysis. The intent was not to make conclusions about relationships between achievement and socioeconomic status statewide. Rather, the purpose was to examine whether, in this particular sample for which scores on two different tests were available, the TAAS functioned in the same way as an alternative test that many other states use, namely, the Stanford 9. The fact that the TAAS and Stanford 9 1 Accepted under the editorship of Sherman Dorn. Send commentary to Casey Cobb (casey.cobb@uconn.edu). Education Policy Analysis Archives Vol. 13 No. 37 2 showed radically different relationships with SES does not tell us anything about what those relationships would be in a larger sample, and we never claim that it does. In particular, we do not, as Toenjes claims, use the 20-schools study to conclude that TAAS lacks validity. Instead, we used the findings with the 20 schools to show that in these particular schools, the scores on a commonly used test of mathematics achievement did not correspond to scores on the TAAS. The difference between these two tests was much larger than can plausibly be explained by differences in content coverage, and therefore raises questions about the meaning of TAAS scores for these schools. As we state clearly, “We are therefore reluctant to draw conclusions from our findings with these schools or to imply that these findings are likely to occur elsewhere in Texas. Nevertheless, they do suggest the desirability of periodic administration of external tests to validate TAAS results. This procedure, which is sometimes referred to as ‘audit testing,’ could have been incorporated into the study of the Metropolitan Achievement Test discussed previously.” This is a much more cautious conclusion than Toenjes claims we make. Third, Toenjes misunderstands the relationship between our study and the Grissmer, Flanagan, Kawata, & Williamson (2000) study. Toenjes states that we concluded that the observed increases in Texas student academic performance reported by Grissmer et al. (2000) were highly suspect. Our paper contains no such conclusion. We focused on whether the TAAS scores were suspect, whereas Grissmer et al. examined gains in NAEP. As we clearly state in our paper, “these studies differed in the questions they investigated, the data they analyzed, and the methodologies they employed.” The studies do not in fact produce conflicting findings; they simply address different questions. References Grissmer, D., Flanagan, A., Kawata, J., & Williamson, S. (2000). Improving student achievement: What state NAEP test scores tell us. Santa Monica, CA: RAND, MR-924-EDU. Haney, W. (2000). The Myth of the Texas Miracle in Education. Education Policy Analysis Archives, 8(41). Retrieved June 21, 2005, from http://epaa.asu.edu/epaa/v8n41/. McCombs, J. S., Kirby, S. N., Barney, H., Darilek, H., & Magee, S. J. (2005). Achieving state and national literacy goals, a long uphill road: A report to Carnegie Corporation of New York. Santa Monica, CA: RAND, TR-180-EDU. Peterson, P. E., & Hess, F. M. (2005). Johnny can read…in some states. Education Next, Summer 2005, 52-53. Retrieved June 21, 2005, from http://www.educationnext.org/20053/pdf/52.pdf. Toenjes, L. A. (2005). What do Klein et al. tell us about test scores in Texas? Education Policy Analysis Archives, 13(36). Available at http://epaa.asu.edu/epaa/v13n36/. Response to "What Do Klein et al. Tell Us About Test Scores in Texas?" 3 About the Authors Stephen P. Klein Laura S. Hamilton Daniel F. McCaffrey Brian Stecher RAND Email: klein@rand.org Dr. Stephen P. Klein is a Senior Research Scientist at RAND, where for the past 30 years he has led studies on health, criminal justice, military manpower, and educational issues. His current projects include analyzing licensing examinations in teaching and other professions, delivering computer performance tests over the Web, and measuring the effects of instructional practices and curriculum on student performance. Dr. Laura S. Hamilton is a Senior Behavioral Scientist at RAND where she conducts research on educational assessment and the effectiveness of educational reform programs. Her current projects include a study of systemic reforms in math and science, an investigation of district and school responses to standards-based accountability, and a study of teachers’ and principals’ use of information from a value-added assessment system. Dr. Daniel F. McCaffrey is is a Senior Statistician at RAND where he works on studies of health and educational issues. His recent research has focused the study of value-added model estimates of teacher effects by evaluating the literature and statistical methods for value-added modeling, by developing new models and algorithms for estimating effects, and by conducting empirical evaluations of the various estimates. He is also studying the effect of a value-added assessment program on student outcomes and educational practice. Dr. Brian Stecher is a Senior Social Scientist in the Education program at RAND. Dr. Stecher's research emphasis is applied educational measurement, including the implementation, quality, and impact of state assessment and accountability systems and the cost, quality, and feasibility of performance-based assessments in mathematics and science. Education Policy Analysis Archives Vol. 13 No. 37 4 EDUCATION POLICY ANALYSIS ARCHIVES http://epaa.asu.edu Editor: Sherman Dorn, University of South Florida Production Assistant: Chris Murrell, Arizona State University General questions about appropriateness of topics or particular articles may be addressed to the Editor, Sherman Dorn, epaa-editor@shermandorn.com. Editorial Board Michael W. Apple University of Wisconsin David C. Berliner Arizona State University Greg Camilli Rutgers University Casey Cobb University of Connecticut Linda Darling-Hammond Stanford University Mark E. Fetler California Commission on Teacher Credentialing Gustavo E. Fischman Arizona State Univeristy Richard Garlikov Birmingham, Alabama Gene V Glass Arizona State University Thomas F. Green Syracuse University Aimee Howley Ohio University Craig B. Howley Appalachia Educational Laboratory William Hunter University of Ontario Institute of Technology Patricia Fey Jarvis Seattle, Washington Daniel Kallós Umeå University Benjamin Levin University of Manitoba Thomas Mauhs-Pugh Green Mountain College Les McLean University of Toronto Heinrich Mintrop University of California, Berkeley Michele Moses Arizona State University Anthony G. Rud Jr. Purdue University Michael Scriven Western Michigan University Terrence G. Wiley Arizona State University John Willinsky University of British Columbia Response to "What Do Klein et al. Tell Us About Test Scores in Texas?" 5 EDUCATION POLICY ANALYSIS ARCHIVES English-language Graduate-Student Editorial Board Noga Admon New York University Jessica Allen University of Colorado Cheryl Aman University of British Columbia Anne Black University of Connecticut Marisa Burian-Fitzgerald Michigan State University Chad d'Entremont Teachers College Columbia University Carol Da Silva Harvard University Tara Donahue Michigan State University Camille Farrington University of Illinois Chicago Chris Frey Indiana University Amy Garrett Dikkers University of Minnesota Misty Ginicola Yale University Jake Gross Indiana University Hee Kyung Hong Loyola University Chicago Jennifer Lloyd University of British Columbia Heather Lord Yale University Shereeza Mohammed Florida Atlantic University Ben Superfine University of Michigan John Weathers University of Pennsylvania Kyo Yamashiro University of California Los Angeles Education Policy Analysis Archives Vol. 13 No. 37 6 Archivos Analíticos de Políticas Educativas Associate Editors Gustavo E. Fischman & Pablo Gentili Arizona State University & Universidade do Estado do Rio de Janeiro Founding Associate Editor for Spanish Language (1998—2003) Roberto Rodríguez Gómez Editorial Board Hugo Aboites Universidad Autónoma Metropolitana-Xochimilco Adrián Acosta Universidad de Guadalajara México Claudio Almonacid Avila Universidad Metropolitana de Ciencias de la Educación, Chile Dalila Andrade de Oliveira Universidade Federal de Minas Gerais, Belo Horizonte, Brasil Alejandra Birgin Ministerio de Educación, Argentina Teresa Bracho Centro de Investigación y Docencia Económica-CIDE Alejandro Canales Universidad Nacional Autónoma de México Ursula Casanova Arizona State University, Tempe, Arizona Sigfredo Chiroque Instituto de Pedagogía Popular, Perú Erwin Epstein Loyola University, Chicago, Illinois Mariano Fernández Enguita Universidad de Salamanca. España Gaudêncio Frigotto Universidade Estadual do Rio de Janeiro, Brasil Rollin Kent Universidad Autónoma de Puebla. Puebla, México Walter Kohan Universidade Estadual do Rio de Janeiro, Brasil Roberto Leher Universidade Estadual do Rio de Janeiro, Brasil Daniel C. Levy University at Albany, SUNY, Albany, New York Nilma Limo Gomes Universidade Federal de Minas Gerais, Belo Horizonte Pia Lindquist Wong California State University, Sacramento, California María Loreto Egaña Programa Interdisciplinario de Investigación en Educación Mariano Narodowski Universidad Torcuato Di Tella, Argentina Iolanda de Oliveira Universidade Federal Fluminense, Brasil Grover Pango Foro Latinoamericano de Políticas Educativas, Perú Vanilda Paiva Universidade Estadual Do Rio De Janeiro, Brasil Miguel Pereira Catedratico Universidad de Granada, España Angel Ignacio Pérez Gómez Universidad de Málaga Mónica Pini Universidad Nacional de San Martin, Argentina Romualdo Portella do Oliveira Universidade de São Paulo Diana Rhoten Social Science Research Council, New York, New York José Gimeno Sacristán Universidad de Valencia, España Daniel Schugurensky Ontario Institute for Studies in Education, Canada Susan Street Centro de Investigaciones y Estudios Superiores en Antropologia Social Occidente, Guadalajara, México Nelly P. Stromquist University of Southern California, Los Angeles, California Daniel Suarez Laboratorio de Politicas Publicas-Universidad de Buenos Aires, Argentina Antonio Teodoro Universidade Lusófona Lisboa, Carlos A. Torres UCLA Jurjo Torres Santomé Universidad de la Coruña, España