Microsoft Word - v13n36.doc EDUCATION POLICY ANALYSIS ARCHIVES A peer-reviewed scholarly journal Editor: Sherman Dorn College of Education University of South Florida Copyright is retained by the first or sole author, who grants right of first publication to the Education Policy Analysis Archives. EPAA is published jointly by the Colleges of Education at Arizona State University and the University of South Florida. Articles are indexed in the Directory of Open Access Journals (www.doaj.org) and by H.W. Wilson & Co. Volume 13 Number 36 August 30, 2005 ISSN 1068–2341 What Do Klein et al. Tell Us About Test Scores in Texas?1 Laurence A. Toenjes United States Citation: Toenjes, L. A. (2005, August 30). What do Klein et al. tell us about test scores in Texas? Education Policy Analysis Archives, 13(36). Retrieved [date] from http://epaa.asu.edu/epaa/v13n36/. Abstract A paper appearing in this journal by Klein, Hamilton, McCaffrey and Stecher (2000) attempted to raise serious questions about the validity of the gains in student performance as measured by Texas’ standardized test, the Texas Assessment of Academic Skills (TAAS). Part of their analysis was based on the results of three tests which they administered to 2,000 fifth grade students in 20 Texas schools. Although Klein et al. indicated that the 20 schools were not selected in a way which would insure that they were representative of the nearly 3,000 Texas schools that enrolled fifth graders, generalizations based upon the results for those schools were nonetheless offered. The purpose of this short paper is to demonstrate just how unrepresentative the 20 schools used by Klein et al. actually were, and in so doing to cast doubt on certain of their conclusions. 1 The author wishes to thank Prof. Jon Lorence for numerous constructive suggestions. Accepted under the editorship of Sherman Dorn. Send commentary to Casey Cobb (casey.cobb@uconn.edu). Education Policy Analysis Archives Vol. 13 No. 36 2 This is a reply to the paper “What do Test Scores in Texas Tell Us?”, by Stephen P. Klein, Laura S. Hamilton, Daniel F. McCaffrey, and Brian M. Stecher (2000). Klein et al. (2000) presented numerous findings questioning the effectiveness of the Texas accountability system. They specifically challenged the validity of the Texas Assessment of Academic Skills (TAAS) test, required of all eligible students in the Texas public school system. Their conclusions were based on several sets of analyses. First they estimated the association between student test scores and student economic status. Other analyses examined the relationships between student performance on the TAAS and three different tests which they administered in the spring of 1997 to some 2,000 fifth graders in 20 schools, described only as “a mix of 20 urban and suburban schools in Texas.” Although results based on individual students were as anticipated, aggregate school-level findings indicated that high TAAS performance was largely unrelated to scores on standardized tests not used in the state accountability system. In addition, Klein et al. did not find the expected negative linear association between the percentages of disadvantaged children and test scores commonly reported in the research literature. Although Klein et al. suggested that the discrepancies “could be due to the unique characteristics of the 20 schools in our study or other factors,” they nonetheless concluded that their findings indicated that observed increases in Texas student academic performance reported by Grissmer et al. (2000) were highly suspect. However, analyses based on all Texas public schools reported in this note reveals that part of the Klein et al. findings were incorrect because the campuses selected for their study were not representative of the population of elementary schools. P ro p o rtio n o f S tu d en ts R ec eiv in g S u b sid ized L u n c h es F ig u re 1 . C a m p u s M ean M ath S c o res V ers u s P ro p o rtio n o f D is a d va n ta g ed S tu d en ts, T ex a s F ifth G ra d ers , 1 9 9 7 T A A S T es t Figure 1 (above) shows the relationship between student TAAS test scores and economic disadvantage based on Texas public schools which tested fifth-grade students in mathematics in the spring of 1997. To be consistent with Klein et al., only 1997 fifth grade TAAS mathematics scores are utilized. The results are shown for non-special education students whose test results were included in the state’s accountability rating system. The scatterplot in Figure 1 is based on 225,433 What Do Klein et al. Tell Us About Test Scores in Texas? 3 students attending 2,825 campuses. A few schools which tested fewer than 20 students were excluded from the analysis in order to obtain reliable estimates of average school performance. The test performance measure, shown on the vertical axis, is the simple average for each campus of the number of questions answered correctly. The horizontal axis represents the proportion of economically disadvantaged tested students as measured by enrollment in the federally-subsidized free or reduced-price lunch programs. Figure 2. Diagram appeared as Figure 5. In Klein, et al. (2000) Although campuses testing fewer than 20 students are omitted, the picture is intended to show the universe from which the twenty campuses used in the study by Klein, et al, were drawn. The corresponding scatterplot based on the 20 campuses analyzed by Klein et al. is reproduced in Figure 2 (above). When the figure from the Klein study is compared with the figure based on all Texas schools, it is apparent how unrepresentative those 20 schools were with respect to characterizing the relationship between performance on the fifth grade math TAAS test and the average proportion of students who were economically disadvantaged. Consequently, the Pearson correlation between average math performance and economic disadvantage based on the 2,825 campuses differs considerably from the correlation Klein et al. reported. Whereas their correlation was only 0.13, and positive, the correlation from the more comprehensive set of campuses shown is a -0.55. This latter figure has the expected large negative value, and is comparable to the correlation coefficients which they found between SES and the non-TAAS tests which they administered. Those correlations ranged between –0.66 to –0.76. Although there is a degree of heteroscedasticity in the figure based on the more representative set of schools, due perhaps to ceiling-effect in the low-poverty schools, there is no indication of the u-shaped association evident in Figure 2, based on the Klein et al. data. In sum, the results derived from the tests Stecher and Klein administered in 1997 to 2,000 students in 20 schools are highly unrepresentative of Texas elementary schools and should not be used to draw conclusions about the relationship between poverty and school performance in Texas. The analysis presented here supports the validity of the TAAS test. Additional, more comprehensive, school-level analyses examining the association between performance on the TAAS Education Policy Analysis Archives Vol. 13 No. 36 4 and other standardized tests are required to ascertain whether the Klein et al. cross-test findings based on 20 elementary campuses can be generalized. There is no intent here to suggest that the Texas accountability system is immune from criticism. Analyses of NAEP data have also raised questions pertaining to the validity of learning gains reported on Texas. To illustrate, even RAND Corporation researchers differ among themselves in their interpretation of the meaning of NAEP test results for Texas. Whereas Klein et al. (2000) contended NAEP findings showed the Texas accountability system is ineffective, Grissmer et al. (2000) suggested that the implementation of stricter state educational requirements in Texas helped improve student academic learning. The current note does not address this issue of validity (but see Toenjes and Dworkin 2002; Toenjes, Dworkin, Lorence and Hill 2002). My findings are more limited, but clearly demonstrate that the absence of a negative relationship between school- level test performance and economic disadvantage reported by Klein et al., based on 20 unrepresentative schools, does not support their claim that the TAAS is an invalid measure of student achievement. References Grissmer, D. W., Flanagan, A., Kawata, J., Williamson, S. (2000), Improving Student Achievement: What Do State NAEP Test Scores Tell Us, RAND, Santa Monica, Calif. Klein, Stephen P., Hamilton, Laura S., McCaffrey, Daniel F., and Stecher, Brian M (2000) “What do Test Scores in Texas Tell Us?”, Education Policy Analysis Archives, 8(49). Retrieved March 22, 2005 from http://epaa.asu.edu/epaa/v8n49/. Toenjes, L.A. and Dworkin, A.G. (2002). “Are increasing test scores in Texas really a myth, or is Haney's myth a myth?”, Education Policy Analysis Archives, 10(17). Retrieved March 22, 2005 from http://epaa.asu.edu/epaa/v10n17/. Toenjes, L.A., Dworkin, A.G., Lorence, J., Hill, A.N. (2002), “High-Stakes Testing, Accountability, and Student Achievement in Texas and Houston”, in Bridging the Achievement Gap, John E. Chubb and Tom Loveless, editors, Brookings, Washington, D.C. About the Author Laurence A. Toenjes Email: ltoenjes@aol.com Laurence A. Toenjes was Research Associate Professor of Sociology and Co-Founder of the Sociology of Education Research Group (SERG) at the University of Houston. He holds a Ph.D. degree in Economics from Southern Illinois University. Retired three years ago, Toenjes recently completed a sailing voyage from Texas to Seattle by way of Hawaii. Interestingly, he noticed the weekly posting of local school testing results on the bulletin board at the small Keaukaha Market near the harbor in Hilo during his stay there. There is no escape. What Do Klein et al. Tell Us About Test Scores in Texas? 5 EDUCATION POLICY ANALYSIS ARCHIVES http://epaa.asu.edu Editor: Sherman Dorn, University of South Florida Production Assistant: Chris Murrell, Arizona State University General questions about appropriateness of topics or particular articles may be addressed to the Editor, Sherman Dorn, epaa-editor@shermandorn.com. Editorial Board Michael W. Apple University of Wisconsin David C. Berliner Arizona State University Greg Camilli Rutgers University Casey Cobb University of Connecticut Linda Darling-Hammond Stanford University Mark E. Fetler California Commission on Teacher Credentialing Gustavo E. Fischman Arizona State Univeristy Richard Garlikov Birmingham, Alabama Gene V Glass Arizona State University Thomas F. Green Syracuse University Aimee Howley Ohio University Craig B. Howley Appalachia Educational Laboratory William Hunter University of Ontario Institute of Technology Patricia Fey Jarvis Seattle, Washington Daniel Kallós Umeå University Benjamin Levin University of Manitoba Thomas Mauhs-Pugh Green Mountain College Les McLean University of Toronto Heinrich Mintrop University of California, Berkeley Michele Moses Arizona State University Anthony G. Rud Jr. Purdue University Michael Scriven Western Michigan University Terrence G. Wiley Arizona State University John Willinsky University of British Columbia Education Policy Analysis Archives Vol. 13 No. 36 6 EDUCATION POLICY ANALYSIS ARCHIVES English-language Graduate-Student Editorial Board Noga Admon New York University Jessica Allen University of Colorado Cheryl Aman University of British Columbia Anne Black University of Connecticut Marisa Burian-Fitzgerald Michigan State University Chad d'Entremont Teachers College Columbia University Carol Da Silva Harvard University Tara Donahue Michigan State University Camille Farrington University of Illinois Chicago Chris Frey Indiana University Amy Garrett Dikkers University of Minnesota Misty Ginicola Yale University Jake Gross Indiana University Hee Kyung Hong Loyola University Chicago Jennifer Lloyd University of British Columbia Heather Lord Yale University Shereeza Mohammed Florida Atlantic University Ben Superfine University of Michigan John Weathers University of Pennsylvania Kyo Yamashiro University of California Los Angeles What Do Klein et al. Tell Us About Test Scores in Texas? 7 Archivos Analíticos de Políticas Educativas Associate Editors Gustavo E. Fischman & Pablo Gentili Arizona State University & Universidade do Estado do Rio de Janeiro Founding Associate Editor for Spanish Language (1998—2003) Roberto Rodríguez Gómez Editorial Board Hugo Aboites Universidad Autónoma Metropolitana-Xochimilco Adrián Acosta Universidad de Guadalajara México Claudio Almonacid Avila Universidad Metropolitana de Ciencias de la Educación, Chile Dalila Andrade de Oliveira Universidade Federal de Minas Gerais, Belo Horizonte, Brasil Alejandra Birgin Ministerio de Educación, Argentina Teresa Bracho Centro de Investigación y Docencia Económica-CIDE Alejandro Canales Universidad Nacional Autónoma de México Ursula Casanova Arizona State University, Tempe, Arizona Sigfredo Chiroque Instituto de Pedagogía Popular, Perú Erwin Epstein Loyola University, Chicago, Illinois Mariano Fernández Enguita Universidad de Salamanca. España Gaudêncio Frigotto Universidade Estadual do Rio de Janeiro, Brasil Rollin Kent Universidad Autónoma de Puebla. Puebla, México Walter Kohan Universidade Estadual do Rio de Janeiro, Brasil Roberto Leher Universidade Estadual do Rio de Janeiro, Brasil Daniel C. Levy University at Albany, SUNY, Albany, New York Nilma Limo Gomes Universidade Federal de Minas Gerais, Belo Horizonte Pia Lindquist Wong California State University, Sacramento, California María Loreto Egaña Programa Interdisciplinario de Investigación en Educación Mariano Narodowski Universidad Torcuato Di Tella, Argentina Iolanda de Oliveira Universidade Federal Fluminense, Brasil Grover Pango Foro Latinoamericano de Políticas Educativas, Perú Vanilda Paiva Universidade Estadual Do Rio De Janeiro, Brasil Miguel Pereira Catedratico Universidad de Granada, España Angel Ignacio Pérez Gómez Universidad de Málaga Mónica Pini Universidad Nacional de San Martin, Argentina Romualdo Portella do Oliveira Universidade de São Paulo Diana Rhoten Social Science Research Council, New York, New York José Gimeno Sacristán Universidad de Valencia, España Daniel Schugurensky Ontario Institute for Studies in Education, Canada Susan Street Centro de Investigaciones y Estudios Superiores en Antropologia Social Occidente, Guadalajara, México Nelly P. Stromquist University of Southern California, Los Angeles, California Daniel Suarez Laboratorio de Politicas Publicas-Universidad de Buenos Aires, Argentina Antonio Teodoro Universidade Lusófona Lisboa, Carlos A. Torres UCLA Jurjo Torres Santomé Universidad de la Coruña, España