EPAA Vol. 10 No. 9 Willson & Kellow: Confusing the Messenger ... 1 of 4 Education Policy Analysis Archives Volume 10 Number 9 January 28, 2002 ISSN 1068-2341 A peer-reviewed scholarly journal Editor: Gene V Glass College of Education Arizona State University Copyright 2002, the EDUCATION POLICY ANALYSIS ARCHIVES . Permission is hereby granted to copy any article if EPAA is credited and copies are not sold. Articles appearing in EPAA are abstracted in the Current Index to Journals in Education by the ERIC Clearinghouse on Assessment and Evaluation and are permanently archived in Resources in Education. Confusing the Messenger with the Message: A Response to Bolon Victor L. Willson Texas A&M University Thomas Kellow University of Houston Citation: Willson, V.L. & Kellow, T. (2002, January 28). Confusing the messenger with the message: A response to Bolon. Education Policy Analysis Archives, 10(9). Retrieved [date] from http://epaa.asu.edu/epaa/v10n9/. Abstract The conclusions by Bolon (2001) based on the relationship between per capita income and school mean grade 10 mathematics scores in Massachusetts and on instability in year-to-year mean school scores are criticized by us. Our concerns focus on the uninterpretable covariation of economic condition with test performance and the limitations in interpreting cross-time variability. We agree with Bolon's conclusions but consider the methodology employed inadequate to support them. We suggest alternative requirements and discuss our own previous efforts in this area. 2 of 4 In an analysis of the Massachusetts graduation examination, Bolon (2001) examined the aggregate grade 10 mathematics test scores for 47 high schools and the demographic characteristics of the communities in which they were situated. From several data analyses, Bolon determined that since the best single predictor of mean high school score was community per capita income, "The state is treating scores and ratings as though they were precise educational measures of high significance. A review of thenth-grade mathematics test scores from academic high schools in metropolitan Boston showed that statistically they are not." Further, when removing the variability due to per capita income, "Large uncertainties in residuals of school-averaged scores, after subtracting predictions based on community income, tend to make the scores ineffective for rating performance of schools. Large uncertainties in year-to-year score changes tend to make the score changes ineffective for measureing performance trends." While we agree with Bolon's concerns, on the whole, we find little support in the evidence he presents to support them. Our discussion below details our concerns. Predicting aggregate test scores One of the problems with regression analysis is that without reasonable theoretical support, all sorts of predictors can be found that produce high correlation. In examining aggregate scores, such as high school test means, it is no secret that for many decades, as Bolon himself pointed out (Bolon, 2000), achievement has been associated with socioeconomic conditions in communities. In earlier eras, when school spending was much more unequal, these differences were more indicative of opportunity to learn for students. In a judicial climate that has tended to minimize, although not eliminate such disparities, it is much less persuasive, although it remains an important area for study. The difficulty with using a community aggregate measure as a predictor is that it is a surrogate for many other indicators, some of which are absurd at face value but interpretable. Variables such as driver's-license passing rate or per capita champagne consumption may predict student achievement as well as community per capita income. We can construct meaningful arguments why they might. For none is the test invalidated using accepted standards (American Educational Research Association, American Psychological Association, and National Council on Measurement in Education, 1999). In other areas of research such aggregation has produced fundamentally misleading conclusions. For example, the literature on intelligence and income is directly parallel to the discussion here. White (1982) demonstrated the difference between using an aggregate measure of SES (school or community) and individual measure in relating SES to intellectual functioning. Since Bolon used school as his unit of analysis, he eliminated proximate measures more appropriate to his analysis. The school-level variables Bolon eliminated are more appropriate than community per capita income on 3 of 4 Copyright 2002 by the Education Policy Analysis Archives The World Wide Web address for the Education Policy Analysis Archives is epaa.asu.edu General questions about appropriateness of topics or particular articles may be addressed to the Editor, Gene V Glass, glass@asu.edu or reach him at College of Education, Arizona State University, Tempe, AZ 85287-2411. The Commentary Editor is Casey D. Cobb: casey.cobb@unh.edu . EPAA Editorial Board Michael W. Apple University of Wisconsin Greg Camilli Rutgers University John Covaleskie Northern Michigan University Alan Davis University of Colorado, Denver Sherman Dorn University of South Florida Mark E. Fetler California Commission on Teacher Credentialing Richard Garlikov hmwkhelp@scott.net Thomas F. Green Syracuse University Alison I. Griffith York University Arlen Gullickson Western Michigan University Ernest R. House University of Colorado Aimee Howley Ohio University Craig B. Howley Appalachia Educational Laboratory William Hunter University of Calgary Daniel Kallós Umeå University Benjamin Levin University of Manitoba Thomas Mauhs-Pugh Green Mountain College Dewayne Matthews Education Commission of the States William McInerney Purdue University Mary McKeown-Moak MGT of America (Austin, TX) Les McLean University of Toronto Susan Bobbitt Nolen University of Washington Anne L. Pemberton apembert@pen.k12.va.us Hugh G. Petrie SUNY Buffalo Richard C. Richardson New York University Anthony G. Rud Jr. Purdue University Dennis Sayers California State University—Stanislaus Jay D. Scribner University of Texas at Austin Michael Scriven scriven@aol.com Robert E. Stake University of Illinois—UC Robert Stonehill U.S. Department of Education David D. Williams Brigham Young University 4 of 4 EPAA Spanish Language Editorial Board Associate Editor for Spanish Language Roberto Rodríguez Gómez Universidad Nacional Autónoma de México roberto@servidor.unam.mx Adrián Acosta (México) Universidad de Guadalajara adrianacosta@compuserve.com J. Félix Angulo Rasco (Spain) Universidad de Cádiz felix.angulo@uca.es Teresa Bracho (México) Centro de Investigación y Docencia Económica-CIDE bracho dis1.cide.mx Alejandro Canales (México) Universidad Nacional Autónoma de México canalesa@servidor.unam.mx Ursula Casanova (U.S.A.) Arizona State University casanova@asu.edu José Contreras Domingo Universitat de Barcelona Jose.Contreras@doe.d5.ub.es Erwin Epstein (U.S.A.) Loyola University of Chicago Eepstein@luc.edu Josué González (U.S.A.) Arizona State University josue@asu.edu Rollin Kent (México) Departamento de Investigación Educativa-DIE/CINVESTAV rkent@gemtel.com.mx kentr@data.net.mx María Beatriz Luce (Brazil) Universidad Federal de Rio Grande do Sul-UFRGS lucemb@orion.ufrgs.br Javier Mendoza Rojas (México) Universidad Nacional Autónoma de México javiermr@servidor.unam.mx Marcela Mollis (Argentina) Universidad de Buenos Aires mmollis@filo.uba.ar Humberto Muñoz García (México) Universidad Nacional Autónoma de México humberto@servidor.unam.mx Angel Ignacio Pérez Gómez (Spain) Universidad de Málaga aiperez@uma.es Daniel Schugurensky (Argentina-Canadá) OISE/UT, Canada dschugurensky@oise.utoronto.ca Simon Schwartzman (Brazil) Fundação Instituto Brasileiro e Geografia e Estatística simon@openlink.com.br Jurjo Torres Santomé (Spain) Universidad de A Coruña jurjo@udc.es Carlos Alberto Torres (U.S.A.) University of California, Los Angeles torres@gseisucla.edu