He, M Citation: Robinson, Daniel H. & Robert A. Bligh. (2012 April 4) Review of Uneducated Guesses: Using Evidence to Uncover Misguided Education Policies by Howard Wainer. Education Review, 15. Retrieved [Date] from http://www.edrev.info/reviews/rev1161.pdf April 4, 2012 ISSN 1094-5296 Education Review/Reseñas Educativas is a project of the National Education Policy Center http://nepc.colorado.edu Follow Education Review on Facebook Wainer, Howard. (2011). Uneducated Guesses: Using Evidence to Uncover Misguided Education Policies. Princeton, NJ: Princeton University Press. Pp. 200 ISBN 0691149283 Reviewed by Daniel H. Robinson University of Texas at Austin & Robert A. Bligh Omaha, Nebraska Howard Wainer writes in the preface to his newest book that he often found himself shouting at the television when proposed policies and the lack of evidence to support them were being broadcast. Wainer openly wonders what decisions were based on before the recent emphasis on “evidence-based decisions.” The title, Uneducated Guesses, accurately describes the book’s contents. The dedication embodies the academic fate of all children and the obligations of all parents. The book provides a model for the development of rational public education policies, something that America needs desperately. The introduction artfully employs education review // reseñas educativas editors: gene v glass gustavo e. fischman melissa cast-brede a multi-lingual journal of book reviews http://www.facebook.com/pages/Education-Review/178358222192644 Education Review http://www.edrev.info 2 a Richard Feynman quotation to support what the book is intended to accomplish: Reason is a tool that has been ignored (or intentionally hidden) by the purveyors of popular education policy. This is foolish; it is unethical; it hurts children; it wastes resources; and it must stop. It is easy to imagine Wainer sitting in front of his television armed with a foam brick. Wainer’s writing is impeccable, and his wit is sharp. He does not take on projects unless he is serious about them. This is a man who has literally swum the English Channel. His latest effort is yet another well-written, incredibly timely book that will improve our thinking about education policy. Not everyone will appreciate the book. Those who might hope that Wainer’s book will be a polemical attack on standardized tests such as the SAT will be disappointed. Using tests to make decisions that are sometimes unpopular has been common practice for some time. The earliest intelligence tests used in France in the 19 th Century by Binet were developed simply to identify those children who would not benefit from schooling. Armed Forces Placement Examinations were designed to place soldiers strategically according to their abilities (e.g., brighter individuals assigned to leadership positions). The SAT is designed to predict who will succeed in college. It is NOT designed to ensure equal performances by sex, race, and the like. As a predictor, as Wainer mentions, it is as good as high school GPA and takes much less time. Much of Wainer’s book is about challenging, with evidence, policies designed to open the admissions floodgates to admit persons who score poorly on college entrance exams. For example, in Chapter 1, he examines the policy of making college entrance exams optional, or even eliminating them. Where such a policy exists, students with low scores often choose not to submit them. Some of these students are admitted. Evidence shows that students with lower SAT scores perform worse in their first year than do those who submit the scores. The bottom line is that entrance exams provide us with useful information in terms of whether students will succeed in college. If we eliminate that information, we increase the risk of choosing to admit students who have a lower likelihood of success than others who were rejected. Is that a good thing? It depends on the relative costs of the two kinds of errors. Wainer argues that before making the Education Review http://www.edrev.info 3 choice we ought to be aware of the size of the potential errors and their associated costs. Permit us offer one view of how this seemingly noble effort can lead to disastrous consequences. One of us has a daughter who recently joined her high school tennis team as a freshman. She plays in tournaments as part of the junior varsity doubles team. Now, this may not seem surprising until you consider that she had NEVER played tennis before this year. No, she is not one of those kids who is a naturally gifted athlete and can quickly pick up any sport. Instead, she attends a small private high school where getting enough people to field a team can be challenging. This is in sharp contrast to the very large public high school that she could attend, where making the tennis team would indeed require years of tennis training and playing. What if, based on her impressive “starting” status as a freshman, her parents decide she should attend a famous tennis academy’s summer camp, in the hopes that she would be a star one day? Thanks to a new admission policy, the academy no longer requires applicants to submit their USTA ratings. The girl is accepted, the parents spend a small fortune in fees, and the summer experience is the worst of her life. She competes against other girls who have played for years and gets crushed in every match. The other girls are resentful that she was admitted to the camp based on her limited skills and she is ostracized. Was the policy of relaxed admissions standards beneficial to anyone involved? Wainer’s book does a nice job of uncovering the absurdities of such policies. Somewhat ironically, at the end of Chapter 4, he argues, “errors of placement should be more in the direction of aiming too high than too low.” Really? At what cost to those who do not make it? Is spending five years and thousands of dollars in tuition and other costs worth it if a student drops out of college and has nothing to show for it? Would the student have been better off being enrolled at a college with lower admissions standards? Balancing the costs of such errors against the putative advantages for those students who beat the odds and succeed requires both data and an understanding of the costs of both kinds of errors. In such a calculation we must also include the costs of excluding Education Review http://www.edrev.info 4 more qualified applicants, rejected because space was limited. In a similar attempt to lower admissions standards, some folks want colleges to consider scores on the SAT II achievement tests instead of the SAT I aptitude scores. In Chapter 2, Wainer discusses the difficulties of comparing student scores on achievement tests in different areas. He sums up the problem with the question “Was Einstein a better physicist than Mozart was a composer?” The difficulty of making such comparisons is confused because tests in disparate subjects like physics and French appear to be scored on the same scale. The complications of allowing students to choose to take achievement tests instead of an overall aptitude test are revealed. Again, suppose a student is admitted based on an extremely high achievement test score that has no correlation with college success. The student eventually drops out due to poor grades. Is this a good for the student, for society, for unknown third parties? Chapter 4 should be a special treat for those who love “teacher” movies. Wainer examines Jaime Escalante’s experiences teaching Advanced Placement calculus to students at Garfield High School in a ghetto in East Los Angeles as dramatized in the film “Stand and Deliver.” Teacher movies contain stories we can all relate to. They all have good (education and civilization) triumphing over evil (ignorance and chaos). They appeal to our desire to believe in the possibility of human salvation. “Stand and Deliver” might just be the best teacher movie of all time. Why? Ironically, because it is different from the others. There is no magic or magical thinking in it. Mr. Escalante got to recruit his students. The students remained in class only because they chose to. Mr. Escalante got the support of their families. He could demand Saturday classes. He could demand vacation classes. He could expel any student at any time. Mr. Escalante was a brilliant teacher. That means that when he was given students who showed up ready to learn, he taught them effectively. If Mr. Escalante had been given a cross section of Garfield students, does anyone think that he would have had similar success? There is absolutely no good reason to think so. Chapter 4 looks at Mr. Escalante’s success through objective eyes. He deserved that and America’s children deserve education policies that are consistent with that truth. Education Review http://www.edrev.info 5 In Chapter 6, Wainer tackles the issue of allowing examinees choice of which questions to answer. On the surface, this seems like a harmless policy and, moreover, in the words of an obviously nice person Wainer quotes it can “show examinees that we care.” Students should have that warm fuzzy feeling while taking tests. The problem is that when you give students choices, the students who would typically score the lowest also are more likely to choose to answer the more difficult items. On the other hand, students who score highest are also smart enough to choose the easier questions. Thus, group differences are increased when choice is allowed. This is yet another example of a seemingly well-intentioned policy that may backfire. The rest of the book is filled with similar policies and Wainer’s uncovering of evidence that challenges them. We won’t completely review all chapters here. But we will add a few personal reflections. Chapter 8 tells the story of a teacher who is accused of helping students to cheat on a state math test by employing the “point and look” method of proctoring. The proctor strolls around the room and when she notices a student doing something incorrectly, she points to the item and looks at the student. Obviously this method constitutes giving hints or clues—something forbidden according to the state’s rules. We won’t give away the ruling of the arbitration hearing, but the argument is worth studying. In Chapter 9, Wainer tackles the increasingly popular policy of using value-added models (VAM) to measure teachers’ and schools’ contributions to student learning. Again, this appears to be an objective way to evaluate teachers and allow us more fairly to reward good teachers and get rid of bad teachers. We agree that the teaching profession has numerous under-appreciated great teachers and also poor teachers who hang on forever. This same problem exists for just about any other profession, one must remember. The problem with using VAM is that they are fraught with problems that may make their use more dangerous than helpful. Wainer details three: (i) test scores unsuited for such a purpose, (ii) massive missing data, and (iii) the heroic untestable assumptions required to make the causal inferences that are at the heart of the Education Review http://www.edrev.info 6 use of value-added scores. Let us add another: regression to the mean. We know that extreme scores on a test are likely to contain more measurement error than scores in the middle (e.g., they don’t represent the student’s true score). So, if a student scores very high one year, that student may score lower the following year simply because the first score was a fluke, and not due to poor teacher performance. Similarly, a student may score rock bottom one year and amazingly higher the next year while having a lousy teacher. Such regressions to the mean have nothing whatsoever to do with the quality of teaching. Moreover, where is the “parent effect” in the equation? We know that things like poverty and parental involvement are also related to student achievement. Once VAM begins accounting for things that explain much more variance in student test scores than do teachers, we will begin to take them a bit more seriously. The fact that many education policies only make sense if you say them fast is an emerging theme in Wainer’s book. Our advice is to say them slowly, read the book, and then consider other policies on which we should unleash the evidence-based approach espoused by Wainer. About the Reviewers Daniel H. Robinson is Professor of Educational Psychology (Learning, Cognition and Instruction) at the University of Texas. His primary research interests involve evaluating the use of technology to enhance learning. He is the editor of Educational Psychological Review, 2006 to present, and serves on the editorial boards of Contemporary Educational Psychology, Educational Technology, Research, & Development, Journal of Educational Psychology, and the Journal of Experimental Education. Dr. Robinson has been named as one of the top 20 most published authors in educational psychology journals from 1991-1996, 1997-2002, and 1991-2002, (Contemporary Educational Psychology, 1998, 2004). Robert A. Bligh is retired from his position as General Counsel of the Nebraska Association of School Boards. His research interest involves the efficacy of the school reform efforts promoted by the Elementary and Secondary Education Act since its original adoption in 1965. He served as Assistant Professor at Doane College and was the first Agency Legal Education Review http://www.edrev.info 7 Counsel at the Nebraska Department of Education immediately following law school. He was author, editor and publisher of the Nebraska School Law Reporter between 1987 and 2003. Copyright is retained by the first or sole author, who grants right of first publication to the Education Review. Education Review/Reseñas Educativas is a project of the National Education Policy Center http://nepc.colorado.edu Editors Gene V Glass glass@edrev.info Gustavo Fischman fischman@edrev.info Melissa Cast-Brede cast-brede@edrev.info