Education Policy Analysis Archives: Vol. 4 No. 1 Stedman "Rev... 1 of 12 Education Policy Analysis Archives Volume 4 Number 1 January 23, 1996 ISSN 1068-2341 A peer-reviewed scholarly electronic journal. Editor: Gene V Glass,Glass@ASU.EDU. College of Education, Arizona State University,Tempe AZ 85287-2411 Copyright 1996, the EDUCATION POLICY ANALYSIS ARCHIVES.Permission is hereby granted to copy any article provided that EDU POLICY ANALYSIS ARCHIVES is credited and copies are not sold. The Achievement Crisis is Real: A Review of The Manufactured Crisis Lawrence C. Stedman State University of New York-Binghamton stedman@bingsuns.cc.binghamton.edu Abstract: In a provocative new book, The Manufactured Crisis, David Berliner and Bruce Biddle make four sweeping claims about U.S. achievement: there never was a test score decline, today's students are "out-achieving their parents substantially" (p. 33), U.S. students "stack up very well" in international assessments (p. 63), and the general education crisis is a right-wing fabrication. As a progressive, I'm sympathetic to their concerns, but as a scholar who specializes in this material, I find their analysis deeply flawed and misleading. They mischaracterize the test score decline data, mishandle the international findings, and fail to acknowledge students' continuing low levels of academic achievement. The Decline Although Berliner and Biddle are generally right that achievement has been stable, they ignored important contradictory evidence and the 1970s decline. They claimed "only 'one' test, the SAT" ever suggested a decline (p. 35). This is remarkable. High school students' NAEP civics scores, for example, dropped substantially between 1969 and 1976 and have been slipping ever since. Their science scores also fell during the 1970s and have only partly rebounded. Several commercial tests, such as CTBS and STEP, showed declines in the 1970s. In the late 1980s, senior high school reading scores declined on the MAT while reading and math scores fell in 2 of 12 many grades on the SRA (Linn, Graue, & Sanders, 1990). In the late 1980s, younger students' NAEP reading and writing performance slipped. (For details, see Stedman & Kaestle, 1991; Stedman, 1993.) They attributed the SAT decline to demographic changes in test takers, yet never reviewed the evidence which shows this explains much, but not all, of the decline. They used "average" SAT scores to claim minority gains, but this masked minority verbal declines in the late 1970s and late 1980s (Stedman, 1994b). Mexican-American, Puerto-Rican, and Asian American verbal scores were about the same in the early 1990s as they were in 1976. Berliner and Biddle made sweeping claims about recent gains on commercial tests. Their handling of the Linn, Graue and Sanders study demonstrates how selective they are with evidence. Their graph omitted Linn, Graue and Sanders' SRA data which showed declines in many grades. They only graphed the elementary school data, which hid the less impressive high school scores, some of which were declining or stagnating. They never mentioned that Linn, Graue and Sanders pondered, "But the more important question is: Has student 'achievement' improved in recent years?" and concluded that the answer was "equivocal" (Linn, Graue & Sanders, 1990, p. 13). Linn, Graue and Sanders determined that recent gains were partly caused by districts' repeated use of the same tests rather than by genuine improvement. The 1980s back-to- basics movements also artificially raised scores by frequent testing and skill-drill approaches (Stedman & Kaestle, 1991). Finally, Berliner and Biddle claimed "virtually all" commercial tests would "show that today's students are out- achieving their parents substantially" (p. 33), yet never presented any evidence to support their claim. They ignored the many reviews of historical trends on equating studies which refute their claim (Stedman & Kaestle, 1987). The best that can be concluded is that this generation of students "generally" performs about the same as earlier ones, but the patterns are complicated and there is contradictory evidence. Given changing school populations and societal conditions, generally stable scores are still a remarkable accomplishment for U.S. schools. This is an important message that the public needs to hear. Nevertheless, the reality is more complicated than they suggested. Although school critics often exaggerated the extent and ramifications of the declines, many did occur (Stedman and Kaestle, 1991). Berliner and Biddle should have admitted that, on several indicators, our students are not performing as well as they once did. International Assessments U.S. performance in the international arena is not as dismal as school critics have asserted, but it certainly is not as glowing as Berliner and Biddle claim. Our students have done well in reading and elementary school science, middling to poor in geography and secondary school science, and last or near-last in mathematics (Stedman, 1994b). Berliner and Biddle offered several arguments to try to explain the weak U.S. performance but, in doing so, they tacitly acknowledged that our international performance often has been poor. Opportunity-to-Learn Berliner and Biddle's opportunity-to-learn argument is a red herring. International researchers pioneered the use of OTL measures and it is already factored into many results. ETS's 1988 international math and science findings, for example, came only from schools in which "more than 75 percent of the students had already had an opportunity to learn the content" (Lapointe et 3 of 12 al., 1989, p. 33). Even so, the U.S. did poorly whether judged by rankings, proficiency levels, or percentage correct. Berliner and Biddle claimed that our students are at a disadvantage because we generally delay algebra until 9th or 10th grade. But U.S. students have done poorly in most math areas, not just algebra. In 1988, for example, our 13-year-olds ranked last in arithmetic and measurement and next-to-last in geometry, data organization, and problem solving (NCES, 1991, p. 395). They also had poor results in 1991 in these areas (NCES, 1992, p. 21). U.S. and Japanese curricula were also more comparable than claimed. In the Second International Mathematics Study, content coverage was similar in arithmetic, geometry, and statistics, yet U.S. students still scored lower (Stedman, 1994a). In a telling analysis, Baker (1993) found that when one considers "only" the test items that U.S. 8th graders were taught during the year, they averaged only 40% correct. Westbury Study The Westbury study was at the heart of their curricular claims, but their handling of it revealed they care more about their argument than the evidence. First, the study has limited implications because it used data that were over a decade old, dealt only with one subject--math, and involved a better-than- usual U.S. 8th grade performance. Second, they did not even report Westbury's comparisons properly. They took his scores for our most advanced 8th grade math classes--the top 25% comprising algebra and pre-algebra--and compared them to the "average" Japanese class! No wonder our algebra classes looked good in their comparison. What Westbury actually did was compare our most advanced 8th grade math classes to the top fifth of Japanese students. Although this was a fairer approach, it still did not "isolate" the effects of the curriculum, but confounded them with selection effects. U.S. students who study algebra in 8th grade are a select group of 14%, differing from other U.S. students in college expectations, math interest, parental support, social class, and academic ethic. Consequently, one cannot tell how much of their performance reflects their algebra curriculum and how much their background advantages. (Using this comparison directly violated their own research precept--the Principle of Control, p. 159.) What did Westbury actually find? Our select students did not do that well. Our pre-algebra classes scored only 56% correct and lagged well behind, by a substantial two standard deviations (Westbury, 1992, p. 21). Our algebra classes scored comparably to the Japanese classes, but this was hardly surprising. They were an elite group of only 14% of our classes compared to a less select 20% of the Japanese students. They were judged only on the algebra portion of the test, yet they had spent more of their time on algebra (formulas and equations), 61% to 26%, and had covered more of the test problems, 88% to 82%, than the Japanese students (Westbury, 1992, p. 20, p. 21). (So much for claims that curricula were equated!). In two other test areas, geometry and measurement, they even scored below the "average" Japanese class (Stedman, 1994a). Finally, our 8th graders were older and had been in school longer--the Japanese students were only 7th graders! Berliner and Biddle ignored Westbury's analysis of U.S. calculus classes, yet this tested the overall quality of our best math programs given to our best students. Our calculus classes fared poorly, however, substantially trailing the "average" Japanese class in every tested area (Stedman, 1994a). Given all this, it was misleading for them to claim that "U.S. teachers and schools are [not] deficient compared with those in Japan" (p. 56) and to conclude that "Many, 4 of 12 perhaps most, of the studies' results were generated by differences in curricula" (p. 63). Variability Argument Berliner and Biddle tried to explain away poor U.S. international performance by claiming our achievement is "a 'lot' more variable" (p. 58) than other countries, but offered no evidence. In fact, the 1991 IAEP math and science studies showed our variability was similar to that of other nations and less than that of Taiwan and Korea, the leading performers (cf. 10th & 90th percentiles, NCES, 1993b, p. 56; NCES, 1993a, p. 415). States-to-Nations Comparison They never mentioned that the states-to-nations comparison they cited was designated "experimental" and technically problematic (see caution, NCES, 1993b, pp. 54, 94). The international scores were projections from a U.S. sample that took both the NAEP and IAEP tests. No international student ever took the NAEP test and it is unclear that the IAEP-NAEP relationship would be the same for students in other countries. Our states had two important advantages. Our students were older--over half were 14-15 years old whereas the international students were 13-year- olds. Our states' scores came from the 1992 NAEP assessment and were higher than what was projected for the U.S. (cf. NCES, 1993c, p. 83; NCES, 1993b, p. 56). Finding that a few select, typically high-scoring mid-Western states did well in the comparison is not surprising. What is staggering is that our best state scores were only the "average" level in Taiwan and Korea! Berliner and Biddle did not report that the same comparison showed that the typical U.S. student was two years behind the average Taiwanese student and scored only around Taiwan's and Korea's 25th percentile (NCES, 1993b, pp. 54, 56). It also showed that only 13-16% of U.S. students reached the proficient level, while 35-43% of Taiwanese and Korean students did (Pashley & Phillips, 1993). Social Inequality Argument Although racism and social inequality have taken a severe toll on many of our students' academic development, this does not explain the poor general performance of U.S. students. The math deficit, for example, is not simply a minority student problem. In 1992, only 30% of "white" U.S. 8th graders demonstrated proficiency in the NAEP math assessment; over a quarter did not even make the basic level (NCES, 1993c, pp. 101-102). Nor are our problems due to low-achievers. Even our top half have not kept pace internationally in math and science (Stedman, 1994a). Although U.S. students do not generally fail in international comparisons, it is misleading for Berliner and Biddle to claim that "they stack up very well" (p. 63). Low Achievement The book's central problem is that Berliner and Biddle tell only part of the story. Although achievement trends, for the most part, have been stable, academic and general knowledge have been at low levels for decades (Stedman, 1993). In math, NAEP analysts recently concluded that "less than half (of high school seniors) appeared to have a firm grasp of seventh-grade content" (Mullis et al., 1991, p. 80). They have trouble even with simple problems involving fractions, decimals, and percents. 5 of 12 Few high school students have done well on NAEP writing tests. Only about a third wrote adequate papers and only a small percentage could write "elaborated" papers. The one bright spot is their competence in basic grammar and punctuation. Our functional illiteracy rate remains around 20-30%--meaning that millions of adults have trouble with common day-to-day reading tasks (Stedman & Kaestle, 1987; Kirsch, 1993). Students lack basic knowledge in history and literature. In the late 1980s, substantial majorities of our 17-year-olds did not recognize that Upton Sinclair was a muckraker, the Scopes trial dealt with evolution, Jim Crow laws segregated blacks, or the time period of the Civil War. A majority did not recognize classics by Shakespeare, Chaucer, Conrad, and Whitman and were unfamiliar with major women and African-American writers. These were straight- forward multiple-choice questions deliberately designed without the usual distractors. Geographical knowledge also has often been poor. In 1988, Gallup repeated a survey given to adults in 1947 and concluded that "Americans' geographic literacy has gotten worse in the last forty years." They found that, "From outline maps, the average American can identify only four of twelve European countries, less than three of eight South American countries, and less than six of ten U.S. states" (Gallup Organization, 1989, p. 162). Rejoinders Instead of reviewing and acknowledging this evidence, Berliner and Biddle offer several rejoinders why such findings don't matter. They suggest that the standards for knowledge are unrealistic and are those of classicists, historians, and test designers. Most people, however, would expect high school seniors to be competent in 7th grade math, literacy, and basic social studies information. Breadth of Experience They argue that U.S. students are focused on a breadth of experience, but this does not excuse our low achievement. Certainly academic achievement is one of our goals and should be one of our strengths. Nor is it clear that U.S. students have a monopoly on breadth or richness of experience. Portraits of Japanese elementary schools clearly show that students are not academic automatons, but are engaged in rich curricular and extra- curricular activities--calligraphy, sewing, hands-on math and science activities, group problem-solving, electronics, dance, musical training, play, reading, physical exercise, cooperative learning, school jobs, etc. (Stevenson & Stigler, 1992). Scaling Problems They rightly argue NAEP scales are flawed, but this does not explain students' poor performance or limited knowledge. Contrary to their assertions, it doesn't require tough questions to generate scale scores or discriminate among U.S. students. The problems at the highest NAEP levels are actually fairly easy. The 300 level in math, for example, includes simple decimal problems and level 350 has "routine problems involving fractions and percents." This is junior high general math, yet 17-year-olds have trouble with it! In history, many 350 level problems required nothing more than simple recognition of basic facts. (For a more detailed look at NAEP findings as well as its scaling problems, see Stedman, 1993.) 6 of 12 Many findings of low performance do not come from traditionally scaled tests. The writing results involve authentic holistic evaluations and thus avoid the scaling problems. The functional illiteracy estimate came from tests of many different designs and was derived through a systematic analysis of individual items not scaled results. The true rate might even be higher because some tests used items that were easier than their real-life counterparts and did not test dropouts, the homeless, prisoners, or non-English speakers (Stedman and Kaestle, 1987). Low levels of civic literacy and general knowledge were revealed in national surveys as well as standardized tests. Details of Low Achievement Careful reviews of individual items and sets of items have avoided many scaling problems and still indicate students struggle with basic material (Carpenter et al, 1988, 1982). Math educators found that students "exhibit serious gaps in their knowledge" and often learn "concepts and skills at a superficial level" They concluded that "students' achievement at all age levels shows major deficiencies" (Carpenter et al, 1988, pp. 40-41). In 1990, for example, only around half the 17-year-olds could convert a decimal to a fraction, find a number given a percent, estimate a square root, and use the properties of triangles (Mullis et al, 1991, pp. 302-309). 34% could not even find the area of a rectangle, given a diagram and the length of two sides (Mullis et al, 1991, p. 306). Although students' geographical knowledge is better than many have asserted, there still are serious problems (Stedman, 1993). 15-40% of high school students had trouble with basic geographical material. Most could not interpret a graph showing birth and death rates. Given the Vietnam War, it is unsettling that 63% of our high school seniors could not locate Southeast Asia on a world map. 64% did not know Saudi Arabia's location, although this was before the Persian Gulf War. Half could not answer such simple questions as the following: The construction of the Panama Canal shortened the sailing time between New York and [London, Port-au-Prince, Rio de Janeiro, San Francisco] Functional literacy tests have produced some disturbing findings. Twenty percent of the population, for example, had trouble reading and understanding dosage information on medicine bottles. Similar percentages had problems with a housing inspection notice, basic coupons, and price per unit weight. About a third failed at figuring out train schedules, how much change should come from a purchase, and which subjects had improved on a report card. Student achievement may be even worse than these findings suggest. The NAEP data do not include dropouts who presumably would score lower. To reach a given NAEP level, students only have to answer correctly 65-80% of its problems. The burden on students is light. Compared to the SATs and achievement tests, which can be half-day or all-day affairs, the NAEP tests are short, only 45 minutes. The tests are predominantly multiple choice, recognition-based rather than open-ended, recall which make them easier for students to do well on. Real-World Relevance Berliner and Biddle argue that findings about low achievement are irrelevant because the tests did not measure real-world problem solving. This is a curious position given that their claims about stable achievement trends came from these same tests! There are several problems with their argument. First, many tests that showed low achievement did measure the knowledge and skills needed in the real world. The functional literacy tests, for example, used real-world tasks 7 of 12 with real-world materials. Math tests have involved calculators. graphing, and open-ended items. NAEP reading tests have used poetry, newspaper articles, and passages from real literature. Second, in-school and out-of-school tasks, although different in many ways, still involve related abilities. Standardized tests give some indication of real-world problem solving ability. One indication of this is the marked correlation between scores on traditional tests and those on authentic assessment measures (Wang, Haertel, and Walberg, 1993, p. 371). Third, "real-world problem-solving" is not our only educational goal. General knowledge, some of which can be measured successfully via multiple-choice testing, is an important goal in itself. We want informed and knowledgeable citizens. Historical knowledge can play a central role in understanding public policy debates. Consider the on-going and highly-charged debates over immigration policy and affirmative action. How can we expect students and young adults to make informed appraisals of the arguments when they are ignorant about the history of race relations in this country? NAEP testing in the 1980s showed that the vast majority of high school students did not know what Jim Crow laws were, what the 3/5ths Compromise was, or what the Emancipation Proclamation actually did. Substantial percentages did not know what Plessy v. Ferguson or Brown v. Board of Education were about. They lacked basic information pertaining to the Civil War, one of our nation's epochal events and a key force in shaping race relations. Sizable majorities were unfamiliar with the Missouri Compromise, nullification, the Dred Scott decision, the dates of Civil War, and the dates of Lincoln's term. Such ignorance is not an artifact of an obscure psychometric scaling procedure. Knowledge does matter. Fourth, there likely will be little comfort in results from more authentic, real-world testing. NAEP is increasingly using performance assessment--the new science test, for example, includes drawing tasks, writing, and open-ended questions. The new reading assessment has longer passages and is 40% open-ended. Students, however, often do more poorly on the open-ended versions of test items. When their understanding of a subject is probed, surprising gaps and confusions often appear (Bridgeman, 1992; Martinez, 1991; NAEP, 1983, p. 32; Rogers & Stevenson, 1988). Future assessments are likely to produce even more disturbing news about low achievement than we have now. Finally, Berliner and Biddle argue that school critics focused more on the imagined economic consequences of low achievement than on the actual achievement evidence. I agree. Soon after the "Nation at Risk" report appeared, I argued that it made too much of a high-skilled, hi-tech future economy as a rationale for reforming education (Stedman and Smith, 1983). But the actual evidence is troubling and Berliner and Biddle did not examine it. The low levels of achievement are unimpressive results for 12 years of schooling. The tests do measure much of what is being taught in our schools and show we are not succeeding in our efforts. A complex, democratic society needs a well-read and knowledgeable citizenry and yet the evidence shows we are not accomplishing this. Teaching Methods and Student Work Habits Our achievement problems are deep-seated, widespread, and long-standing. But this is not the only reason for fundamental and far-reaching school reform. Teaching methods and student work habits also leave much to be desired (Stedman, 1993). Although there are a few bright spots, such as the frequent use of demonstrations in science classes, the portrait is troubling. NAEP analysts found math instruction 8 of 12 "continues to be dominated by teacher explanations, chalkboard presentations, and reliance on textbooks and workbooks. More innovative forms of instruction--such as those involving small group activities, laboratory work, and special projects--remain disappointingly rare." (Dossey et al, 1988) History and civics classes are dominated by textbooks, tests, quizzes, and short-answer questions. It is unusual to find students working in groups or writing long papers. Writing instruction in the schools is also limited and is focused on mechanics. Only about a fourth of 8th graders report that their teachers spend more than an hour a week on writing. Interest in science has not been sparked. In 1986, fewer than a fourth of 11th graders reported working on science-related hobbies or talking with friends about science. Only about a third reported going to a science museum or trying to fix something electrical or mechanical. Students do little schoolwork. The data on homework and TV watching are revealing. In 1990, only about a third of our 17- year-olds reported spending over an hour a day on homework, whereas half reported watching 3 or more hours of TV daily! Reading has been shortchanged. In 1986, over half the 11th graders reported reading on their own less than once a week; about a fifth reported they never did! One cannot look over this information without a sense that our schools are not what they should be. Over the past decade, thought-provoking ethnographies and school profiles by Boyer, Fine, Goodlad, Oakes, Sizer, and others have portrayed a school system in crisis. What we're seeing, particularly at the high school level, is that students are often disengaged, teachers' work is often factory-like, and intellectual life is often poor. These accounts were hardly the products of right-wing ideologies (cf. Berliner & Biddle, Chapter 4, pp. 140-141). Reformers have been busy. They know that the schools are not better than ever, but rather, more than ever, they need to be different than they are. Teachers and other educators who are intimately involved in the life of schools recognize there is a serious problem. There are major reform efforts affecting every major aspect of education: curriculum, evaluation, funding, governance, pedagogy, and school organization. Local educators are not mere pawns in a conservative political chess game, but have been responding actively to real needs and problems. The Scope of Reform Fixing the schools is a crucial part of solving our long- standing academic problems. But we also need to create a society that values scholarship and learning over commercialism and entertainment. This will require a major political and economic transformation. Educators must challenge the vested interests that are more interested in profits than the welfare of communities and civil society. We must fight the economic displacements that disrupt families, produce violence, and undermine students' development. We must take on the media conglomerates that are focused more on selling products than nurturing our cultural and intellectual life. We must change a system that values the bombastic broadsides of radio talk show hosts and political candidates over reasoned and civil discourse. To succeed in our most troubled communities, we will need to overhaul school financing systems and break down powerful, entrenched bureaucracies. But school reform is no substitute for job creation, income redistribution, and political empowerment. We must make our educational efforts part of a broader social and political agenda, one that promotes full employment, 9 of 12 community revitalization, and civic participation. Conclusion In the 1980s, school critics often exaggerated the size and extent of the test score decline. In spite of enormous changes in society and school populations, U.S. achievement has been remarkably stable for many decades. But it remains inadequate and at low levels. Ignoring this evidence or arguing it is a right- wing fabrication hampers much needed school reform. The crisis is real, what is actually being manufactured here is a new mythology about U.S. student achievement. Interested readers can find an in-depth--and balanced-- treatment of the achievement evidence in several of my recent articles. I examine the NAEP data in the November 1993 Phi Delta Kappan, the Sandia Report and the SAT data in the January-February 1994 Journal of Educational Research, and the international data in the October 1994 Educational Researcher. References Baker, D. P. (1993). Compared to Japan, the U.S. is a low achiever Really. Educational Researcher, 22(3), 18-20. Berliner, D.C. and Biddle, B.J. (1995). The Manufactured Crisis: Myths, Fraud, and the Attack on America's Public Schools. Reading, MA: Addison-Wesley. Bridgeman, B. (1922, Fall). A comparison of quantitative questions in open-ended and multiple-choice formats. Journal of Educational Measurement, 269+. Carpenter, T. P., Corbitt, M. K., Kepner, H. S., Jr., Lindquist, M. M., & Reys, R. E. (1982). Student performance in algebra: Results from the National Assessment. School Science and Mathematics (October), 514-531. Carpenter, T. P., Lindquist, M. M., Brown, C. A., Kouba, V. L., Silver, E. A., & Swafford, J. O. (1988). Results of the fourth NAEP assessment of mathematics. Arithmetic Teacher (December), 38-41. Dossey, J. et al. (1988). The Mathematics Report Card. Princeton, New Jersey: Educational Testing Service. Gallup Organization (1989). The Gallup Poll public opinion 1988. Wilmington, Delaware: Scholarly Resources. Kirsch, I. S. (1993). Adult literacy in America. Washington, D.C.: National Center for Education Statistics. Lapointe, A., Mead, N., & Phillips, G. (1989). A world of differences. Princeton, New Jersey: Educational Testing Service. Linn, R. L., Graue, M. E., & Sanders, N. M. (1990). Comparing state and district results to national norms: The validity of claims that "everyone is above average". Educational Measurement: Issues and Practice (Fall), 5-14. Martinez, M. (1991, Summer). A Comparison of Multiple-Choice and Constructed Figural Response Items. Journal of Educational Measurement, 131-45. 10 of 12 Mullis, I. V. S., Dossey, J. A., Owen, E. H., & Phillips, G. W. (1991). The Sate of Mathematics Achievement. Washington, D.C.: U.S. Department of Education. NAEP (1983). The Third National Mathematics Assessment. Report No. 13-MA-01. Denver, Colorado: Education Commission of the States. National Center for Education Statistics (1993a). Digest of education statistics 1993. Washington, D.C.: U.S. Department of Education. National Center for Education Statistics (1993b). Education in States and Nations. Washington, D.C.: U.S. Department of Education. National Center for Education Statistics (1993c). NAEP 1992 Mathematics Report Card for the Nation and the States. Washington, D.C.: U.S. Department of Education. Pashley, P. J. & Phillips, G. W. (1993). Toward world-class standards: A research study linking international and national assessments. Princeton, New Jersey: Educational Testing Service. Rogers, V. & Stevenson, C. (1988, February). How Do We Know What Kids Are Learning in School? Educational Leadership, 68-75. Stedman, L. C. (1993). The condition of education: Why school reformers are on the right track. Phi Delta Kappan, 75 (3), 215-225. Stedman, L. C. (1994a). Incomplete explanations: The case of U.S. performance in the international assessments of education. Educational Researcher, 23(7), 24-32. Stedman, L. C. (1994b). The Sandia Report and U.S. achievement: An assessment. Journal of Educational Research (January- February), 133-146. Stedman, L. C. (1995). The new mythology about the status of U.S. schools. Educational Leadership, 52(5), 80-85. Stedman, L. C. & Kaestle, C. F. (1987). Literacy and reading performance in the United States, from 1880 to the present. Reading Research Quarterly (Winter), 8-46. Stedman, L. C. & Kaestle, C. F. (1991). The great test score decline: A closer look. In C. F. Kaestle, H. Damon-Moore, L. C. Stedman, K. Tinsley, & W. V. Trollinger (Eds.), Literacy in the United States (Chapter 4). New Haven: Yale University Press. Stedman, L. C. & Smith, M. S. (1983). Recent reform proposals for American education. Contemporary Education Review, 2(2), 85- 104. Stevenson, H. & Stigler, J. (1992). The learning gap: Why our schools are failing and what we can learn from Japanese and Chinese education. New York: Simon and Schuster. Wang, M, Haertel, G., & Walberg, H. (1993). Toward a knowledge base: Why, how, for whom? Review of Educational Research, Westbury, I. (1992). Comparing American and Japanese achievement: Is the United States really a low achiever? Educational Researcher, 21(5), 18-24. About the Author 11 of 12 Lawrence C. Stedman stedman@binghamton.edu Lawrence C. Stedman is Associate Professor of Education at the State University of New York at Binghamton. His Ph.D. is from the University of Wisconsin at Madison in Educational Policy Studies with a minor in Sociology. He has worked as a school district policy analyst, secondary school teacher, VISTA volunteer, and educational researcher. He has a keen interest in equal opportunity and school reform. His dissertation and early articles centered on effective schools research and the reform reports of the early 1980s. He has helped evaluate ESL, minority achievement, merit pay, and dropout intervention programs. More recently, his research has focused on the general condition of education and its implications for policy-making. He has written articles on the test score decline, literacy trends, the international assessments, and the Sandia Report. He is currently investigating historical trends in students' and adults' general knowledge. It is the outgrowth of a book he helped author with Carl Kaestle and others on the history of the U.S. reading public (Literacy in the United States: Readers and Reading Since 1880, Yale University Press, 1991). This new research has been funded by a SUNY Faculty Research Grant and Fellowship and by a National Academy of Education Spencer Foundation post-doctoal fellowship. Copyright 1996 by the Education Policy Analysis Archives EPAA can be accessed either by visiting one of its several archived forms or by subscribing to the LISTSERV known as EPAA at LISTSERV@asu.edu. (To subscribe, send an email letter to LISTSERV@asu.edu whose sole contents are SUB EPAA your-name.) As articles are published by the Archives, they are sent immediately to the EPAA subscribers and simultaneously archived in three forms. Articles are archived on EPAA as individual files under the name of the author and the Volume and article number. For example, the article by Stephen Kemmis in Volume 1, Number 1 of the Archives can be retrieved by sending an e-mail letter to LISTSERV@asu.edu and making the single line in the letter read GET KEMMIS V1N1 F=MAIL. For a table of contents of the entire ARCHIVES, send the following e-mail message to LISTSERV@asu.edu: INDEX EPAA F=MAIL, that is, send an e-mail letter and make its single line read INDEX EPAA F=MAIL. The World Wide Web address for the Education Policy Analysis Archives is http://seamonkey.ed.asu.edu/ Education Policy Analysis Archives are "gophered" in the directory Campus-Wide Information at the gopher server INFO.ASU.EDU. To receive a publication guide for submitting articles, see the EPAA World Wide Web site or send an e-mail letter to LISTSERV@asu.edu and include the single line GET EPAA PUBGUIDE F=MAIL. It will be sent to you by return e-mail. General questions about appropriateness of topics or particular articles may be addressed to the Editor, Gene V Glass, Glass@asu.edu or reach him at College of Education, Arizona State University, Tempe, AZ 85287-2411. (602-965-2692) Editorial Board John Covaleskie jcovales@nmu.edu Andrew Coulson andrewco@ix.netcom.com Alan Davis adavis@castle.cudenver.edu Mark E. Fetler fetlerctc.aol.com 12 of 12 Thomas F. Green tfgreen@mailbox.syr.edu Alison I. Griffith agriffith@edu.yorku.ca Arlen Gullickson gullickson@gw.wmich.edu Ernest R. House ernie.house@colorado.edu Aimee Howley ess016@marshall.wvnet.edu Craig B. Howley u56e3@wvnvm.bitnet William Hunter hunter@acs.ucalgary.ca Richard M. Jaeger rmjaeger@iris.uncg.edu Benjamin Levin levin@ccu.umanitoba.ca Thomas Mauhs-Pugh thomas.mauhs-pugh@dartmouth.edu Dewayne Matthews dm@wiche.edu Mary P. McKeown iadmpm@asuvm.inre.asu.edu Les McLean lmclean@oise.on.ca Susan Bobbitt Nolen sunolen@u.washington.edu Anne L. Pemberton apembert@pen.k12.va.us Hugh G. Petrie prohugh@ubvms.cc.buffalo.edu Richard C. Richardson richard.richardson@asu.edu Anthony G. Rud Jr. rud@sage.cc.purdue.edu Dennis Sayers dmsayers@ucdavis.edu Jay Scribner jayscrib@tenet.edu Robert Stonehill rstonehi@inet.ed.gov Robert T. Stout stout@asu.edu 1 of 2 Contributed Commentary on Volume 4 Number 1: Stedman The Achievement Crisis is Real: A Review of The Manufactured Crisis 11 April 1996 Eugene Bartoo ebartoo@CECASUN.UTC.EDU On April 10, 1996 Andrew Coulson wrote: Like Berliner's 1993 EPAA paper, The Manufactured Crisis is a fanciful and selective romp through the data on public schooling. Anyone who is currently relying on or quoting this material will do themselves a great favor by reading Larry Stedman's recent detailed critique of their treatment of the academic achievement data (also published in the Education Policy Analysis Archives). OK, I haven't read B & B's book, but read Stedman I, B & B reply, and Stedman II. And Andrew's post. Some impressions: (1) I suspect that many foreign students study more languages for longer periods than our students do, but I also think, as has been said by others here, that many foreign students get to use those languages more often and hence the language study takes. There are vast numbers of american students that have studied French for years and find themselves speechless when confronted with a French waiter. The opportunity and need to converse in another language is just not there and thus has an effect, I think, on the force of insisting on more rigor and time in learning languages. The argument that it broadens one's horizons may have been lost years ago. (2) I am engaged in a project in a local private girls school. I sit in on a precalculus class of junior girls; very bright, quick, high achievers. They will do very well later; most will go to good schools; get good jobs; have good lives. This morning I asked them if any of them knew where the Pyrennes were [I read Stedman II last night; I mulled over the data on geography from the NAEP]. Only three girls knew where it was. I'm not sure I knew where it was when I was sixteen. They had geography as seventh graders - I imagine that that experience was more like "this is where Africa is". Is this a sobering finding? At my university we have one faculty member whose specialty is geography; one! In the teacher ed. program which I helped design some few years ago, we have a geography concentration for social science ed. folks. It has few majors; has had fewer graduates. They have trouble getting the courses; most are taught by adjuncts. For most people in the US, the closest thing they have ever gotten to geography was that yellow-bordered magazine that is popular at garage sales. I was not surprised at the findings of the NAEP. I don't know if that is something with which to be concerned. (3) I think Stedman is basically right. We do have poor achievement. The achievement levels in this country have been poor forever. Most of our fellow citizens do not have a large store of knowledge as measured by the NAEP studies [or others like them]. Our children's achievement patterns are about what their parents' achievement patterns are. Perhaps the school 2 of 2 is culpable, but I didn't get the sense that Stedman took that view. (4) The SAT issue is basically not controversial anymore. There seems to be general agreement that those data have been stable for twenty some years; that earlier declines were due to a variety of factors, the largest percentage of variance being SES changes in test takers. And who cares? (5) There has just been another summit on educational matters and the reaffirmations of the need for rigorous standards and assessment were given. Shanker concurred in his latest column, stating that it was time to take control of the content of the curriculum out of local hands. In some areas around here those are fightin' words. And so our dream is to raise ourselves by the bootstraps of our children so that, for example, those bright girls would know where the Pyrennes were [and I might know how to spell it]? Just some thoughts. Thanks for your indulgence. 1 of 1 Contributed Commentary on Volume 4 Number 1: Stedman The Achievement Crisis is Real: A Review of The Manufactured Crisis 10 April 1996 John F. Covaleskie JCOVALES@NMU.EDU It is good to see Andrew attacking public schools again. If B & B are somewhat selective in their presentation of evidence, they are no more than following the lead, if not to the extent, of those who have been unfairly attacking schools for the past two decades. As I understand the core of their argument, it is that there is much evidence that schools are doing a remarkably good job with an increasingly diverse population. To cite one example, SAT scores have been declining for some time, and this is pointed to as a failure of schools. However, disagreggated data show that scores are actually rising for all subgroups, while the decline is due to the expansion of the test-taking pool among those who score lower, and were previousl;y excluded. The central claims of B & B seems unrefuted: (1) schools are not perfect, and need to made better, but they are hardly failures as they are painted, and (2) to make the case that schools are failing requires ignoring a great deal of data, and (3) the media have been complicit with those who, for ideological reasons, oppose public funding of education, quite regardless of its success or failure. Opposition to public education is hardly a matter of evidence; never has been, probably never will be. Support for public education is the same. The issue is what, if anything we owe collectively to our children. 1 of 1 Contributed Commentary on Volume 4 Number 1: Stedman The Achievement Crisis is Real: A Review of The Manufactured Crisis 10 April 1996 Larry S. Bowen lbowen@OSF1.GMU.EDU In addition to what John Stone has contributed to this thread, I believe it important to recognize that heightened expectations for public education are at work in societal concern over our schools--a phenomenon that likely will persist and guarantee continuing frustration. And with over 2 million teachers in more than 15,000 school systems, criticisms of "the schools" and their teachers will vary enormously in validity. This is of course common sense, but it is oft lacking when looking for actions to deal with real and severe problems. Attempts to deal with glaring inequities in funding are too few, even within school districts where a PTA of one school purchases large numbers of computers and software while another in a poor neighborhood languishes. The general effects way of thinking about the process of education is terribly unfortunate, I believe, with both the layperson and educator unable to develop a differentiated way of thinking and approaching the state of schooling and all its complexities. In the Washington, D.C. area, for example, there is strong evidence re a number of large districts which are viewed as either "excellent" or "terrible" by the media and the public in general. Outstanding schools in a terrible district are rarely acknowledged (or even known about by most people, I'm persuaded in listening), and in at least one highly-praised district there is mediocrity that fits the mantra of how "schools are failing." Until we are willing to seriously deal with the inequities in public education in those schools that are lousy and with teachers of children who are not attaining functional literacy the issue of public education quality will not be adequately addressed. I would suggest that those who are concerned with achieving excellence for all learners in a given school system begin by asking what the inequities are and proceed with determining the particular problems of the setting and insist that those problems are the problems for the school division and community as a whole, necessitating a "village" response. But that is hard work, especially for those who have "made it" and believe theirs is theirs. 1 of 2 Contributed Commentary on Volume 4 Number 1: Stedman The Achievement Crisis is Real: A Review of The Manufactured Crisis 10 April 1996 Andrew Coulson andrewco@IX.NETCOM.COM Dan Cline takes me to task for including a personal anecdote in my recent post criticizing The Manufactured Crisis. In response, I'd just like to say: Good for you! I threw in that personal comment because, in my experience, it's true, and because I was interested in getting a debate going, but it does not play even a minor role in my overall verdict on The Manufactured Crisis. In my post I mentioned Larry Stedman's critique, and my own very brief assessment of their treatment of the IQ data and the comparative spending and efficiency data (neither of which Dan mentioned). I have also, independently from Larry, examined their achievement data and arguments. When I complete my review of their case I plan to incorporate it into a book I'm writing on school reform. I do not intend to include my personal opinions on the educational breadth of European vs. U.S. students unless I can provide some concrete, broad-based data to support them. Berliner and Biddle, however, saw no such need for empirical evidence in making their own claim about American students being more "broadly educated." They write, without even a token piece of supporting evidence, that: By comparison then, American teenagers probably have more nonacademic interests and a wider knowledge base than do students from countries that stress narrow academic concerns (p52). They do not bother to prove that the emphasis on academic achievement enjoyed by other countries is indeed "narrow," nor do they bother to show that foreign students fail to enjoy all the same activities they list for American youth. The possibility that "narrow" concerns such as literature, history, geography, civics, might widen people's "knowledge base" also appears not to have occured to them. I trust that readers of The Manufactured Crisis, whether or not they like what B. & B. are saying, will be just as quick to discount their use of unsupported anecdote as Dan was to discount mine. ---------- An aside on the "educational breadth" issue: This is such a vague term, and such a, well, "broad" concept, that I never would have initiated it myself if I were trying to make an international comparison. Still, since we're on the topic, I'll just address a few of Dan's comments on the subject. While I don't disagree with Dan's suggestion that learning second, third, or fourth languages is to some degree related to the child's environment, schooling is clearly a significant factor in Europe and in other regions, such as the Canadian province of Quebec. In Quebec it is eminently practical to become bi/multi-lingual, and schools generally teach whichever languages 2 of 2 (French or English or both) are not native to the student. Success seems to depend on how intensively the languages are taught, and how much use the student makes of them (in and out of school). According to the OECD (Education at a Glance, 1995), foreign language teaching in 15 European countries takes up an average of 13% of teaching time. This is second only to mathematics, at 16%. It ranges from a low of 9% to a high of 26%. According to the Digest of Education Statistics. (1993), U.S. students spend roughly 6.5% of their class time on foreign languages--half of the European average, and less than the lowest rate of the 15 European countries studied by the OECD. Of course it remains to be argued that foreign languages add to one's "breadth" of knowledge, but I think that case can fairly easily be made. 1 of 1 Contributed Commentary on Volume 4 Number 1: Stedman The Achievement Crisis is Real: A Review of The Manufactured Crisis 10 April 1996 John Stone STONEJ@EDUSERV.EAST-TENN-ST.EDU On April 9, 1996 Dan Cline wrote: However, I think responses to these works by policy analysts and scholars of whatever pursuasion, even the critics of the schools among them, should perhaps rise above conclusions based just on hunches derived from personal encounters. (snip) I am not sure whether Berliner & Biddle were making an empirical knowledge claim or a simple assertion that American students are more broadly educated, but in either case, an examination of the claim or assertion is more enlightened by empirical evidence, rather than recitation of impressions from personal encounters alone - the latter can certainly lead to hypotheses to be subjected to empircal test. I agree that we need to go beyond personal experience in assessing this issue, but on this list, I see arguments buttressed by personal experience almost daily. Andrew does not need me to defend him but I am confident that he could have been much more specific and data based about Berliner & Biddle. There is a mountain of credible empirical evidence that points in the opposite direction from Berliner & Biddle. Their claim that the larger public's perception of the schools is the product of a conspiracy or right wing extremism or mere bad publicity is preposterous. I agree with Andrew that Berliner & Biddle are doing the cause of public education no favor by trying to dismiss its critics and their arguments. I know that the criticisms, the unflattering studies, and the negative reports in the media are distasteful--especially to people who are working hard to do what they feel is the right thing. However, as one who follows that which is being said in the larger public arena, I think we of the education community will either have to move in the direction of that which the public wants or progressively be replaced. People who are as committed to public education as Al Shanker have been saying the same thing for years. Professor Stedman's refutation of Berliner & Biddle may be painful to contemplate but if it moves us to confront reality rather than quibbling about its existence, he has, in my opinion, done us a great favor. 1 of 1 Contributed Commentary on Volume 4 Number 1: Stedman The Achievement Crisis is Real: A Review of The Manufactured Crisis 9 April 1996 Dan Cline DHC@PAWNEE.ASTATE.EDU Berliner & Biddle have taken some heavy hits for the Manufactured Crisis and for their response to Stedman's first response to the book. His second response, "Respecting the evidence..." is apparently being received on this list as a balanced and correct refutation of the major claims made by Berliner & Biddle and I suppose it is -- it is certainly a significant and thoughtful contribution on its own. However, I think responses to these works by policy analysts and scholars of whatever pursuasion, even the critics of the schools among them, should perhaps rise above conclusions based just on hunches derived from personal encounters. Andrew Coulson, for example, characterizes as "unsupported" and "far fetched" Berliner & Biddle's contention that American students are more broadly educated than their foreign counterparts based on his observation that in "personal travels" in Europe, "it is common to encounter high school and college students who speak two, three or even four languages fluently" and are well versed in international events (by what measure, not stated). As long as we are on personal experience, I was raised in the Dakotas speaking German, English and Russian, as were my mother and her sixteen brother and sisters, out of economic necessity -- farmers couln't do business with farmers and businesses from other communities in the area who spoke other languages unless they were conversant in those languages. Since none of them (myself being the exception) attended school beyond the eighth grade, I wonder to what extent being multilingual is a measure of being "broadly educated." I wonder if being multilingual in Europe isn't more a function of close proximity of nations (like New York and Pennsylvania) and ready access to television broadcasts in many languages rather than a function of curriculum in the schools. I am not sure whether Berliner & Biddle were making an empirical knowledge claim or a simple assertion that American students are more broadly educated, but in either case, an examination of the claim or assertion is more enlightened by empirical evidence, rather than recitation of impressions from personal encounters alone--the latter can certainly lead to hypotheses to be subjected to empirical test. 1 of 1 Contributed Commentary on Volume 4 Number 1: Stedman The Achievement Crisis is Real: A Review of The Manufactured Crisis 8 Apr 1996 Andrew Coulson andrewco@IX.NETCOM.COM Like Berliner's 1993 EPAA paper, The Manufactured Crisis is a fanciful and selective romp through the data on public schooling. Anyone who is currently relying on or quoting this material will do themselves a great favor by reading Larry Stedman's recent detailed critique of their treatment of the academic achievement data (also published in the Education Policy Analysis Archives). For those wondering if only the academic achievement data is mistreated by Berliner and Bracey, the answer appears to be no. My bedtime reading over the past week has been a combination of the Manufactured Crisis and two telephone-book sized tomes: The Handbook of Human Intelligence (Edited by Sternberg), and the 1993 issue of the Digest of Education Statistics (Dept. of Ed.). Based on the research compiled in the Handbook, B. & B.'s coverage of the IQ data is very poor. They draw causal conclusions from correlational data on numerous occations, and are prone to offer unsupported conjecture as fact. They infer from the positive correlation between years of schooling and higher IQs that schooling increases IQ, but specialists in the field have argued the converse--that students who have an affinity for academics and schooling and a high tested intelligence tend to choose to stay in school for longer than students who perform poorly. Their figures for per-pupil education spending are outdated (1985) and misleading. They appear to combine private and public school spending, without mentioning that private schools spend less per-pupil, thus bringing down the average. The research on cost-effectiveness of public schools is given short shrift, and one of the key meta-analyses, conducted by Childs and Shakeshaft, is not mentioned at all. Furthermore, their belief that American students are more broadly educated than their foreign counterparts is entirely unsupported, not to mention far-fetched based on my personal travels in Europe and North America. In Europe it is common to encounter high-school and college students who speak two, three, or even four languages fluently, and are well versed in current events in the international arena. This (again in my own experience) is much less common in the United States. The Manufactured Crisis, however, is not a total loss. As Larry points out in the introduction to his critique, people have a tendency to research their opinions only until they find support. Berliner and Biddle act as devils' advocates for those, like myself, who are critical of public schooling, pushing us to support every aspect of our arguments with hard evidence. Because B. & B.'s beliefs are so poorly defended, however, this ends up reinforcing the case for radical reform. Probably not the outcome they had in mind.