INFORMAL LOGIC XVI.2, Spring 1994 The Zen of Argument Analysis: Reflections on Informal Logic's Argument Evaluation Contest DON S. LEVI University of Oregon Key Words: Audience; technical terms of rhe- torical analysis; sequences of premises; experts in argument analysis; analysis as a contribution to argument; issue raised by argument; evalua- tion-matrix (check-list); rhetorical context. Abstract: Argument analysis should contribute to the controversy from which the argument is taken. I suggest that each entry in the argument evaluation contest be judged by considering whether it addresses a lay audience, makes clear why its criticisms matter and how the argument might be improved, and gives us something to think about. I see the judging deadlock as sig- nificant because of what it reveals about the evaluation-matrices of the judges. I conclude by suggesting that the contest is valuable as a re- minder of how there are many perspectives from which good argument analysis can be given. I regret missing the announcement of the argument evaluation contest because I would have enjoyed the competition. Now that I am writing about my thoughts on the contest, I have to guard against the temp- tation to show how I could have done bet- ter, especially when I know that I would be aided in doing so by my reading of the entries and the judges' comments. To guard against it I need to be clear about what I am doing in submitting these reflections. The question I find myself asking is whether we who teach or write about the critical analysis of argumentation are ex- perts in argument analysis. There are rea- sons for thinking that we might not be. For one thing, when we analyze an actual ar- gument in the classroom or study we do not offer our analyses as contributions to that controversy, but illustrations of lessons or theoretical principles. For another, we seem arrogant in our presumption that we can evaluate the argument without having to do any research about the issues in the debate from which the argument is taken. The participants in Informal Logic's Argu- ment Evaluation Contest offer competent and perceptive analyses. But the question arises as to how much of each analysis is dictated by a certain pedagogy of argument analysis, and how much is dictated by what the analysis of the argument requires. 1. The Audience for the Analysis The rules for the contest explicitly re- quire that the entry address a "reasonably well-informed, non-specialist, general pub- lic." Only David Hitchcock's entry satis- fies this requirement. Surprisingly, Marie Secor, who is the most sensitive of the con- testants to the rhetorical dimension of ar- gumentation, seems insensitive to the needs of her intended audience. Not only does she make use of technical rhetorical terms such as "formal enthymemic topic," but she says, without a trace of irony, that the descrip- tion of her intended audience - "non-spe- cialist, general public" - fits the readers of Informal Logic! One of the judges, Michael Scriven, also remarks on the fact that the entries other than Hitchcock's would not be clear to non- specialists. He is thinking primarily of the use of technical language, such as Chai'm Perelman's terms and concepts (in Secor and Richard Fulkerson's entries). However, he does not seem to be aware of how un- readable an analysis like Jonathan Adler's 88 Don S. Levi is because of its restatement of the argu- ment as twelve premises or Fulkerson's is for restating it as a sequence of eight propo- sitions. He even praises Fulkerson's use of a tree diagram for not being as "hard to decipher as Mayan sign-language," despite the fact that that is just how it will seem to its intended audience. The entries should address those who will want to know why they should be in- terested in analyses of the argument. Of course, the entries are written for the judges of the contest, and this explains why most of the analysts confine their analysis to the merits and demerits of the argument with- out considering how the argument may be improved or what really is at issue. Hitchcock, alone among the contestants, does participate in the controversy from which the argument is taken. As one of the judges, Henry Johnstone, Jr. points out, Hitchcock's entry demonstrates the "clear- est awareness" of the concerns of the ar- guer, something that is necessary when the analysis is directed to an audience that will want to know why the argument is being analyzed. Johnstone also praises Hitchcock's analysis for considering how the argument might more effectively pro- mote the concerns of the arguer. Hitchcock thinks of himself as making a contribution to the controversy, whereas the other contest- ants seem to write from the point of view of a teacher correcting a student's exercise. 2. Ranking the Entries My rankings are: Hitchcock first, fol- lowed by Secor, Fulkerson and Adler. I do not offer them because they are so differ- ent from the rankings of any of the judges - they are the same as Johnstone's - but because explaining how I arrived at them will enable me to further develop my ideas about the critical analysis and evaluation of argumentation. Hitchcock seems to be the only one of the entrants who asked himself where he stands on the question of quotas in hiring the disabled. This explains why he was the only one to think about the kind of quota being proposed - whether it is to be de- fined in terms of the percentage of the disa- bled in the population as a whole, as the argument indicates, or the percentage of disabled among the qualified applicants, which is more compatible with the argu- er's desire not to be hired if unqualified. Unlike the others, he seems to have tried to find out about the issue, as the reference to Judge Abella's 1984 report suggests. Al- though other contestants also have com- plaints about the argument's use of statis- tics, he alone seems to have considered what turns on the use of the statistics. The more fruitful the analysis of an ar- gument in giving us something to think about, the better it is - this should be an article of faith in argument evaluation. Hitchcock gives us something to think about - whether requiring that the percent- age of the disabled hired match the percent- age of the disabled among qualified appli- cants requires reverse discrimination, and whether that discrimination can be justified. Hitchcock's entry is to be preferred over the others because it gives us more to think about than they do. But considering what turns on the use of statistics is not sufficient; the issues the analysis raises should be raised by the ar- gument that is being analyzed. Hitchcock supposes that a quota system defined in terms of percentages of the disabled in the applicant pool would result in reverse dis- crimination. The arguer explicitly rejects this as a possibility. Hitchcock seems right when only a small number of hirings are involved because then the requirement that the number of disabled hired be propor- tional to their numbers in the applicant pool is likely to result in more qualified people being overlooked. But this is by no means obvious when large numbers of jobs are involved, as there would be with a big city police or public works department. Whether or not Hitchcock is right seems an issue raised by his claim that the quota system does require reverse discrimination rather than by the argument he is analyzing. This point also applies to another issue raised by his defense of reverse discrimi- nation, that the end of remedying past in- justices justifies the means. Hitchcock could have found issues that are raised by the argument by asking whether the weaker quota system could be justified even if it does not require reverse discrimination. Consider, for example, the elasticity of the concept of qualification. That elasticity may enable employers to use it to hire fewer of the disabled. I am think- ing here of how job descriptions can be worded to include qualifications that work against the disabled. Is a factory worker with cerebral palsy unqualified because her fellow workers find it a struggle to under- stand her? Is a blind academician less quali- fied because she has not published as much as her sighted colleagues or because she may need some help in grading papers or even because she cannot see what is going on in her classroom? That elasticity also enables the concept to be stretched to pro- vide seeming evidence of reverse discrimi- nation when what really is involved is the prejudiced perception of the capabilities of the disabled. Also elastic is the category of the "disa- bled" which includes a lot more than those who, like the arguer, are confined to a wheelchair. It also includes victims of blindness, epilepsy, mental illness, head injury, AIDS, cerebral palsy, autism, and perhaps even old age. The fact that it in- cludes so many different types of disabili- ties raises questions about whether a quota system will be easy to define or implement. It also raised questions, as Secor points out, as to whether the expense of administering it will draw funds away from the needed social services now being provided for the disabled. Among these services are stipends given to employers to pay for part of the salaries of disabled workers who cannot work as quickly or efficiently as the non- disabled. If a quota system is implemented The Zen of Argument Analysis 89 with an emphasis on being "qualified", perhaps some of these incentives for hir- ing the disabled would be withdrawn. I mention these problems not because I think that they are insurmountable but be- cause they are a sample of what an analy- sis of the argument should prompt us to consider. A good test of whether a critical analysis of an argument is valuable is whether it enables us to identify the things that are at issue and helps us to think about the best ways to address those issues. Al- though Hitchcock's analysis does satisfy this requirement, it would have been even better if the issues raised by his analysis were raised by the argument itself. Even though it is not suitable for a lay audience, I rank Secor's entry as second best. She finds at least as many flaws in the "microstructural linkages" of the argu- ment as do the others. Moreover, her entry is much better on how the argument gets the reader to identify with the arguer's problems. She is especially good on how the argument's "macrostructural" strengths will make it persuasive for a certain kind of audience, and how the emotional appeals at the beginning and the end of the essay make the arguer's fear and victimization "present" to the audience. Her conclusion that the argument's strengths are macrostructural and its weaknesses microstructural seems to betray an unwill- ingness to consider what is really at issue and how the argument might better address that issue. But her argument is superior to the other two in its perceptive treatment of both logical and rhetorical elements. Adler's entry I rank as third best. Instead of questioning the validity of the statistics, he wonders what follows from them. He suggests that certain disabilities may justly disqualify people from holding certain jobs, and that the disparities revealed by the sta- tistics may be due to the fact that disabled people are so demoralized by their disabili- ties that they do not seek jobs as assidu- ously as non-disabled people. He also won- ders whether the costs of administering a 90 Don S. Levi quota system may be too high, whether it may be a form of reverse discrimination, and whether being hired because of a quota will undermine self-esteem. Some of these worries make me won- der whether Adler is familiar with the lit- erature on the subject; much of which is concerned with challenging the idea that there may be sound reasons for the appar- ent discrimination against the disabled. People who work in vocational rehabilita- tion know that their clients can become dis- couraged, but often that is because they are having so much difficulty getting jobs be- cause of discrimination! Adler's worries about a loss of self-esteem and about re- verse discrimination seem misplaced when it comes to a weak quota system, such as the one Hitchcock considers, that is con- fined to insuring that the percentage of disa- bled people hired corresponds to the per- centage of qualified disabled people in the job pool. (Adler does not seem to have thought about what kind of quota system is being proposed or why the arguer would think that it does not involve reverse dis- crimination, a point that also applies to Secor's and Fulkerson's entries.) So, his entry is flawed because it is inaccessible to the audience for which it was intended, because it seems not to have given much thought to what is being proposed, and be- cause the issues it raises seem so oblivious to the literature on discrimination in the hiring of the disabled. The strength of Fulkerson's entry is in its identification of flaws in the argument's use of statistics. However, its weakness is that it does not consider how the use of sta- tistics might have been improved or whether anything really turns on their use. Granted that the statistics from a 1982 study comparing the numbers of disabled and non-disabled college graduates that are employed seem suspect when the effective- ness of a 1986 employment equity law is in question - a problem overlooked by the other contestants. Granted, too, that the disabled and non-disabled college gradu- ates may differ in significant respects - grades; major fields of study; work experi- ence, etc. - and so the comparison may be gUilty of fallaciously confusing a correla- tion with a causal relationship. But Fulkerson never really considers the sig- nificance of these omissions. Is the only issue raised by the argument whether the statistics are reliable? If so, then he might have looked into the matter to see what the available data really discloses. Does Fulkerson want to suggest that there really is no discrimination against the disabled? There is nothing to indicate what he thinks about it, and so there is little to explain to a reader with no investment in certain tech- niques of argument analysis why these problems are being identified. There are other problems with his analy- sis. One is that he complains that the argu- ment proposes that there should be man- datory quotas for other groups, but presents evidence only about the disabled. This complaint seems perverse because it is so obvious that the argument is concerned only with the disabled. When coupled with the fact that his statement of the conclu- sion in his layout of the argument is that an anti-discrimination law should be passed when the argument is proposing a quota system, his entry seems doomed to failure almost from the beginning. He, like Adler and Secor, complains about the fact that there may be disanalogies between the situ- ation in the U.S. and Canada, but he does nothing to suggest what they might be or why they are significant. These problems with his analysis also explain why I have ranked his entry lower than the others. I do not want to make too much of my rankings. I could have ranked Fulkerson's entry ahead of Adler's. Adler makes no obvious mistakes, but Fulkerson is more perceptive than Adler about the problems with the arguer's use of statistics, and his entry offers more food for thought. The is- sues the argument raises for Adler is whether there may be more satisfactory responses to discrimination than a quota system or explanations other than discrimi- nation for the fact that proportionally fewer of the qualified disabled are employed. Adler does not indicate what these alter- natives might be; nor does he do anything to make his explanations more plausible. Consequently, he does not give us much to think about. By contrast, the issue the argument raises for Fulkerson is how a quota system can avoid reverse discrimination. Fulkerson argues that the quota system cannot avoid it on the grounds that if the arguer's pro- posal is implemented then a company that does not comply with the quota will be forced to hire the disabled before hiring anyone else until it is in compliance. Al- though the argument does not say that the penalties for non-compliance would take the form that Fulkerson imagines, his criti- cism does invite a consideration of how the penalties for non-compliance would not result in reverse discrimination. The problems with these last three en- tries are symptomatic of what might be called "the blackboard disease," an ap- proach to argument analysis, fostered by the artificial conditions of the classroom or study, that seems to encourage sitting in judgement over an argument without a suf- ficient interest in what is at issue in the ar- gument. Each of them makes good points about the merits and defects of the argu- ment; I rank none of them as high as Hitchcock's because none seems to have been written as a contribution to the de- bate. 3. The Judging of the Contest The deadlock in the judging raises ques- tions about the significance of the differ- ences in the judges' approaches. Johnstone and I see argument analysis as a contribu- tion to the controversy from which the ar- gument is taken. Perhaps this explains why we agree in our rankings. That the other judges, Scriven, and the editors of Infor- mal Logic, Anthony Blair and Ralph The Zen of Argument Analysis 91 Johnson, disagree with us, may be due to differences from us in their points of view and emphases. Scriven uses an evaluation-matrix that led to his recommending that the two en- tries which Johnstone and I have ranked as least satisfactory be "regarded as tying for the prize." He does not say that the two entries received exactly the same score, a result which would be hard to believe given the number of criteria the matrix incorpo- rates. He offers little opportunity to sec- ond guess how he filled in the matrix, ex- cusing himself by saying that the details of his scoring would take too much space and his report is too long as it is. Most of that length is devoted to explaining about the matrix rather than discussing the entries. Since my rankings are different from Scriven's, and since I know him to be a very subtle analyst of argumentation, I wonder whether the differences in the results may be due to his questionable investment in the use of an argument evaluation matrix. I say this even though I have no real objections to any of his eleven criteria. In- deed, I seem to have criteria of my own, such as whether the analysis is written for lay people, makes clear what turns on its objections, considers how the argument might be improved, or helps us to think more clearly about the issues. But I am bothered by the assumption that a quanti- tative measure is possible. Scriven criti- cizes Hitchcock's overall support for the argument on the grounds that that "conclu- sion did not appear to follow well from the balance of the preceding considerations." What Scriven seems to have in mind is that Hitchcock's conclusion is based more on his support for the conclusion than on what the arguer is actually able to offer in sup- port of it. I made a similar point earlier when I suggested that Hitchcock uses the argument to raise questions that interest him rather than to consider the issues actually raised by the argument. Scriven and I seem to weigh this deficiency, if it is one, differ- ently. The other contestants, content as they 92 Don S. Levi are to consider the merits of what was ac- tually argued, did not address the question of whether a good argument can be given for a quota system. Should we deduct even more points from their "syntheses" because of their failure to do so? I ask the question not because of how I think the points should be distributed, but because I question the validity of any quantitative measure of an analysis. Another problem with criteria such as Scriven's is with the assumption that the same criteria apply to every analysis, re- gardless of whether the analysis is done by inexperienced students or accomplished teachers such as the participants in this con- test, and regardless of the particular prob- lems that the argument being analyzed presents. The criteria I am employing are designed to apply to the contest analyses, and they are justified to the extent that they help to clarify the problems we experience in using these analyses to think about the argument. They are not to be justified by showing how they should be used to judge any argument analysis. I can only speculate on why Scriven's rankings are so at odds with Johnstone's or mine. Knowing what I do about Scriven's convictions about argument analysis I do know that he emphasizes the importance of identifying the assumptions upon which the argument depends. Perhaps this empha- sis explains why he rated highest the en- tries by Adler and Fulkerson, which are so difficult to read, because of how much im- portance they place on listings all of the premises or assumptions of the argument. Still another point of view that may be taken in judging the contest is based on considering the argument as statistical. Consequently, that entry will be judged best that offers the most perceptive analysis of the strengths and weaknesses in the uses of statistics. From such a point of view, which might be the one adopted by Johnson and Blair, Hitchcock and Fulkerson's analyses may be regarded as the best. Johnson and Blair tried to break the deadlock between Johnstone and Scriven by using a "fine-grained check-list aimed at identifying both the quantity and the quality of each assessment." I find it dis- concerting to be told that the check list gave (exactly?) the same score to their two top entries, or that it gave (exactly?) the same score for the other two entries. Even if these worries are overlooked, the questions raised about Scriven's use of a matrix also arise in connection with their fine-grained check-list, namely whether any quantita- tive measure can be of value in judging the critical analysis of an argument. Another judge may put more emphasis than I do on the mistakes made by the ana- lyst. By this criterion perhaps Secor's en- try should be preferred over Hitchcock's. While she avoids any obvious errors, Hitchcock's use of anecdotal evidence seems a bit of a stretch. Despite what Hitchcock says, the arguer does not say that he or she actually missed out on an inter- view because the building where it was held had no wheelchair access. And a single example of an employer expressing preju- dice against hiring the disabled hardly con- stitutes an "ostensive demonstration" of the existence of significant discrimination against the disabled in the workplace. I did not give much weight to such apparent weaknesses in Hitchcock's analysis; an- other judge might have weighted them more heavily. The deadlock in judging the contest is not surprising when we consider the dif- ferent points of view that may be brought to the judging. As further support for this point consider the different weight that those trained in the rhetorical rather than the philosophical tradition will give to the fact that Secor and Fulkerson discuss the persuasiveness of the argument, whereas Hitchcock and Adler do not. Of course, there is nothing wrong with approaching the judging with a point of view. On the contrary, I think it is unavoidable. What is important is that the judge be prepared to defend that point of view, something I have tried to do when I argued for the need to evaluate an argument in terms of the kind of contribution it makes to the controversy from which the argument has been taken. 4. The Framing of the Contest Let me comment on the design of the contest. Johnstone makes a criticism of it that I do not think is very serious. He com- plains about calling it an "argument evalu- ation" contest and then asking contestants for a "critical analysis" of the argument. What bothers him is that a critical analy- sis, because its focus is on the components of the argument and their logical relation- ships, need not consider the concern being expressed by the argument, or how the ar- gument might be improved to more effec- tively implement that concern. I do not agree with Johnstone that a critical analy- sis does not need to consider such matters. But even if a contestant did agree with Johnstone, I suspect that the contestant would not have supposed that a critical analysis called for anything different from an argument evaluation. A more serious problem with the contest is that the argu- ment that is the subject of the contest is supplied with an incomplete rhetorical con- text, something that may be due to the fact that the argument was "adapted," to avoid the messy process of getting copyright per- mission. We are told only that the argument was taken from a public debate on the Ca- nadian government's 1986 employment equity program - we are not told why it was given, to whom or what it was a re- sponse. We are not even told whether it was taken from unrehearsed oral remarks made in a question-and-answer period or from a prepared speech. To see why a knowledge of the rhetori- cal context is important for other reasons, consider the question raised by Adler about what the arguer does to anticipate the ob- jection that there may be better alternatives than a mandatory quota system. Two alter- natives are dismissed as not effective, the The Zen of Argument Analysis 93 existing mandatory federal employee eq- uity system, and other existing voluntary programs. Adler adds as a missing assump- tion his premise eleven: "The alternatives criticized are the only realistic alterna- tives." He seems to think it should be in- cluded because he thinks that the argument requires it if is to be valid. But, the use of the criterion of validity seems warranted only if the argument is understood as a proof or demonstration, and there seems to be no warrant for doing so. Does the argument do anything more than suggest that a quota system may be the best hope of overcoming discrimina- tion against the handicapped? I have just suggested that it is not designed as a proof or demonstration. But I do not know what the argument is a response to, what has just been said or why it has been said. If the argument is an opening statement, then perhaps it may be understood to pose a question - isn't a quota system our best hope? If so, then that analysis would be best that considers the issues raised by the sug- gestion. Unfortunately, most of the contest- ants interpreted the argument differently. They understood it to be a proof or dem- onstration that a quota system was supe- rior to any other alternative, despite the fact that the author only considers existing al- ternatives and offers no evidence that the situation is not improving. Knowledge of the rhetorical context would seem to be necessary in order to determine whether their interpretations are correct. Perhaps the most serious problem with the contest was that it was a contest be- cause contestants were likely to write for their peers who were judging the contest rather than for a lay audience who would want to know why the analysis was being given or what significance the points be- ing made have. As Hitchcock's entry dem- onstrates, it was possible for contestants to write from the point of view of someone making a contribution to the analysis of the issues. But, as the other entries indicate, the contest format seems to encourage per- 94 Don S. Levi fonning the analysis from the standpoint of a critic rather than a participant. 4. Conclusion Despite the problems with the way the contest was introduced or with its being a contest at all, the argument evaluation con- test was a good idea. It is to be commended for its emphasis on the importance of analyzing actual arguments. More signifi- cantly, by printing all of the entries the con- test also offered a reminder of how there can be many different points of view from which an argument can be critically analyzed. Because of this contest I would urge that we should incorporate into our teaching or texts samples of different criti- cal analyses of an actual argument, to il- lustrate not how the analysis can go wrong, but how there can be sound analyses of the same argument that are written from very different points of view. Although I think that the contest was a good idea, my take on the entries and the judging is that they would have been even better if everyone involved were clearer on the need for an analysis to be written as a contribution to the argument. It seems to be an occupational hazard of teaching and writing about critical thinking or logic that the analysis of an actual argument becomes an occasion for the illustration of certain concepts or principles, the application of certain procedures or methods. We have our differences when it comes to the les- sons we teach. Whatever the value of these lessons, the real interest in the contest, I want to suggest, is that it reveals the im- portance of another lesson. That lesson is that we need to develop a zen of argument analysis, that enables us to find the right things to say about an argument because making a contribution to the controversy from which the argument has been taken requires that these things be said. DONS. LEVI DEPARTMENT OF PHILOSOPHY UNIVERSITY OF OREGON EUGENE, OR, U.S.A. 97403-1295 0