Teaching Note INFORMAL LOGIC XIV.2&3, Spring & Fall 1992 A Thirty-First Way to Mess Up a Critical Thinking Test: A Critical Response to Facione H. HAMNER HILL Southeast Missouri State University Peter Facione's generally well thought- out set of suggestions for messing up a critical thinking test l overlooked one sig- nificant option for the construction of de- fective multiple-choice (hereinafter MC) tests: require the impossible. 2 Perhaps Facione was being intentionally devious, leaving this option to be discovered by the intrepid reader, but it appears that one of the MC items he uses to exemplify a different strategy for failure is guilty of requiring the' impossible by not listing a right answer. If, however, Facione was not being devious, then a genuine issue arises concerning the use of sophisticated and complex multiple choice items to test critical thinking. Counseling that one avoid MC items requiring careful analysis, contrary to fact reasoning, hypothetical reasoning and the like, he provides the following example: Q6: Consider the "krendalog" relation- ship. It can be defined as follows: "Every human being now living has krendalogs. Nobody can be their own krendalog. If someone is your krenda- log, then all of that person's krenda- logs are your krendalogs too. If someone is your krendalog, then you cannot be that person's krendalog. Jacob and Kathy were the first hu- mans to exist in the whole world." Which of the following must be true, if all of the above are true. A=Either Jacob or Kathy has no krendalogs. B=Jacob and Kathy are krendalogs 10 each other. C=Jacob and Kathy each are their own krendalogs. D= There is a krendalog who is no krendalog to Jacob and Kathy. *E=AII humans are krendalogs to Jacob or Kathy.3 This particularly devious problem, involv- ing something like an ancestral relation, is made even more so by Facione's skillful use of a distracting asterisk before option E as other uses of an asterisk within the p~per mark the right answer, but I will re- turn to that issue shortly. It is fairly clear that neither B nor C could be true, since each of these options directly contradicts one of the initial conditions (and makes each of them suspect as distractors). A, on the other hand, is consistent with,4 but not implied by, the initial set, as the skillful construction of different inter- pretations reveals. To eliminate A as a pos- sible right answer one must show that A is not implied .by the initial set. To show this, one needs to construct an interpretation ac- cording to which the negation of A is con- sistent with the initial set. One strategy for constructing such an interpretation follows. As the negation of A is "There is an x such that x is a krendalog of Jacob and there is a y such that y is a krendalog of Kathy, "5 and as the initial data set is a fi- nite set of quantified schemata. we know from Lowenheim's theorem6 that if this set plus the negation of A is consistent in some non-empty universe, then it has a true interpretation in the universe of 190 H. Hamner Hill positive integers. Conversely, any finite set of quantified schemata that can be given a consistent interpretation in the universe of positive integers (i.e., that can be mapped, if you will, onto the positive integers) is it- self consistent. Thus, the negation of A is shown to be consistent with the initial data by constructing a consistent interpretation of the initial set plus the negation of A in the universe of the positive integers. Let each human being equal exactiy one posi- tive integer. Let Jacob and Kathy equal 1 and 2, respectively. Finally, let "is a krend~ alog of' be defined as "is a successor of," noting that these two predicate terms have exactly the same formal properties, viz. transitivity, irreflexivity, and asymmetry. Clearly the initial data set plus the negation of A comes out true under this interpreta- tion (as both 1 and 2 have successors), demonstrating that A is not implied by the initial set so it cannot be the (or even a) correct answer. D, like A, is consistent with,7 but not implied by, the initial set, as can be demon- strated through the same method used for A. The negation of D states "Every krenda- log is a krendalog to either Jacob or Kathy." Using the same interpretation as was used for A, the negation of D is seen to be consistent with the initial set (every number which is a successor is a successor to either 1 or 2). So D, not being implied by the initial data set, is not correct. Which leaves only E, the answer Facione marked with an asterisk usually reserved for the correct answer. E has a surface plausibility to it, and one that would likely attract all but the best stu- dents, but that plausibility is misleading. Not only is E not implied by the initial set, E is inconsistent with it. If all humans are krendalogs to either Jacob or Kathy, then both Jacob and Kathy, having been defined as humans in the initial data set, must be krendalogs to either Jacob or Kathy. Such a requirement violates either the asymme- try or irreflexivity conditions of the krend- alog relation. Consider Jacob. He cannot be a krendalog to himself given irreflexivi- ty, so he must be a krendalog to Kathy. Now consider Kathy, she cannot be her own krendalog so she must be a krendalog to Jacob, but this is impossible since Jacob is a krendalog to her. Parallel reasoning ap- plies if one begins with Kathy. So, despite the distracting asterisk, E is not correct either. As things stand, this putative exam- ple of a sophisticated MC item requires the student to do the impossible: select the correct answer from a set containing no correct answer. To be sure, slight modifications to this problem would make it a useful, if even more devilishly difficult, MC item. For in- stance, by eliminating options Band C, and redesignating A, D, and E as I., II., and IlL, respectively, one could change the question so as to ask: "Which of the following could be true if all of the above are true?" I. Either Jacob or Kathy has no krendalogs. II. There is a krendalog who is no krendalogto Jacob and Kathy. III. All humans are krendalogs to Jacob or Kathy. A. I only. B. II only. C. III only. D. Both I and II. E. Both I and III. and the right answer would be D. Using as a distractor the option "III only," would likely guarantee a very high error rate. No one familiar with the level of so- phistication achieved by tests like the LSAT should doubt either the power or the subtlety possible for MC tests. However, it is incumbent upon anyone wishing to take advantage of the possibilities of MC tests to pay very careful attention to detail and nuance. This is particularly the case when CT items, like Q6, stray perilously close to formal logic. My complaint here is not so much with Facione's use of MC items, but rather with the suggestion that an item like Q6 tests students' abilities in critical thinking or informal logic (as those terms are general- ly understood). To be sure, even without training in formal techniques, some students will find the correct answer. But A Critical Response to Facione 191 the innumeracy of many of our students is well known, and it strikes me as foolish to invite students to rely on their untutored mathematical intuitions to solve items like Q6. Given rigorous formal trammg, however, sophisticated MC items do pro- vide an appropriate avenue for assessing a student's abilities. Notes Peter Facione, "Thirty Great Ways to Mess Up a Critical Thinking Test," Informal Logic, Vol. XII, No.2 (Spring 1990), pp. 106-112. 2 One might, in the alternative, treat this as a spe- cial case of Facione's suggestion # 19: Provide no correct answers or give multiple correct an- swers. However one conceptualizes the option, one obvious avenue for its use it to have the stem say, "Which of the following must be true" when, in fact, none of the options must be true although a few could, but need not, be true. 3 Facione, "Thirty Great Ways to Mess Up a Critical Thinking Test." p. 111. This example is given to show that one can do more with a MC test than critics generally admit. While I agree with Facione's basic claim, I find serious flaws with the example. 4 A can be shown consistent with the initial data in several ways. Perhaps the easiest way it to define "is a krendalog of' as "is a predecessor of' and, clearly, I has no predecessors. Anoth- er strategy requires resort to an infinite uni- verse or to assumptions that are unsupported by the initial data set (e.g., that neither Jacob nor Kathy is currently living). Requiring stu- dents to think in terms of infinite universes may well be a good strategy for messing up critical thinking tests, particularly when the students lack rather detailed training in meth- ods of formal logic. But that is another story. 5 Different quantifiers are appropriate here as there is no requirement that the krendalog of Jacob be the same krendalog as the krendalog of Kathy in order to falsify A. 6 For an excellent treatment of Lowenheim's theorem, its uses in analyses like the one pro- vided here, and its relation to the more power- ful Lowenheim-Skolem theorem ( one not needed here), see W. V. Quine, Methods of Logic, 4th ed. (Cambridge: Harvard University Press, 1982), Ch. 33, pp. 209-212. 7 As with A, a demonstration of the consistency of D with the initial data set is possible, but tricky. Perhaps the most promising avenue for demonstrating consistency involves construct- ing a satisfying interpretation in a finite uni- verse of discourse. Such an interpretation can be constructed in a four element universe, but the details of such a construction are not es- sential here. What is important, however, is that the techniques needed for such a construc- tion are in no way informal-they require skillful formal instruction. H. HAMNER HIU DEPARTMENT OF PHILOSOPHY & RELIGION SOUTHEAST MISSOURI STATE UNIVERSITY CAPE GIRARDEAU, MO 63701-4799 Q