31.1StephensPub © Christopher Stephens. Informal Logic, Vol. 31, No. 1 (2011), pp. 56-65. A Bayesian Approach to Absent Evidence Reasoning CHRISTOPHER STEPHENS Department of Philosophy University of British Columbia 1866 Main Mall E370 Vancouver, BC Canada V6T 1Z1 chris.stephens@ubc.ca Abstract: Under what conditions is the failure to have evidence that p, evidence that p is false? Absent evidence reasoning is common in many sciences, includ- ing astronomy, archeology, biology and medicine. An often-repeated epistemo- logical motto is that “the absence of evidence is not evidence of absence.” Analysis of absent evidence reasoning usually takes place in a deductive or fre- quentist hypothesis-testing framework. Instead, a Bayesian analysis of this motto is explored and it is shown that, under plausible assumptions about the nature of evidence, the absence of evidence is evidence of absence. Résumé: Under what conditions is the failure to have evidence that p evidence that p is false? Absent evidence reasoning is common in many sciences, includ- ing astronomy, archeology, biology and medicine. An often-repeated epistemo- logical motto is that “the absence of evidence is not evidence of absence.” Analysis of absent evidence reasoning usually takes place in a deductive or fre- quentist hypothesis-testing framework. Instead, a Bayesian analysis of this motto is explored and it is shown that, under plausible assumptions about the nature of evidence, the absence of evidence is evidence of absence. Keywords: absence of evidence, arguments from ignorance, Bayesianism, evi- dence, evidence of absence 1. Introduction An often-repeated epistemological motto is that “the absence of evidence is not evidence of absence”.1 Here is Donald Rumsfeld, former Secretary of Defense of the U.S.A., commenting on the United Nations Weapons Inspection Team’s failure to find weap- ons of mass destruction in Iraq: There’s another way to phrase that and that is that the absence of evidence is not the evidence of absence. It is basically saying the same thing in a different way. Sim- ply because you do not have evidence that something 1 The motto is common in criminology and archaeology; for examples see Turvey (2002, p. 275) and Bryant (2001, p. 177). Baysian Approach to Absent Evidence Reasoning 57 does exist does not mean that you have evidence that it doesn’t.2 Carl Sagan (1996) uses the motto to criticize arguments such as the following: we don’t have any evidence of intelligent life on other planets; hence, there is no intelligent life elsewhere. The fact that there is no evidence of intelligent life is not, according to Sagan, evidence that they do not exist.3 The motto has also come under criticism4. Jonathan Adler (2002, p. 107) says that the motto is “false in both detail and in spirit.” Adler gives several counterexamples to the motto, including one from Arthur Conan Doyle’s famous story ‘Silver Blaze’. Sher- lock Holmes notices that a dog does not bark (absent evidence) and concludes that this is evidence that there is no criminal who is a stranger—the culprit must have known the dog. Similarly, the lack of evidence of alien abductions is evidence that there are none. The absence of bear tracks in the snow around my cabin is evidence that no bears have visited recently, and so on. Sagan uses the motto to criticize examples of reasoning from ignorance, which he marks as fallacious. Douglas Walton (1996), however, correctly notes that some arguments from ignorance are not fallacious, and can be represented as a kind of modus tollens: I don’t have any evidence that p. (CP1) If p were true, I’d have evidence that p is true. Therefore, p is false. This explains why the examples about absent evidence reasoning in the cases of alien abduction and bear tracks are convincing. If (CP1) is true, then absent evidence is evidence of absence. On the other hand, the following form of absent evidence rea- soning is fallacious: I don’t have any evidence that p. (CP2) If p were true, I would not have evidence that p is true. Therefore, p is false. 2 These remarks were made at a press conference at NATO headquarters in Brussels, Belgium. See U.S. Department of Defense News Transcript for June 6, 2002. 3 Sagan (1996, p. 213). To be fair to Sagan, when he describes the argu- ment as fallacious, he characterizes it in a deductive way: one cannot conclude that p must be true because p hasn’t been proved false (or vice versa). This is surely right, but it is natural to wonder what one should say in more nuanced cases where the relationship between the evidence and hypothesis is probabilistic. 4 See also Thornton and Kimmel-Lake (2006, p. 202). Christopher Stephens 58 Thus, Walton (1996) concludes that the truth of the conditional premise (CP1) or (CP2) is crucial for evaluating whether or not ar- guments from ignorance are fallacious. His analysis of such argu- ments is done in a non-probabilistic framework. This paper exam- ines the motto in cases where evidence is a matter of probability.5 In such cases, neither (CP1) nor (CP2) is true. 2. An introduction to Bayesian epistemology Bayesian epistemologists use a theorem from probability theory to explicate various notions of evidence and confirmation6. Before one makes a new observation, one assigns a certain probability to a given hypothesis H—it may be low, middling or high. After one makes a new observation O, one then updates the probability one has assigned to H. The probability that H has before the observa- tion O is known as the prior probability and is written as Pr(H). The probability that H has in light of the observation is known as the posterior probability, which is written as Pr(H|O). Bayes’ theo- rem tells us how the posterior and prior probabilities are related. It can be derived from a simple definition of conditional probability. Following Kolmogorov (1933), conditional probability is defined as: Pr(H|O) = Pr(H & O)/Pr(O) where ‘Pr(H|O)’ is understood as the probability of H, on the as- sumption that (or given that) O. This definition makes sense. Sup- pose a player catches a glimpse of a card dealt at random from a standard deck as it is passed it to her neighbour: she sees only that it is red. She can then ask: what is the probability that it is the ace of hearts, given that it is red? The definition of conditional prob- ability tells us that it is equal to the probability that it is red and that it is the ace of hearts (which is 1/52) divided by the probability that it is red (which is ½). This is equal to 1/26, which is the intuitively 5 Oaksford and Hahn (2004) and Hahn and Oaksford (2006) were the first to apply a Bayesian approach to thinking about arguments from igno- rance. They focus more on using Bayesian epistemology to measure the strength of arguments from ignorance; my focus is more directly on the absence of evidence motto. But there are several important points of con- tact with their work; this paper should be viewed as complementary to theirs. Sober (2009) also examines absent evidence reasoning in a prob- abilistic framework in the context of debates about fossil evidence and fine-tuning arguments. 6 For a more complete introduction to Bayesian epistemology, see How- son and Urbach (1989). Baysian Approach to Absent Evidence Reasoning 59 right answer for the probability that a card drawn from a fair deck is the ace of hearts, on the assumption that it is red. By switching the order of H and O in the definition of condi- tional probability, it is also true that Pr(O|H) = Pr(O & H)/Pr(H), and since Pr(O & H) = Pr(H & O), we can define Pr(H & O) in two ways: Pr(H & O) = Pr(H|O)Pr(O) = Pr(O|H)Pr(H) From this, one can obtain Bayes’ theorem: Pr(H|O) = Pr(O|H)Pr(H)/Pr(O) Bayesians think of confirmation in the following way: observation O confirms hypothesis H if and only if the probability of H condi- tional on the observation is greater than the unconditional probabil- ity of the hypothesis, that is: Pr(H|O) > Pr(H). It should be kept in mind, however, that even if some observation O confirms H, it does not necessarily mean that the hypothesis should be believed. Even if the probability of H in light of the observation is higher than be- fore the observation, H can still be very improbable. Besides a notion of diachronic confirmation, one can also use Bayesianism to compare two hypotheses in light of the same obser- vations. Applying Bayes’ theorem to both a hypothesis and its ne- gation, we get: Pr(H|O)>Pr(¬H|O) if and only if Pr(O|H)Pr(H)>Pr(O|¬H)Pr(¬H) This tells us that whether a hypothesis or its negation has a higher posterior probability depends on both the prior probabilities in question, as well as the probabilities that each hypothesis confers on the observations.7 Finally, there is a third aspect of confirmation that can be captured in the Bayesian framework. In some cases, one is inter- ested simply in what the impact of the observation is on a pair of hypotheses (or on a hypothesis and its negation). One is not inter- ested in whether one hypothesis has a higher prior probability than another. In such cases, we can think of confirmation as: H is better supported than ¬H by O if and only if Pr(O|H)>Pr(O|¬H) 7 A more general formulation of this would allow you to compare any two hypotheses, even if they are not exhaustive. Christopher Stephens 60 3. A Bayesian approach to absent evidence reasoning What does the Bayesian approach to epistemology say about the traditional motto that the absence of evidence is not evidence of absence? There are two parts to the motto: the absence of evidence and the evidence of absence. My suggestion is that we understand the absence of evidence as the claim that one looked for some evi- dence—suspicious looking buildings, bear tracks in the snow or the sound of a dog barking—but failed to find such evidence. This con- stitutes absence of evidence. Our examples involve hypotheses that make claims that something exists—whether it is weapons of mass destruction in Iraq, a bear outside my cabin or a stranger passing by the backyard. We can explicate evidence of absence as the claim that some obser- vation decreases the probability of these hypotheses. So if an ob- servation lowers the probability that there are weapons of mass de- struction in Iraq, that there is a bear outside of my cabin, or that a stranger walked by the yard, these observations would count as “evidence of absence”—evidence, that is, that WMDs, bears or strangers are absent. Let H be one such hypothesis that makes an existence claim (e.g., that there are WMD in Iraq; that a bear has visited my cabin; that a stranger passed by the yard) and Pr(H) = p is the prior prob- ability of H. If, as the motto suggests, the absence of evidence is not evidence of absence, then conditionalizing on the absence of evidence8 should not decrease one’s probability in H; on the other hand, if the absence of evidence is evidence of absence, condition- alizing on the absence of evidence should decrease one’s probabil- ity in H (that there are WMDs; that a bear has visited my cabin, or that a stranger passed by the yard). Under some very plausible as- sumptions, one can see that the absence of evidence is evidence of absence. Suppose that there is some observational evidence E (satel- lite photos of suspicious looking warehouses of a certain size (Rumsfeld’s case) or bear tracks in the snow, or the barking of a dog (Sherlock Holmes’ case) that would support, but not guarantee, the truth of H. That is, rather than a deductive relationship between E and H, imagine that there is merely a probabilistic correlation between E and H. Three crucial assumptions of my model: 8 There is a delicate issue about what exactly the ‘absence of evidence’ means. Presumably absent evidence should be time indexed—e.g., “no bear tracks within the past week” or “no suspicious looking buildings big enough to house weapons of mass destruction in the last year”. Baysian Approach to Absent Evidence Reasoning 61 (1) Uncertainty in the relevant hypothesis: Pr(H) = p < 1 Other- wise, one might think: why bother looking for evidence if one is already certain of the hypothesis? (2) Genuine Evidence: Pr(E|H) > Pr(E|¬H), so that the evidence we’re looking for is evidence for H. It follows that Pr(¬E|¬H) > Pr(¬E|H). (3) Total Evidence: the agent takes into all known evidence9 and relevant background knowledge K. It is possible for Pr(E|H & K) = Pr(E|¬H & K), even though assumption (2), that Pr(E|H) > Pr(E|¬H), is true. Consider again the bear track example. Ac- cording to (2), we are assuming that the probability of observ- ing bear tracks is greater if there is a bear than if there is not. But suppose one has the background knowledge that my friend who likes to cover up real bear tracks is staying in the cabin with me. In this case, the probability of there being no bear tracks is the same regardless of whether or not a bear has visited recently. In such a case, one would be mistaken to think that the absence of “evidence” (no bear tracks) is evidence of absence (no bear). But in doing so one would be ignoring a crucial bit of knowledge about the relationship between bear tracks and the existence of bears. Once one does pay attention to the further evidence that one is staying with a friend who always covers up any bear tracks that appear, then condition (2) fails. The ab- sence of bear tracks is no longer evidence that no bear has vis- ited recently. Of these three assumptions, the Genuine Evidence condition plays the most important epistemic role. Condition (2) rules out the possibility that the evidence is misleading or false. If a bear is actu- ally more likely to be around when there are no bear tracks, then the absence of bear tracks is not evidence that no bear is around, and the motto holds true: the absence of evidence is not evidence of absence. To assess our motto, we want to know what the probability of H is conditional on searching but failing to find evidence. One might worry that if an agent doesn’t even search, then “absent evi- dence” is easy to come by. However, if I don’t even look for bear tracks, then Pr(I see bear tracks|Bear visited) = Pr(I see bear tracks|No bear visited) ≈ 0, and condition (2) fails. So the model only applies to cases where the agent searches for evidence. It is a well known consequence of Bayesian confirmation theory that if Pr(E|H) > Pr(E|¬H), then E confirms H. It similarly follows that if Pr(¬E|¬H) > Pr(¬E|H), then ¬E confirms ¬H. The Genuine Evidence condition states that the antecedent of this latter 9 This is sometimes known as the principle of total evidence. See Good (1967). Christopher Stephens 62 conditional is true, and so in such cases we should expect that ¬E confirms ¬H. But this is just to say that the absence of evidence (¬E) is evidence of absence (¬H). If the dog fails to bark, then this is evidence that there is no stranger in the yard. If no bear tracks are found, this is evidence that there are none. Both of these claims are true if the above conditions hold. Similarly, if the probability of getting positive evidence (suspicious looking buildings of a certain size) for weapons of mass destruction is greater if weapons of mass destruction really exist than if they don’t, then the absence of evi- dence is evidence of absence. Looking and failing to find weapons of mass destruction—if these conditions are met—is evidence that they don’t exist. Notice that it doesn’t matter what value you initially assign to the prior probability that weapons of mass destruction exist in Iraq—as long as it isn’t a probability of one—and the evidence meets the minimal reliability condition that I’ve referred to as Gen- uine Evidence (which is an intuitive way of what it means to say something is evidence), then the slogan is false. In Sagan’s case, he thought that the failure (so far) to have found intelligent life on other planets (absence of evidence for in- telligent life) was not evidence that there is none (evidence of ab- sence). So for the motto to be true in the situation that Sagan de- scribes, it must be the case that one of our assumptions is not met— perhaps random radio signals are actually more likely to be from an intelligent agent—if so, then one of our assumptions (that Pr(¬E|¬H) > Pr(¬E|H)) would be violated. But if this is not plausi- ble, then we should also conclude, in Sagan’s case, that absence of evidence of a certain non-random radio signal is evidence that there is no intelligent life. Thus it appears we have vindicated Adler’s suspicions about the motto. 4. Conclusions Three caveats are in order. First, this Bayesian approach only tells us that the absence of evidence should increase our degree of be- lief—it doesn’t say by how much.10 Second, it doesn’t tell us that all things considered we ought to believe that there are no weapons of mass destruction or intelligent life elsewhere. We might have lots of other evidence, for or against these claims. Finally, it should be noted that perhaps Rumsfeld and Sagan would both claim that they were thinking of a deductive framework—the absence of evi- dence does not logically entail the evidence of absence. This is true 10 There is considerable controversy among Bayesians about how best to measure degree of confirmation—see Fitelson (1999) for discussion. Baysian Approach to Absent Evidence Reasoning 63 enough, though both of their examples seem naturally treatable in a probabilistic fashion. The Bayesian argument I have given of the motto’s limita- tions is straightforward. So what accounts for the persistence and popularity of the saying that the absence of evidence is evidence of absence? One possibility is that defenders of the motto are thinking, like Sagan, of the logical point that you cannot deduce that p must be true (or false) from the fact that p has not been proved false (or true).11 While this may be part of the explanation for its popularity, I think part of its attraction also comes from thinking about statisti- cal inference in more frequentist, Neyman-Pearson setting. For ex- ample, just because a study shows that there is absence of evidence of a difference between two hypotheses—one of which involves a control (placebo) and another that involves some drug treatment, this doesn’t mean one can conclude that there is evidence that there is no difference between the placebo and the drug in question. If the study is not well designed, it is misleading to characterize its results as “negative” evidence because one cannot calculate the power of a test from its significance. In other words, one should not infer from the fact that one has evidence that one has not committed a type- one error (falsely rejecting the null) that one has not committed a type-two error (falsely failing to reject the null).12 There is another important sense in which the motto is ar- guably true: if one has failed to even look for evidence of the harm- ful side effects of some drug, one should not conclude from this that there is (even slight) evidence that the drug has no harmful side effects. This is why it is important to keep in mind the fact that the agent must actually search for (and fail to find) the evidence in question. If the agent does not even look for evidence for the hy- pothesis in question, then, as I argued in section 3, the Genuine Ev- idence condition will fail. The Genuine Evidence condition will also fail if the kind of “evidence” for H that the agent searches for is actually more likely to be an indication of ¬H than of H. Relat- edly, there is an important distinction between the claim that it is not the case that there is evidence of H and the claim that there is 11 Woods, Irvine and Walton (2000, p. 61) characterize the ad ignoran- tiam fallacy as resulting from confusing refutations in the strong sense (where p has been shown to be false) with refutations in the weak sense (where some argument for p has been shown to be problematic). 12 See Hartung, Cottrell and Giffin (1983) and Altman and Bland (1995) for discussion of the motto in a frequentist setting. To say that there is controversy about the foundations of statistical inference is something of an understatement, with at least three major positions represented by Bayesians, frequentists and likelihoodists. This debate is obviously be- yond the scope of the present paper. For an introduction to the Bayesian approach, see Howson and Urbach (1989), for frequentism, see Mayo (1996), and for Likelihoodism, see Royall (1997). Christopher Stephens 64 evidence of ¬H; the original motto is plausible if “absent evidence” is understood in the former, but not the latter, way. Merely not hav- ing evidence of H is not enough to confirm ¬H. In many ordinary cases of absent evidence reasoning, however, the agent in question has searched for but failed to find the relevant evidence. The U.N. weapons inspection team did search for weapons of mass destruc- tion, the SETI project has searched for intelligent life, and Sherlock Holmes did search (listen) for the sound of the barking dog. In each of these cases, conditions (1) through (3) are plausible, and the motto is false: the absence of evidence is evidence of absence. Acknowledgements: Thanks to Paul Bartha, John Beatty, Andrew Irvine and an anonymous referee for extremely helpful comments on previous drafts. References Achinstein, Peter. (2000). “Why philosophical theories of evidence are (and ought to be) ignored by scientists,” Philosophy of Science 67, Vol. 3: S180-S192. Adler, Jonathan. (2002). Belief’s Own Ethics. Cambridge, MA: MIT Press. Altman, D.G. and J.M. Bland. (1995). “Absence of evidence is not evidence of absence” British Medical Journal 311: 485-6. Bryant, Edwin. (2001). The Quest for the Origins of Vedic Culture: The Indo-Aryan migration. Oxford: Oxford University Press. Fitelson, Branden. (1999). “The Plurality of Bayesian Measures of Confirmation and the Problem of Measure Sensitivity” Phi- losophy of Science 66: S362-S378. Good, I. J. (1967). “On the Principle of Total Evidence” The British Journal for the Philosophy of Science 17: 319-321. Hahn, Ulrike and Mike Oaksford. (2006). “A Normative Theory of Argument Strength,” Informal Logic 26: 1-24. Hartung, J., J.E. Cottrell and J.P. Giffin. (1983). “Absence of evi- dence is not evidence of absence” Anesthesiology 58.3: 298- 300. Howson, Colin and Peter Urbach. (1989). Scientific Reasoning: The Bayesian Approach. Chicago: Open Court Publishing. Kolmogorov, A.N. (1933). Foundations of the Theory of Probabil- ity, 2nd English edition (1956). Translated and edited by N. Morrison. New York: Chelsea Publishing. Mayo, Deborah. (1996). Error and the Growth of Experimental Knowledge. Chicago: The University of Chicago Press. Baysian Approach to Absent Evidence Reasoning 65 Mayo, Deborah. (2000). “Experimental Practice and an Error Sta- tistical Account of Evidence,” Philosophy of Science 67, No. 3, S193-S207. Oaksford, Mike and Ulrike Hahn. (2004). “A Bayesian Approach to Arguments from Ignorance,” Canadian Journal of Experi- mental Psychology 58: 75-85. Royall, Richard. (1997). Statistical Evidence—A Likelihood Para- digm. Dordrecht: Chapman and Hall. Sagan, Carl. (1996). The Demon-Haunted World: Science as a Candle in the Dark. New York: Ballantine Books. Sober, Elliott. (2009). “Absence of Evidence and Evidence of Ab- sence: Evidential Transitivity in Connection with Fossils, Fishing and Fine-Tuning and Firing Squads” Philosophical Studies 143: 63-90. Thornton, John and Donna Kimmel-Lake. (2006). “Trace evidence in crime reconstruction” in Jerry Chisum and Brent Turvey (Eds.), Crime Reconstruction, pp. 197-214. New York: Aca- demic Press. Turvey, Brent. (2002). “Stage Crime Scenes” in Brent Turvey (Ed.), Criminal Profiling: An introduction to Behavioral Evi- dence Analysis, pp. 249-287. New York: Academic Press. Walton, Douglas. (1996). Arguments from Ignorance. University Park, PA: The Pennsylvania State University Press. Woods, John, Andrew Irvine and Douglas Walton. (2004). Argu- ment: Critical Thinking, Logic and the Fallacies. Toronto: Prentice Hall.