KEY EVENTS On November 23, 2021, Dr. David Mandel presented Communicating Uncertainty in Warning Intelligence at the 2021 CASIS West Coast Security Conference. The primary concepts of Dr. Mandel’s presentation centered on the utilization of verbal versus numeric probabilities, the variability in understandings of verbal probabilities, and the relationship between confidence levels and event probabilities. Dr. Mandel’s presentation was followed by a question and answer period directed at a group of panelists allowing the audience and CASIS Vancouver executives to directly engage with the content of each speaker’s presentation. NATURE OF DISCUSSION Presentation The overarching theme of Dr. Mandel’s presentation focused on how uncertainty is communicated and how shortcomings of this communication have impacts on decision makers using intelligence assessments. Dr. Mandel outlined the inconsistencies in what is understood by verbal probability terms both between and within contexts and individuals. Despite popular guidance to analysts that states event probabilities and confidence levels are distinct, Dr. Mandel illustrated the indistinct nature of how confidence levels affect understandings of probability and vice versa. Thus, Dr. Mandel demonstrated challenges to communicating uncertainty and stressed the inefficacy of intelligence assessments if they cannot be communicated clearly. COMMUNICATING UNCERTAINTY IN WARNING INTELLIGENCE Date: November 23, 2021 Disclaimer: This briefing note contains the encapsulation of views presented by the speaker and does not exclusively represent the views of the Canadian Association for Security and Intelligence Studies. David Mandel 134 The Journal of Intelligence, Conflict, and Warfare Volume 4, Issue 3 Question Period The question period highlighted the difficult nature of suggesting one specific model for improving the communication of uncertainties because of the multitude of requirements, contexts, and scenarios in which intelligence assessments are utilized by practitioners. The misunderstanding of what it means to communicate using numeric probability was also discussed. BACKGROUND Presentation Dr. Mandel’s presentation began with a depiction of the ‘communication mode preference paradox,’ which states that senders of information prefer to use verbal explanations of probabilities, whereas receivers prefer numeric probabilities. As demonstrated by multiple studies, there is variability in what is understood by verbal probability terms, including those that are often used in intelligence assessments: likely, highly likely, probable, etc. This variability is visible between individuals, as well as within certain individuals across different contexts. Dr. Mandel terms this the ‘illusion of shared understanding,’ leading individuals to often use these terms without confirming with the other party a common understanding or definition. Dr. Mandel presented the standards used by the Office of the Director of National Intelligence (USA), the Professional Head of Intelligence Assessment (UK), and NATO joint intelligence doctrine, wherein each body maps a numeric probability range onto a curated set of verbal probability terms. Although these bodies share the common approach of assigning numeric ranges to each linguistic term, the upper and lower bounds for each term do not line up across all three bodies, thus creating barriers to interoperability. For example, there is a potential breakdown in communication when for the United States the term ‘likely’ means roughly 55%-85% chance but for NATO the term ‘likely’ means 60%-90% chance. Furthermore, research indicates that people’s interpretations and uses of verbal probability do not comply with the given schemes provided by these bodies. Dr. Mandel went on to iterate the importance of being able to garner meaning from multiple different probability assessments. This might occur when decision makers are given an assortment of estimates for one event, where probabilities don’t align and they need to average them. Another scenario could be when trying to work out the probability of a threat that is comprised of multiple factors, all of which are necessary conditions. With this in mind, Dr. Mandel discussed David Mandel 135 The Journal of Intelligence, Conflict, and Warfare Volume 4, Issue 3 experiments undertaken with three other colleagues to discern whether people were more successful navigating these scenarios when given verbal or numeric probabilities. One of the results of this study indicated that analysts were able to work more accurately when given numeric probabilities rather than verbal. Another significant finding from a related study he presented is that the variability of probability terms has the potential for very real and damaging costs to the accuracy of strategic intelligence forecasts. Further expanding the discussion on the uncertainty of events, Dr. Mandel’s presentation examined how analytic confidence is communicated. While analysts are often guided to assess event probabilities and analytic confidence as separate entities, Dr. Mandel urged that the two are not so easily separated and this principle is itself inherently problematic. In reality, the likelihood, or lack thereof, of an event puts constraints on the available breadth of a reasonable confidence interval. For example, if it is stated an event has a 99% chance of occurring, only 1% is left before the upper bound is reached, which impacts what one might reasonably declare as the confidence level for this assessment. In addition, Dr. Mandel explored the psychological connection between event probabilities and confidence levels. When asked, a pool of analysts indicated that when the term ‘likely’ was coupled with ‘low confidence’ this was interpreted to mean around a 41% chance, but when ‘likely’ was coupled with ‘moderate confidence’ this meant approximately a 57% chance, and when ‘likely’ was coupled with ‘high confidence’ this meant approximately 75% chance. These findings illustrate the role that psychology plays in assessing probabilities due to the quite drastic difference in interpretation of the verbal probability term that came with the shift in confidence levels. In conclusion, Dr. Mandel related that even the best intelligence cannot inform decision makers if it is not communicated clearly. While there is emphasis on improving the quality of assessments, analytic quality will be underutilized if it cannot be communicated clearly. The research findings of Dr. Mandel and others illustrated during this presentation demonstrate that the current standards with which uncertainty is communicated is not effective. Although it is more challenging than using vague language, quantifying both probabilities and confidence levels would provide information that is better suited to aid decision makers. David Mandel 136 The Journal of Intelligence, Conflict, and Warfare Volume 4, Issue 3 Question Period During the question and answer period, the audience asked whether Dr. Mandel had a specific model for changing or improving the communication taking place between analysts and policy makers. Dr. Mandel asserted that suggesting one specific model or method of communication is challenging because of the variety of contexts in which intelligence is provided to those who use this information to make decisions. For example, it may not be necessary to know the exact numeric probability of an event occurring if the only information requested is whether the likelihood is more or less than 50%. Dr. Mandel successfully illustrated the inefficacy of current models in communicating probabilities rather than presenting one specific solution to the plethora of contexts in which this communication poses challenges. There is push-back against using numeric probabilities that is rooted in the idea that numeric values provide a false sense of precision. Dr. Mandel suggested that using numeric probabilities does not have to be precise, it merely has to be clear and not vague. Probabilities can be presented as a range with an upper and lower bound to demonstrate the imprecision of the given conclusion. Presenting probability as a numeric range is not precise but it is very clear, thereby helping decision makers to utilize assessments to the best of their abilities. KEY POINTS OF DISCUSSION Presentation • There is no consensus in what is understood by verbal probability terms. • Dr. Mandel illustrated a recent study of his that demonstrates there is no cost when verbal probability cues are lacking, but there is a cost when numeric probability cues are lacking. Further, there may be a benefit of only presenting numerical probabilities. • Event probabilities and analytic confidence are not separate entities. The two are linked and each has the ability to influence the interpretation of the other. • Dr. Mandel found that in his experiment, the pool of analysts were equally split between preferring numeric versus verbal probability. He found that his general participants preferred numeric probabilities. However, Dr. Mandel found that analysts declared numeric probability to be more informative. • Clear communication of uncertainties allows decision makers to make well- informed decisions. David Mandel 137 The Journal of Intelligence, Conflict, and Warfare Volume 4, Issue 3 Question Period • It is difficult to provide one specific solution to communicating uncertainties because uncertainties are communicated across diverse conditions and using varying requirements. • There is resistance to the suggestion of using numeric probabilities because of the assumption that numeric values imply a scientific precision. • In actuality, numeric probabilities can be presented as being anywhere within a certain range in order to communicate its imprecision clearly without being vague or precise. This work is licensed under a Creative Commons Attribution-Non- Commercial-NoDerivatives 4.0 International License. © (DAVID MANDEL, 2022) Published by the Journal of Intelligence, Conflict, and Warfare and Simon Fraser University Available from: https://jicw.org/