International Journal of Computational Intelligence Systems Vol. XX(Z); Month (Year), pp. xx–yy DOI: 10.1080/XXXXXXXXXXXXXX; ISSN XXXX–XXXX online https://www.atlantis-press.com/journals/ijndc Journal of Risk Analysis and Crisis Response Vol. 9(3); October (2019), pp. 128–133 DOI: https://doi.org/10.2991/jracr.k.191024.002; ISSN 2210-8491; eISSN 2210-8505 https://www.atlantis-press.com/journals/jracr Research Article A Critique of Pandemic Catastrophe Modeling Daniel J. Rozell* Department of Technology and Society, 1432 Computer Science, Stony Brook University, Stony Brook, NY 11794-3760, USA 1. INTRODUCTION Over the past three decades, catastrophe modeling has become an essential tool of the insurance industry for estimating poten- tial financial losses from disasters such as earthquakes and hurri- canes [1]. The basic premise of catastrophe modeling is to combine stochastic hazard models with asset models to assess financial risk exposure from large events. While the original intent was to assess systemic risk, the influence of catastrophe models eventually expanded into related practical tasks—setting rates, allocating cap- ital reserves, selecting markets, etc. Considering the substantial variation in the nature of hazards, one would also expect considerable variation in the utility of catastro- phe models depending on the availability of data and science. That is, more common events should generally yield more reliable pro- jections. This requires a skeptical eye when evaluating new models, yet catastrophe models have become widely adopted and viewed as established (and therefore implicitly reliable) technology. It is normal to want to replicate success in one area to new situations. But the temptation to apply existing catastrophe modeling tech- niques to more uncertain events has gotten ahead of methodologi- cal development. The result is inevitable disappointment. A prime example is pandemic catastrophe modeling where the hazard is a pandemic-inducing pathogen that results in substan- tial life and health insurance losses. While the insurance industry should be commended for tackling such an important topic, the uncertainties are daunting. The difficulty of quantitative pandemic risk assessment is summarized by Harvey Fineberg, the former president of the U.S. National Academy of Medicine, “Major flu pandemics arise on average only about three times every century, which means scientists can make relatively few direct observations in each lifetime and have a long time to think about each observa- tion. That is a circumstance that is ripe for over-interpretation” [2]. Major limitations of hurricane catastrophe modeling—primarily in the form of hidden subjectivity—have already been well described [3]. Many of these limitations are generalizable to similar catastro- phe models: flooding, earthquakes, etc., yet there is a new class of models with exceptional levels of uncertainty that require their own discussion. This critique explores the additional limitations unique to pandemic catastrophe modeling that further limit the value of these types of models for estimating financial exposure, disaster response, or social policymaking. 2. LESSONS FROM EBOLA While pandemic catastrophe modeling has appropriately focused on avian influenza—the most likely cause of a pandemic—the current state of pandemic risk assessment can be evaluated using another recent deadly outbreak—the Ebola virus disease epi- demic that emerged in West Africa in late 2013 [4]. Much like avian influenza, Ebola virus disease has zoonotic origins, but in this case the primary natural reservoir is bats rather than birds. In addition to the much higher mortality rate, Ebola is much less transmissible than influenza. These combined factors tend to limit outbreaks and the pandemic potential of Ebola is much lower than influenza [5]. Despite its more modest growth charac- teristics, Ebola outbreaks provide some insight into the current state of pandemic forecast modeling. During the Ebola outbreak, multiple epidemiological models (Table 1) that made short-range predictions in 2014 considerably overestimated its severity— some by an order of magnitude. Unless the World Health Organization (WHO) underreported cases (which would be a separate serious problem in itself ), the A R T I C L E I N F O Article History Received 12 August 2019 Accepted 30 August 2019 Keywords Catastrophe modeling pandemics uncertainty insurance risk assessment A B S T R A C T Catastrophe modeling is a popular risk assessment tool for the insurance industry and has been applied to a variety of natural disaster events. More recently, catastrophe modeling techniques have been extended to events, such as pandemics, where the range of possible scenarios is less understood due to the complexity of the hazard and the dependence of event magnitude to human response. Some general limitations of catastrophe modeling are discussed in the context of pandemics—such as the failure to distinguish natural variability from incertitude and the difficulty of ensuring a representative model—along with recommendations for minimizing surprises. © 2019 The Authors. Published by Atlantis Press SARL. This is an open access article distributed under the CC BY-NC 4.0 license (http://creativecommons.org/licenses/by-nc/4.0/). *Corresponding author. Email: Daniel.Rozell@stonybrook.edu https://doi.org/10.2991/jracr.k.191024.002 https://www.atlantis-press.com/journals/jracr http://creativecommons.org/licenses/by-nc/4.0/ Daniel.Rozell%40stonybrook.edu D.J. Rozell / Journal of Risk Analysis and Crisis Response 9(3) 128–133 129 general inaccuracy of these predictions suggests that the progres- sion of an epidemic is still difficult to model. Pandemics move quickly and the early stages are sensitive to assumptions regarding public health interventions. Likewise, failure to understand all the transmission routes (e.g., sexual transmission of Ebola by recov- ered patients) can make the end stages of a pandemic equally diffi- cult to predict [10,11]. And it is not that the Ebola epidemic modeling was obviously too alarmist in overestimating the severity of the outbreak. For exam- ple, in July 2014, a Liberian man infected with Ebola flew to Lagos, Nigeria. If that case had not been quickly detected and contained, the epidemic could have entered a metropolitan area of 21 million people. Had that happened, the WHO mortality projections most likely would have been grossly underestimated. The difficulty is that these model estimates are strongly dependent upon many smaller events within the larger epidemic. One potential solution is to move toward agent-based modeling—computer simulations that estimate the aggregated consequences of the independent behavior of many individuals [12]. However, this approach is still very resource inten- sive, complex to build, and challenging to calibrate [13]. In addition, the translation of Ebola transmission risk studies performed in less-industrialized nations to more-industrialized nations presents a considerable challenge [14]. The response to the Ebola outbreak among the most industrialized nations was not uni- form and there was even a noticeable difference in how scientists and public officials estimated risk. Because the transmissibility of Ebola is relatively low, medical experts recommended against the quarantine of U.S. medical workers returning from West Africa. However, the cost and effort of tracking and monitoring poten- tial exposures led public officials to lean toward precautionary quarantines that varied from state-to-state. Estimating the human response is challenging and averaged data tell a very incomplete story. Any useful pandemic risk model would need to account for local and regional conditions that affect an outbreak’s progression. The financial effects of a pandemic may be even more difficult to model. For example, the WHO’s estimate of the cost of con- trolling the West African Ebola outbreak jumped from less than 5 million USD$ in April 2014 to almost 1 billion USD$ in September 2014 [4,15]—an increase by a factor of 200 within 6 months. This demonstrates how difficult it can be to estimate the cost of a poten- tial outbreak when it is not known at what stage the outbreak will be discovered or contained. The secondary economic impacts of a pandemic would also be expected to change rapidly and would need to include a range of extraneous factors that would only increase the uncertainty. One might assume that after all the analysis of the West Africa Ebola outbreak [16], at least Ebola would now be easier to model. Yet the ongoing Ebola outbreak in the Democratic Republic of Congo (DRC) has been extremely difficult to predict. Initial opin- ion on the outbreak was cautious [17], but optimistic because the previous outbreak had made the world realize that Ebola was not just a self-limiting African disease, but had pandemic potential. Thus, the DRC’s prompt call for assistance and the faster WHO response—along with a new highly effective vaccine—all suggested rapid containment of the outbreak [18]. However, the outbreak has continued longer than expected due to multiple factors: persistent regional conflict [19], rampant disinformation [20], widespread use of inadequate health clinics [21], and even the limited availabil- ity of diagnostic tests [22]. The important lesson from this discussion of Ebola is that there is an extra layer of complexity inherent to pandemic modeling that is missing from most other natural disasters. The severity of damage from an earthquake is dependent on infrastructure built before the event, but the magnitude of the event itself is essentially independent of human activity. However, the magnitude of a pan- demic is also dependent on the human response after the begin- ning of the disaster. Political inaction or inadequate public health systems will not make an earthquake larger, but can allow a small local outbreak to spread into a pandemic. Likewise, changing indi- vidual behaviors, such as increased social distancing or just more frequent hand-washing can substantially influence the progression of an infectious disease outbreak [23]. Because social conditions vary widely by region and can change unpredictably, a useful pan- demic catastrophe model is much more difficult to construct than other natural hazard models. A realistic accounting of uncertainty in a pandemic risk model should show that our current incerti- tude is so large as to preclude standard probabilistic quantitative decision-making. 3. UNDERLYING PROBLEMS WITH CATASTROPHE MODELS So, if we are asking too much of pandemic catastrophe model- ing, what is wrong with the models and how do we fix them? The simple answer is that overconfidence in models is the under- lying problem, but this overconfidence has some specific causes. The following outlines two of these problems and proposes mitigating actions. 3.1. The Mistreatment of Uncertainty It is difficult to specifically critique commercially available pandemic catastrophe models because they are proprietary. However, the documentation for several hurricane catastrophe models is available for public inspection as a requirement of the Florida Commission on Hurricane Loss Projection Methodology. Table 1 | Accuracy of Ebola virus disease predictions made in late 2014 Model Timeframe (weeks) Prediction Actual (WHO data) Total cases in Liberia 10 9400–47,000 [6] <7000 Daily cases in Liberia 15 249–545 [7] <50 Reported cases in Liberia 12 20,471–94,143 [8] 7819 Total confirmed and probable cases 6 5740 in Guinea 1820 in Guinea 9890 in Liberia 4240 in Liberia 5000 in Sierra Leone [9] 4602 in Sierra Leone 130 D.J. Rozell / Journal of Risk Analysis and Crisis Response 9(3) 128–133 Given the basic similarities of catastrophe models, some general remarks can be made regarding their attempts to characterize uncertainty by aggregating scenarios through a large number of stochastic simulations. In these cases, the numbers of simulations or aggregation methods are not the issue, but the method of representing uncertainty. There is no current consensus on the best way to represent uncer- tainty or when to use one form over another [24]. Furthermore, there is no consensus on the number of forms of uncertainty. Uncertainty has been categorized across a broad spectrum that ranges from certainty to nescience (unresolvable ignorance) [25]. Bayesians [26,27] generally argue that there is only one type of uncertainty—a measure of belief irrespective of its source—and that probability is the best way to express it [28]. At the other extreme, complex uncertainty typologies have been created that distinguish it by location, nature, and level with multiple subtypes [29,30]. The utility of this level of detail is debatable when there is no method for treating each distinct form of uncertainty [31,32]. While uncertainty representation is an unsettled matter, distin- guishing between incertitude (i.e., lack of knowledge) and natural variability does have known advantages over a single treatment methodology. For example, when an analyst has no knowledge of the value of a parameter within a bounded range, it is common to represent that interval using a uniform distribution (a frequently used “uninformed prior” in Bayesian inference). However, treating interval data as uniform distributions requires an equiprobability assumption—an idea that traces back to Bernoulli’s and Laplace’s ‘principle of insufficient reason’ and more recently critiqued by Keynes under the name ‘principle of indifference’ [33]. Under this assumption, each possible value is considered to be equally likely and thus the interval can be represented by the interval’s mid- point—the mean and median of a uniform distribution. While the uniform distribution approach is computationally expedient and easy to understand, it can also disguise uncertainty. One result of the Central Limit Theory is that any distribution with finite vari- ance, such as a uniform distribution, will converge to a normal distribution approximation when repeatedly convolved [34]. As shown in Figure 1, adding two uniform distributions results in a triangle distribution that would appear to have more certainty than the original uniform distributions. As the number of distributions increases, the resulting normal approximation does spread, but the assumed central tendency is maintained. By comparison, the addi- tion of two intervals of [0, 1] by interval arithmetic yields the even wider interval [0, 2]. Repeating the process yields ever wider and more uncertain intervals without an artifact of precision. The practical implication of this phenomenon is that standard Bayesian and Monte Carlo techniques have an important limita- tion. These approaches will yield valid results when the model inputs represent normally distributed natural variability. However, when the uncertainty due to lack of knowledge is large compared to the natural variability, the uncertainty in the model output will be under-represented. Unfortunately, this situation frequently occurs in rare natural disasters, such as pandemics. In practice, uncertainty representation may be based on extraneous factors such as familiarity, academic tradition, or lack of knowl- edge of alternatives. Catastrophe modelers have been warned in the past to avoid becoming enamored with extreme precision that may not have a basis in reality [35], but they need further guid- ance. Fortunately, there has been considerable work in the area of intervalized probabilities [36,37], second-order Monte Carlo tech- niques, and robust Bayesian or Bayesian sensitivity analysis [38,39] which allows for the recognition of incertitude within Bayesian inference [36]. Models that more explicitly account for incertitude can sometimes yield results that seem disappointingly vague, but an honest portrayal of uncertainty is always informative. However, catastrophe model users typically come from the world of financial modeling where data are abundant and historical prices are precisely known. It is not surprising that they prefer data anal- ysis methods that are not optimal for addressing scenarios with sparse data and large incertitude. Adopting new methods may be a difficult transition despite their utility. 3.2. Failures of Imagination In a workshop on the West Africa Ebola epidemic, virologist Daniel Bausch noted how prior relatively small outbreaks influenced expectations, “I think if you’d asked [Ebola virus experts] … a year and a half ago ‘Are we going to have 25,000 cases of Ebola in West Africa?,’ most of us would have not said that was a likelihood” [4]. It is common for modelers to claim that their models have been Figure 1 | Convolution of n uniform distributions with bounds [0, 1]. D.J. Rozell / Journal of Risk Analysis and Crisis Response 9(3) 128–133 131 “validated” because they can replicate past data. However, this is history matching, not validation [40,41]. Acknowledging this distinction creates less surprise if we encounter extreme events not previously experienced or we discover that a model contains non-stationary processes that invalidate a model for future use. In terms of pandemic catastrophe models, diseases can behave unpredictably due to evolving pathogens or changing demo- graphic conditions. For example, the 2017 outbreak of plague in Madagascar—normally a relatively controllable bacterial disease— was exacerbated by its spread to growing urban centers. The user’s trust in any catastrophe model depends on assurances of validation (i.e., history matching) which may be acceptable in the case of relatively stable processes, such as earthquakes, but more tentative for pandemic catastrophe models that depend on non- stationary processes [42]. After catastrophic events, modelers have an opportunity to check their model against actual losses, so it’s not surprising that catastrophe model outputs will change after these recalibrating updates. However, it can be rather difficult to detect changes in the probabilities of extreme events caused by non-sta- tionary processes [43]. One practical output of a catastrophe model is an exceedance prob- ability curve, which summarizes the annual probability of exceed- ing particular losses—most importantly, a loss that would lead to the insolvency of an insurer. To generate these curves, catastrophe models will typically simulate many potential scenarios that include both historical data as well as scenarios that attempt to address lim- itations in the representativeness of the historical data set. This is especially necessary for models that include known non-station- ary processes such as sea level rise, changing building technology, increased coastal development, etc. However, the Ebola epidemic serves as an example of a previously unforeseen risk that would not be captured merely by modestly expanding the range of a catastro- phe model’s simulations. Another lesson from the Ebola epidemic is that the various out- breaks within the larger epidemic had very different transmission dynamics depending on the level of cooperation and trust between the international public health workers, government officials, and affected communities. The impact of these social factors is critical to any risk assessment and difficult to characterize. Furthermore, new factors and previously unknown relationships in epidemic evo- lution are emerging as large epidemiological data sets are analyzed [44]. It would seem that a truly representative catastrophe model of a real-world epidemic would tend towards substantial complexity. One thoughtfully constructed pandemic catastrophe model [45–47] considers a myriad of factors including: wild and domestic viral reservoirs; the virulence and transmissibility of potential pandemic pathogens; the age and density distribution of populations; seasonal impacts; social factors such as air travel, work commutes and travel restrictions; and the availability of quality health care, antiviral medi- cations, and vaccines. Whether these are enough factors is debatable. Estimating the insurance losses from a hurricane is fairly well-known and straightforward by comparison, yet building construction and risk mitigation details frequently ignored in standard catastrophe models can have a substantial impact on risk exposure estimates [48]. Furthermore, the impact, timing, and magnitude of each factor must be estimated along with any correlation with other factors. In the end, we are left with the question of whether to pursue a more detailed pandemic catastrophe model that might be unusably complex or a simpler model that may be missing critical factors. There is a common, but often untested, assumption that complex problems call for complex solutions and that it is always better to use complex meth- ods that make use of all available information [49]. However, there is a history of studies showing that very simple predictive models can often have better performance than complex models [50,51]. The question of best modeling approach has been explored in recent years in the epidemiological community in a series of infectious dis- ease forecasting challenges [52]. For example, in 2015, the Research and Policy for Infectious Disease Dynamics (RAPIDD) Ebola fore- casting challenge compared the predictive abilities of eight Ebola epidemiological models against a synthetic set of data over four sce- narios at multiple points in an outbreak [53]. Models varied in com- plexity and ranged from 2 to 27 input parameters. The results did not point toward an “ideal” level of complexity because one of the best individual performers was a two-parameter stochastic model, while the worst performer was a two-parameter logistic growth model. The overall best performer was a Bayesian average of all the models which suggests that using multiple independent models is the current best approach. However, the last challenge scenario of an uncontrolled Ebola outbreak with noisy data was poorly predicted by all models, so even multi-model averaging has its limitations. In a similar forecasting challenge for influenza in the U.S. over seven flu seasons, almost three-quarters of the 22 models evalu- ated made better seasonal predictions than the historical base-line model and the best performer was again a weighted combination of other models [54]. However, none of the models replicated the observed data more than 50% of the time and the challenge did not include an influenza pandemic—the models were optimized to predict seasonal flu. Current epidemiological modeling for emerging infectious dis- eases tends toward using relatively simple models that only require the estimation of a few parameters primarily due to the scarcity of data [55,56]. More complex tools for assessing the pandemic risk of influenza [57,58] have not been widely adopted—one reason being their substantial data requirements. Ultimately, choosing the level of model detail is a professional judge- ment in the art of modeling until the state of the art is improved. The current best solution to the problem of potentially non-representa- tive data, processes or models is more attitudinal than technical. In the face of ignorance, the best approach is to use multiple competing models and constantly question their assumptions and results. In the case of epidemic modeling, real-time prediction remains a difficult challenge [59]. Furthermore, longer-term forecasting and predicting the emergence of new pathogens with pandemic potential will likely remain beyond our capabilities for the near future [60]. Given the dynamic nature of factors underlying epidemic progression, there may be limits on long-term predictability of infectious disease out- breaks and new analytical methods may be needed [61]. 4. CONCLUSION Catastrophe models can err in disaster forecasting for a variety of reasons. Here I have outlined one reason that is essentially an error of mathematical method and another based on an important psychological bias related to natural human optimism. However, there are others. For example, a pernicious cause of overconfidence in modeling is the human tendency toward excessive admiration of our own creations [62,63]. 132 D.J. Rozell / Journal of Risk Analysis and Crisis Response 9(3) 128–133 Ultimately, savvy users may know what level of confidence to put in their models, but shortcomings in current practices in uncertainty quantification require that model interpretation remain an art as much as a science until more progress is made. Although the focus of this discussion was pandemic modeling, catastrophe models are tackling other complex hazards with strong sociological com- ponents, such as cybersecurity and terrorism risks. Furthermore, standard catastrophe models are expanding into secondary areas such as business interruption losses [64]. Given the potential for overreach, catastrophe model users will need to practice shrewd professional judgment for some time to come. Despite the critiques presented here, it is important to clarify that planning for disasters using data and quantitative methods remains an essential approach that can provide significant insight. While there is danger in putting too much confidence in a model’s results, failing to consider modeling at all presents the much larger danger of miss- ing critical trends and vulnerabilities hidden within an overwhelm- ing mountain of data. The only way forward is to keep improving the models while continuing to use them with a healthy skepticism. CONFLICTS OF INTEREST The author declare they have no conflicts of interest. ACKNOWLEDGMENT I would like to thank Sheldon Reaven for insightful questions and comments regarding this work. REFERENCES [1] Grossi P, Kunreuther H, Windeler D. An introduction to castro- phe models and insurance. In: Grossi P, Kunreuther H, editors. Catastrophe modeling: a new approach to managing risk. Boston, MA: Springer; 2005, pp. 23–42. [2] Fineberg HV. Swine flu of 1976: lessons from the past. An inter- view with Dr Harvey V Fineberg. Bull World Health Organ 2009;87:414–5. [3] Weinkle J, Pielke R. The truthiness about hurricane catastrophe models. Sci Technol Human Values 2017;42:547–76. [4] Mack A, Snair MR, Mundaca-Shah C. The ebola epidemic in West Africa: proceedings of a workshop. Washington, DC: National Academies Press; 2016, p. 120. [5] Adalja AA. Ebola in West Africa: a familiar pattern? Biosecur Bioterror 2014;12:161–2. [6] Gomes MF, Pastore Y Piontti A, Rossi L, Chao D, Longini I, Halloran ME, et al. Assessing the international spreading risk associated with the 2014 West African ebola outbreak. PLoS Currents 2014;6. [7] Pandey A, Atkins KE, Medlock J, Wenzel N, Townsend JP, Childs JE, et al. Strategies for containing ebola in West Africa. Science 2014;346:991–5. [8] Lewnard JA, Ndeffo Mbah ML, Alfaro-Murillo JA, Altice FL, Bawo L, Nyenswah TG, et al. Dynamics and control of ebola virus transmission in Montserrado, Liberia: a mathematical modelling analysis. Lancet Infect Dis 2014;14:1189–95. [9] WHO Ebola Response Team, Aylward B, Barboza P, Bawo L, Bertherat E, Bilivogui P, et al. Ebola virus disease in West Africa—the first 9 months of the epidemic and forward pro- jections. N Engl J Med 2014;371:1481–95. [10] Enserink M. Infectious diseases. In Guinea, a long, difficult road to zero Ebola cases. Science 2015;348:485–6. [11] Hayden EC. Ebola virus and its legacy linger on. Nature 2016;537:291–2. [12] Macal C, North M. Introductory Tutorial: Agent-based Modeling and Simulation. In: Proceedings of the Winter Simulation Conference (WSC) 2014. Savanah, GA, USA: IEEE Press; 2014, pp. 6–20. [13] Venkatramanan S, Lewis B, Chen J, Higdon D, Vullikanti A, Marathe M. Using data-driven agent-based models for forecast- ing emerging infectious diseases. Epidemics 2018;22:43–9. [14] Haas CN. The role of risk analysis in understanding ebola. Risk Anal 2015;35:183–5. [15] Butler D, Morello L. Ebola by the numbers: the size, spread and cost of an outbreak. Nature 2014;514:284–5. [16] Sabeti P, Salahi L. Outbreak culture: the ebola crisis and the next epidemic, 1st ed., Cambridge, MA: Harvard University Press; 2018, p. 288. [17] Frieden TR. Still not ready for Ebola. Science 2018;360:1049. [18] Cohen J. Congo rapidly curtails Ebola. Science 2018;361:211–12. [19] Maxmen A. Violence propels Ebola outbreak towards 1,000 cases. Nature 2019;567:153–4. [20] Spinney L. In Congo, fighting a virus and a groundswell of fake news. Science 2019;363:213–4. [21] Maxmen A. Ebola detectives race to identify hidden sources of infection as outbreak spreads. Nature 2018;564:174–5. [22] Cnops L, De Smet B, Mbala-Kingebeni P, van Griensven J, Ahuka-Mundeke S, Ariën KK. Where are the Ebola diagnostics from last time? Nature 2019;565:419–21. [23] Eksin C, Paarporn K, Weitz JS. Systematic biases in disease fore- casting - the role of behavior change. Epidemics 2019;27:96–105. [24] Flage R, Aven T, Zio E, Baraldi P. Concerns, challenges, and direc- tions of development for the issue of representing uncertainty in risk assessment. Risk Anal 2014;34:1196–207. [25] Tannert C, Elvers H-D, Jandrig B. The ethics of uncertainty. In the light of possible dangers, research becomes a moral duty. EMBO Rep 2007;8:892–6. [26] Apostolakis G. The concept of probability in safety assessments of technological systems. Science 1990;250:1359–64. [27] Winkler RL. Uncertainty in probabilistic risk assessment. Reliabil Eng Syst Safety 1996;54:127–32. [28] O’Hagan A, Oakley JE. Probability is perfect, but we can’t elicit it perfectly. Reliabil Eng Syst Safety 2004;85:239–48. [29] van Asselt MBA, Rotmans J. Uncertainty in integrated assessment modelling. Climatic Change 2002;54:75–105. [30] Walker WE, Harremoës P, Rotmans J, van der Sluijs JP, van Asselt MBA, Janssen P, et al. Defining uncertainty: a conceptual basis for uncertainty management in model-based decision support. Integr Assess 2003;4:5–17. [31] Norton JP, Brown JD, Mysiak J. To what extent, and how, might uncertainty be defined? Comments engendered by “Defining uncertainty: a conceptual basis for uncertainty management in model-based decision support” (Walker et al., 2003). Integr Assess 2006;6:83–8. [32] Krayer von Krauss MP, Walker WE, van der Sluijs JP, Janssen P, van Asselt MBA, Rotmans J. Response to “To what extent, and https://doi.org/10.2471/blt.09.040609 https://doi.org/10.2471/blt.09.040609 https://doi.org/10.2471/blt.09.040609 https://doi.org/10.1177/0162243916671201 https://doi.org/10.1177/0162243916671201 https://doi.org/10.1089/bsp.2014.1563 https://doi.org/10.1089/bsp.2014.1563 https://doi.org/10.1371/currents.outbreaks.cd818f63d40e24aef769dda7df9e0da5 https://doi.org/10.1371/currents.outbreaks.cd818f63d40e24aef769dda7df9e0da5 https://doi.org/10.1371/currents.outbreaks.cd818f63d40e24aef769dda7df9e0da5 https://doi.org/10.1371/currents.outbreaks.cd818f63d40e24aef769dda7df9e0da5 https://doi.org/10.1126/science.1260612 https://doi.org/10.1126/science.1260612 https://doi.org/10.1126/science.1260612 https://doi.org/10.1016/S1473-3099%2814%2970995-8 https://doi.org/10.1016/S1473-3099%2814%2970995-8 https://doi.org/10.1016/S1473-3099%2814%2970995-8 https://doi.org/10.1016/S1473-3099%2814%2970995-8 https://doi.org/10.1056/NEJMoa1411100 https://doi.org/10.1056/NEJMoa1411100 https://doi.org/10.1056/NEJMoa1411100 https://doi.org/10.1056/NEJMoa1411100 https://doi.org/10.1126/science.348.6234.485 https://doi.org/10.1126/science.348.6234.485 https://doi.org/10.1109/WSC.2014.7019874 https://doi.org/10.1109/WSC.2014.7019874 https://doi.org/10.1109/WSC.2014.7019874 https://doi.org/10.1109/WSC.2014.7019874 https://doi.org/10.1016/j.epidem.2017.02.010 https://doi.org/10.1016/j.epidem.2017.02.010 https://doi.org/10.1016/j.epidem.2017.02.010 https://doi.org/10.1111/risa.12361 https://doi.org/10.1111/risa.12361 https://doi.org/10.1038/514284a https://doi.org/10.1038/514284a https://doi.org/10.1126/science.aau3345 https://doi.org/10.1126/science.361.6399.211 https://doi.org/10.1038/d41586-019-00805-7 https://doi.org/10.1038/d41586-019-00805-7 https://doi.org/10.1126/science.363.6424.213 https://doi.org/10.1126/science.363.6424.213 https://doi.org/10.1038/d41586-018-07618-0 https://doi.org/10.1038/d41586-018-07618-0 https://doi.org/10.1038/d41586-019-00212-y https://doi.org/10.1038/d41586-019-00212-y https://doi.org/10.1038/d41586-019-00212-y https://doi.org/10.1016/j.epidem.2019.02.004 https://doi.org/10.1016/j.epidem.2019.02.004 https://doi.org/10.1111/risa.12247 https://doi.org/10.1111/risa.12247 https://doi.org/10.1111/risa.12247 https://doi.org/10.1038/sj.embor.7401072 https://doi.org/10.1038/sj.embor.7401072 https://doi.org/10.1038/sj.embor.7401072 https://doi.org/10.1126/science.2255906 https://doi.org/10.1126/science.2255906 https://doi.org/10.1016/S0951-8320%2896%2900070-1 https://doi.org/10.1016/S0951-8320%2896%2900070-1 https://doi.org/10.1016/j.ress.2004.03.014 https://doi.org/10.1016/j.ress.2004.03.014 https://doi.org/10.1023/A:1015783803445 https://doi.org/10.1023/A:1015783803445 https://doi.org/10.1076/iaij.4.1.5.16466 https://doi.org/10.1076/iaij.4.1.5.16466 https://doi.org/10.1076/iaij.4.1.5.16466 https://doi.org/10.1076/iaij.4.1.5.16466 D.J. Rozell / Journal of Risk Analysis and Crisis Response 9(3) 128–133 133 how, might uncertainty be defined” by Norton, Brown, and Mysiak. Integr Assess 2006;6:89–94. [33] Keynes JM. A treatise on probability. London, UK: Macmillian and Co.; 1921, p. 539. [34] Grinstead CM, Snell JL. Introduction to probability. Rhode Island: American Mathematical Society; 1997, p. 510. Available from: http://www.dartmouth.edu/~chance/teaching_aids/books_ articles/probability_book/amsbook.mac.pdf. [35] Jablonowski M. Do catastrophe models mislead? Risk Manage 2005;52:32–8. [36] Walley P. Statistical reasoning with imprecise probabilities. London: Chapman and Hall; 1991, p. 706. [37] Ferson S, Ginzburg LR. Different methods are needed to propagate ignorance and variability. Reliabil Eng Syst Safety 1996;54:133–44. [38] Berger JO. Statistical decision theory and Bayesian analysis, 2nd ed., New York: Springer Science & Business Media; 1985, p. XVI, 618. [39] Berger JO, Moreno E, Pericchi LR, Jesús Bayarri M, Bernardo JM, Cano JA, et al. An overview of robust Bayesian analysis. Test 1994;3:5–124. [40] Bredehoeft JD. From models to performance assessment: the conceptualization problem. Ground Water 2003;41:571–7. [41] Oreskes N, Shrader-Frechette K, Belitz K. Verification, valida- tion, and confirmation of numerical models in the Earth sci- ences. Science 1994;263:641–6. [42] Manheim D. Questioning estimates of natural pandemic risk. Health Secur 2018;16:381–90. [43] Ceres RL, Forest CE, Keller K. Understanding the detectability of potential changes to the 100-year peak storm surge. Climatic Change 2017;145:221–35. [44] Dalziel BD, Kissler S, Gog JR, Viboud C, Bjørnstad ON, Metcalf CJE, et al. Urbanization and humidity shape the intensity of influ- enza epidemics in U.S. cities. Science 2018;362: 75–9. [45] AIR Worldwide. The AIR pandemic flu model; 2014. Available from: https://www.air-worldwide.com/Publications/AIR-Currents/2014/ The-AIR-Pandemic-Flu-Model/. [46] AIR Worldwide. AIR Worldwide Expands Its Global Pandemic Model; 2016. Available from: https://www.globenewswire.com/ news-release/2016/07/25/858573/10164236/en/AIR-Worldwide- Expands-Its-Global-Pandemic-Model.html. [47] AIR Worldwide. The AIR pandemic model; 2018. Available from: https://www.air-worldwide.com/Models/Life-and-Health/. [48] Karl JB, Medders LA, Maroney PF. The effects of revealed infor- mation on catastrophe loss projection models’ characterization of risk: damage vulnerability evidence from Florida. Risk Anal 2016;36:1224–50. [49] Goldstein DG, Gigerenzer G. Fast and frugal forecasting. Int J Forecast 2009;25:760–72. [50] Makridakis S, Hibon M, Moser C. Accuracy of forecasting: an empirical investigation. J R Stat Soc Ser A 1979;142:97–145. [51] Makridakis S, Taleb N. Decision making and planning under low levels of predictability. Int J Forecast 2009;25:716–33. [52] Viboud C, Vespignani A. The future of influenza forecasts. Proc Natl Acad Sci U S A 2019;116:2802–4. [53] Viboud C, Sun K, Gaffey R, Ajelli M, Fumanelli L, Merler S, et al. The RAPIDD ebola forecasting challenge: synthesis and lessons learnt. Epidemics 2018;22:13–21. [54] Reich NG, Brooks LC, Fox SJ, Kandula S, McGowan CJ, Moore E, et al. A collaborative multiyear, multimodel assessment of sea- sonal influenza forecasting in the United States. Proc Natl Acad Sci U S A 2019;116:3146–54. [55] Metcalf CJE, Lessler J. Opportunities and challenges in modeling emerging infectious diseases. Science 2017;357:149–52. [56] Tuite AR, Fisman DN. The IDEA model: a single equa- tion approach to the Ebola forecasting challenge. Epidemics 2018;22:71–7. [57] Trock SC, Burke SA, Cox NJ. Development of an influenza viro- logic risk assessment tool. Avian Dis 2012;56 4 Suppl:1058–61. [58] Cox NJ, Trock SC, Burke SA. Pandemic preparedness and the Influenza Risk Assessment Tool (IRAT). Curr Top Microbiol Immunol 2014;385:119–36. [59] Kraemer MUG, Cummings DAT, Funk S, Reiner RC, Faria NR, Pybus OG, et al. Reconstruction and prediction of viral disease epidemics. Epidemiol Infect 2018;147:1–7. [60] Holmes EC, Rambaut A, Andersen KG. Pandemics: spend on surveillance, not prediction. Nature 2018;558:180–2. [61] Scarpino SV, Petri G. On the predictability of infectious disease outbreaks. Nat Commun 2019;10:898. [62] Festinger L. A theory of cognitive dissonance. Redwood City, CA: Stanford University Press; 1957, p. 291. [63] Norton MI, Mochon D, Ariely D. The IKEA effect: When labor leads to love. J Consumer Psychol 2012;22:453–60. [64] Rose A, Huyck CK. Improving catastrophe modeling for business interruption insurance needs. Risk Anal 2016;36:1896–915. https://doi.org/10.1016/S0951-8320%2896%2900071-3 https://doi.org/10.1016/S0951-8320%2896%2900071-3 https://doi.org/10.1016/S0951-8320%2896%2900071-3 https://doi.org/10.1007/978-1-4757-4286-2 https://doi.org/10.1007/978-1-4757-4286-2 https://doi.org/10.1007/BF02562676 https://doi.org/10.1007/BF02562676 https://doi.org/10.1007/BF02562676 https://doi.org/10.1126/science.263.5147.641 https://doi.org/10.1126/science.263.5147.641 https://doi.org/10.1126/science.263.5147.641 https://doi.org/10.1089/hs.2018.0039 https://doi.org/10.1089/hs.2018.0039 https://doi.org/10.1007/s10584-017-2075-0 https://doi.org/10.1007/s10584-017-2075-0 https://doi.org/10.1007/s10584-017-2075-0 https://doi.org/10.1126/science.aat6030 https://doi.org/10.1126/science.aat6030 https://doi.org/10.1126/science.aat6030 https://doi.org/10.1111/risa.12524 https://doi.org/10.1111/risa.12524 https://doi.org/10.1111/risa.12524 https://doi.org/10.1111/risa.12524 https://doi.org/10.1016/j.ijforecast.2009.05.010 https://doi.org/10.1016/j.ijforecast.2009.05.010 https://doi.org/10.1016/j.ijforecast.2009.05.013 https://doi.org/10.1016/j.ijforecast.2009.05.013 https://doi.org/10.1073/pnas.1822167116 https://doi.org/10.1073/pnas.1822167116 https://doi.org/10.1016/j.epidem.2017.08.002 https://doi.org/10.1016/j.epidem.2017.08.002 https://doi.org/10.1016/j.epidem.2017.08.002 https://doi.org/10.1073/pnas.1812594116 https://doi.org/10.1073/pnas.1812594116 https://doi.org/10.1073/pnas.1812594116 https://doi.org/10.1073/pnas.1812594116 https://doi.org/10.1126/science.aam8335 https://doi.org/10.1126/science.aam8335 https://doi.org/10.1016/j.epidem.2016.09.001 https://doi.org/10.1016/j.epidem.2016.09.001 https://doi.org/10.1016/j.epidem.2016.09.001 https://doi.org/10.1637/10204-041412-ResNote.1 https://doi.org/10.1637/10204-041412-ResNote.1 https://doi.org/10.1007/82_2014_419 https://doi.org/10.1007/82_2014_419 https://doi.org/10.1007/82_2014_419 https://doi.org/10.1017/S0950268818002881 https://doi.org/10.1017/S0950268818002881 https://doi.org/10.1017/S0950268818002881 https://doi.org/10.1038/d41586-018-05373-w https://doi.org/10.1038/d41586-018-05373-w https://doi.org/10.1038/s41467-019-08616-0 https://doi.org/10.1038/s41467-019-08616-0 https://doi.org/10.1016/j.jcps.2011.08.002 https://doi.org/10.1016/j.jcps.2011.08.002 https://doi.org/10.1111/risa.12550 https://doi.org/10.1111/risa.12550