Vol8No1Paper5 KSS To cite this article: Søilen, K.S. (2018) Book review: Superforecasting: The art and science of prediction. Journal of Intelligence Studies in Business. 8 (1) 46-53. Article URL: https://ojs.hh.se/index.php/JISIB/article/view/286 This article is Open Access, in compliance with Strategy 2 of the 2002 Budapest Open Access Initiative, which states: Scholars need the means to launch a new generation of journals committed to open access, and to help existing journals that elect to make the transition to open access. Because journal articles should be disseminated as widely as possible, these new journals will no longer invoke copyright to restrict access to and use of the material they publish. Instead they will use copyright and other tools to ensure permanent open access to all the articles they publish. Because price is a barrier to access, these new journals will not charge subscription or access fees, and will turn to other methods for covering their expenses. There are many alternative sources of funds for this purpose, including the foundations and governments that fund research, the universities and laboratories that employ researchers, endowments set up by discipline or institution, friends of the cause of open access, profits from the sale of add-ons to the basic texts, funds freed up by the demise or cancellation of journals charging traditional subscription or access fees, or even contributions from the researchers themselves. There is no need to favor one of these solutions over the others for all disciplines or nations, and no need to stop looking for other, creative alternatives. Journal of Intelligence Studies in Business Publication details, including instructions for authors and subscription information: https://ojs.hh.se/index.php/JISIB/index Book review: Superforecasting: The art and s c i e n c e o f p r e d i c t i o n Klaus Solberg Søilena aDepartment of Engineering, Natural Sciences and Economics, Faculty of Marketing, Halmstad University, Halmstad, Sweden; klasol@hh.se Journal of Intelligence Studies in Business PLEASE SCROLL DOWN FOR ARTICLE Book review: Superforecasting: The art and science of prediction. Crown Publishers, New York, NY. Tetlock. E. Philip, Gardner, Dan (2015) Klaus Solberg Søilena aDepartment of Engineering, Natural Sciences and Economics, Faculty of Marketing, Halmstad University, Halmstad, Sweden *Corresponding author: klasol@hh.se Throughout this review I write “the authors” referring to everything that is written in the book, even though I suspect that Tetlock is the leading theorist. Gardner is a journalist, it says on the dust jacket. I do not exactly know what that means when it comes to whose ideas are in the book – who has contributed with what – and I do not want to speculate. Philip E Tetlock is a scholar of psychology with an impressive number of publications and citation, so expectations are high right from the start. And, this is a good book, but not for any of the reasons that it pretends to be one; in fact, it is the opposite. I will get back to this at the end of this review but concentrate on the critique. Forecasting is another word for intelligence work or guessing about the future. When we talk about forecasting we normally think about scientific methods that imply using more quantitative methods, on problems where such methods are thought to be of real use, as in weather forecasting. As we shall see throughout the book, the methods used for actually predicting events in this book are not quantitative but qualitative. That by itself is a problem when the term ‘forecasting’ is chosen, as it is confusing to intelligence professionals. The ‘super’ in ‘Superforecasting’ sounds like something that is made up to sell extra copies of a book. For two authors who place so much value on modesty (as they describe in chapter 12) it’s an odd contradiction to throw the word “super” around in so many forms through a book about the activities one is doing oneself (for example, superforecasters, superteams, superquestions, supersmart, superquants and supernewsjunkies). I guess all professional like to be “super”, but super is something that others say about our work, not something we use to describe our own work and it is difficult to find any irony most of the time when the prefix is being used about how well the authors/project/project members did. It’s quite possible that the authors thought that the ambiguity and playing with irony would go over well with the reader, but it does not. The subtitle is the ‘art and science’. It’s a popular subtitle in English but does not say much as it suggests everything (both a science and an art), thus nothing. What is normally more interesting to know is if the authors see something as a science or an art and why. Again, the impression is one of selling more copies of the book. Chapter one throws arounds names and parallels like Bill Gates and his anthropological work and Tom Friedman and his thesis about the flat world. The project the authors work with is “The Good Judgement Project,” which sounds like something pulled out of a commentary to the bible. More interesting, the authors explain how their wok is supported by the American Intelligence Community (IC) and that its participants have outperformed other analysts. This is a claim throughout the book which is never explained in any detail. We are not told much about how the actual competition was arranged, for example how the answers were graded. We are only given some example of questions asked and presented with names of some participants Journal of Intelligence Studies in Business Vol. 8, No. 1 (2018) pp. 46-53 Open Access: Freely available at: https://ojs.hh.se/ 47 that used the authors methods (the superforecasters) and how well they did compared to others. The book promises that the key to becoming a good forecaster using the method is not math skills, or an abundance of reading or excellent knowledge of history or geography, but comes down to some simple methods of psychology. In other words, good predictions all come down to how you think (not what you know), the authors claim. It is about thinking in a way that is “open-minded, careful, curious and – above all – self-critical. It also demands focus” (p. 20). Now, If I had been a few decades younger I would have been very excited at this point in the book with the promise of a quick solution, a method available to everyone (who reads the book), but these personal qualities, as much as they are required, are just the beginning of good forecasting. At this point in the book I get the feeling that I just saw an infamous gambler ride into town. Chapter two cannot wait to provoke with its title: “Illusions of Knowledge”. We are told some quick, smart stories from the history of medicine where the moral is that we should be critical, as in scientific rigor. Then we should think about how we think, a favorite idea among psychologists. The chapter goes on to talk about Kahnemann and Tversky (colleague) and abruptly ends without every really explaining what illusions are found in knowledge or having ever come close to treating the topic of knowledge more than superficially. By this time it is unclear whether or not it is worth reading the rest of the book. The suspicions from having read the title and the few introductory pages are confirmed. In chapter three, entitle “Keeping Score,” we are introduced to an old legend, the historian Sherman Kent, who was one of the first people in modern times to introduce science into intelligence work. To ensure his analysts were using the same language, Kent defined 100% certainty as “certain”, 95% as “almost certain”, 75% as “probable”, 50% as “chances about even”, 30% as “probably not”, 7% as “almost certainly not” and 0% as “impossible”. The idea that this would help analysts use the same measure, thus increasing accuracy in predictions. The idea was also good, but never became widespread. One could object that if you use a Likert scale of seven it would make more sense to set the percentages with 14.3 percentage point intervals (for example, 100- 85.7%). To allow for the 50% mark, it would make more sense with a five-grade Likert scale. The authors do not comment on this but conclude that the system was never adopted. What they do note is that what is here presented as objective statements is subjective. That in itself is a strange comment as it excludes the possibility that some observations are facts (100% and “certain”), and that all statements are subjective. The authors go on to say that at the end all these estimates can only be presented as opinions, which depends entirely upon what kind of questions the scale goes on to measure (for example, natural facts or predicting human behavior at time t). What did remain in IC after Kent was the use of probabilities, such as when IC told Obama that there was a 70% or 90% probability that the man in the Pakistani compound was Osama bin Laden (p. 59). What that implies is more disturbing, that Obama decided to lead a military operation into a foreign country (a military ally) without even consulting their government when there was a 10-30% probability that they were wrong. The same logic goes to explain why so many civilians are killed with drones and other air strikes; the US has a policy of bombing targets when they are not quite sure who the targets are. The authors go on to argue for the value of the Brier score that measures the accuracy of probabilistic predictions. But they fail to note that the Brier score becomes inadequate for very rare (or very frequent) events, because it does not sufficiently discriminate between small changes in forecasts. The authors fail to see the fundamental difference between predicting the weather with fewer and easier variables to measure and predicting human behavior which consists of many more variables that are more difficult to measure and that frequently vary under the same conditions, such as when a customer suddenly decides not to buy an ice-cream on a hot day even though he did so a week ago under similar conditions. Not to mention the unreliability of the rationality assumptions, which are largely avoided in the book. Too many analysts think ideologically, and try to fit their observations with their beliefs. What does not fit is treated as an irrelevant distraction. They are also likely to declare things “impossible” or “certain”, the authors remind us. This brings us to a key element in the method that is presented, that the “superforecasters” are taught to express themselves more carefully. This is illustrated 48 in the allegory of the fox and the hedgehog by Isaiah Berlin. The foxes win by “playing it safe with 60% and 70% forecasts where hedgehogs boldly went with 90% and 100%” (p. 69). This is the same in Obama’s dilemma presented above. What actually happens is that the risk of mistake is transferred from the intelligence analyst to the decision maker. The decision maker is tempted to give the go-ahead if he is presented with something that has a 60% or higher probability. If things go wrong then the intelligence analyst can always say it was not his fault as there was a 40 or 30% chance of failure or mistake. Does this mean we have a better method for intelligence analysis? No, of course not. It is only transferring the risk of fault from the person who is doing to analysis to the person who is requesting it or making the decision. To the extent to which it is not possible to be more certain of course then 60 or 70% likelihood will have to do. The question then becomes if the decision maker should make a decision to engage at all, given the risks. Are the risks sufficiently explained to the decision maker? In the case of Osama in the bunker the answer is not clear. So, is this better intelligence work and is it a better method for intelligence analysis? I think the book offers some good advice in terms of rules of thumb, which we shall come back to, but so far the suggestions made imply that the analysts have just become smarter fencing off potential criticism for potential mistakes. If this is how the authors won the competition against their colleagues in the IC – by giving vaguer answers - then that is no real victory, but a statistical trick. This would also explain why they do not focus on knowledge, as they are not so concerned with ideas, but more with careful expressions. So far into the book this seems to be the essence, and a better title may have been “the art of careful expressions”. The question remains what kind of people you would like to fill your intelligence department with, well-read experts or people who have learned that careful expressions will put you in the right more often? Note again that in the Obama case the analysts are not really helping Obama by saying that there is a 70% or 90% possibility bin Laden is in that house in Pakistan. It’s also odd to say “70% or 90%”, 70 to 90% would at least make some sense, but 70 or 90 is like giving two different answers. As if we are free to choose. Obama is faced with two choices: to bomb/attack or not to bomb/attack, it is either or, but the answers given him are in terms of a percentage likelihood of bin Laden being in that house, which is not what he needs. In other words he is not being given the intelligence he requested. If it was difficult to be sure, why not wait until they were more certain? The analysts figure Obama will bomb/attack because there is only a 30% chance that bin Laden is not in that house, but Obama could also have reasoned that it is not worth bombing/attacking as there is a 30% chance someone else (innocent people) will be killed. Another technique used by foxes is to analyze the problems using many methods/analyses and synthesize it into one answer at the end, something the authors call aggregation, but others call redundancy in method. It is a well-used method in the social sciences, so there is nothing new about it. Chapter four starts with the horrifying story of how the intelligence community made up of 20,000 intelligence analysts supported a claim from the White House that Iraqis had a nuclear weapons program that produced weapons that was a threat to the US and NATO countries (National Intelligence Estimate 2002-16HC). One explanation was that the IC had been bullied by the White House to come up with documents that suggested a war. With the authors method, the IC should have said that there was a 70% likelihood or similar, but then the results would probably have been the same anyway. This just proves how dangerous the method of transferring the risk to the decision maker is. The authors struggle to find the right answer to the question. They do not start by saying that maybe the IC should have listened to Dr. Hans Blix, the IAEA Director General from 1981 to 1997, who was experience with these issues and guided the Agency through the Chernobyl disaster. Dr. Blix was against the invasion from the start, as there was no evidence to suggest that the claim was true. Thus, it is disheartening to see how the authors stay with their initial method in this example, they should have said 60-70%. Then they would not have been completely wrong and that, the authors think, would have been better. For whom? For the estimated 1 million Iraqis who died as a result of the conflict? Another example that is used in the book is the use of math to make predictions on Wall Street. The authors suggest that the answer to intelligence is statistics and math, just like for the study of economics (probability). But how well did the quantitative analysts really do for their investors? What about the consequences 49 of the failed banks and all the pensioners who lost their retirement funds? The authors never go down that road. In general, has the study of finance succeeded with math? If one had asked that question 20 years ago most colleagues would have said yes, but today large part of quantitative finance is left behind as irrelevant, including option pricing models. Some of those who received the Nobel Prize for their “inventions” in finance have since been discredited. Chapter five is about IQ and intelligence. Much of the chapter and chapters in general are case allegories, small cases with no clear conclusion, as in the example of the cause of death of Yasser Arafat (pp. 114-117). The case is picked up in later chapters as a to-be- continued ploy for the reader to find the content interesting, it seems. Chapter six is entitled “Superquants”. We are told that Superforcasters are not like the quants (quantitative analysists) of Wall Street, they don’t use that much math. It’s more careful thought-out and nuanced answers (p. 129). The authors return to the Obama – bin Laden example, citing Mark Bowden, who confirms what Obama thought about the intelligence estimates he received. Obama got “probabilities that disguised uncertainty as opposed to actually providing you with useful information“ (p. 135). Obama acknowledged that he was left with a gamble, as we commented on earlier in the review. Obama himself is quoted as having said it was a “fifty- fifty”. Then a whole analysis follows about what this comment means; if it was to be interpreted literally or not. Was he being sarcastic, critical or just stating a fact? It’s easier to say for those who were in the room. He may have thought that the figures presented insufficient information. One interpretation says that Obama would have attacked the facility no matter how small the odds were for finding bin Laden. If that is true it borders to an almost bizarre example of decision making that resembles gambling, which may or may not be what he meant. The authors and those consulted in the book cannot agree what Obama was thinking when he said “fifty-fifty” or what I meant, which is not much more comforting. Chapter seven in entitled “Supernewsjunkies”. Just the idea that extensive reading makes someone a “junkie” is offensive but fits well with the authors’ idea that it is not what you know but how you think. The chapter starts by unfolding more of the “superforecasting” method, leaving the reader puzzled as to why the method is spread around the book in small pieces. It makes the book seem scientifically unfriendly, again, as it is all about selling books and consultant services. The suggestion is to “unpack the question into components” distinguish between unknown and known and leave no assumption unscrutinized (p. 153). Fair enough, but this is much more difficult than it seems and poorly explained on the following pages. “Adopt the outside view and put the problem into a comparative perspective that downplays its uniqueness and treats it as a special case of a wider class of phenomenon”. “Also explore the similarities and differences of your own views and those of others…” (p. 153). The author’s method consists of synthesizing these two views and the views of the crowd. This is questionable. First of all, if I am not well-read on a topic why include my opinion at all? And surely the opinion of the crowd is a function of the information spread in mass media, whatever that may be. Thus to find some sort of average (another statistical ploy) on these three positions is ludicrous. Why should this method bring you any closer to anything truthful? What it will give us is what the social truth is, but the social truth is very often different from the truth per se as will be obvious, for example, to anyone asking people about which religion is right. The authors go on to say that this process of gathering the three views takes time and is only the beginning of the method (but by now the reader is a bit tired of the sales talk). The reader is annoyed by the probability figures the authors keep throwing around in the chapters, like the 60% probability that polonium would be found in Arafat’s body (p. 153). The authors should for once tell the reader how the analyst got to that figure, as that calculation is the cornerstone of the whole method suggested in this book. It’s not explained anywhere. The time frame of a decision is very important of course. The authors talk about “scope”, an effect that may give an answer of no today, but yes in a month or two, so the answer depends on the point in time. The “superforcasters” know this so they update their information much more frequently, on average, than regular forecasters, we are told. It makes you wonder who the regulars are, analysts at IC? I am sure they must be thrilled 50 to read how badly they do their work, all 20 000 of them. By now the reader is also annoyingly interested in learning about all the facts of the “tournament” where the “superforecasters did so well and so much better than the rest. What were the questions? Who set them up? How much time was given to each question? And more fundamentally, how were they graded? I do not want to speculate but I suspect that the best answer was not in terms of right or wrong answers, but the answer that comes closest to the truth as that would favor those answer with vague answers. It should have all been explained clearly at the start, not as loose sentences spread around the book like bait to turn another page. On the other hand I guess that is how bestseller books are written, they are exiting partly because the reader hopes to know what it’s all about and keeps flipping those pages. The point about updates also makes you wonder if the “superforcasters” won because they updated their information more frequently. The article continues on the Arafat question, and Bill Flac (one of the superforcasters) updates his estimate from 60% to 65% yes as he thinks that the delay in time the Swiss laboratory has with publishing the results has to do with the operation they may be testing to rule out lead as the source of death. Another issue that is interesting here is the calculation that increases with 5% likelihood. That calculation is never shown. Why not? Surely if focus is on psychology it would be interesting to learn about the cognitive processes that makes the difference of 5%, not least the biases if there is no clear calculation but more of a feeling. In a book dedicated to this essential topic how come the calculations are not shown? I am not saying it is easy, but others have tried and it is the central theme of the book. Instead the authors talk about the Briar score again, which is used as a measure of success for predictions, not for the calculation of estimates. In fact, about the only thing the method presented in the book has in common with forecasting is the Briar score. The randomness of the method is clear in another example about Republican voters in Colorado: … “So you think that the maximum you should raise your forecast is 10%. It’s now between 1% and 10%” (p. 168) “Finally you settle on 4%”. This shows clearly that this is what we call a rule of thumb, which by itself is fine, but then it should say so clearly, and there is nothing new with this approach. Maybe that is the most critical part about this book: that it pretends to be about forecasting but is instead a good collection of rules of thumb. It’s a method by which new information leads to small adjustments in the estimates. Another methodological problem is that if you go with a certain hypothesis and gather a large amount of information in that direction, then you are likely to get a high likelihood of true or false because each new piece of information could lead to a small adjustment. It will also depend on the information you happen to find in the language(s) you can read. There will be plenty of information that you do not see or find, there will be some stories you tend to go with so in reality this incremental approach by which likelihoods are increasing or decreasing with percentage points is not that straight forward to use. Chapter nine is entitled “Superteams”. It starts by telling the disastrous story of the Bay of Pigs Invasion (1,400 terrorists were surrounded by 20,000 soldiers when they tried to invade a foreign country) and how that lead to the Cuban Missile Crisis. Much of this is true but the authors forget to mention that the Russian placement of missiles was also a reaction to the American placement of missiles in Turkey. That in itself is an argument for the importance of knowing history. And if you did not know that it does not help to put you into a team of other superforecasters in a superteam asking superquestions. The result is just going to look even more wrong. Chapter ten raises a relevant topic for anyone who has read this far, how it is possible to be a good leader and make accurate decisions if all you are getting are vague estimates. The answer suggested by the authors seems to be based on Moltke, the Prussian general. The reason Moltke is largely implied is because he said that everything in war is uncertain. So, don’t trust your plan. An officer should be calm and assuring, and knows that he needs to make a decision in a fog of uncertainty. As often is in these kinds of books, there is the introduction of a German magic word that is supposed to explain it all (other examples in other books: “gestalt” or “verstehen”). The word this time is ‘Auftragstaktik’, or mission command in English. As valuable as the idea may be, I am not sure it is going to be a consolation for Obama when he is asked to take the risk of attacking a house just outside of a Pakistani 51 army base. It is not going to give me more confidence in intelligence analysts. Chapter eleven is the second to last and is called “Are they really so super”? So, through the whole book they have been telling me how super they are and now they are about to say that they are not super? As could be expected, the authors do not give a clear answer. This is not unusual in these kinds of bestseller books either. Instead, there is an insinuation, a hint to the reader to draw his own conclusion that they are in fact super because their predictions are best, which is a claim that can never be tested. The chapter goes on to talk about conversations with General Mikael Flynn who was the National Security Advisor for Donald Trump for 24 days, the shortest in the office's history. (He pleaded guilty to lying to the FBI over his contact with the Russian government during the Trump presidential transition). Flynn tells the author that he thinks “societal conflicts” are at unprecedented levels. The reader thinks that he must have forgotten about the race riots of the 1960s and the American Civil War. Maybe he meant during the past generation, in the US, but it does not say so. The authors criticize Flynn for falling for the “oldest trick in the psychology book”, assuming that what is presented to you is all there is. Flynn’s inbox is full of reports that confirm this view. The authors argue that facts show that interstates conflicts have been declining since the 1950s: it’s enough to google the question and you will see. What the authors fail to mention is that googling a question is often a poor source of information, but otherwise they may be right. Much of the information found on webpages is false and most good information is not freely available. That is one reason why books continue to be so important. Not to mention a good general education. Then there is a lot of Kahnemnan and Tverksy again, but few other references to psychologists’ research. There is also a comparison between the authors and Kahnemann and Taleb’s ideas about predictions, where the authors claim to be right. An interesting replica of a strategic memo written by Linton Wells II (not Linto Wells, who was his father and a well-known American foreign correspondent) is presented. It was from 2001. In it, Wells II shows examples from the past hundred years of how fast foreign relations have changes, thus drawing the conclusion that the US should plan for something unexpected, that that is the best overall strategy. Another good citation here is from Eisenhower, “plans are useless, but planning is indispensable” (P. 244). The memo from Rumsfeld citing Wells II says nothing about what England, and later the US, actually knew or how good their guesses were about the future at that time. It just assumes that they were surprised, which is probably close to the truth for most of the examples listed. At the same time, it’s a bit like saying that the US was not very good at predictions at the time (not that any other powers are recorded has having gotten it right more often, to my knowledge). Wells II’s response was to plan for adaptability and resilience as a way to meet the unexpected. This is also close to what the US has done with its continuous massive military buildup. One problem has been that there has not been any money for this buildup, so the government has turned to massive borrowing during the past administrations. (It is often forgotten, but Obama borrowed more money and engaged in more wars than any of his predecessors since the Vietnam War). The US has also not been able to make money on its wars, which is the other major problem. Today they are in a squeeze needing to borrow more money to keep the military strong so as not to have to repay their foreign debt, which cannot be paid. In Wells II’s defense, we can say that he did not imagine the financing part of his strategy. Unfortunately for the US and its allies the US military is failing both with adaptability and resilience. The authors then go on to speculate about why China may not become the world’s leading economic power by comparing it to Japan. Many thought Japan would become the leader, but it did not happen, they reason. The authors do not discuss the fact that China’s population is growing to ten times the size of Japan’s, the fact that China has been a world economic power for most of the past 2-3 millennia, except since the mid-1600s (the Enlightment). They do not discuss cultural similarities or differences either, I assume again because they do not look at knowledge but how you think. Sure, China may face great difficulties and may even decline as a result, but the authors are too light on this question. The simplicity with which this parallel is treated is symptomatic of the whole book when it comes to questions of history, geography and culture. Their approach is a combination of psychology 52 studies and basic statistics, good enough, but not enough by itself. Chapter twelve is the last chapter. It highlights the credo, “keep scores”. It also says to analyze results, but how to do this is not shown with any clarity anywhere in the book (p. 259). Keeping scores, or evaluations of past performances, is a key part of any intelligence cycle (that is why it is drawn as a cycle), which is the most basic model any intelligence analyst is shown for how to work. That evaluations are not done in the American IC (or in many other countries, I am sure) is not surprising, but that is more a question of professionalism within the working corps. It’s a fact, the “sharpest knives in the box” don’t become intelligence analysts, not yesterday and not today. The IC is not McKinsey or KPMG, not yet at least. A useful rule of thumb mentioned in the book is to try to solve the larger questions by breaking them into many small questions. A parallel is made to the technique of pointillism (p. 263), where a painter makes a painting by adding a greater number of dots on the canvas. A few dots do not look like anything, but as more dots are added we see an image emerge, the larger picture or question. Of course a painter knows what he is setting out to make so no dots are wasted. An intelligence analyst may collect the wrong dots, or dots belonging to another painting and it is far from certain that enough dots or the right dots are collected to get the larger picture so the parallel is merely suggestive. Towards the end the author reminds us that his friend Tom Friedman (who is mentioned on every other page or so it feels) was for the invasion of Iraq because he thought that Iraq was the way it was because of Saddam Hussein. Another possibility is that Saddam Hussein was the way he was because of Iraq. Friedman decided upon the first alternative. The authors point to the fact that the conclusion and his reasoning was not correct. To present the conflict in such simplistic terms is shocking, to say the least. Anyone with a minor grip on history will analyze this conflict from a Shia-Sunni perspective, which could also explain why the Sunnis felt desperate enough to form the Islamic State after their defeat. It was the American-led invasion that created ISIS. Actually US foreign policy is to blame for most of the disruption of the Arab world and the Middle East, which started with the First Gulf War but whose history goes back to the beginning of the American-Saudi relationship at the end of WW2, a relationship they inherited from the British. At the end the authors explain that superforcasters are more humble than other forecasters, analysts or experts; they do not show off and know their limitations (they do not need to go to Davos, but leave that to others). They can do this because they have the support of a proven record of predictions. With the Briar score they ride into the sunset. Somehow I was never impressed but I know some of my colleagues are. Conclusion There are many things that are good about this book. Philip E Tetlock is a scholar with an impressive number of publications and citation. The book is well-written and easy to read, but that is also the best that can be said. The book falls into a long line of bestselling books that have an extravagantly attractive title that has little to do with the content, and a first chapter that is all about promises of what is to be delivered in the following pages. As such, this is all too common in the management literature in general as we have known it since the early 1980s, maybe even earlier. It throws around the names of famous people and stories people can relate to. But what is the problem with that, the reader may ask. Well the problem is that these types of management books continue to have a significant influence on practice, much more so than scientific articles or more instrumental books on intelligence analysis. This is not a new phenomenon either but has been going on since “In search of Excellence” or maybe even longer. For the most part though these books are being discredited in the long run, but then it is too late, as their content has already been put into practice. For one thing there is nothing that has been presented in the book that helps explain why the project was better at predicting events than anybody else, if we are to believe that that is true. More worryingly, the book does not say how the authors and the project beat the other analysts, if it was by simply using a more vague language in its estimates or by the way correct answers were calculated. The rules of these competitions are never explained, at least not in the book. The main idea in the book is that if you give precise questions and ask for answers expressed in numbers for specific time frames, then you can also sit back and wait to measure the results. You will then know how good you 53 are. That by itself is not a bad idea. Instead we are led on a series of loose threads and assumptions, by the authors who are expert analysts because they did so – “it took years” - and won. It seems like a proven way to sell consultancy, but does not convince a reader who is even half awake. Clearly psychology is important for decision making and forecasting, especially when confronted with social situations where an outcome is the result of the interaction and the expectations of several individuals with different interests and values. Some of these problems can be modelled using game theory, but the authors fail to see that this is only one half of the equation. The other half is what you actually know. The intelligence reality of Mr Tetlock is much like that of a psychologist in a poker game. He does not know what the other person knows but tries to guess it based on his behavior. That is a much riskier way of solving a problem than using resources to actually find out. Good intelligence is about finding out what hand was actually dealt. This will give us certainty to know how we could win the game, or at least avoid losing more money than what was in the pot. Psychology is important in knowing how the player will behave. It is this other part of the equation—that the psychological insights are valuable—that Tetlock introduces in this book. It’s a good suggestion to test or check guesses to learn from them, but it’s hardly a new or novel idea. It’s true that it is “astonishing” how many organizations do not check the intelligence they produce or buy, but it’s hardly a new problem or even surprising. The book is one in a long tradition of “hype” books which are so popular and not only in the Anglo-Saxon world, similar to Nassim Taleb’s book “Black Swan”, which the authors also refer to. You take something that is merely common sense and present it in an appealing way, such as that complete unknowns are like black swans. The reader will not have learned anything new, but old wisdom is frightfully well packaged, thus appealing. It does not help that the authors disagree with Taleb in that they think that many swans that people say are black are in fact grey (another metaphor of the same type). I said at the beginning that this is a good book. The reason for this is that it contains many good rules of thumb. Unfortunately, they are not listed in any single place in the book. We should break large questions into many small questions. We should make scorekeeping an integral part of intelligence analysis (p. 259). That is a simple but important lesson. Thus the book is worth reading.